Draft:3D Gaussian Splatting for Real-Time Radiance Field Rendering

Introduction

The field of Real-Time Radiance Field Rendering has been revolutionized by the usage of 3D Gaussian Splatting since the introduction of the methodology in this seminal ACM Transactions on Graphics Paper in 2023. The work enables the creation of high-quality real-time (1080p, >=30 fps) novel-view scenes by stringing together multiple photos or videos, which had historically been a big challenge.

As of September 2024, the paper has 1000+ citations.

Contributions

The paper introduced several creative ideas to the field of novel-view generation, towards improving the speed and quality of generation, especially over Neural Radiance Field based approaches.

1. For the first time, anisotropic 3D Gaussians were used to represent radiance fields.
2. A new optimization method was introduced.
3. A fast rendering approach, catered to usage of GPUs was proposed.

Overview

The approach can be summarized as follows: (insert image)

• Input: Set of images of a static scene along with camera positions, expressed as a sparse point-cloud.
• 3D Gaussians: Define mean, covariance matrix and opacity.
• Color: Represented by spherical harmonics.
• Algorithm: Optimize the above mentioned parameters.
• Rasterizer: Fast sorting and fast backward pass.

Method

This diagram illustrates the working of the proposed algorithm.

The approach has the following salient points:

• Usage of Differentiable 3D Gaussian Splatting, that is unstructured and explicit, hence allowing for rapid rendering, and also can be projected to 2D splats. Intuitively the covariance of the gaussian's can be thought of as configurations of an ellipsoid, which can be mathematically broken down into a scaling matrix and a rotation matrix. For all parameters, the gradients are derived explicitly to overcome any overhead due to autodiff.

• The optimization creates a dense set of such 3D Gaussians which represent the scene as accurately as possible. Each step of rendering is followed by a step of comparison to the training views available in the dataset. Stochastic Gradient Descent is used to optimize the following loss function: ${\mathcal {L}}=(1-\lambda ){\mathcal {L}}_{1}+\lambda {\mathcal {L}}_{D-SSIM}$ inspired by the Plenoxels work. The trick is to control the number of Gaussians and the density via this procedure.

• The blending of different Gaussian components is then achieved via a tile-based rasterizer. The approach can handle scenes with varying complexities without explicitly tuning the hyperparameters.

Results and Evaluation

• Authors tested their algorithm on 13 real scenes from previously published datasets and the synthetic Blender dataset.
• They compared their method against state-of-the-art techniques like Mip-NeRF360, InstantNGP, and Plenoxels.
• Quantitative evaluation metrics used were PSNR, L-PIPS, and SSIM.
• The authors report results for two configurations of their method: 7K and 30K iterations.
• Their fully converged model (30K iterations) achieves quality on par with or slightly better than Mip-NeRF360, but with significantly reduced training time (35-45 minutes vs. 48 hours) and faster rendering (real-time vs. 10s/frame).
• At 7K iterations (5-10 minutes of training), their method achieves comparable quality to InstantNGP and Plenoxels.
• For synthetic bounded scenes (Blender dataset), they achieved state-of-the-art results even with random initialization, starting from 100K uniformly random Gaussians.

Limitations

• Elongated artifacts or "splotchy" Gaussians in some areas.
• Occasional popping artifacts due to large Gaussians created by the optimization, especially in regions with view-dependent appearance.
• Higher memory consumption compared to NeRF-based solutions, though still more compact than previous point-based approaches.
• May require hyperparameter tuning (e.g., reducing position learning rate) for very large scenes.
• Peak GPU memory consumption during training can be high (over 20GB) in the current unoptimized prototype.

Authors note that some of these limitations, such as popping artifacts and memory consumption, could potentially be addressed through future improvements like better culling approaches, antialiasing, regularization, and compression techniques.

Applications of 3D Gaussian Splatting

Below are a few examples of how Gaussian Splatting has been adapted and extended across various computer vision and graphics applications, from dynamic scene rendering to autonomous driving simulations and 4D content creation:

• 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering:
  • Extends 3D Gaussian Splatting to dynamic scenes (4D representation)
• Text-to-3D using Gaussian Splatting:
  • Applies 3D Gaussian Splatting to text-to-3D generation
• End-to-end Autonomous Driving: Challenges and Frontiers:
  • Mentions 3D Gaussian Splatting as a data-driven sensor simulation method for autonomous driving
  • Highlights its ability to generate realistic novel views of a scene
• SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering:
  • Proposes a method to extract precise and fast meshes from 3D Gaussian Splatting
• SplaTAM: Splat, Track and Map 3D Gaussians for Dense RGB-D SLAM:
  • Applies 3D Gaussian-based radiance fields to Simultaneous Localization and Mapping (SLAM)
  • Leverages fast rendering and rich optimization capabilities of 3D Gaussians
  • Achieves state-of-the-art results in camera pose estimation, map estimation, and novel-view synthesis
• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models:
  • Uses dynamic 3D Gaussians for 4D content creation from text
    

Authors

The authors of the 3D Gaussian Splatting for Real-Time Radiance Field Rendering paper are Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.