Summary of their method: A sequence-to-sequence model, the Transformer model, is trained on various speech processing tasks, such as multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are represented as a sequence of tokens for the decoder to predict, enabling a single model to replace multiple stages of a traditional speech processing pipeline. The multitask training format employs specific tokens that act as task identifiers or classification targets.

Core Idea

At its core, Whisper is an encoder-decoder transformer model trained on 680,000 hours of diverse audio data. Unlike traditional Automatic Speech Recognition (ASR) systems that rely on complex pipelines with separate components for acoustic modeling and language processing, Whisper unifies these tasks into a single neural network. This approach allows it to handle accents, background noise, and multilingual inputs more robustly.

The model’s versatility comes from its ability to perform multiple tasks—like transcription, translation, and language identification—using a unified architecture. By conditioning the decoder with special tokens (e.g., <|en|> for English or <|translate|> for translation), Whisper dynamically adapts to the user’s needs.

Architecture: From Sound Waves to Text

Step 1: Processing Audio into Features

Whisper begins by converting raw audio into a log-Mel spectrogram, a mathematical representation that captures frequency patterns over time. It transforms chaotic sound waves into a structured format the model can analyze. This spectrogram is generated by:

1. Resampling: Audio is standardized to 16,000 Hz.
2. Spectrogram Creation: Splitting the waveform into 25-millisecond windows with a 10-millisecond stride, then converting these windows into 80 Mel-frequency bins (mimicking human hearing sensitivity).
3. Normalization: The spectrogram is scaled to a range of [-1, 1] to stabilize training.

Step 2: The Encoder – Understanding Speech

The encoder outputs a high-dimensional representation of the audio, capturing both acoustic and linguistic features (good blog talking about it link).

Convolutional Layers

The spectrogram first passes through two 1D convolutional layers (kernel size 3), which act as a “stem” to detect local acoustic patterns like phonemes or formants. These layers:

• Reduce dimensionality while preserving temporal resolution.
• Use GELU activation to introduce non-linearity, enabling the model to learn complex relationships in the audio.

Positional Embeddings

Since audio is a time-series, Whisper adds sinusoidal positional embeddings to the spectrogram. These embeddings encode the position of each 40ms audio frame, allowing the model to understand temporal order—crucial for distinguishing sequences like “cat” vs. “act”.

Transformer Blocks

The core of the encoder is a stack of 32 transformer blocks (for large models). Each block includes:

• Multi-head self-attention: Identifies relationships between distant audio segments. For example, it links pronouns (“he”) to their later references (“ran”) even if separated by seconds of speech.
• Feed-forward networks: Refine features using learned transformations.
• Pre-activation layer normalization: Stabilizes training by normalizing inputs before attention and feed-forward operations.

Through these layers, the encoder gradually builds a high-dimensional representation (1,500 tokens of 1,280 dimensions for 30-second audio) that encapsulates both acoustic details (e.g., pitch) and linguistic context

Step 3: The Decoder – Generating Text

The decoder generates text one token at a time using an autoregressive process:

The decoder predicts text tokens autoregressively, one word at a time, using:

1. Task-Specific Tokens: Instructions like <|transcribe|> or <|translate|> guide the decoder49.
2. Cross-Attention: The decoder focuses on relevant parts of the encoder’s output while generating each token.
3. Language Modeling: Whisper’s training on vast text data helps it produce grammatically correct sentences, even in noisy environments1036.

For example, when translating French to English, the decoder might process: <|startoftranscript|><|fr|><|translate|> → “Bonjour” → “Hello”

Inference

1. Chunking: Long audio is split into 30-second segments (or padded to fit).
2. Batch Processing: Multiple segments are processed in parallel for efficiency.
3. Timestamp Prediction: Optional timestamps mark when each word was spoken.

This pipeline ensures Whisper handles everything from podcasts to phone calls seamlessly.

Simple Usage Example

Using Whisper is straightforward with Open AI whisper Library

Installation and Setup

To use Whisper, install it via pip:

pip install openai-whisper

Basic Transcription

Here’s how to transcribe an audio file in Python:

import whisper
# Load the model (choose 'tiny', 'base', 'small', 'medium', or 'large')
model = whisper.load_model("base")
# Transcribe audio
result = model.transcribe("audio.mp3")
print(result["text"])

• Translation: Use model.transcribe(..., task="translate") to convert non-English speech to English text.
• Timestamps: Enable word-level timestamps with model.transcribe(..., verbose=True).

Subtitle Generator Example

You can reference the link on Kaggle

1. Introduction and Problem Statement

Existing state-of-the-art (SOTA) methods for generalizable 3D scene reconstruction, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), require accurate camera poses as input. These poses are typically obtained through Structure-from-Motion (SfM) methods like COLMAP. However, this requirement is impractical in many real-world scenarios, especially when dealing with sparse views (e.g., just a couple of images) or in textureless areas where pose estimation is unreliable.

Limitations of Existing Methods

Current methods that attempt to jointly estimate poses and reconstruct scenes suffer from a compounding effect: errors in initial pose estimates degrade reconstruction quality, which in turn further degrades pose accuracy. This creates a feedback loop that limits the effectiveness of these approaches.

NoPoSplat’s Solution

NoPoSplat introduces a feed-forward network that directly predicts 3D Gaussian parameters within a canonical space from sparse, unposed multi-view images. This eliminates the need for explicit pose estimation during reconstruction. By anchoring the first input view’s local camera coordinates as the canonical space, NoPoSplat avoids the need to transform Gaussians from local to global coordinates, thus bypassing the errors associated with pose estimation.

2. Related Works

2.1. Generalizable 3D Reconstruction and View Synthesis

NeRF (Neural Radiance Fields) and 3DGS (3D Gaussian Splatting) have significantly advanced 3D reconstruction and novel view synthesis, but these methods typically require dense posed images as input and per-scene optimization limits their practical application

Recent approaches focus on generalizable 3D reconstruction from sparse inputs. These methods typically use task-specific backbones that leverage geometric information to enhance scene reconstruction. Examples include:

• MVSNeRF and MuRF: These methods build cost volumes to aggregate multi-view information
• pixelSplat: This method employs epipolar geometry for improved depth estimation

However, these geometric operations often require camera pose input and sufficient camera pose overlap among input views. In contrast, NoPoSplat uses a Vision Transformer (ViT) without any geometric priors, making it pose-free and more effective in handling scenes with large camera baselines.

2.2. Pose-Free 3D Scene Reconstruction

Classical methods rely on accurate camera poses of input images, typically obtained through Structure from Motion (SfM) methods like COLMAP, which complicates the overall process.

Some recent works attempt to jointly optimize camera poses and neural scene representations, but they still require rough pose initialization or are limited to small motions. Other methods adopt incremental approaches, but they only allow image/video sequences as input

For generalizable sparse-view methods, requiring camera poses during inference presents significant challenges, as these poses are often unavailable in real-world applications during testing. While two-view pose estimation methods can be used, they are prone to failure in textureless regions or when images lack sufficient overlap. Some recent pose-free novel view synthesis methods break the task into two stages: first estimate camera poses, then construct the scene representation. However, this two-stage process lags behind pose-required methods due to noise from the initial pose estimation.

In contrast, NoPoSplat completely eliminates camera poses by directly predicting 3D Gaussians in a canonical space, avoiding potential noise in pose estimation and achieving better scene reconstruction

Splatt3R, a concurrent work, also predicts Gaussians in a global coordinate system but relies on the frozen MASt3R model for Gaussian centers and requires ground truth depth during training, which makes it unsuitable for novel view synthesis and limits its ability to leverage widely available video data

3. Methodology

3.1. Problem Formulation:

The goal is to reconstruct a 3D scene from a set of unposed multi-view images and their corresponding camera intrinsics, represented as ${I^v, k^v}$

Here, $I^v$ is the input image for view $v$, and $k^v$ represents the camera intrinsic parameters for that view.

The network, denoted as $f_θ$ with learnable parameters $θ$, maps these inputs to 3D Gaussians within a canonical 3D space. The mapping is defined as:

• $\mu$: the center position of the Gaussian primitive in $R^3$
• $\alpha$: the opacity of the Gaussian primitive in $R$.
• $r$: the rotation factor in quaternion in $R^4$
• $s$: the scale of the Gaussian primitive in $R^3$
• $c$: the spherical harmonics (SH) coefficients of the Gaussian primitive in $R^k$

The method assumes that camera intrinsics (k) are available, which is a common assumption as they are generally available from modern devices.

The method can generalize without any optimization. The results 3D Gaussians in canonical space enabble two tasks

1. Novel view synthesis: Given a target camera transformation relative to the first input view, the model renders a novel view.
2. Relative pose estimation: The model estimates the relative camera poses between input views.

3.2. Overall Pipeline

The NoPoSplat pipeline consists of three main components:

1. Encoder: A Vision Transformer (ViT) that processes the input images and camera intrinsics.
2. Decoder: Another ViT that integrates multi-view information through cross-attention.
3. Gaussian Parameter Prediction Heads: Two prediction heads that output the Gaussian parameters (center positions and other attributes).

Both the encoder and decoder use Vision Transformer (ViT) architectures without any geometric priors, which is a key difference from other methods that rely on geometric priors like epipolar constraints or cost volumes. The use of a pure ViT structure allows the model to be more flexible and effective when the overlap between input views is limited.

3.2.1 Vision Transformer (ViT) Encoder and Decoder

The RGB images are converted into sequences of image tokens, and these are concatenated with an intrinsic token (Detail in Section 3.4). These tokens are fed into a ViT encoder separately for each view. The encoder shares weights across all input views.

The output features of the encoder are passed to a ViT decoder, which integrates multi-view information through cross-attention layers, allowing the features from each view to interact with those from all other views.

3.2.2 Gaussian Parameter Prediction Heads

DPT

Two prediction heads are used to predict the Gaussian parameters, based on the DPT architecture.

• The first head focuses on predicting Gaussian center positions ($µ$) and takes features exclusively from the transformer decoder.
• The second head predicts the other Gaussian parameters (α, r, s, c) and uses both the ViT decoder features and an RGB image shortcut as input.

The RGB image shortcut allows for the direct flow of texture information which is crucial for capturing fine details. The ViT decoder features are downsampled, so they lack detailed structural information which the RGB shortcut compensates for

3.3. Canonical Gaussian Space

Instead of predicting Gaussians in each local camera coordinate system and then transforming them to a world coordinate system using camera poses, NoPoSplat directly outputs Gaussians in a canonical space (Figure 5 and 6)

The local camera coordinate of the first input view is anchored as the canonical space, meaning that the camera pose for the first view is considered to be [U 0], where $U$ is a unit/identity matrix for rotation and 0 is a zero translation vector.

All Gaussians are predicted relative to this canonical space. The network predicts the set

where the superscript $v→1$ denotes the Gaussian parameters corresponding to pixel $p_j$ in view $v$, under the local camera coordinate system of view 1.

This approach eliminates the need for camera poses and allows the network to learn the fusion of different views directly within the canonical space, which results in a more cohesive global representation

3.4. Camera Intrinsics Embedding

The camera intrinsics ($k = [f_x, f_y, c_x, c_y]$) are crucial for resolving the scale ambiguity

Three methods for embedding camera intrinsics are compared

1. Global Intrinsic Embedding - Addition: Camera intrinsics are fed into a linear layer to obtain a global feature, which is added to the RGB image features after the patch embedding.
2. Global Intrinsic Embedding - Concatenation: The global feature is treated as an intrinsic token and concatenated with the image tokens.
3. Dense Intrinsic Embedding: Per-pixel camera rays are converted to higher-dimension features using spherical harmonics and are concatenated with the RGB image.

The intrinsic token method (concatenation) performs the best. This method is not only effective at injecting camera intrinsic information but also gives the best performance.

3.5 Training and Inference

3.5.1 Training

The 3D Gaussians are used to render images at novel viewpoints, and the network is trained end-to-end using ground truth target RGB images as supervision. The loss function is a linear combination of MSE (Mean Squared Error) and LPIPS (Learned Perceptual Image Patch Similarity) loss with weights 1 and 0.05, respectively.

3.5.2 Relative Pose Estimation

The 3D Gaussians in the canonical space are directly used for relative pose estimation. The pose estimation is approached in two steps:

1. Coarse Pose Estimation: The an initial pose estimate is obtained using the PnP (Perspective-n-Point) algorithm with RANSAC, given the Gaussian centers.
2. Refinement: The initial pose is refined by rendering the scene using the estimated pose and optimizing the alignment with the input view using photometric loss and the structural part of the SSIM loss.

The camera Jacobian is calculated to reduce computational overhead during optimization.

3.5.3 Evaluation Time Pose Alignment

“”However, 3D scene reconstruction with just two input views is inherently ambiguous as many different scenes can produce the same two images. As a result, though the scene generated by our method successfully explains the input views, it might not be exactly the same as the ground truth scene in the validation dataset.””

For evaluation purposes, the camera pose for the target view is optimized to match the ground truth image by freezing the Gaussians and optimizing the target camera pose using photometric loss. This is done to compare with other baselines, but is not required for real world use (Figure 7)

4. Experiment Results and Analysis

The experiments use the RealEstate10k (RE10K) and ACID datasets, and also test zero-shot generalization on DTU and ScanNet++ datasets. To handle input images with varying camera overlaps, they generate input pairs for evaluation that are categorized based on the ratio of image overlap

The evaluation metrics used include PSNR, SSIM, and LPIPS for novel view synthesis, and AUC (Area Under the Curve) for pose estimation.

4.1. Novel View Synthesis

NoPoSplat significantly outperforms all state-of-the-art pose-free methods in novel view synthesis. Methods such as DUSt3R and MASt3R struggle to fuse input views effectively due to their reliance on per-pixel depth loss, and Splatt3R inherits this limitation

NoPoSplat achieves competitive performance with state-of-the-art pose-required methods like pixelSplat and MVSplat, and outperforms them particularly when the overlap between input images is small.

This improved performance is attributed to NoPoSplat’s 3D Gaussian prediction in a canonical space, as opposed to the transform-then-fuse strategy used by other methods.

Qualitative results in Figure 4 demonstrate NoPoSplat’s ability to achieve more coherent fusion from input views, superior reconstruction with limited image overlap, and enhanced geometry reconstruction in non-overlapping regions.

4.2. Relative Pose Estimation

NoPoSplat shows strong performance in relative pose estimation on the RE10k, ACID, and ScanNet-1500 datasets. The method is trained on either RE10K (denoted as Ours) or a combination of RE10K and DL3DV (denoted as Ours*).

The results demonstrate that performance consistently improves when scaling up training with DL3DV. NoPoSplat demonstrates superior zero-shot performance on ACID and ScanNet-1500, even outperforming RoMa, which was trained on ScanNet. The authors state that this indicates the effectiveness of the pose estimation approach and the quality of the 3D geometry produced.

4.3. Geometry Reconstruction

NoPoSplat outputs improved 3D Gaussians and depths when compared to state-of-the-art pose-required methods.

Figure 5 shows that MVSplat suffers from misalignment in the intersection regions of input images, and distortions in areas with limited overlap. These problems are attributed to the transform-then-fuse pipeline8.

NoPoSplat’s direct Gaussian prediction in canonical space addresses these issues. The results show that even without camera poses as input, NoPoSplat generates higher quality 3D Gaussians, resulting in improved color and depth rendering

4.4. Cross-Dataset Generalization

NoPoSplat exhibits superior zero-shot performance on out-of-distribution data, when trained exclusively on RealEstate10k and tested on ScanNet++ and DTU datasets. The model’s minimal geometric priors help it to adapt effectively to different scene types.

As shown in Table 4 and Figure 6, NoPoSplat outperforms state-of-the-art pose-required methods on out-of-distribution data.

Notably, NoPoSplat outperforms Splatt3R, even on ScanNet++ where Splatt3R was trained.

4.5. Model Efficiency

NoPoSplat can predict 3D Gaussians from two 256x256 input images in 0.015 seconds (66 fps) on an RTX 4090 GPU. This speed is approximately 5 times faster than pixelSplat and 2 times faster than MVSplat, which shows the benefits of using a standard ViT without additional geometric operations.

4.6. Real Application

NoPoSplat can be directly applied to in-the-wild unposed images, including images taken with mobile phones and frames extracted from videos generated by Sora. The results indicate potential applications of NoPoSplat in text/image to 3D scene generation pipelines.

Figure 7 shows results from in-the-wild data, demonstrating the method’s applicability to sparse image pairs from mobile phones and frames from Sora-generated videos.

5. Abalation Studies

5.1. Ablation on Output Gaussian Space

The authors compare NoPoSplat’s canonical Gaussian space prediction with the transform-then-fuse pipeline used by pose-required methods.

The transform-then-fuse method predicts Gaussians in each local camera coordinate system and then transforms them to a world coordinate system using camera poses. For a fair comparison, both methods use the same backbone and head, but differ in the prediction of Gaussian space.

Results in row (f) of Table 5 show that the pose-free canonical space prediction method outperforms the transform-then-fuse approach. Figure 8 shows that the transform-then-fuse strategy leads to ghosting artifacts in the rendering due to difficulties in aligning Gaussians from different input views.

5.2. Ablation on Camera Intrinsic Embedding

Three intrinsic encoding strategies are studied, along with a scenario where no intrinsic information is provided. The results in Table 5 row (b) and Figure 8 show that not using intrinsic encodings leads to blurry results due to scale ambiguity.

The intrinsic token embedding consistently performs the best among the three proposed methods, and is used as the default setting. The three methods compared are (b) (c) (d)

5.3. Importance of RGB Shortcut

The impact of using an RGB image shortcut to the Gaussian parameter prediction head is studied. In addition to low-resolution ViT features, RGB images are input into the Gaussian parameter prediction head.

Figure 8 shows that without the RGB shortcut, rendered images are blurry in texture-rich areas

5.4. Extend to 3 Input Views

The method is extended to use three input views to show how additional views improve performance. The third view is added between the two original input views.

Row (g) of Table 5 shows that performance significantly improves with the inclusion of an additional view

6. Closing

Reliance on Known Camera Intrinsics: NoPoSplat, like other pose-free methods, assumes that camera intrinsics are known. Although the authors note that heuristically set intrinsic parameters work well for in-the-wild images, they acknowledge that relaxing this requirement would improve the method’s robustness in real-world applications.

Non-Generative Model: As a feedforward model, NoPoSplat is not generative. This means it cannot reconstruct unseen regions of a scene with detailed geometry and texture. This limitation is apparent in the 2-view model, where areas not covered by the input views may not be accurately reconstructed. The authors suggest that incorporating additional input views could potentially mitigate this issue.

Limited Training Data: The current training data is limited to the RealEstate10K, ACID, and DL3DV datasets. This constrains the model’s ability to generalize to diverse in-the-wild scenarios. The authors propose training the model on larger, more diverse datasets in the future to address this.

Static Scenes: The method is currently limited to static scenes.

However, it points out two key problems: achieving high visual quality requires computationally expensive neural networks for training and rendering, and faster methods often sacrifice visual quality for speed. The problem is further compounded when rendering unbounded scenes at 1080p resolution, where no existing method can achieve real-time display rates.

This paper has become a hot topic in the novel-view synthesis field. It surpasses Mip-NeRF360 (2022), the SOTA (state-of-the-art) method in rendering quality for high-resolution (1920x1080) outputs, and even reduces training time compared to InstantNGP (2022), the SOTA method for speed.

In the diagram above, the values in parentheses represent rendering speed (fps). It shows how much improvement has been achieved compared to existing SOTA studies. The prospect of real-time services using NeRF has come one step closer. (Although there was previously a method called FastNeRF, capable of rendering at 200 FPS, it had slow training times and lower quality.)

The reason this paper has gained significant attention is its revolutionary fast rendering speed (exceeding approximately 100 FPS), in addition to the advantages mentioned above.

Why use 3D Gaussian ?

To introduce two methods of representing 3D models:

1. The most common methods are Meshes and Points. These are optimized for rasterization (the process of converting 3D into 2D images) based on GPU/CUDA.
2. In recent NeRF techniques, scenes are represented as continuous scenes, which are well-suited for optimization. However, during rendering, the stochastic (probability-based) sampling process requires significant computation and may produce noise.

Let’s explain this from a rendering perspective from existing NeRF:

1. Draws a ray for each pixel of the image and samples several points along the ray.
2. Calculates the color and volume density for each sampled point.
3. Summates these values along the ray to render the image.

This paper combines the advantages of the aforementioned 3D model representation methods and proposes a novel approach called 3D Gaussian.

• Supports differentiable volumetric representation,
• Allows explicit representation (unlike neural networks, which use implicit representations with hidden structures),
• Efficiently performs 2D projection of 3D models and α-blending (an additive operation for transparency values), enabling fast rendering.

Note: 2D projection and α-blending are parts of the rasterization process

From a computational perspective, the rendering operations required for 3D Gaussian Splatting are significantly lower compared to conventional NeRF.

Overview

Let’s first take a look at the overall pipeline. Look at it to get understanding the big picture.

1. Initialization: Using an SfM (Structure-from-Motion) algorithm like COLMAP, not only the camera poses but also the point cloud data can be obtained. These point clouds are used as the initial values for the 3D Gaussians.
2. Projection: The 3D Gaussians are projected onto the image plane (a plane at a distance of 1 unit along the z-axis from the camera) to form 2D Gaussians. This step is performed to update parameters by comparing with ground truth (GT) input images.
3. Differentiable Tile Rasterizer: A differentiable tile rasterization process generates the 2D Gaussians into a single image.
4. Gradient Flow: The generated image and the GT image are used to calculate the loss, and gradients are backpropagated accordingly.
5. Adaptive Density Control: Based on the gradients, the shape of the Gaussians is adjusted.

This paper review will be explained based on pseudocode. While it might seem rigid, this approach is chosen as it is the most effective way to explain unfamiliar concepts. The actual paper also feels like reading a written manual.

The overall process can be divided into three main parts:

1. The first part (red): This is the variable initialization step. Although it may appear simple from a coding perspective, it is critical as it pertains to model design.
2. The second part (blue): This structure is familiar in the ML field. It involves taking inputs, performing inference, calculating the loss, and updating parameters.
3. The third part (green): This part directly manipulates the Gaussians. At specific iterations, Gaussians are cloned, split, and removed.

Initialization

M, S, C, and A are the learning parameters. Below is the explanation of the meaning and initial values of each parameter:

$M$ (mean): represents the point cloud obtained using SfM (Structure-from-Motion). A 3D Gaussian is composed of a mean and a covariance. The points in the point cloud are used as the initial mean values of the 3D Gaussians. As a result, the number of 3D Gaussians generated is equal to the number of points in the point cloud.

$S$ (Covariance Matrix): is the covariance matrix of the 3D Gaussian, which is a $3 \times 3$ matrix. In the mathematical formulation in the paper, it is described as $\Sigma$, composed of a Scale Matrix (S) and a Rotation Matrix (R). (Note: the notation differs between the pseudocode and the equations in the paper, as shown below.)

\begin{equation}\Sigma = R SS^TR^T\end{equation}

• The design separates the scaling factor $s$ and the quaternion $q$ (one of the ways used for rotation representation) into two independent factors for optimization.
• The scaling vector $s$ contains information about scaling along the $x$, $y$, and $z$ axes.
• The quaternion representation $q$ (shape: $4 \times 1$) is converted into a rotation matrix $R$ (shape: $3 \times 3$)
• The reason for this design is to ensure that the covariance matrix is positive definite when projecting the 3D Gaussian into a 2D Gaussian for rendering. A positive definite matrix ensures that all variables have values greater than 0.

$C$ (color) represents the color value of the 3D Gaussian, and color is designed using the Spherical Harmonics (SH) function. Each 3D Gaussian is designed to find the optimal SH coefficient according to the view direction

$A$ (transparency) represents the transparency (alpha) value of the 3D Gaussian and is a single scalar value.

Differentiable 3D Gaussian Splatting

The 3D Gaussians are projected to 2D splats for rendering. . This projection is done using a viewing transformation, and a 2x2 covariance matrix $\sigma$ is obtained that has similar properties as if the method started from planar points with normals. \begin{equation}\Sigma’ = JW\Sigma W^T J^T\end{equation}

• Projective Transformation: The Jacobian matrix $J$ (a matrix of partial derivatives) for converting from the camera coordinate system to the image coordinate system.
• Viewing Transformation: The transformation matrix $W$ for converting from the world coordinate system to the camera coordinate system.
• Covariance Matrix: $\Sigma$, the covariance matrix in the world coordinate system.

Optimization

From this point, we delve into the operations executed within the train loop.

Line 1: Simply reads the target image $\hat{I}$ and its corresponding camera pose information $V$.

Line 2: The inputs— $M$(mean = $xyz$), $S$ (covariance), $C$(color), $A$ (transparency), and $V$ (camera pose)—are passed to the Rasterizer, which generates the predicted image.

Line 3: The predicted image is compared with the ground truth (GT) image to compute the loss. The loss function is designed as a combination of L1 loss and D-SSIM, with $\lambda = 0.2$. Additionally, for $M$(mean = $xyz$), the standard exponential decay scheduling is applied, similar to the Plenoxel approach.

\begin{equation}L = (1-\lambda) L_1 + \lambda L_{DSSIM}\end{equation}

Line 4: The $M$, $S$, $C$, and $A$ values are updated using the Adam optimizer.

Rasterizer

The algorithm takes as input:

• Image dimensions (w, h),
• Gaussian means and covariances in world space (μ, Σ),
• Gaussian colors and opacities (c, α),
• Camera pose for the image (V).

Cull Gaussian:

The view frustum is a 3D volume representing the visible area of the camera, and the culling process determines which 3D Gaussians fall within this volume.

This step performs frustum culling to remove 3D Gaussians that are not within the current camera’s view

The algorithm uses a 99% confidence interval of each Gaussian, keeping only the Gaussians that intersect the view frustum, and using a guard band to reject those at extreme positions. This avoids unnecessary computation on invisible or problematic Gaussians

Screen Space Gaussian

Transforms the 3D Gaussian parameters from world space to screen space, preparing them for 2D rasterization.

The 3D Gaussian means (μ) and covariances (Σ) are transformed into camera coordinates using a viewing transformation (V) and the Jacobian of the affine approximation of the projective transformation. The result, $M’$ and $S’$, represent the means and covariances in screen space. The original 3D covariance matrix (Σ) is reduced to a 2x2 covariance matrix during the projection.

This step involves matrix multiplication and affine transformations to change coordinate systems, specifically projectively transforming the 3D Gaussians into a 2D view space.

Create Tiles

This step divides the image space into smaller regions to enable parallel processing of different screen region. Creates a grid of tiles over the image, splitting the screen into 16x16 pixel tiles for parallel processing

This is a straightforward partitioning of the screen into equally sized rectangular tiles.

Duplicate With Keys

Each Gaussian in the scene ($M’$, $S’$) is duplicated for each tile it overlaps. A key is generated for each instance. This key combines the view space depth of the Gaussian and the tile ID. The key is constructed so that the lower bits represent the projected depth and the higher bits the tile ID.

The resulting list contains the Gaussian instances and their corresponding keys (L, keys).

Sort by Keys

Sorts the Gaussian instances based on the generated keys for each tile.

A fast GPU Radix Sort is used to sort the Gaussian instances (L) using the keys (keys), ensuring the Gaussian instances are ordered by depth within each tile. The Radix sort is a non-comparative integer sorting algorithm that sorts data with integer keys by grouping the keys by individual digits that share the same significant position and value. It provides a fast, parallel way of sorting.

This sorting is done once for each frame and is a global sort across all tiles.

Identify - Read Tile Ranges

Creates per-tile lists of Gaussians, by identifying start and end indices within the sorted keys array. By comparing neighboring elements of the sorted keys array, it identifies the start and end indices for each tile. The result, R, is a set of per-tile lists of Gaussians.

Read the range r for each tile in the image.

Blend In Order

Blends the colors and opacities of the sorted Gaussians for each pixel within the tile to create the final pixel color. The Gaussians in the tile’s range, as defined in range, are blended front-to-back. The color and opacity of each Gaussian that overlaps the current pixel are combined with existing pixel color and opacity using alpha-blending. If the accumulated opacity of the pixel is close to 1, the loop is terminated early.

The contribution of a Gaussian is determined by a 2D Gaussian function with covariance $Σ’$, and a learned per-point opacity.

During the rasterization process, the algorithm also ensures numerical stability by

• Skipping blending updates with alpha values lower than a small threshold.
• Clamping the accumulated alpha values.
• Stopping the blending operation early if the accumulated opacity of the pixel exceeds a threshold.

Adaptive Control of Gaussians

This is the stage where 3D Gaussians are adaptively modified to suit the scene. While the previously mentioned parameters $M, S, C, A$ are updated in each iteration, the processes in the green section are executed every 100 iterations. In this stage, 3D Gaussians undergo removal, splitting, or cloning, collectively referred to as densification.

Remove Gaussian

Gaussians with an alpha value ($\alpha$, transparency) below a specific threshold ($\varepsilon$) are removed. In the code, this threshold is set to 0.005

After removing Gaussians step, the system addresses regions where geometric features are not well-captured (=Under-reconstruction) and regions where Gaussians have modeled overly broad areas (=Over-reconstruction). Specifically:

• Both under-reconstruction and over-reconstruction regions are identified as having a high view-space positional gradient.
• If the average magnitude of the view-space positional gradient exceeds a certain threshold (set at 0.0002), the Gaussians in these regions are subjected to cloning or splitting.

Clone Gaussian

For under-reconstructed regions:

• Small Gaussians (Gaussians with low covariance) are duplicated.
• The cloned Gaussians are placed along the direction of the positional gradient.

Split Gaussian

For over-reconstructed regions:

• Large Gaussians (Gaussians with high covariance) are split into smaller Gaussians.
• A single Gaussian is divided into two Gaussians, and the scale is reduced by a factor of 1.6 (determined experimentally).
• The positions of the split Gaussians are assigned based on the probability density function of the original Gaussian.

Optimization

In the case of cloning, both the number of Gaussians and the scene’s volume increase. On the other hand, with splitting, the total volume is maintained while the number of Gaussians increases. As with other volumetric techniques, floaters appear in regions closer to the camera, manifesting as randomly scattered Gaussians.

To address this, a strategy is employed where the alpha values are reset to 0 every 3000 iterations. During the stage where $M$, $S$, $C$, and $A$ are optimized, the alpha values transition from 0 to non-zero values over 100 iterations. After these 100 iterations, the unwanted values are removed through the Remove Gaussian operation during the Gaussian densification phase.

Another benefit of this approach is that it also addresses cases where 3D Gaussians overlap. By periodically resetting the alpha values to 0, instances of large Gaussians overlapping are effectively eliminated.

The strategy of periodically resetting alpha values to 0 plays a significant role in controlling the overall number of Gaussians.

Evaluation

Mip-NeRF360 utilized 4 A100 GPUs, whereas the others used A6000 GPUs. FPS refers to the rendering speed. Compared to Instant-NGP, 3D Gaussian Splatting achieves a similar training speed but higher PSNR and, most importantly, significantly faster rendering speed.

From the perspective of rendering speed, Instant-NGP’s ~10 FPS might seem acceptable, but this speed was achieved using high-spec GPUs. If run on low-cost GPUs, stuttering could occur.

Drawbacks:

• Unlike previous methods, this approach consumes a significant amount of memory.
• During training for very large scenes, GPU memory usage reached up to 20GB.
• The source code explicitly recommends using GPUs with at least 24GB of VRAM.
• For smaller scenes, GPUs with less memory can be used.

Ablation Study

Initialization from SfM

This section addresses experiments on the initialization of 3D Gaussians using the SfM point cloud. When creating a cube three times the size of the input camera’s bounding box and sampling it uniformly, it was found that the method performed relatively well without completely failing, even in the absence of SfM points. As shown in the figure below, performance degradation is mainly observed in the background.

In areas not covered by the training views, initializing with random values results in floaters that cannot be eliminated through optimization. From another perspective, synthetic NeRF datasets do not exhibit this issue because they lack backgrounds, and the input camera pose values are well-defined.

Densification

They evaluated the densification method from the perspectives of cloning and splitting. Each method was disabled individually, while the others were optimized without modification. As seen in the figure below, splitting large Gaussians helps reconstruct the background effectively, whereas cloning small Gaussians enables faster convergence.

Limitations

Artifacts can occur in sparse scenes with insufficient input images. While anisotropic Gaussians offer many advantages, they can sometimes result in elongated artifacts or blotchy Gaussians. (This phenomenon has also been observed in other studies.) When creating large Gaussians through optimization, artifacts occasionally appear, primarily in areas that exhibit different appearances depending on the viewing pose.

• One reason for these artifacts is the trivial rejection of Gaussians during the rasterizer stage through a guard band. Using a more theoretical culling approach could mitigate these artifacts.
• Another reason lies in the simple visibility algorithm, which can lead to abrupt switching in depth/blending order of Gaussians. This issue could potentially be addressed with anti-aliasing, which has been left as a topic for future research.

The algorithm proposed in this paper currently cannot incorporate any form of regularization. Introducing regularization could help better handle unseen areas and regions prone to artifacts.

Although the same hyperparameters are used across all evaluations, initial experiments suggest that reducing the position learning rate might be necessary for convergence in very large scenes (e.g., urban datasets).

Compared to prior point-based approaches, this method is highly compact, but it still uses significantly more memory than NeRF-based solutions. Training large scenes can cause peak GPU memory usage to exceed 20 GB. However, implementing low-level optimization logic could substantially reduce this. Rendering a trained scene requires sufficient GPU memory to store the entire model, with additional memory demands for the rasterizer ranging from 30 to 5000 MB depending on the scene size and image resolution.