🚀 NeuralGS: Bridging Neural Fields and 3D Gaussian Splatting for Compact 3D Representation

arXiv Code(coming soon)

Zhenyu Tang1*, Chaoran Feng1*, Xinhua Cheng1, Wangbo Yu1, Junwu Zhang1,
Yuan Liu2†, Xiaoxiao Long2, Wenping Wang3, Li Yuan 1†,
1Peking University
2Hong Kong University of Science and Technology
3Texas A&M University
*These authors contributed equally to this work.
†Corresponding author.

We propose NeuralGS, a novel framework that effectively adopts the neural field representation to encode the attributes of 3D Gaussians with multiple tiny MLPs, only requiring a small model size even for a large-scale scene.

📌 Abstract


Recently, 3D Gaussian Splatting (3DGS) has gained popularity, demonstrating superior quality and rendering speed in novel view synthesis. However, 3DGS requires millions of 3D Gaussians, each with extensive associated attributes, resulting in significant storage and transmission costs. In contrast, neural fields like NeRF can represent complex 3D scenes with Multi-Layer Perceptron (MLP) neural networks using only a few megabytes. In this paper, we present a novel Gaussian compression method NeuralGS that effectively adopts the neural field representation to encode the attributes of 3D Gaussians with MLPs, requiring a small storage size even for a large-scale scene.However, naively fitting the Gaussian attributes with an MLP network leads to severely degenerated quality. To address this, we adopt a clustering strategy and fit the Gaussians with multiple tiny MLPs for different clusters, based on importance scores of Gaussians as fitting weights.We validate our approach on multiple datasets, achieving a 38x average model size reduction without harming the visual quality.

💡 Compression Method

We propose a simple yet effective framework NeuralGS, which adopts the neural field representation to encode the attributes of 3D Gaussians with MLPs to enable a compact 3D representation. Specifically, as shown in Figure 1, we use a designed criterion to assess the importance of each Gaussian, allowing us to prune Gaussians that have minimal impact on renderings. To reduce variations among Gaussians, we cluster these 3D Gaussians based on their attributes, ensuring similarity within each cluster. Then, each cluster is then assigned a tiny MLP that fits the attributes of its 3D Gaussians. Given the varying contributions of each Gaussian to the renderings, we apply Gaussian’s importances as the fitting weights in the MLP fitting. We also incorporate a fine-tuning stage along with frequency loss to restore quality and preserve high-frequency details. We address the storage and rendering issue of 3D Gaussian Splatting (3DGS) by compressing the reconstructed scene parameters and rendering the compressed representation via GPU rasterization. To compress the scenes, we first analyze its components and observe that the Spherical Harmonics (SH) coefficients and the multivariate Gaussian parameters take up the majority of storage space and are highly redundant. Our compression pipeline consists of three steps:


Compression Pipeline
Figure 1: Overview of NeuralGS.

🔗 Results

We evaluate our method on four datasets for comparison on NVIDIA A100 GPUs.


Qualitative Comparisons

Compression Pipeline
Figure 2: Qualitative comprison on the mentioned four datasets.

Quantitative Comparisons

Table 1. Quantitative results evaluated on Mip-NeRF 360, Tanks&Temples, and Deep Blending datasets. We highlight the best-performing results in red and the second-best results in yellow for all compression methods
Compression Pipeline

Table 2. Quantitative results of the proposed method evaluated on the NeRF-Synthetic dataset. We highlight the best-performing results in red and the second-best results in yellow for all compression methods.
Table 2

BibTeX

@InProceedings{neuralgs,
                author    = {Zhenyu Tang, Chaoran Feng, Xinhua Cheng, Wangbo Yu, Junwu Zhang, Yuan Liu, Xiaoxiao Long, Wangping Wang, Li Yuan},
                title     = {NeuralGS: Bridging Neural Fields and 3D Gaussian Splatting for Compact 3D Representation},
                booktitle = {},
                month     = {February},
                year      = {2025},
                pages     = {}
            }