DN-Splatter + AGS-Mesh This repo implements research papers (DN-Splatter and AGS-Mesh) for depth and normal supervision of Gaussian splatting models for improved novel-view synthesis using smartphone data (iPhones) and mesh reconstruction. Pipelines: DN-Splatter AGS-Mesh Quick demo: https://github.com/maturk/dn-splatter/assets/30566358/9b3ffe9d-5fe9-4b8c-8426-d578bf877a35 Table of Contents Installation Usage Mesh…
This repo implements research papers (DN-Splatter and AGS-Mesh) for depth and normal supervision of Gaussian splatting models for improved novel-view synthesis using smartphone data (iPhones) and mesh reconstruction.
29.11.2024: We release AGS-Mesh method, which improves mesh reconstruction using a novel depth and normal filtering strategy, and octree-based isosurface extraction method. We release code for Splatfacto based AGS-Mesh model and 2DGS based AGS-Mesh model. We release AGS-Mesh plug in code for both DN-Splatter and Inria 2DGS version. Please switch to AGS-Mesh-2dgs for Inria 2DGS version or open the AGS-Mesh for more details. 📣
17.10.2024: FusionSense improves DN-Splatter in sparse settings for robotic tactile applications! 🚀
04.09.2024: Support Open3d TSDF to extract mesh, support Patch-based Depth Correlation Loss from SparseGS for monodepth supervision, support visualizing normal estimates from the Gaussian geometry and estimated surface normal from depths, support colmap SFM point cloud initialization for MuSHRoom dataset.
14.06.2024: Support gsplat v1.0.0 🚀. Faster training and better memory consumption. Training with --pipeline.model.predict_normals is about 20% slower than without.
16.04.2024: Support for DSINE monocular normal supervision.
20.04.2024: dn-splatter-big: a variant featuring less aggressive Gaussian culling threshold, which leads to an increased number of Gaussians in the optimized scene. On certain datasets, this can lead to better novel-view synthesis results.
Installation
Method 1. Using Conda and Pip
Follow installation instructions for [Nerfstudio](https://docs.nerf.studio/quickstart/installation.html). This repo is compatible with a `nerfstudio` conda environment.
Method 2. Using Pixi
Download the [pixi package manager](https://pixi.sh/latest/), this will manage the installation of cuda/pytorch/nerfstudio for you
Clone and install DN-Splatter
git clone https://github.com/maturk/dn-splatter
cd dn_splatter/
pixi install
To run an example
pixi run example
To activate conda enviroment
pixi shell
Data Preparation Scripts
Generate pseudo ground truth normal maps
The `dn-splatter` model's predicted normals can be supervised with the gradient of rendered depth maps or by external monocular normal estimates using the flag `--pipeline.model.normal-supervision (mono/depth)`. To train with monocular normals, you need to use an external network to predict them.
We support generating low and hd monocular normal estimates from a pretrained omnimodel and from DSINE.
Then generate normal maps using the following command:
python dn_splatter/scripts/normals_from_pretrain.py
--data-dir (path to data root which either has a transforms.json or images/ folder)
--img-dir-name (if transforms.json file is not found, use images in this folder (default /images))
--resolution {low, hd} (low default and reccommended)
We highly reccommend using low res normal maps, since generating HD versions from omnidata (that match the dataset image size) is very time consuming.
2. DSINE normals:
To generate normals from DSINE, run the following command:
Default save path of generated normals is data_root/normals_from_pretrain
NOTE: different monocular networks can use varying camera coordinate systems for saving/visualizing predicted normals in the camera frame. We support both OpenGL and OpenCV coordinate systems. Each dataparser has a flag --normal-format [omnidata/dsine] to distinguish between them. We render normals into the camera frame according to OpenCV color coding which is similar to Open3D. Some software might have different conventions. Omnidata normals are stored in OpenGL coordinates, but we convert them to OpenCV for consistency across the repo.
Convert dataset to COLMAP format
If your dataset has no camera pose information, you can generate poses using COLMAP.
If your dataset has no sensor depths, and you have a COLMAP processed dataset, we provide a script to generate scale aligned monocular depth estimates. Scale alignment refers to solving for the scale ambiquity between estimated monocular depths and the scale of your input COLMAP poses.
This script generates sfm_depths/ and mono_depth/ directories in the data root:
TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int64'>, <class 'numpy.int64'>]
Downgrading from Torch 2.1.2 to 2.0.1 solves the issue.
Generate only mono depth estimates skipping SfM alignment
To skip SfM alignment and just render monocular depths for your dataset, use the following script:
The AGS-Mesh method filters inconsistent depth estimates using a depth->normal consistency check. To generate the depth masks, use the following script:
where --data-dir is the root dataset path (e.g. room_datasets/vr_room/iphone/long_capture) and --transforms-name is the name of the json file containing dataset data (e.g. transformations_colmap.json for the mushroom dataset) and --normal-format refers to what model was used to generate normal maps. The omnidata and dsine networks have different coordinate systems, so this has to be carefully dealt with.
Usage
This repo registers two models called dn-splatter and ags-mesh with various additional options:
Whether to use monocular or rendered depths for normal supervision. ‘mono’ default.
–pipeline.model.two-d-gaussians (True/False)
Encourage 2D gaussians
Please check the dn_model.py for a full list of supported configs (some are only experimental).
Model overviews
dn-splatter is the baseline model that implements depth and normal supervision within 3DGS. ags-mesh improves dn-splatter with a novel depth and normal filtering strategy. Please see dn-splatter and ags-mesh research papers for more details regarding model architectures. We provide a 2DGS based version of AGS-Mesh in the ags-mesh-2dgs branch and the ags_mesh repo.
Recommended settings:
For larger indoor captures with sensor depth data (e.g. MuSHRoom / ScanNet++ datasets):
# to run the dn-splatter model
ns-train dn-splatter --pipeline.model.use-depth-loss True \
--pipeline.model.depth-lambda 0.2 \
--pipeline.model.use-normal-loss True \
--pipeline.model.use-normal-tv-loss True \
--pipeline.model.normal-supervision (mono/depth)
<dataparser_name (normal-nerfstudio, mushroom, scannetpp, etc)> --data PATH_TO_DATA
# to run the ags-mesh model
ns-train ags-mesh --pipeline.model.use-depth-loss True \
--pipeline.model.depth-lambda 0.2 \
--pipeline.model.use-normal-loss True \
--pipeline.model.normal-supervision (mono/depth)
<dataparser_name (normal-nerfstudio, mushroom, scannetpp, etc)> --data PATH_TO_DATA --load-depth-confidence-masks True
dn-splatter-big:
We also provide a dn-splatter-big variant that increases the number of Gaussians in the scene which may enhance the quality of novel-view synthesis. This increases training time and hardware requirements. Simply replace the dn-splatter keyword with dn-splatter-big in the above commands.
Supported Depth Losses
To train with a specific depth loss, use the flag: --pipeline.model.depth-loss-type DepthLossType where DepthLossType is one of ["MSE", "LogL1", "HuberL1", "L1", "EdgeAwareLogL1", "PearsonDepth"]
For sensor depth supervision we reccommend EdgeAwareLogL1 loss. For monocular depth supervision, we recommend the relative Pearson correlation loss PearsonDepth.
Export a mesh with the gs-mesh --help command. The following mesh exporters are supported.
gs-mesh
Description
Requires normals?
gs-mesh dn
Backproject depth and normal maps to Poisson
Yes
gs-mesh tsdf
TSDF Fusion algorithm
No
gs-mesh o3dtsdf
TSDF Fusion algorithm used in 2DGS paper
No
gs-mesh sugar-coarse
Level set extractor from SuGaR (Sec 4.2 from the paper)
Both
gs-mesh gaussians
Use Gaussian xyzs and normals to Poisson
Yes
Use the --help command with each method to see more useful options and settings.
For very small object captures, TSDF works well with 0.004 voxel size and 0.02 SDF trunc distance.
Custom RGB-D Smartphone (Android/iPhone) Data
For casually captured RGB-D streams, consider using SpectacularAI SDK for iPhone/Android or Oak/RealSense/Kinect sensor streams. For LiDaR enabled smartphones, download the app from the Apple/Play store and capture your data.
Once you have gathered the data, process the inputs into a Nerfstudio suitable format (calculate VIO poses and create a transforms.json file with poses and depth frames):
For other custom datasets, use the Nerfstudio conventions and train with the above command.
Datasets
Other preprocessed datasets are supported by dataparsers with the keywords mushroom, replica, scannetpp, nrgbd, dtu, coolermap.
To train with a specific dataset, the command follows the sequence:
Dataparsers have their own options, to see the full list use ns-train dn-splatter [mushroom/replica/scannet/nrgbd/dtu/coolermap] --help. Please use the --help flag to see all supported commands for each dataset (they are all a bit unique in how they process data). Some useful ones are:
--depth-mode : ["sensor","mono","all", "none"] determines what depths to load.
--load-normals : [True/False] whether to load normals or not.
--normal-format : ["omndiata", "dsine"] What monocular normal was used to generate normal maps (there are slight coordinate differences).
--load-pcd-normals : [True/False] initialise gaussian scales/rotations based on estimated SfM normals.
--load-depth-confidence-masks : [True/False] used for the filtering strategy proposed in AGS-Mesh.
Supported dataparsers:
COLMAP datasets
For arbitrary COLMAP processed datasets, we expect the following directory structure
Please see dn_splatter/eval/eval_instructions.md for more details.
To run DN-Splatter on an entire dataset of sequences (potentially in parallel with multi-GPU cluster), you can use the dn_splatter/eval/batch_run.py script.
For evaluating rgb, depth, and pointcloud metrics (optional), run the following command:
–transformation_file path is the path of the transform.json generated when training scannetpp here.
–dataset_path is the path to the dataset folder, e.g. datasets/scannetpp/data/b20a261fdf/iphone
Acknowledgements
I want to thank Tobias Fischer, Songyou Peng and Philipp Lindenberger for their fruitful discussions and guidance, especially concerning mesh reconstruction. This project is built on various open-source software, and I want to thank the Nerfstudio team for their great efforts maintaining and extending a large project allowing for these kinds of extensions to exist.
Citation
If you find this work useful in your research, consider citing DN-Splatter:
@InProceedings{turkulainen2024dnsplatter,
title={DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing},
author={Matias Turkulainen and Xuqian Ren and Iaroslav Melekhov and Otto Seiskari and Esa Rahtu and Juho Kannala},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year={2025}
}
and AGS-Mesh:
@InProceedings{ren2024agsmesh,
title={AGS-Mesh: Adaptive Gaussian Splatting and Meshing with Geometric Priors for Indoor Room Reconstruction Using Smartphones},
author={Xuqian Ren and Matias Turkulainen and Jiepeng Wang and Otto Seiskari and Iaroslav Melekhov and Juho Kannala and Esa Rahtu},
booktitle={International Conference on 3D Vision (3DV)}
year={2025},
archiveprefix={arxiv},
eprint={2411.19271},
}
Contributing
We welcome any bug fixes, issues/comments, or contributions. Also, if you have any suggestions or improvements, let me know!
I want to thank Pablo Vela for contributing the DSINE normal predictor into this project and allowing for an alternative (much easier) installation with Pixi.
评论