M2Mapping: Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems (ICRA 2025)

Created on June 11, 2026

2026   ·   mapping   neural-rendering   lidar-visual   ·   paper-review

Item Details
Authors Jianheng Liu, Chunran Zheng, Yunfei Wan, Bowen Wang, Yixi Cai, Fu Zhang (HKU MaRS Lab, The University of Hong Kong)
Venue ICRA 2025 (arXiv:2409.05310, 2024-09-09)
Links arXiv · Code

One-line Summary

M2Mapping is a mapping framework that couples the distance field built from LiDAR (NDF, structure) and the radiance field built from the camera (NeRF, appearance) through a single SDF-to-density transform. The core is a bidirectional complementarity: the LiDAR point cloud pins down the structure, and the photometric error of NeRF flows back to fill in blurry structure that LiDAR missed. As a result, it achieves high-precision watertight surface reconstruction and extrapolated rendering simultaneously, even on free-view trajectories where multi-view coverage around the region of interest is scarce.

  • The starting point is NeRF (Mildenhall 2020). It revolutionized novel view synthesis by learning a neural radiance field through volume rendering, but its density field is ambiguous and cannot define surfaces clearly.
  • Surface lineage: VolSDF (Yariv 2021), NeuS (Wang 2021), and UniSurf (Oechsle 2021) use an SDF as the structure field and reconstruct multi-view surfaces through volume rendering. This paper inherits this “volume rendering of SDF” pipeline as is.
  • Acceleration and representation lineage: Instant-NGP (Müller 2022, multi-resolution hash encoding + tiny MLP) is the backbone of this method. Neuralangelo (Li 2023, numerical-gradient curvature and progressive detail) and Mip-NeRF 360 (Barron 2022, unbounded background contraction) are also borrowed.
  • Point-cloud SDF mapping lineage (robotics): Voxblox, VDBFusion (TSDF), iSDF, SHINE-Mapping, and H2Mapping learn an SDF/NDF from LiDAR/RGBD point clouds. This paper compares against these as structure baselines.
  • Competing representation: 3D Gaussian Splatting (Kerbl 2023) is explicit, fast, and high-quality, but manifold surface representation is difficult and there is no pipeline for refining structure in reverse. This paper points to this as a weakness and chooses the SDF volume-rendering side.

To summarize its position, it is a LiDAR-visual fusion work that fills the gap between “image-only NeRF surfacing (VolSDF/NeuS)” and “point-cloud-only SDF mapping (SHINE/H2)”. The differentiator is that LiDAR depth is not used as depth supervision in 2D image space, but supervises the NDF directly in 3D space.

Problem and Motivation

Robots easily obtain multi-modal data with cameras and low-cost LiDAR, but LVI-SLAM (LVI-SAM, R3LIVE, FAST-LIVO) only outputs colored raw point clouds, so resolution, density, and accuracy are bound to the LiDAR. Digital twins require watertight surfaces and photorealistic rendering.

Explicit meshing is vulnerable to noise and registration error, and image-only NeRF surfacing requires rich multi-view coverage at every location, making it unsuitable for free-view trajectories. LiDAR provides direct samples on the surface and can thus reduce the multi-view dependence. Hence the goal is a unified framework that reconstructs appearance and structure simultaneously from “posed images + low-cost LiDAR + an arbitrary casual trajectory”.

The key difficulty is the structural visibility gap between NeRF and NDF. The LiDAR FOV and camera FOV differ and LiDAR is sparse, so there is much space visible to the camera but not captured by LiDAR.

Key Ideas and Method

Visible-aware Occupancy Map

A standard occupancy grid (free / occupied / unknown) is built by LiDAR raycasting, and image pixel rays are additionally cast to split unknown in two. Cells visible to the camera with no occupied cell in front are marked “visible unknown”. The neural field is trained only inside occupied and visible-unknown cells, so free space is skipped for efficiency, and regions visible to the camera but absent from LiDAR are filled in, enabling complete rendering. As an analogy, a highlighter marking “the camera saw this” is layered over the map drawn by LiDAR, and only places seen by at least one of the two are studied.

NDF geometry supervision: directly from the point cloud

The geometry network \(f_S(\mathbf{x}) = (\hat{s}, \hat{\beta})\) outputs both the SDF value \(\hat{s}\) and a spatially varying scale \(\hat{\beta}\) via hash encoding and a tiny MLP. For each point uniformly sampled along a LiDAR ray, the “remaining distance to the endpoint” is taken as the ray distance \(s\), and the ground-truth occupancy is obtained through the sigmoid \(\Phi\).

\[o = \Phi(-s, \hat{\beta})\]

This is trained with a binary cross-entropy loss to build the SDF. The point cloud itself is the supervising teacher of the distance field. VolSDF/NeuS learn the SDF from images only, whereas here LiDAR points supervise directly in 3D, avoiding the multi-view sparsity problem.

NeRF appearance supervision and the SDF-to-density transform

The color network \(f_C(\mathbf{x}, \mathbf{d})\) concatenates hash encoding (position) and spherical-harmonics encoding (direction) and outputs view-dependent color through an MLP. The pixel color is composed with the standard volume-rendering equation (transmittance × opacity × color) and trained with an L2 photometric loss.

The key link is the VolSDF-style transform in which the slope of the SDF along the ray direction produces the density.

\[\hat{\sigma} = \max\left(-\frac{\Phi'(-\hat{s}, \hat{\beta})}{\hat{\beta}}\left(\frac{\partial \hat{s}}{\partial \mathbf{x}}\cdot\mathbf{d}\right),\ 0\right)\]

The photometric error flows backward through this transform and volume rendering all the way to the NDF via backpropagation. That is how NeRF fills in blurry or missing structure.

Spatially varying scale: the core novelty

Previously the scale \(\beta\) was a global constant, a single learned value, or a computed value. Here \(\hat{\beta}\) is predicted per location, so the SDF becomes a stochastic distance field and \(\beta\) plays the role of the network’s “confidence”. A small \(\beta\) gives a sharp transition, i.e. high density near the surface (almost surface rendering), producing crisp detail. A large \(\beta\) gives a smooth transition that considers more points, which suits uncertain and blurry structures such as leaves. Crisp walls are rendered like a blade, fluttering leaves like fog, adapting automatically to the granularity across the scene.

Structure-aware Sampling: based on sphere tracing

NeRF’s photometry-driven sampling creates artifacts in mid-air and overfits to images. Instead, sphere tracing (jumping toward the surface using the SDF value as the radius) is borrowed for sampling, and each sphere-tracing step is used directly as a sample. Samples are dense near the surface and sparse far from it. The adaptive step is stabilized with a filtered slope \(m\) (relaxation factor \(\gamma = 0.7\)).

\[\delta_i = \lvert s_i\rvert\cdot\frac{2}{1-m}\]

Tracing proceeds until transmittance \(T < 0.001\), and free space is skipped using the visible-aware map. Since sphere tracing naturally validates the slope, the slope needed for the SDF-to-density transform is obtained without expensive analytic/numerical gradient computation.

Training regularization and practical tricks

  • Eikonal loss (\(\lVert\nabla f\rVert = 1\), enforcing the distance-field property) and curvature loss (preventing overfitting and zero-everywhere solutions, with progressively shrinking numerical differentiation).
  • Outlier removal: periodically, the inferred SDF of LiDAR points is checked and points more than \(\epsilon\) away from 0 are removed. Dynamic objects (people, cars) are erased from the static background field and also removed from rendering.
  • Directional embedding scheduler: the spherical-harmonics degree is increased progressively from 0 (view-independent diffuse) to 4 to mitigate color degradation in extrapolated views.
  • Scene contraction (Mip-NeRF 360 style): the unbounded background is contracted outside the boundary and painted as a false surface.

The combined loss is as follows. \(\lambda_{rgb}\) increases linearly from \(10^{-4}\) to 10, with \(\lambda_{eik} = 0.1\) and \(\lambda_{curv} = 5\times10^{-4}\).

\[L = L_{sdf} + \lambda_{rgb}L_{rgb} + \lambda_{eik}L_{eik} + \lambda_{curv}L_{curv}\]

Experiments and Results

The pipeline builds a visible-aware occupancy map (raycasting) from posed images + LiDAR point cloud, trains the NDF from the point cloud, lets the NDF provide structure-aware sampling and density for NeRF volume rendering, and refines the NDF by backpropagating the photometric error. The implementation uses Instant-NGP-style multi-resolution hash encoding (16 levels, \(2^5\) to \(2^{21}\)), geometry/appearance MLPs each 3 layers of width 64, and LibTorch + CUDA. Rays 8192, point batch fixed at 256000, 20000 iterations (outlier removal every 2000), on an Intel i7-13700K + a single NVIDIA RTX 4090 GPU.

The baselines are the structure methods VDBFusion, iSDF, SHINE-Mapping; the appearance methods InstantNGP; and the depth-assisted appearance methods 3DGS† (dense point-cloud initialization), H2Mapping, and MonoGS. The datasets are Replica (indoor RGBD simulation, with a separately generated extrapolation evaluation set) and FAST-LIVO2 (real-world camera + LiDAR, three trajectory types FF/OC/Free: Campus/Sculpture/Culture/Drive). The metrics are Chamfer-L1 (cm) and F-Score (under 2cm) for geometry, and SSIM, PSNR, LPIPS for rendering (interpolation I and extrapolation E reported separately).

The three key results are as follows.

  1. Overwhelming advantage in extrapolated rendering. The average Replica PSNR(E) of 38.23 is 4 to 6dB higher than the runners-up MonoGS (32.59) and InstantNGP (34.34). LPIPS(E) of 0.078 is the lowest (MonoGS 0.120). Structure and texture are preserved even from viewpoints not seen in training (ceilings, etc.).
  2. Surface reconstruction SOTA. The average Replica Chamfer-L1 of 0.499cm is the lowest among all baselines (H2Mapping 0.580, VDBFusion 0.626), and the F-Score of 98.673% is the highest. It captures thin-object detail while maintaining smoothness.
  3. Surpasses 3DGS on free-view real-world data. The average PSNR on the FAST-LIVO2 dataset is 26.19, above 3DGS† (25.76) and InstantNGP (24.24). The gap is especially large on free-view sequences with insufficient co-visibility (Culture PSNR 24.70 vs 3DGS 21.76). Interpolated rendering is competitive with 3DGS (average SSIM 0.764 vs 0.780).
Metric (Replica) M2Mapping Comparison
PSNR (extrapolation E) 38.23 InstantNGP 34.34, MonoGS 32.59
LPIPS (extrapolation E) 0.078 MonoGS 0.120
PSNR (interpolation I) 41.15 3DGS† 41.01, MonoGS 40.03
Chamfer-L1 (cm) 0.499 H2Mapping 0.580, VDBFusion 0.626
F-Score (%) 98.673 Highest

For interpolated (I) rendering, the Replica PSNR(I) of 41.15 is on par with 3DGS† and MonoGS, and LPIPS(I) of 0.050 is among the lowest. The consistent picture is: on par for interpolation, superior for extrapolation.

The ablation showed the following. The spatially varying \(\beta\) balances the over-smoothing of a large \(\beta\) and the noise overfitting of a small \(\beta\) (room-2 with 2cm depth noise). Structure-aware sampling concentrates on the surface without missing blurry structure, compared to occupancy/PDF sampling and surface rendering. Outlier removal was verified to remove dynamic objects. The directional scheduler improves extrapolated color consistency with almost no effect on interpolation (wo dir. PSNR 26.24 ≈ 26.19). Visual-aided structure showed that NeRF completes the structure from sparse point clouds.

Efficiency is about 20 minutes of training per Replica scene and about 0.07 seconds (13Hz) per 1200×680 render. The bottleneck is the imbalance of sphere-tracing steps, where unconverged rays delay the batch. Ray-oriented rendering is proposed as a future improvement.

Checking the claimed contributions: the open-source framework claim is valid given the GPL-2.0 release and the experiments. The efficiency and completeness claims of the visible-aware occupancy prior are clearly motivated, but the acceleration ablation of the visible-aware map alone is weak in the main text and relies on supplementary material. The spatially varying scale NDF and the sphere-tracing structure-aware sampling are the core novelties, most robustly proven by qualitative and quantitative ablation. Honestly reporting the sphere-tracing bottleneck raises credibility, but a considerable part of the content is pushed into supplementary material, so the supplement is needed for standalone reproduction.

Conclusion and Significance

M2Mapping unifies the strengths of image-only surfacing (VolSDF/NeuS) and point-cloud-only SDF mapping (SHINE/H2) on a single SDF-to-density transform, presenting a practical path to elevate LiDAR-visual SLAM output (colored point clouds) into watertight meshes and photorealistic NVS. The core insight is that “structure (LiDAR) and appearance (camera) refine each other when they share the same SDF scale field”. It greatly improved extrapolated-rendering robustness under the realistic data-collection conditions of real robots, namely free-view, casual trajectories.

From a robotics practice perspective, the takeaways are as follows.

  • Where it can be used: it suits a post-processing stage that takes the posed images + colored point cloud output by a LIVO such as FAST-LIVO2 and produces a watertight mesh and free-view rendering for digital twins. The practical strength is that extrapolated views do not collapse even for data from a single casual pass through a multi-floor indoor building or unstructured outdoor terrain. On a consumer-grade single GPU (RTX 4090), training takes about 20 minutes per scene and rendering runs at about 13Hz.
  • Simulation asset generation: since a watertight surface and photorealistic appearance are obtained together, it can be considered for scanning real sites and transferring them into environment assets such as a quadruped simulation testbed. Note that it is an offline training approach, not real-time mapping.
  • Adoption conditions: the code depends on C++/CUDA/LibTorch plus Open3D and ROS Noetic (optional for visualization), so the build is non-trivial. Whether pretrained checkpoints are provided has not been confirmed; per-scene from-scratch training is the default. Quality depends on LiDAR-camera extrinsic calibration and accurate poses (the paper uses FAST-LIVO2 results as ground-truth poses), and robustness to pose noise is unverified. Reproduction difficulty is moderately high.
  • Limitations: the efficiency bottleneck of sphere-tracing step imbalance, the static-scene assumption (dynamic objects are only erased, not modeled), and the trade-off of being slightly behind 3DGS on high-frequency texture interpolation. The GPL-2.0 license requires a copyleft review for commercial integration. Combining the explicit appearance of 3DGS with this method’s SDF structure, as well as city-scale scalability and memory, remain open questions.

References

  • Paper: arXiv:2409.05310
  • Code: hku-mars/M2Mapping (GPL-2.0)
  • VolSDF: Volume Rendering of Neural Implicit Surfaces (NeurIPS 2021) — direct parent of the SDF-to-density transform
  • NeuS (NeurIPS 2021) — single learned scale predecessor, contrasted with the spatially varying scale
  • Instant-NGP (2022) — backbone architecture
  • SHINE-Mapping (2022) — direct predecessor of BCE-based geometry supervision
  • 3D Gaussian Splatting (SIGGRAPH 2023) — the main competing representation