Geometry-Preserving in 3D Gaussian Splatting for LiDAR-Camera Extrinsic Calibration

How 3DGS Calibrates a LiDAR–Camera Pair

Gaussians are initialized on LiDAR points and rendered into the camera view, so the extrinsic $\mathbf{T}_{cl}$ can be optimized by minimizing image-domain losses. Two losses do the work.

Photometric Alignment  $\mathcal{L}_{pho}$

Aligns the extrinsics by matching the rendered image to the observed one. The signal reaches $\mathbf{T}_{cl}$ through the learned Gaussian color.

Cross-view Consistency  $\mathcal{L}_{rep}$

Warps pixels into an adjacent frame with the rendered depth and compares ground-truth intensities. The signal is mediated by depth, not color.

Both losses reach the extrinsics through the renderer. The Jacobian $\partial\Phi(\mathcal{G},\mathbf{T}_{cw})/\partial\mathbf{T}_{cl}$ depends on the current proxy state $\mathcal{G}$ — so every extrinsic update is only as trustworthy as the geometric fidelity of the Gaussian proxy.

GeoP-Calib

Two components that keep the Gaussian proxy metrically faithful while the extrinsics are still being refined. They cooperate but stay gradient-decoupled: DDA governs where the geometry may sit, GD decides what is allowed to move it.

Dense Depth Anchoring (DDA)

A single scan leaves most of the image unconstrained, so we accumulate every LiDAR observation into a global cloud $\mathcal{P}_{global}=\bigcup_t (\mathbf{T}^t_{lw})^{-1}\mathcal{P}^t$ and project it back into the current view. The resulting dense anchor $D^{dense}_l$ covers far more of the image than one scan and pins the Gaussian spatial parameters $(\mu,\Sigma)$ even where the current frame has no returns.

Volumetric Soft Mask (VSM)

Accumulation introduces occlusion ambiguity — points seen from a distant viewpoint land on foreground objects. We use the rendered depth itself as a continuous visibility prior, down-weighting anything that sits far behind the Gaussian surface:

$W_{vis}(\mathbf{p}) = \sigma\!\left(\beta \cdot \big(D_{rend}(\mathbf{p})(1+\tau) - D^{dense}_l(\mathbf{p})\big)\right)$

Gradient Decoupling (GD)

We apply a stop-gradient to the spatial parameters inside the color rendering path, so texture-driven residuals can no longer deform the geometry:

$\mathcal{L}^{decoupled}_{pho} = \mathcal{D}_{photo}\Big(I^t,\, \Phi_{color}(\text{sg}[\mu,\Sigma],\alpha,c,\mathbf{T}_{cl}\mathbf{T}^t_{lw})\Big)$

Crucially, GD is applied only to $\mathcal{L}_{pho}$. The reprojection loss keeps its gradients to $(\mu,\Sigma)$: it compares ground-truth intensities through depth-induced correspondences without ever touching the learned Gaussian color, so it encourages view-consistent structure rather than per-view texture fitting.

$\mathcal{L}^{GeoP}_{total} = \mathcal{L}^{decoupled}_{pho} + \lambda_{rep}\mathcal{L}_{rep} + \lambda^{sparse}_{depth}\mathcal{L}^{sparse}_{depth} + \lambda^{dense}_{depth}\mathcal{L}^{dense}_{depth}$

Calibration in Motion

Every sequence optimized from a perturbed initialization. The estimated camera pose converges onto the ground truth, the projected LiDAR locks onto the image, and the rotation and translation errors fall in step. Each clip is a 5 × 3 grid — full screen is worth it.