Computational Robotics Group, Harvard RSS 2025 Best Systems Paper Award Awarded to our conference version, XM: Building Rome with Convex Optimization →

XM4D Non-Rigid ConveX Structure from Motion with Embedded Deformation Graphs for 4D Reconstruction

Haoyu Han1 Jiahui Lei2 Jack Benarroch Jedlicki1 Heng Yang1

1Harvard University 2UC Berkeley

Paper Code BibTeX

Overview · drag the slider or let it play

Certified geometry first, rendering second

From (a) 2D tracks and monocular depth, a solver with certified global optimality recovers (b) the cameras, static geometry, and dynamic motion, which initialize (c) 4D Gaussian splatting for rendering.

Step 1 of 4 · Input

A casual monocular video

An off-the-shelf point tracker gives 2D tracks, and a monocular depth model gives per-frame depth. Masks split the tracks into static and dynamic ones.

Step 2 of 4 · Preprocess

Lift every track to 3D

Each 2D observation is lifted to a 3D point with its predicted depth. The depth of each frame is only known up to scale, so the solver also estimates one scale per frame. The clip lifts one frame this way: every pixel slides along its camera ray to its depth.

Step 3 of 4 · Solve

Certified cameras, then certified motion

Our GPU solver certifies a scaled bundle adjustment over the static tracks as globally optimal, then the deformation graph's motion. The clip replays it: each frame's lifted depth falls into place as the cameras settle over eight iterations, then the dynamic tracks play.

Step 4 of 4 · Render

4D Gaussian splatting

The certified cameras and graph motion initialize dynamic Gaussian splatting, which refines appearance and geometry for novel-view rendering. The clip is that model rendered from the panel's viewpoint, with the same tracked points.

XM4D splits 4D reconstruction into a static and a dynamic stage, and solves each one with a certifiably globally optimal GPU solver before any photometric training.

Inside the solver

Both stages are solved by the same GPU Burer–Monteiro solver. These clips replay its iterates, from a copy of the solver that saved every one.

Static solve: scaled bundle adjustment over the static tracks. Every frame’s lifted depth is placed by that iterate’s camera and depth scale, and comes into register as the cameras settle.

Deformation graph: one segment’s rotation-only problem. Each node’s translations follow from its rotations in closed form. The clip starts from zero translation, with every node at its rest position, and from identity rotations, shown as the faint frames. The inset plots the cost above the optimum.

Static solve 50 frames, 8 iterations; the walkthrough’s horse.
Deformation graph, one segment Frames 0–10, 125 nodes, 33 iterations. The rank-3 solution fails the certificate, so the solver lifts to rank 4, where it certifies the optimum.

Applications

XM4D’s motion is explicit: a graph of nodes, each with a position and a rotation per frame, that carries the dynamic Gaussians. The reconstruction can therefore be edited and re-animated. A motion can drive a robot, parts can be posed by hand, and the moving object can be left out. The static solver on its own also tracks a known object in 6D.

Open and close each door yourself

For the fridge, the graph is one rigid node per door. Each door’s hinge, an axis and a point on it, is fitted to its solved poses (within a few millimetres), and the doors’ Gaussians are its part. Drag a slider to turn that door about its hinge, including combinations of door angles the video never showed.

Loading…

Drag to orbit · right-drag or shift-drag to pan · scroll to zoom

Doors
Show
View
Demo: each door in turn

Remove the moving object

Leaving out the dynamic Gaussians and rendering every frame from its own camera removes the car from the video. The road behind it comes from the frames where it was seen.

DAVIS car-roundabout, frames 36–74

How it works

Lift tracks with predicted depth

Point trackers and a monocular depth model give 2D tracks and per-frame depth. Every tracked point is lifted to 3D, and the tracks are split into static and dynamic ones.

Certified cameras

A scaled bundle adjustment over camera poses, per-frame depth scales, and static landmarks reduces to a QCQP over scaled rotations. Its SDP relaxation is solved by a GPU Riemannian staircase that returns a certificate of global optimality.

Certified non-rigid motion

Dynamic motion is an embedded deformation graph whose node transformations are expressed relative to one rest frame. Tracking, rigidity, and smoothness then reduce to a rotation-only QCQP that the same certified solver handles.

4D Gaussian splatting

The certified cameras and graph motion initialize dynamic Gaussian splatting, which refines appearance and geometry for novel-view rendering.

From XM to XM4D

XM4D is the journal extension of XM, our RSS 2025 solver for static structure from motion. It keeps XM's certifiable optimization and carries it over to dynamic scenes.

RSS 2025 Best Systems Paper Award

XM: Building Rome with Convex Optimization

  • Scaled bundle adjustment that lifts 2D keypoints to 3D with learned depth.
  • A tight SDP relaxation, solved by a CUDA Burer–Monteiro solver with a certificate of global optimality.
  • Static scenes, scaling to 10,155 frames solved in about one hour.
Project page Paper Code

This work · journal extension

XM4D: what is new

  • A uniform sphere constraint on the depth scales replaces XM's first-frame anchor.
  • A dynamic stage: an embedded deformation graph whose tracking, rigidity, and smoothness terms reduce to a rotation-only QCQP, handled by the same certified solver.
  • The certified cameras and motion initialize 4D Gaussian splatting for novel-view rendering.

Abstract

Accurate 4D reconstruction remains challenging: jointly estimating camera motion, scene geometry, and non-rigid deformation entails highly nonconvex optimization, making global optimality difficult to guarantee. We present XM4D, a framework for accurate and efficient 4D reconstruction built upon certifiably globally optimal geometric optimization. By separating camera motion estimation from non-rigid deformation, we formulate both stages as structured quadratically constrained quadratic programs (QCQP) that admit semidefinite programming (SDP) relaxations and verifiable global optimality certificates of the resulting SDPs. For camera motion estimation, we lift static point tracks into 3D using foundational depth prediction models and formulate a scaled bundle adjustment problem that jointly estimates camera poses, depth scales, and static landmarks. By analytically eliminating translations and landmarks, we reduce the problem to a QCQP over scaled rotations. For non-rigid reconstruction, we represent dynamic motion using an embedded deformation graph (EDG) and show that its tracking, rigidity, and temporal regularization objectives similarly reduce to a rotation-only QCQP. Both problems admit SDP relaxations that we solve efficiently with a GPU-accelerated Burer–Monteiro solver based on Riemannian optimization. The recovered geometry further initializes dynamic Gaussian splatting for photometric refinement. Extensive experiments across seven benchmarks demonstrate accurate camera motion estimation on both static and dynamic scenes, efficient optimization that scales to thousands of frames, and state-of-the-art novel-view rendering quality.

Citation

@article{han2026xm4d,
  title   = {{XM4D}: Non-Rigid Convex Structure from Motion with Embedded
             Deformation Graphs for {4D} Reconstruction},
  author  = {Han, Haoyu and Lei, Jiahui and Benarroch Jedlicki, Jack and Yang, Heng},
  year    = {2026}
}

If you use the static solver, please also cite XM:

@inproceedings{han2025xm,
  title     = {Building Rome with Convex Optimization},
  author    = {Han, Haoyu and Yang, Heng},
  booktitle = {Robotics: Science and Systems (RSS)},
  year      = {2025}
}