00original
Harvard Computational Robotics Group
School of Engineering and Applied Sciences, Harvard University
School of Engineering and Applied Sciences, Harvard University
Preprint
2026
2026
SceneAgent
An agentic pipeline for converting 3D captures to simulation environments with predictive physics for policy evaluation, online planning, and policy training
1 Harvard Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University
* Equal contribution
* Equal contribution
[coming soon]
SceneAgent converts 3d captures of real-world scenes into simulation environments that can be used for policy evaluation, online planning, and policy training or finetuning. The pipeline primarily focuses on converting environments captured with 3d Gaussian Splatting (3DGS) but also supports scenes captured with photogrammetry and LiDAR. It incorporates 3d semantic features, object segmentation, predictive per-gaussian physics properties, object decomposition, deformability, and articulation. To increase policy generalization, it is also able to generate “digital sisters” or similar versions of each object within an environment. We are still evaluating how well policies trained in this environment perform in the real world, and initial results show promising success rates similar to or improving on other recent papers such as SimFoundry and PolaRiS.
From Real-world Scenes into Simulation
Our pipeline supports converting 3D capture data from 3DGS, photogrammetry, and LiDAR into usable scenes that can be used across a wide range of simulators, such as Isaac Lab, MuJoCo, and Unreal Engine.
We initially infer semantic features for each 3d Gaussian and save them in a codebook for rapid lookup, inspired by the method from LangSplat. We then segment the foreground objects in the scene from the background and crop their 3d Gaussians from the background. We then predictively infill these cropped regions. For each object, we predict a physics material with a combination of semantic features, inference against a per-gaussian predictive physics model, and use VLM to estimate features such as friction coefficient, rigidity, mass, and density.
From this data, we then bake a physics material for each object and then decompose that object into smaller pieces. We then add articulations to the objects as necessary so they can perform actions as they would in real life. Finally, we generate versions of each object with slight variations–instead of digital twins, we term these digital sisters. We then use a VLM to inspect and review each of these steps as well as the transform of the converted objects in the scene.
Clamp
SceneAgent
Processes each 3D object for predictive physics, decomposition, and object properties
013DGS
Convert input RGB or other 3D data
02Semantic features
Use GroundingDINO + SAM to assign semantic features to each 3D Gaussian
03Predictive physics
Generate predictive per-Gaussian physics priors to bake physics material
04Object decomposition
Decompose each object into individual parts as needed
05Articulation
Articulate any movable parts within each object
06Object sisters
Generated object variations with geometric or visual differences
07 Package scene in Universal Scene Description format
Share usable variations of the full scene as USDZ for NVIDIA Isaac Lab, MuJoCo, Unreal Engine, and other simulators.
1.2
08 Interactive viewer
Figure 1 Pipeline architecture. We support converting input RGB image sequences (from video or as images), pre-existing 3DGS or photogrammetry assets, and LiDAR scans. Output is in Universal Scene Description format (USDZ) and able to be used in a wide range of simulation tools such as Isaac Lab, MuJoCo, and Unreal Engine.
1.3
After we convert the scene, we created an interactive viewer built with Three.js to assist in positioning the robot(s) as needed within the scene and making further edits. This allows for adding objects from NVIDIA Omniverse or further cropping and modifying the background 3DGS as well as fixing any imperfections with conversion.
The final converted scene is exported in Universal Scene Description format so that it can be used in modern simulation environments, such as Isaac Lab, MuJoCo, and Unreal Engine. We tested on the Franka Emika Panda, following a modified DROID setup with the Universal Manipulation Interface gripper, so our featured simulation examples use this setup.
with SceneAgentwithout
CodexGPT-6 Astra · xhigh
Claude CodeOpus 5.5 · extra
ZCodeGLM 5.3 · max
Grok BuildGrok 4.7 · xhigh
AntigravityGemini 3.8 Flash · high
MuseMuse Spark 1.3 · xhighCodexGPT-6 Astra · xhighClaude CodeOpus 5.5 · extraZCodeGLM 5.3 · maxGrok BuildGrok 4.7 · xhighAntigravityGemini 3.8 Flash · highMuseMuse Spark 1.3 · xhigh1.4
SceneAgent / GPT-6 Astra (Codex)
SceneAgent converted a short input video from the CollectData Messy Kitchen capture into a simulation that could be used for policy evaluation and training in Isaac Lab, MuJoCo, and Unreal Engine.
The renders below come from the other harnesses. Each ran without SceneAgent.
1.5
4DCodeBench
We also ran SceneAgent on 4DCodeBench, a benchmark in which coding agents rebuild a dynamic scene from a video as a simulation program.
1.6
The goal of 4DCodeBench is to convert a single input video into an executable program that regenerates the dynamic scene, with its geometry, motion and appearance. Each coding agent watches the video in a sandboxed harness and writes the program. The program is then rendered and scored against the video on appearance, dynamic IoU, optical flow, point tracks, depth and 3D trajectories. Its 200 tasks cover rigid, deformable, fluid and fracture behavior in 100 real clips and 100 synthetic scenes.
1.7
We measured SceneAgent with frontier models on the videos that performed worst in 4DCodeBench and saw improvements.
Predictive Per-Gaussian Physics Prior
We infer predictive physics features for each simulated object via a combination of semantic features, object decomposition, and VLM labeling.
2.1
For example, the pipeline processes a dry erase marker with the object decomposition of the cap, infill generation of the tip, and predictive friction coefficient holding the cap onto the marker tip.
Similarly, we are able to use this workflow to create materials in the OpenUSD standard that represent physics properties heterogeneously distributed throughout an object. For example, this hat has a higher rigidity around the brim.
3d Capture
Rigidity in the physics material in the OpenUSD. Higher rigidity is red, lower is blue.
We predict physics properties for unlabeled capture with VLMs and semantic per-Gaussian features. The prior predicts material, mass, friction and a 0–1 rigidity for every Gaussian.
2.2
Workflow from input images to physics properties on segmented objects
2.3
Similar to rigidity, we infer destruction mechanics from the predictive physics properties of each object. This is important for accomplishing a wider range of common tasks that require deforming scene objects, such as cooking and cleaning.
A Dexmate robot with two Sharpa hands pulls parchment from a roll, tears a sheet free, and lays it over a bowl of fruit.
2.4
Beyond rigid and articulated objects, we also simulate liquids, soft solids, cutting and fracture. Milk is simulated with the material point method (MPM) and position-based fluids, while the jelly and watermelon use cohesive finite elements in Newton and the salmon uses a cohesive volumetric network over the region the knife cuts.
Milk spill
Jelly under hand contact
Watermelon impact and fracture
Salmon cutting
Totoro with Panda and Sharpa hands
2.5
Each run is simulated forward from contact, and Isaac Lab renders the recorded states with RTX path tracing. When a cut or a fracture opens an object, its scanned appearance follows the new surfaces. The material parameters are priors that we have not yet calibrated against measured interactions. The Totoro render shows the scanned appearance of a plush toy; its fabric, seam and stuffing mechanics have not been evaluated.
Agentic Pipeline for SceneAgent
We provide a team of agents to convert many types of 3d captures into fully usable simulation scenes ready for training in Isaac Lab, MuJoCo, or Unreal. The easiest way to convert a 3d capture is by an agent swarm that processes each step of the pipeline and visually inspects and reviews it for errors.
Figure 4 · An overview of the full agentic pipeline with various tools and skills for processing 3d capture data into usable simulation scenes
3.2
Our agentic workflow processes 3D capture data of diverse origins into fully usable simulation scenes in Isaac Lab, MuJoCo, or Unreal Engine. The agents are helpful in this case specifically because of the diversity of data formats of 3d data and error-prone nature of object conversion and decomposition models. For example, a common failure mode is that the 3d object segmentation will incorrectly transform the scale, rotation, or placement of the objects in the scene, and agents can take camera renders from within Isaac Lab to visually inspect the conversion results and rapidly correct such small errors.
Sony CFS-43, photogrammetry of a radio cassette player and articulations of its door, keys, knobs, and selector
A 3dgs of a stegosaurus skeleton from the AMNH, fully articulated and walking (without hand-created animations)
A jet engine with processed turbofan, articulated and running based on engine schematics
A Bencini cine camera converted into a usable simulation asset, with usable frame-rate dial and battery compartment
A Cessna cockpit, converted into simulation asset with extreme detail in its instrument panel, switches, and placards
A motorcycle converted from 3d capture into a usable simulation asset, including many details
3.3
One of the most unique and beneficial aspects about our agentic workflow is that it can support converting multiple different data types from diverse sources with useful results. For example, it can similarly process input RGB images and align them with structure-from-motion with colmap and then train a 3DGS scene with NerfStudio’s splatfacto big and inspect, review, and correct any results, such as the per image color calibration of the input images. The same pipeline will equally process public internet 3d capture data from photogrammetry or pre-existing 3DGS .ply files.
A patterned textile draped over a garment rack, orbited to read the folds from every side
A hoodie lifted from a surface and left to drape under gravity
Three towels hung on a wall rack, orbited to read the folds from every side
A coiled firefighter hose unwinding under soft-body dynamics
A Liquid Death corduroy cap with soft-body physics, deforming under contact dynamics
An extension cord uncoiling from its wound state
3.4
Likewise, it is able to turn synthetic datasets from generative 3DGS models (such as World Labs Marble, NVIDIA Lyra, or TenCent Hy-World). One can prompt the pipeline to convert the full scene into segmented 3d objects with predictive physics properties or instead focus only foreground objects. The first is much more time-consuming and error prone at this point, but we believe that future iterations of the pipeline will easily improve this.
We imagine a future application of this agentic workflow running on individual in-home or in-factory robots that can 3d capture a space, process that space on-device only with an NVIDIA Jetson Orin or Thor or equivalent, and then train a finetune of its policy on the results with no data leaving device. We hope that this will continue the use of 3DGS in the future to be part of the solution to the data problem of generalist robotic policy development in the future.
Hybrid Scenes with 3d Captures and Digital Art
We convert individual 3d captured objects into simulation assets with our SceneAgent pipeline and incorporate them into synthetic scenes prepared by 3d artists.
Hybrid scenes from available assets in Isaac Lab, a Kitchen scene picking fruit, similar to RoboCasa
Contact rich tasks picking tightly packed vertical cylinders
Chemistry lab, sorting vials into beakers
Picking and plugging cables into a data center server rack
Two arms folding a cardboard packaging in simulation, packing a deformable object
Hybrid digital art and 3d capture scene with a Franka Emika Panda arm watering a succulent with an eyedropper
3.2
Because many backgrounds and environments are redundant across common tasks, we combine specific 3d captured objects processed with our SceneAgent pipeline with other existing art assets created by 3d artists, such as in the NVIDIA Omniverse asset repository. This allows for rapid development of common scenes for policy evaluation along with scenes with specific objects for individual tasks.
For example, when a manipulator may need to learn how to use a specific part, say for a new healthcare device or a new cooling fan in a server rack, that specific part individually could be 3d captured and processed in our pipeline and then incorporated into a pre-existing hospital or data center scene such as the default environments that accompany NVIDIA Isaac Lab.
A Sharpa dexterous hand scooping to cup a delicate flower in its palm
Two dextrous Sharpa hands clipping a GPU into place into a motherboard
Sharpa hand picking up and unfolding sunglasses then putting them on a face
A Dexmate mobile robot with dual Sharpa dexterous hands serving a pizza slice onto a plate and clearing the dishes to the sink
Two Sharpa dexterous hands lifting and unfolding a pair of sunglasses, then setting them on a bust
Dual robot arms lifting a marker into a Sharpa dexterous hand, uncapping it, and writing Hello World on a whiteboard
3.3
The generated environments can be adapted for policy evaluation and training for multiple embodiments, and we believe that they will be especially useful for training dextrous hands. Whereas manipulators like the UMI gripper on the Franka Emika Panda can benefit from more accurate physics properties on objects in simulation, dextrous hands must learn to accomplish much more complex tasks with deformable fabrics, plastics, sticky surfaces, and similar challenges. We hope in future simulations to have a pipeline with increased accuracy around specifically supporting packing in plastic bags and applying stickers to diverse surfaces.
Alternative methods of course may use generative diffusion- or autogressive-based world models to generate the multi-view video used for policy evaluation and finetuning. Our method may be beneficial because it is much less computationally intensive than methods relying on generative video models and can be precise for specific environments or parts.
Digital Twins and Sisters
In order to increase training generalizability from simulation back to the real world, we create slight variations of each object in the scenes converted by the pipeline.
5.2
Instead of digital twins, these are rather “digital sisters,” not exact replicas of reality but similar variations that reflect the range of diversity that a manipulator may encounter in real-world scenarios. We also randomize placement of objects and lighting to continue to improve policy performance in real-world rollouts.
In our initial testing, this increases real-world task completion rates after training in our converted simulation digital sister scenes, but we are continuing to test and evaluate how different lighting, placement, and generative 3d models influence policy performance from simulation back to the real world.
We chose to randomize each of these factors when composing full digital sister scenes, but future work could test each in isolation to verify its influence on policy generalizability during finetuning to study which is best to spend more iterations on.
Contact rich tasks with predictive physics, such as rolling a red rubber ball from one side of the suitcase to the other
Picking up delicate items such as the flower 3d capture from the previous greenhouse scene in hybrid digital art and 3d capture environments
5.3
Along with the typical tasks being trained in simulation policy, we are seeking to understand better how the predictive physics properties can help us achieve a series of contact-rich tasks, such as rolling a red rubber ball from one side of the suitcase to another with a gripper on the Franka arm or picking a columnar object from a tightly packed box.
In future iterations, we also hope to study further how small perturbations of the physics properties may also influence training in simulation. For example, if an object’s mass, friction coefficient, or rigidity are varied slightly between versions, will this help our policy during finetuning more accurately grasp and manipulate deformable objects or objects with heterogeneous physics properties (such as shoes, hats, cardboard boxes, and similar).
Agentic Policy Evaluation and Training
Automate evaluations and repair failure modes in digital sisters of Tasks and Scenes
SceneAgent Simulations: steps 02 through 05
01Real-world evaluation
Review robot recordings and state to identify failure modes. In this example, a stem pinch leaves the flower in the glass.
02Digital twin + sisters
Vary task and scene in simulation from 3D capture or 4D reconstruction. Change objects, placement, or lighting to test likely causes of failure.
03Policy evaluation
Test the current policy in simulation with scene and task sisters. Review the rollouts to find where the failure persists.
04Expert revision
An agent revises controller code and reruns the task. Here, the correction supports the flower from below with the palm and fingers, with the stem between them.
- 01 Approach
- 02 Support
- 03 Lift
05Policy training
Fine-tune on simulated recordings that pass the task's success check. Compare the new and previous checkpoints on the same validation trials.
06Real-world improvement
Re-evaluate the new policy and note any failure modes to restart the cycle. Bring the new recordings back into the next round of evaluation and training.
new failure modes
From real-world failures to simulation, policy training, and real-world re-evaluation
5.4
Policy performance across digital sisters
We evaluated pi0.5 on a simple task across perturbations of lighting, geometry, and visual appearance of objects.
Place in bowl success10 paired trials per cell
0%25%50%75%100%
| Object sister ↓ Lighting → | Neutral | Strong warm | Strong cool | Very dim | Very bright | Hard left | Hard right | Backlit | Overhead spot |
|---|---|---|---|---|---|---|---|---|---|
| Original objectsReference objects | |||||||||
| Box length -5%Geometry | |||||||||
| Box length +5%Geometry | |||||||||
| Box coolColor | |||||||||
| Box fiberTexture | |||||||||
| Box printTexture |
5.5
Digital sisters of scenes and tasks let agents systematically vary objects, layouts, and goals to expose policy failure modes. Agents can automate these evaluations, repair failures in simulation, and generate successful demonstrations for training before testing the updated policy on the real robot.
SceneAgent pipeline: Policy training
The policy training part of the SceneAgent pipeline starts from a reconstructed, interactive simulation of a robot's workspace: a Gaussian-splat scene with segmented object meshes and predicted physics. Agents use the digital twin and its sisters to evaluate policies, repair failure modes, and generate demonstrations for training before returning to the real robot.
The SceneAgent pipeline generally consists of two major blocks: (i) a demonstration factory that generates thousands of expert episodes with photo-realistic observations inside the twin; (ii) policy training on these episodes alone, and zero-shot transfer to real world environment.
The demonstration factory creates demonstrations for policy training with two scalable blocks: initial state randomization; and a physics-bounded scripted expert for generating motion control. Randomization creates possible distributions where the reality may be in at large scales: the objects are placed anywhere in the region on the table that the real task uses, the manipulated object's size varies, and the camera and robot base are perturbed around their calibrated poses. Sampled layouts are also physics-bounded: objects are dropped and settled under physics, so every initial state is one that the real table could actually present.
The scripted expert supplies the actions: it plans from prescient ground-truth object poses inside the simulator and pairs a pick-and-place routine — locate, approach, descend, grasp, lift, carry, lower, release, retreat — to the sampled poses. Finally, a success gate checks whether the object truly ended up in the container and discards episodes that fail. In each recorded episode, observations are rendered by depth-compositing a mesh foreground (robot and objects) over the Gaussian-splat background of the deployment environment.
The model-agnostic training uses the simulation-generated demonstration episodes for imitation-learning: camera streams are paired with the expert's actions in a standard format, and each episode carries a natural-language instruction sampled from many phrasings of the task. The pipeline then finetunes a pretrained vision language action model on the demonstration dataset via LoRA.
SceneAgent Evaluation in Progress
After fine-tuning a Vision-Language-Action (VLA) model policy in simulation only, we run the policy in the real world and see improvement in performance. We are still completing the evaluation of our pipeline and will update this with our results soon.
Policy evaluation in realworld on the Franka Emika Panda after finetuning in simulation.
7.2
7.3
After finetuning generalist VLAs such as Physical Intelligence’s pi0.5 and NVIDIA’s GR00T 1.6 in simulation only on the digital twin and sisters, we see similar performance in the real world as in simulation. Initial results show a positive increase in task completion rates in the real world after training for 10,000 episodes in simulation of our lab space and example objects processed through our SceneAgent pipeline.
We are still evaluating the results and are optimistic about the results of real world performance improvements after training in converted 3DGS scenes to use in simulation with predictive physics. We will update our work with the full results soon.
Citation
@misc{sceneagent2026,
title = {SceneAgent: 3D Capture-Derived Scenes with Predictive
Physics for Policy Evaluation and Training Environments},
author = {Luke Hollis and Tianxing Fan and Heng Yang},
year = {2026},
note = {Preprint. Computational Robotics Group, Harvard University}
}The code, processed scenes, and example converted objects will be released with our paper soon.
Featured Art and 3D Captures
- [1]CollectDataIO, “Tidy Up: U.S. Kitchens Before and After Cleaning,” Hugging Face.
- [2]3dhdscan, “Rudbeckia flower,” Sketchfab, CC BY 4.0.
- [3]JordanFry, “Vintage Sony radio CFS-43,” Polycam.
- [4]TechnologicalFrostfang2470, “Totoro,” Polycam.
- [5]netgis, “Chinese statue,” Sketchfab, CC BY-NC 4.0.
- [6]Lassi Kaukonen, “Asus Z170-P motherboard,” Sketchfab, CC BY 4.0.
- [7]Igor.Jop, “RX 480 GPU,” Sketchfab, CC BY 4.0.
- [8]MTP, “Stegosaurus composite skeleton, American Museum of Natural History,” Polycam.
- [9]redline3678, “Bencini Comet Universal 444 camera (1977),” Polycam.
- [10]wlanfr3ak, “Cessna C172R cockpit,” Polycam.
- [11]ZanyFalcon4404, “Motorcycle,” Polycam.
- [12]Pandora, “Towel rack,” Polycam.
- [13]AutomatedStarfish942, “Red towel,” Polycam.
- [14]Artist, “Dept Artisté anorak,” Polycam.
- [15]drewjosh, “Firefighter hose,” Polycam.
- [16]sisamm, “Electric extension lead,” Polycam.
- [17]stevieburd, “Liquid Death hat,” Polycam.