00original
Harvard Computational Robotics Group
School of Engineering and Applied Sciences, Harvard University
School of Engineering and Applied Sciences, Harvard University
Preprint
2026
2026
SceneAgent
An agentic pipeline for converting 3D captures to simulation environments with predictive physics for policy evaluation, online planning, and policy training
1 Harvard Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University
SceneAgent converts 3d captures of real-world scenes into simulation environments that can be used for policy evaluation, online planning, and policy training or finetuning. The pipeline primarily focuses on converting environments captured with 3d Gaussian Splatting (3DGS) but also supports scenes captured with photogrammetry and LiDAR. It incorporates 3d semantic features, object segmentation, predictive per-gaussian physics properties, object decomposition, deformability, and articulation. To increase policy generalization, it is also able to generate “digital sisters” or similar versions of each object within an environment. We are still evaluating how well policies trained in this environment perform in the real world, and initial results show promising success rates similar to or improving on other recent papers such as SimFoundry and PolaRiS.
From Real-world Scenes into Simulation
Our pipeline supports converting 3D capture data from 3DGS, photogrammetry, and LiDAR into usable scenes that can be used across a wide range of simulators, such as Isaac Lab, MuJoCo, and Unreal Engine.
We initially infer semantic features for each 3d Gaussian and save them in a codebook for rapid lookup, inspired by the method from LangSplat. We then segment the foreground objects in the scene from the background and crop their 3d Gaussians from the background. We then predictively infill these cropped regions. For each object, we predict a physics material with a combination of semantic features, inference against a per-gaussian predictive physics model, and use VLM to estimate features such as friction coefficient, rigidity, mass, and density.
From this data, we then bake a physics material for each object and then decompose that object into smaller pieces. We then add articulations to the objects as necessary so they can perform actions as they would in real life. Finally, we generate versions of each object with slight variations–instead of digital twins, we term these digital sisters. We then use a VLM to inspect and review each of these steps as well as the transform of the converted objects in the scene.
Flower (Rudbeckia hirta)
SceneAgent
Processes each 3D object for predictive physics, decomposition, and object properties
013DGS
Convert input RGB or other 3D data
02Semantic features
Use GroundingDINO + SAM to assign semantic features to each 3D Gaussian
03Predictive physics
Generate predictive per-Gaussian physics priors to bake physics material
04Object decomposition
Decompose each object into individual parts as needed
05Articulation
Articulate any movable parts within each object
06Object sisters
Generated object variations with geometric or visual differences
07 Package scene in Universal Scene Description format
Share usable variations of the full scene as USDZ for NVIDIA Isaac Lab, MuJoCo, Unreal Engine, and other simulators.
1.2
08 Interactive viewer
Loading the stage
Figure 1 Pipeline architecture. We support converting input RGB image sequences (from video or as images), pre-existing 3DGS or photogrammetry assets, and LiDAR scans. Output is in Universal Scene Description format (USDZ) and able to be used in a wide range of simulation tools such as Isaac Lab, MuJoCo, and Unreal Engine.
1.3
After we convert the scene, we created an interactive viewer built with Three.js to assist in positioning the robot(s) as needed within the scene and making further edits. This allows for adding objects from NVIDIA Omniverse or further cropping and modifying the background 3DGS as well as fixing any imperfections with conversion.
The final converted scene is exported in Universal Scene Description format so that it can be used in modern simulation environments, such as Isaac Lab, MuJoCo, and Unreal Engine. We tested on the Franka Emika Panda, following a modified DROID setup with the Universal Manipulation Interface gripper, so our featured simulation examples use this setup.
Digital Twins and Sisters
In order to increase training generalizability from simulation back to the real world, we create slight variations of each object in the scenes converted by the pipeline.
2.2
Instead of digital twins, these are rather “digital sisters,” not exact replicas of reality but similar variations that reflect the range of diversity that a manipulator may encounter in real-world scenarios. We also randomize placement of objects and lighting to continue to improve policy performance in real-world rollouts.
In our initial testing, this increases real-world task completion rates after training in our converted simulation digital sister scenes, but we are continuing to test and evaluate how different lighting, placement, and generative 3d models influence policy performance from simulation back to the real world.
We chose to randomize each of these factors when composing full digital sister scenes, but future work could test each in isolation to verify its influence on policy generalizability during finetuning to study which is best to spend more iterations on.
Contact rich tasks with predictive physics, such as rolling a red rubber ball from one side of the suitcase to the other
Picking up delicate items such as the flower 3d capture from the previous greenhouse scene in hybrid digital art and 3d capture environments
2.3
Along with the typical tasks being trained in simulation policy, we are seeking to understand better how the predictive physics properties can help us achieve a series of contact-rich tasks, such as rolling a red rubber ball from one side of the suitcase to another with a gripper on the Franka arm or picking a columnar object from a tightly packed box.
In future iterations, we also hope to study further how small perturbations of the physics properties may also influence training in simulation. For example, if an object’s mass, friction coefficient, or rigidity are varied slightly between versions, will this help our policy during finetuning more accurately grasp and manipulate deformable objects or objects with heterogeneous physics properties (such as shoes, hats, cardboard boxes, and similar).
Hybrid Scenes with 3d Captures and Digital Art
We convert individual 3d captured objects into simulation assets with our SceneAgent pipeline and incorporate them into synthetic scenes prepared by 3d artists.
Hybrid scenes from available assets in Isaac Lab, a Kitchen scene picking fruit, similar to RoboCasa
Contact rich tasks picking tightly packed vertical cylinders
Chemistry lab, sorting vials into beakers
Data center, routing a cable between racks
Two arms folding a cardboard packaging in simulation, packing a deformable object
Hybrid digital art and 3d capture scene with a Franka Emika Panda arm watering a succulent with an eyedropper
3.2
Because many backgrounds and environments are redundant across common tasks, we combine specific 3d captured objects processed with our SceneAgent pipeline with other existing art assets created by 3d artists, such as in the NVIDIA Omniverse asset repository. This allows for rapid development of common scenes for policy evaluation along with scenes with specific objects for individual tasks.
For example, when a manipulator may need to learn how to use a specific part, say for a new healthcare device or a new cooling fan in a server rack, that specific part individually could be 3d captured and processed in our pipeline and then incorporated into a pre-existing hospital or data center scene such as the default environments that accompany NVIDIA Isaac Lab.
A Sharpa dexterous hand scooping to cup a delicate flower in its palm
Two dextrous Sharpa hands clipping a GPU into place into a motherboard
Sharpa hand picking up and unfolding sunglasses then putting them on a face
3.3
The generated environments can be adapted for policy evaluation and training for multiple embodiments, and we believe that they will be especially useful for training dextrous hands. Whereas manipulators like the UMI gripper on the Franka Emika Panda can benefit from more accurate physics properties on objects in simulation, dextrous hands must learn to accomplish much more complex tasks with deformable fabrics, plastics, sticky surfaces, and similar challenges. We hope in future simulations to have a pipeline with increased accuracy around specifically supporting packing in plastic bags and applying stickers to diverse surfaces.
Alternative methods of course may use generative diffusion- or autogressive-based world models to generate the multi-view video used for policy evaluation and finetuning. Our method may be beneficial because it is much less computationally intensive than methods relying on generative video models and can be precise for specific environments or parts.
Agentic Pipeline for SceneAgent
We provide a team of agents to convert many types of 3d captures into fully usable simulation scenes ready for training in Isaac Lab, MuJoCo, or Unreal. The easiest way to convert a 3d capture is by an agent swarm that processes each step of the pipeline and visually inspects and reviews it for errors.
Figure 4 · An overview of the full agentic pipeline with various tools and skills for processing 3d capture data into usable simulation scenes
4.2
Our agentic workflow processes 3D capture data of diverse origins into fully usable simulation scenes in Isaac Lab, MuJoCo, or Unreal Engine. The agents are helpful in this case specifically because of the diversity of data formats of 3d data and error-prone nature of object conversion and decomposition models. For example, a common failure mode is that the 3d object segmentation will incorrectly transform the scale, rotation, or placement of the objects in the scene, and agents can take camera renders from within Isaac Lab to visually inspect the conversion results and rapidly correct such small errors.
Sony CFS-43, photogrammetry of a radio cassette player and articulations of its door, keys, knobs, and selector
A 3dgs of a stegosaurus skeleton from the AMNH, fully articulated and walking (without hand-created animations)
A jet engine with processed turbofan, articulated and running based on engine schematics
A Bencini cine camera, orbited and inspected down to its frame-rate dial and battery compartment
A Cessna cockpit, orbited and inspected down to its instrument panel, switches, and placards
A motorcycle, orbited and inspected down to its forks, spoked wheel, and handlebar controls
4.3
One of the most unique and beneficial aspects about our agentic workflow is that it can support converting multiple different data types from diverse sources with useful results. For example, it can similarly process input RGB images and align them with structure-from-motion with colmap and then train a 3DGS scene with NerfStudio’s splatfacto big and inspect, review, and correct any results, such as the per image color calibration of the input images. The same pipeline will equally process public internet 3d capture data from photogrammetry or pre-existing 3DGS .ply files.
A patterned textile draped over a garment rack, orbited to read the folds from every side
A hoodie lifted from a surface and left to drape under gravity
Three towels hung on a wall rack, orbited to read the folds from every side
A coiled firefighter hose unwinding under soft-body dynamics
A corduroy cap turned in place, its brim and panels deforming
An extension cord uncoiling from its wound state
4.4
Likewise, it is able to turn synthetic datasets from generative 3DGS models (such as World Labs Marble, NVIDIA Lyra, or TenCent Hy-World). One can prompt the pipeline to convert the full scene into segmented 3d objects with predictive physics properties or instead focus only foreground objects. The first is much more time-consuming and error prone at this point, but we believe that future iterations of the pipeline will easily improve this.
We imagine a future application of this agentic workflow running on individual in-home or in-factory robots that can 3d capture a space, process that space on-device only with an NVIDIA Jetson Orin or Thor or equivalent, and then train a finetune of its policy on the results with no data leaving device. We hope that this will continue the use of 3DGS in the future to be part of the solution to the data problem of generalist robotic policy development in the future.
SceneAgent pipeline: Policy training
The policy training part of the SceneAgent pipeline starts from a reconstructed intractable simulation environment, which is a Gaussian-splat scene with segmented object meshes and predicted physics, of a robot's workspace. It then converts the simulation environment into manipulation policies that can be deployed on the real robot.
The SceneAgent pipeline generally consists of two major blocks: (i) a demonstration factory that generates thousands of expert episodes with photo-realistic observations inside the twin; (ii) policy training on these episodes alone, and zero-shot transfer to real world environment.
The demonstration factory creates demonstrations for policy training with two scalable blocks: initial state randomization; and a physics-bounded scripted expert for generating motion control. Randomization creates possible distributions where the reality may be in at large scales: the objects are placed anywhere in the region on the table that the real task uses, the manipulated object's size varies, and the camera and robot base are perturbed around their calibrated poses. Sampled layouts are also physics-bounded: objects are dropped and settled under physics, so every initial state is one that the real table could actually present.
The scripted expert supplies the actions: it plans from prescient ground-truth object poses inside the simulator and pairs a pick-and-place routine — locate, approach, descend, grasp, lift, carry, lower, release, retreat — to the sampled poses. Finally, a success gate checks whether the object truly ended up in the container and discards episodes that fail. In each recorded episode, observations are rendered by depth-compositing a mesh foreground (robot and objects) over the Gaussian-splat background of the deployment environment.
The model-agnostic training uses the simulation-generated demonstration episodes for imitation-learning: camera streams are paired with the expert's actions in a standard format, and each episode carries a natural-language instruction sampled from many phrasings of the task. The pipeline then finetunes a pretrained vision language action model on the demonstration dataset via LoRA.
SceneAgent Evaluation in Progress
After fine-tuning a Vision-Language-Action (VLA) model policy in simulation only, we run the policy in the real world and see improvement in performance. We are still completing the evaluation of our pipeline and will update this with our results soon.
Policy evaluation in realworld on the Franka Emika Panda after finetuning in simulation.
6.2
6.3
After finetuning generalist VLAs such as Physical Intelligence’s pi0.5 and NVIDIA’s GR00T 1.6 in simulation only on the digital twin and sisters, we see similar performance in the real world as in simulation. Initial results show a positive increase in task completion rates in the real world after training for 10,000 episodes in simulation of our lab space and example objects processed through our SceneAgent pipeline.
We are still evaluating the results and are optimistic about the results of real world performance improvements after training in converted 3DGS scenes to use in simulation with predictive physics. We will update our work with the full results soon.
Citation
@misc{sceneagent2026,
title = {SceneAgent: 3D Capture-Derived Scenes with Predictive
Physics for Policy Evaluation and Training Environments},
author = {Luke Hollis and Tianxing Fan and Heng Yang},
year = {2026},
note = {Preprint. Computational Robotics Group, Harvard University}
}The code, processed scenes, and example converted objects will be released with our paper soon.