Articulated digital twins from monocular video

I kept seeing GPT-6 Astra modelling results and got curious enough to build the thing myself: an agentic pipeline that turns a plain RGB video into 3D assets. The living room for robots at MIT CSAIL.

Input: one handheld phone walkthrough of our lab kitchen (about 20 seconds). Nothing else: no depth sensor, no CAD, no asset library. About one day (I actually slept overnight), including human-in-the-loop.

Live: frank-zy-dou.github.io/kitchen-twin

It is still not perfect. Thin and shiny things are still weak, a few objects are drafts, and the room shell needs another pass.

The loop, roughly

• Monocular video → near-metric scan (ViPE): camera poses and depth. This is the measuring instrument, not the output.
• GPT-6 lists what should exist as separate objects, then open-vocabulary detection and tracking gives per-object masks; each object is fused and measured in metres.
• For every asset, GPT-6 works in its own sandbox with tools: it writes the object as a program in a small Blender DSL (closed solids, PBR colours, hinges/drawers), builds it, renders it over the original video frames, compares it with the scan points in 3D, and iterates.
• A separate GPT-6 session is the verifier. It can render the model on any frame it wants and must justify every complaint with a frame. The modeller never grades its own work.
• A completeness pass renders the whole modelled scene from the video's own cameras, puts it next to the real frames, and says what is still missing. What the detector keeps missing (a row of identical cabinets) gets placed geometrically instead.
• Out comes MJCF/URDF with joints.

From geometry to mechanics

The exported joints are a starting point for simulation. A drawer can look right in a render and still have the wrong travel limits, friction, or collision geometry. I would first check a few simple interactions against the real kitchen: open a drawer, swing a cabinet door, place an object on a counter. Those checks would tell me which parts of the twin are ready for robot use.

What visual verification can establish

I use agreement with the video as evidence of reconstruction quality, not as a measurement of every physical property. Surfaces hidden throughout the walkthrough remain underconstrained; a plausible completion should be marked as an assumption. Camera and depth estimates also provide a shared reference, so agreement between a model and its scan may preserve errors in that reference.

The separate verifier reduces direct self-evaluation, but it is not independent ground truth. I would reserve additional viewpoints and a few physical measurements for evaluation, rather than feed them back into modelling. Reporting those checks separately would distinguish fitting the observed video from reconstructing a scene that remains consistent outside it.

My original experiment on X.

ViPE: Huang, J., et al. (2025). ViPE: Video Pose Engine for 3D Geometric Perception. NVIDIA Research Whitepapers.

@misc{dou2026kitchentwin,
  author = {Dou, Zhiyang},
  title = {Articulated digital twins from monocular video},
  year = {2026},
  month = sep,
  howpublished = {Research Notes},
  url = {https://frank-zy-dou.github.io/blog/kitchen-twin-from-video/}
}