Generated multi-view video
Two synchronized views depict the prompted interaction.
Abstract
We present KinoGen, a framework that synthesizes customizable humanoid loco-manipulation references through kinematically grounded video generation. KinoGen conditions a pretrained video generator on calibrated renders of the target robot and object, then fits the same models to the generated two-view video. The resulting reference lies in the robot configuration space without a human body proxy or retargeting. By correcting a single keyframe, KinoGen prescribes a goal object pose, regenerates the surrounding motion, and recovers a reference that satisfies that pose exactly. Across eight tasks, KinoGen reduces penetration and foot skating, preserves over 90% of annotated contacts, and supports tracking-policy deployment on a Unitree G1 with dexterous hands.
Method
Tracking policies for humanoid loco-manipulation require motion references that specify whole-body, hand, and object trajectories in the configuration space of the target robot. Video generation offers a scalable source of such behavior. However, generated videos provide pixels rather than trajectories, and they give no guarantee that a prescribed object pose is reached.
Our key insight is that a video generator and a physics simulator can be coupled through rendering and reconstruction when both operate on the same robot and object models. This coupling leads to three design choices:
- Conditioning on the rendered initial state. Calibrated renders of the initial robot and object state condition the video generator, which places the generated behavior in the exact embodiment and scene. Fitting the same models to the generated video returns the behavior to the simulator. This yields references in the robot configuration space without a human body proxy or retargeting.
- Multi-view reconstruction with motion priors. Two synchronized views of the same interaction reduce depth ambiguity. Whole-body, hand, and object priors constrain the motion that image evidence alone cannot resolve.
- Keyframe-based correction. A corrected simulator keyframe prescribes the goal. The generator regenerates the surrounding motion, and reconstruction enforces the keyframe as a hard constraint. The reference reaches the prescribed configuration exactly while the motion stays coherent.
Kinematic Grounding
Our key insight is that a pretrained video generator and a physics simulator can be coupled through rendering and reconstruction using the same robot and object models. The paired videos illustrate the reconstruction side of this coupling, showing the generated interaction alongside its recovered robot–object reference.
Reconstructed robot–object reference
KinoGen recovers the corresponding body, hand, and object trajectories.
Diverse Generated Reference
Qualitative references at 1× speed span lifting, placement, sitting, and other environmental interaction. These examples show the diversity and quality of the generated motion references.
Pick Up the Dumbbell
Pick Up the Barbell
Pick Up the Basket
Pick Up the Monitor
Pick Up the Basketball
Pick Up the Tennis Ball
Pick Up the Chair
Sit on the Chair
Sit on the Stool
Mop the Floor
Place the Bottle on the Table
Place the Box on the Shelf
Real-World Policy Deployment
Tracking policies trained on KinoGen references execute diverse whole-body tasks on a Unitree G1 equipped with dexterous hands. Videos are shown at 1× speed. The deployments demonstrate that the generated references support hardware policy learning across whole-body lifting, placement, sitting, and environmental interaction.
Place the Flower on the Table
Pick Up the Basket
Sit on the Stool
Pick Up the Chair — Left
Pick Up the Chair — Right
Pick Up the Tennis Ball
Pick Up the Basketball
Mop the Floor
Pick Up the Box and Put It Down
Comparison with Baselines
All methods reconstruct the same generated interaction. KinoGen preserves a plausible grasp and ground contact, while the comparison methods exhibit penetration or floating artifacts. This yields a more physically coherent reference for downstream control.
Reference Correction
By correcting the keyframe that specifies the bottle pose, KinoGen regenerates the surrounding motion and reconstructs it with the corrected state as a hard constraint, using the video model as a behavior-aware interpolator. KinoGen reaches the prescribed targets exactly while producing substantially less penetration than direct interpolation.
Quantitative Evaluation
KinoGen achieves the best penetration, foot-skating, contact-preservation, and downstream RL performance among the video-based reference pipelines evaluated in the paper.