Figure 1. Coordinated by a persistent SUN Program, the Kuafu harness equips the agent with physics-based verification, policy learning, and data-production capabilities. The program supervises the simulation controller and its demonstrations; downstream visual policies learn from recorded observations and actions alone. We evaluate nine tasks and transfer visual policies for six of them to Franka FR3 and Kinova Gen3 robots.
Abstract
Model-based control can directly execute specified objectives, while learning can amortize such behaviors into reactive policies, making their combination a natural solution to multi-stage manipulation. We introduce Semantically UNified (SUN) Programs, typed executables that compile grounded relations into aligned optimal control objectives, satisfaction predicates, and learning rewards. Our harness, Kuafu, equips a foundation model as a task-level agent to orchestrate scene preparation, verification, residual RL, and data production. The agent uses program feedback to repair candidate programs and training diagnostics to calibrate relative reward weights, retaining accepted task semantics across tool calls. Across nine multi-stage manipulation tasks, Kuafu achieves 82.03% average success rate, significantly outperforming all learned baselines. Its learned controllers generate demonstrations at 10.57× the human-teleoperation rate, yielding data that improve visual-policy success by 23.6 percentage points over the strongest baseline. The policies transfer zero-shot to physical Franka and Kinova robots, demonstrating sim-to-real generalization.
Full project video · 2:11
Program-guided manipulation in simulation and deployment on physical robots. Audio is available through the player controls.
01
One program, across control and learning.
SUN · the task program
A Semantically UNified (SUN) Program compiles grounded relations into aligned MPC objective terms, satisfaction predicates, and RL learning costs. The accepted relations keep their identity, grounding, satisfaction boundaries, and stage membership across tool calls.
Kuafu · the agent harness
A robotics execution and learning harness equips a task-level agent to prepare scenes, verify and repair programs, and acquire policies. Physical feedback guides program repair; training diagnostics guide scalar calibration within the accepted task definition.
MPC verification supplies successful demonstrations for Stage-BC. A frozen behavioral prior anchors bounded residual RL, while the agent uses training feedback to calibrate permitted scalars. The resulting simulation controller produces demonstrations for a visual policy.
Inputs Language instructions and object videos, together with a robot model, simulation configuration, and typed operator library.
Outputs A simulation policy for data production and a visual policy for physical deployment. The visual learner receives recorded observations and actions alone.
02
From object videos to simulation assets.
These simulation assets were created by a GPT-6 Astra agent using object videos, language descriptions, and reconstruction tools. Compare the source clips with 360° previews of the updated models, which use multi-view texture fusion and lighting correction.
Sage holder
Input videoTextured model · 360°
Emergency-stop box
Input videoTextured model · 360°
Bamboo organizer
Input videoTextured model · 360°
Bamboo tray
Input videoTextured model · 360°
03
Data Generation
From learned behavior to useful robot data.
Evaluated on nine multistage manipulation tasks, with downstream visual-policy learning and zero-shot deployment on Franka and Kinova robots.
Simulation policy success
82.03%
Task-macro success of the primary privileged controller across nine tasks.
Data-production throughput
10.57×
Successful trajectory time per active hour relative to human teleoperation, after controller training.
Downstream visual-policy gain
+23.6 pp
Over SkillGen with the same DP3 learner and 500 successful trajectories per task.
In the matched-data simulation study, DP3 reaches 46.02% success with Kuafu demonstrations, versus 22.44% with SkillGen. For physical deployment, DP3 trained on 1,000 trajectories per task achieves 34.0% success over 106 trials across six tasks and three robot–gripper configurations, without real-world fine-tuning.
More data per production hour
From the paper · Table IV, panel A · Section IV-C
Human baselines were collected via simulation teleoperation using an AgileX PIKA gripper, with measured poses defining the simulated end-effector targets. Once the final state controller is trained, Kuafu produces 10.57× as much successful trajectory time per active production hour as human teleoperation.
Accounting for controller training cost
From the paper · Table IV, panel B · Section IV-C
Pricing human teleoperation at the U.S. federal minimum wage of $7.25/h and A6000 rental for both controller training and data production at $0.51/GPU h, Kuafu reaches cost parity after 35 successful demonstrations per task on average.
From the paper · Table IV, panel B · Section IV-C
Including controller training, Kuafu catches up with human teleoperation in cumulative successful demonstration count before reaching 1000 demonstrations on every task.
Citation
@misc{wang2026sun,
title = {{SUN}: Agentic Robot Policy Learning with Persistent Task Programs},
author = {Wang, Weiqi and Li, Zhi and Lei, Yudong and Martinez, David and
Gao, Xiaofeng and Jiang, Yuxin and Jiang, Chenfanfu and Wu, Yingnian and
Terzopoulos, Demetri and Gong, Ran},
year = {2026},
note = {Preprint}
}