Recent world action models provide visual foresight and future prediction for robot manipulation, but lack explicit mechanisms for compositional skill reuse. Agentic robotics combines code-based planning and reasoning with reusable skills, but often lacks predictive grounding in physical dynamics.

We introduce AW-0, an agentic world action model framework that integrates code-based planning with joint video and action prediction. We jointly model code, future video, and robot actions as three coupled generative streams within a unified mixture-of-transformers architecture. Within this architecture a code expert generates parameterized skill routines and control logic while video and action experts generate future visual states and low-level motor commands. Code serves two complementary roles: it structures multi-step prediction and control within the world model and represents skills as reusable programs for subsequent agentic execution.

We further introduce an automated curation pipeline that labels robot and human video trajectories with aligned programs and consolidates recurring routines into reusable skill libraries. Evaluations on RoboTwin 2.0 and physical bimanual manipulation tasks show improved task performance over VLA and WAM baselines, alongside gains in language following, semantic reasoning and robustness, demonstrating that agentic code enables connection between planning, prediction and execution within WAMs.

Learning-based robotic manipulation has progressed from small task-specific visuomotor policies over a decade ago to broadly supervised foundation models that connect perception, language, and action (Levine et al., 2016; Zitkovich et al., 2023; Kim et al., 2024; Niu et al., 2024; Black et al., 2024; Bjorck et al., 2025; Niu et al., 2025; Liu et al., 2025; Black et al., 2025; Liu et al., 2026). In parallel, self-supervised video prediction has long been valued as a learning signal to model the world: image inpainting extended to video prediction led to action-conditioned video predictors that can imagine the future before committing to an action (Pathak et al., 2016; He et al., 2021; Tong et al., 2022; Finn et al., 2016; Assran et al., 2025; Ye et al., 2026).

World Action Models (WAMs) couple visual prediction with visuomotor policies, grounding control in representations of how the physical world evolves (Ye et al., 2026; Kim et al., 2026; Yuan et al., 2026). This predictive grounding is particularly valuable for complex manipulation, where successful execution depends on anticipating object motion, contact, and the consequences of low-level actions. Yet predicted video and action chunks do not by themselves specify how a long-horizon task should be decomposed: which reusable skills to invoke, how to parameterize them, or how to compose them across subtasks. Predicting the world and/or actions in general can also be expensive, especially as it can be difficult to establish limits on how many degrees of freedom need to be “world modeled” (or teleoperated, in the supervised setting).

Agentic robot systems provide complementary structure to WAMs: a programmatic harness that selects reusable skills and tools, binds them to the current scene, tracks progress, and replans as observations change (Liang et al., 2023; Fu et al., 2026; Zhang et al., 2026; Xiao et al., 2026; Lu et al., 2026). This programmatic representation makes the structure of multi-step tasks explicit, specifying which skills and tools to invoke, with what arguments, and in what sequence. Yet agentic robotics often lacks predictive grounding in real-world physical dynamics and does not leverage a learned model of how the world will change as the robot acts.

We advocate for an architecture that combines the foresight and action execution of WAMs with the cognitive efficiency, reusable skills, and potential for future self-improvement of agentic robotics. We propose AW-0, an agentic WAM framework that incorporates code-based planning and generation directly into world action modeling via a grounded code expert that harnesses broad programming priors to synthesize modular skills featuring explicit parameters and control logic. Code guides synchronized video and control modules within a mixture-of-transformers architecture, simultaneously denoising discrete code structures, predicted visual observations, and low-level motor commands.

We first show that such agents can be realized by adding a code expert to a WAM, leveraging multisensor action and code generation in a planning stage. We develop a modular generation framework: planning via code generation significantly improves WAM performance, by appropriately conditioning the long-horizon generations required for complex robotic tasks. An automated curation pipeline extracts systematic routines into reusable program libraries, mapping code segments directly onto robotic and human video trajectories. This approach bridges high-level semantic reasoning with predictive physical dynamics, establishing a unified language for compositionality across distinct environments and tasks. We further demonstrate agentic skill reuse via a hybrid deployment strategy that selects between raw action chunk execution and programmatic skill invocation. Empirical evaluations across RoboTwin 2.0 benchmarks and physical bimanual setups show substantial performance gains compared to standard VLA and WAM baselines, highlighting programmatic code as an effective structural backbone for predictive planning, skill discovery, and embodied robotic control.

Our primary contribution is the conceptualization and realization of agentic robotic skills within a world action model framework, showing that code generation improves action prediction in the world model, and that the code generated from the world model can be re-used in an agentic framework. Specific novel contribution elements include: a mixture-of-transformers framework integrating a code expert into world action models via simultaneous denoising of code, video, and control; an automated agentic pipeline designed for program annotation and skill discovery; a dataset of such structured skill programs; and a hybrid execution policy that dynamically toggles between programmatic skill invocation and direct deployment of action chunks from the WAM action expert.

We formulate AW-0 as a unified conditional generative model over code, video, and actions, which respectively describe structured skill plans, future scene dynamics, and continuous robot motion. We represent a skill program as a discrete token sequence C_0=(c_1,\ldots,c_{N_c}), future observations as continuous video latents Z_0 encoded by a pretrained variational autoencoder, and robot motion as an action chunk A_0=(a_t,\ldots,a_{t+H-1}) over a horizon of H steps.

The model is conditioned on the current multi-view observations \mathcal{O}_t, a language instruction L, the current robot state S_t, and a compact memory M_t of the preceding program slices. Together, these inputs define the conditioning context for our joint generative model:

The unified conditional generation proceeds by jointly denoising the three modalities. Given masked code \widetilde C_r and noisy continuous states Z_{\sigma_v} and A_{\sigma_a}, the unified model predicts where r is the code-mask ratio, \sigma_v and \sigma_a are the video and action noise levels, \ell_\theta^c are code-token logits, and v_\theta^v,v_\theta^a are the video and action velocity predictions. The formulation unifies structured skill programming, future dynamics prediction, and continuous control in one generative process.

As shown in Figure 2, AW-0 adopts a Mixture-of-Transformers (MoT) (Liang et al., 2025) architecture with three experts for skill planning, future visual prediction, and continuous control. Each expert keeps its own attention projections (W_Q,W_K,W_V,W_O), layer normalizations, and FFNs, communicating through masked joint attention at every transformer layer. The attention mask is bidirectional within each expert and blockwise causal across experts, following the order of code, video and action. We provide additional model and training details in Appendix A.

Code Expert. The code expert augments a pretrained language backbone with a frozen Qwen3-VL (Bai et al., 2025) ViT and adopts LLaDA-style masked-token diffusion (Nie et al., 2025). Current multi-view observations are encoded into visual tokens and prepended to the instruction and program tokens to ground skill planning in the scene.

Video and Action Experts. The video expert uses a video DiT initialized from Cosmos-Predict2.5 (NVIDIA, 2025b) to model VAE-encoded future frames conditioned on observed frames, while a separate action DiT generates continuous action chunks. Both experts receive language embeddings through cross-attention, with additional robot-state conditioning for the action expert.

At each denoising step, the MoT jointly predicts code logits and continuous velocities as defined in Eq. 2. The code stream supplies planning context to the video and action experts, while the action expert also reads video representations. These connections operate throughout generation, allowing all three modalities to evolve within the same iterative process.

Modality-Specific Corruption. Video and action generation use continuous flow matching. For m\in\{v,a\}, with X_0^v=Z_0 and X_0^a=A_0, we construct noisy inputs as The original video sampling distribution p_{\mathrm{pre}} is induced by u_v=s b, where b\sim\mathrm{Beta}(7,1) and s=0.999; actions use u_a\sim U(0,1). Both shifts are \gamma_m=5. Code instead uses discrete absorbing-mask corruption. We sample a mask ratio r\sim U(0,1) and corrupt each token independently: Observed frames, visual prefixes, and language conditions remain uncorrupted. Training samples the modality-specific noise levels independently. The continuous experts regress the velocity target \epsilon_m-X_0^m, while the code expert minimizes cross-entropy over masked program positions.

Noise Resampling for Predictive Guidance. To provide more informative video features for action prediction, we broaden the high-noise pretrained distribution with a uniform component: Objective-specific importance weights balance video learning and predictive guidance: Here, \ell_v,\ell_a are flow-matching losses, and \ell_c is masked-token cross-entropy; bars denote clipped weights normalized to unit batch mean. We use \alpha=0.5 and leave code-mask sampling unchanged.

Automatic Annotation Pipeline. We propose an automatic annotation pipeline (Figure 3) that annotates the training data with skill planning code in three stages. We first sample one episode per task and use Gemini 3.1 Pro (Google DeepMind, 2026) to write subtask-level skill programs against predefined declaration-only APIs, while Claude Code running Claude Opus 4.8 (Anthropic, 2026) acts as a coordinator that distills recurring, task-general skills. We then freeze the skill library and re-annotate the sampled episodes with the finalized vocabulary, forming a cache of reusable subtask-to-program mappings. To annotate the remaining corpus, we reuse a cached program when the subtask's contact and motion pattern matches an existing entry; otherwise, the coding agent generates a new annotation and adds it to the cache. See Appendix K for further pipeline details.

Code Usage in Model. Annotated programs define the interface through which the code expert plans. During training, the code stream concatenates a conditioning prompt and the active slice, a short block of primitive or skill calls. Only the slice is corrupted by the absorbing-mask process of Eq. 4 and supervised by the masked-token loss. At each replanning point, the policy rebuilds the prompt and appends a canvas of G mask tokens. The canvas is decoded jointly with the video and action states over the T denoising steps, committing the G/T most confident masked positions per step. By default, the decoded slice is not executed. It serves as a subtask plan that conditions the action expert through joint denoising, while the predicted action chunk drives the robot.

Memory Cache. Since a single observation is often insufficient to reveal task progress, the code expert receives a compact cache of the slices generated so far. The history slices are compressed into K learnable memory slots and read back as prefix tokens before the prompt in the code stream. During training, the buffer holds the preceding annotated slices, replaced with probability p by the model's own predictions. At inference, it is reset per episode and grows with each decoded slice.

Hybrid Actions. We additionally explore an inference mode in which the decoded code slice is executed in the manner of Code-as-Policies (CaP) against a library of perception and control primitives, and the executed actions are integrated with the direct actions of the action expert. At deployment, the two channels are arbitrated at every replanning point: the decoded slice takes over the current stage only if it passes a set of validity and progress checks; otherwise the action chunk from the same forward pass drives the robot until the next replan. See Appendix I for the full algorithm.

We first pretrain AW-0 on approximately 2,000 hours of YAM robot data and approximately 1,000 hours of egocentric human manipulation data to learn a general unified generative prior. We apply the automatic annotation pipeline in Section 2.3 to extract two embodiment-specific skill libraries containing 50 skill entries in total (23 for YAM and 27 for egocentric human demonstrations) and annotate demonstrations with temporally aligned program slices. The annotated data support joint pretraining of code, video, and action generation, connecting structured skill plans with visual dynamics and continuous control across embodiments and scenarios. We further include the details of dataset statistics, preprocessing, and state-action representations in Appendix B. For each downstream task, we reuse the same annotation pipeline under the frozen skill library to label task-specific demonstrations and jointly fine-tune AW-0 on the resulting code, video, and action targets using the objective in Eq. 6.

We evaluate AW-0 on RoboTwin 2.0 (Chen et al., 2025) and 8 real-world tasks spanning parallel-gripper and dexterous-hand embodiments, assessing language following, semantic reasoning, and skill execution, as well as the robustness benefits of hybrid execution in variant scenarios.

Baselines. We compare four baselines: (1) \pi_{0.5} (Black et al., 2025), a state-of-the-art VLA with strong semantic reasoning; (2) EgoScale (Zheng et al., 2026), a VLA pretrained on large-scale egocentric human data; (3) DreamZero (Ye et al., 2026), a WAM jointly generating future video and actions; and (4) WLA (Yang et al., 2026), an autoregressive model jointly predicting textual subtasks, subgoal images, and actions. All baselines and the default AW-0 configuration use the same task-specific demonstrations for post-training and follow the same evaluation protocol. The simulation hybrid configuration uses separately collected CaP-generated demonstrations, as described below. Baseline implementation details are provided in Appendix C.

Benchmarks. We evaluate AW-0 on RoboTwin 2.0. Following the training-data requirements specified on the official RoboTwin 2.0 leaderboard (footnote: https://robotwin-platform.github.io/leaderboard), we train on 50 clean demonstrations per task (2,500 episodes in total) and evaluate under the benchmark's two settings: clean, with a clean tabletop, plain background, fixed lighting, and no distractors; and random, with domain randomization over background and tabletop textures, lighting, tabletop height, and distractor objects. We report the mean success rate over 50 rollouts per task.

Hybrid Evaluation. We additionally evaluate AW-0 in the hybrid mode, in which the training demonstrations are collected by executing CaP programs written against a set of CaP-X-style (Fu et al., 2026) primitive APIs that we develop for RoboTwin 2.0. The programs are written automatically by Claude Opus 4.8, following the agentic framework of ASPIRE (Lu et al., 2026).

Main Results. Table 1 and Figure 4 report the results on RoboTwin 2.0. The two inference modes exhibit complementary strengths. Overall, AW-0 achieves the highest average success rate of 50.7% across both settings, outperforming all baselines including models designed for broad generalization. The advantage comes mostly from the clean setting, where AW-0 reaches 80.8%, exceeding the strongest baseline WLA (64.8%) by 16.0 points. Under visual randomization, performance drops to 20.6%, consistent with the challenges of pixel-level O.O.D prediction discussed in prior work (Yan et al., 2026). It falls below the VLA baseline \pi_{0.5} (27.7%) but still outperforms the strongest WAM baseline DreamZero (17.6%) and remains far ahead of EgoScale (2.9%) and WLA (3.6%).

We further observe that hybrid execution improves robustness under distribution shift. In the random setting, it raises the success rate from 20.6% to 29.2%, outperforming all baselines including the strong VLA baseline \pi_{0.5} (27.7%). Relative to its clean-setting performance, the hybrid mode retains 46\% of its success rate under randomization, compared with 25% for the default mode. This robustness comes at a cost in the clean setting, where the hybrid mode reaches 63.1%, on par with the strongest baseline, but 17.7 points below the default mode. For further analysis, see Appendix J.

Robot Platforms and Action Space. We conduct real-world experiments on two platforms: the YAM station, a fixed-base bimanual setup with two 6-DoF arms and parallel grippers, and the Tianji–Sharpa platform, a bimanual robot with two 22-DoF Sharpa dexterous hands. Both platforms observe three RGB views and act in the unified 64-D state–action space defined in Appendix A. Further details on the robot setups are provided in Appendix D.

Evaluation Protocol and Metrics. We use 100 demonstrations per task for post-training and evaluate on the 8 real-world tasks in Table 2, 4 on each platform, with 3\times16 rollouts per task (3 training seeds, 16 evaluation rollouts per seed) under randomized initial object poses. Task definitions, evaluation settings, and per-task scoring rubrics are provided in Appendix E.

Quantitative and Qualitative Analysis. As shown in Table 2, AW-0 achieves the highest average rubric score across both embodiments, outperforming the strongest baseline by 25.1 points. Two observations emerge. Language following and semantic reasoning: AW-0 outperforms the baselines strongly on Arrange Fruits and Flip Scoreboard, which require instruction-specific object selection and counting, respectively. The VLA baselines, particularly \pi_{0.5}, also show relatively strong semantic understanding, while the WAM baselines are less reliable on these properties. These results suggest that code-based planning strengthens the language-following and semantic reasoning of world action models. Robust skill execution: On contact-rich and dexterous tasks, DreamZero outperforms \pi_{0.5} on Replace Lightbulb and Deal Poker, where predicting future states provides stable physical-dynamics features for action prediction. AW-0 further outperforms these baselines by combining guidance from generated skill programs with predictive representations of physical dynamics, aligning skill-level planning with continuous control. See Appendix G for further analysis.

Hybrid Execution in the Real World. We further evaluate hybrid execution on Fold and Stack Towels with the YAM platform based on Algorithm I, comparing it against action-chunk execution under the same post-trained model and evaluation conditions. Under the same task rubric, hybrid execution achieves 90.6%, compared with 84.4% for action-chunk execution, while the direct execution steps of code further enhance the model's reasoning capabilities in distinguishing stacking order and action stability. Further visualizations are provided in Appendix G.3.

We organize our ablation study mainly around three research questions: 1) how to integrate a visually grounded code expert into a WAM framework, 2) which components of the planning representation contribute to task performance, and 3) how joint denoising affects task performance and inference efficiency. All the variants are post-trained and evaluated under the protocol of Section 3.2.

Q1: How should the code expert be integrated into a WAM? We first compare three blockwise causal attention orders during joint denoising: V\to A (w/o code), V\to C\to A, and C\to V\to A (Figure 5). The two variants use the same pretraining data, training budget, and optimization settings as AW-0. Adding code through V\to C\to A improves performance by grounding the predicted dynamics to action, but restricts its role to reasoning over video predictions without guiding or correcting them. With C\to V\to A, both video and action generation attend to code, allowing semantic task plans to shape future prediction throughout denoising. The performance gain of this ordering supports introducing semantic reasoning directly into video generation, which improves language following and provides more reliable guidance for stable action generation.

We then compare a dedicated ViT against reused Wan2.2 VAE features for conditioning the code expert (Table 3). ViT conditioning improves performance on all 8 tasks, with substantial gains on Pack Backpack (64.6% vs. 39.6%) and Solve Rubik's Cube (72.1% vs. 53.9%), which require fine-grained semantic reasoning about object attributes and task states. VAE features are optimized for pixel-level reconstruction (Yan et al., 2026) and thus poorly support the semantic distinctions needed to select and compose skills, whereas ViT features enable the code expert to translate visual observations into more accurate skill plans.

Q2: What makes code an effective planning representation? To assess the planning content produced by our annotation pipeline, we compare (1) full annotated programs, (2) code without comments, (3) code without skills, and (4) natural-language subgoals in Figure 7. Full programs outperform comment-free code on all 8 shared tasks, suggesting that explanatory language helps connect program structure with task intent. Removing skill calls also reduces performance on all 8 evaluated tasks, especially on skill-required tasks such as Deal Poker (58.3% vs. 42.0%). Since this variant still uses structured code, the drop supports the value of reusable skill abstractions beyond primitive-level API composition. Full programs also beat language-only subgoals on 7 of 8 shared tasks, showing that structured semantic guidance can provide more action-centric representation as condition. Together, these comparisons support combining explanatory language, reusable skills, and explicit program structure to provide effective planning guidance.

Q3: How does joint denoising affect performance and efficiency? As shown in Table 4, joint denoising outperforms AR on 7 of 8 tasks. For AR models trained with teacher forcing, the mismatch between ground-truth training prefixes and generated inference prefixes can cause errors to accumulate during inference and affect downstream video prediction. Joint denoising reduces token-by-token serial dependence by reconstructing code alongside video and actions, providing semantic guidance throughout generation. We measure end-to-end inference latency on an NVIDIA RTX 5090, with settings detailed in Appendix A. Joint denoising takes 0.23 s versus 0.55 s for AR, a 2.39\times speedup (Figure 6). Compared with the AW-0 w/o code (0.19 s), it adds 0.04 s (21.1%) while improving performance on all 8 shared tasks (Figure 5).

World Action Models (WAMs). World models predict future observations and have recently emerged as a powerful paradigm for robot manipulation, serving as learned simulators and data generators for policy learning (NVIDIA, 2025a; NVIDIA, 2025b; Gao et al., 2026; Yang et al., 2024; Jang et al., 2025). Building on this progress, World Action Models (WAMs) unify future prediction and action generation, through joint video-action diffusion (Ye et al., 2026; Zhu et al., 2025), expert-based architectures (Bi et al., 2025; Cai et al., 2026), or direct adaptation of pretrained video models into policies (Kim et al., 2026). Recent WAMs further couple autoregressive sub-task prediction with future-state modeling and action synthesis (Yang et al., 2026) or dynamically update subtask instructions to condition video and action generation (Tang et al., 2026). However, existing WAMs lack structured semantic guidance and explicit skill invocation, limiting their ability to perform complex semantic planning and structure predicted dynamics into actionable behavioral abstractions.

Agentic Robotics. Early work composed perception and control APIs into executable closed-loop behaviors without policy training (Ahn et al., 2022; Liang et al., 2023; Singh et al., 2023). Later systems generate full Python programs over perception APIs (Huang et al., 2023; Mu et al., 2024; Goldberg et al., 2025), and executable code has proven a stronger action space for LLM agents than text (Wang et al., 2024). With the rise of coding agents, program synthesis has become explicitly agentic, with recent systems iteratively writing, executing, and repairing control programs (Lu et al., 2026; Fu et al., 2026). Frontier models now also act as controllers themselves, reasoning over instructions and feedback and driving robots directly through native tool calls and learned primitives (Shi et al., 2025; Li et al., 2026; Gemini Robotics Team, 2025a; Gemini Robotics Team, 2025b). In all of these systems, the agent controls the robot from outside the policy and is bounded by the primitives it can call. We instead bring code inside an end-to-end model as a high-level planning interface and further explore how the generated code can be executed to complement the model's own actions at test time.

We present AW-0, an agentic world action model that integrates code-based skill planning with predictive dynamics and continuous control. Through joint denoising, the code expert provides structured semantic guidance for video and action generation. An automated annotation pipeline enables learning this planning interface from large-scale robot and human demonstrations. Experiments on RoboTwin 2.0 and real-world manipulation with YAM and dexterous hands demonstrate improvements in language following, semantic reasoning, and skill execution, with hybrid execution further enhancing robustness. These results establish a unified foundation for future work on skill reuse and iterative interaction between programmatic planning, world prediction, and execution, providing a potential agentic system foundation that continually interacts with and learns from environments.

Acknowledgments. Zhuoyang Liu thanks Prof. Shanghang Zhang at Peking University for her mentorship and support.AI use statement. We used Gemini 3.1 Pro and Claude Code with Claude Opus 4.8 in our automatic annotation pipeline to generate demonstration-aligned program labels and distill reusable skill libraries (Section 2.3). Claude Opus 4.8 was also used to generate executable programs for collecting simulation demonstrations (Section 3.1). Additionally, we used GPT-6 Astra for language polishing and stylistic refinement, as well as part of figure preparation.

Ethics statement. This work studies code-guided world action models for robotic manipulation using robot and egocentric human demonstrations. All pretraining datasets are used in accordance with their respective licenses and data-use terms. All post-training demonstrations are collected by our team for the tasks evaluated in this work.Reproducibility statement. We provide detailed information in Appendix A regarding the model architecture design, parameter settings, and computational resources. Regarding training data, Appendix B details the composition of the pre-training dataset. Section 2.3 introduces the complete data annotation pipeline. While prompts for the agentic framework and performance analysis of the pre-trained model are provided in Appendix K.2 and Appendix F.1. For downstream tasks, we also present standardized data generation criteria and methods to facilitate extension to any further tasks.

We implement the AW-0 architecture using three modality-specific experts coupled by masked joint attention. All experts have 28 transformer layers and project their queries, keys, and values into a common 2,048-dimensional attention space. The code expert receives current visual observations and language instructions, the video expert attends to the code stream, and the action expert attends to both. Tables 5 and 6 summarize the architecture and the pretraining configuration.

State and Action Representation. Both YAM robot data and egocentric human data are mapped into a shared 64-dimensional state–action space. Each side contains a 9-dimensional end-effector pose (3D of x,y,z position and a 6D rotation representation), 22 dexterous-hand joint angles, and one parallel gripper width, giving 2\times(9+22+1)=64 dimensions. States are all absolute; while action chunks use end-effector pose deltas relative to the first frame of the chunk, with absolute gripper targets for YAM and absolute hand-joint targets for human demonstrations/dexterous hand. Dimensions unused by an embodiment—the hand-joint blocks for YAM and the gripper dimensions for human demonstrations—are set to zero and left unnormalized. All remaining dimensions are normalized using per-dimension 1st and 99th percentiles.

Initialization and Trainable Components. We initialize the video DiT from Cosmos-Predict2.5-2B (NVIDIA, 2025b) and the action DiT from scratch before joint pretraining. The code expert initializes its token embeddings, layer normalizations, FFNs, and vocabulary head from SDAR-1.7B (Cheng et al., 2025) and freezes them, while training its MoT attention projections and modulation parameters. The Qwen3-VL (Bai et al., 2025) vision tower is frozen, with a trainable merger mapping visual features to code embeddings. We replace the original VAE with a frozen Wan2.2 (Wan et al., 2025) VAE with 48 latent channels and reinitialize the video DiT's input and output projections. The video DiT uses 1\times2\times2 latent patches and multi-view 3D RoPE. Video and action experts use precomputed Cosmos-Reason1 language embeddings. Downstream adaptation transfers shape-compatible parameters from joint pretraining and reinitializes action and state projections when embodiment dimensions change.

Observation and Program Representations. Images are sampled at timestep offsets 0,4,\ldots,32, giving nine frames per view; only the first frame is used as observation conditioning. The code expert's visual encoder also receives only this first frame from each view. At 224\times320 resolution, spatial merging produces 70 visual tokens per view. The code expert uses the Qwen3-family tokenizer of the SDAR checkpoint. Its 512-token training budget applies to the instruction and program together, with the 210-token visual prefix added separately. At inference, the default program canvas contains 128 tokens. Visual-prefix, instruction, and padding positions are excluded from the standard masked-program loss.

Joint Training and Decoding. Video noise resampling follows Section 2.2. The pretrained variable is u_{\mathrm{pre}}=0.999b, where b\sim\mathrm{Beta}(7,1); training samples are drawn from q=0.5p_{\mathrm{pre}}+0.5U(0,1). The video objective uses the pretrained-density weight, while action and code losses use the uniform-reference weight. Action noise levels and code-mask ratios are sampled independently of the video variable. At least one program token is masked in every nonempty training target. The pretraining recipe additionally replaces the instruction and program inputs with mask tokens for 15% of training batches, retaining the visual prefix. The default decoder progressively commits high-confidence code predictions across the full joint trajectory. Early completion is controlled by \rho<1, a separate decoding-time choice that does not change the training mask distribution. Latency is measured end to end on an NVIDIA RTX 5090, including visual encoding, joint denoising, video decoding, and action postprocessing.

Downstream Optimization. Downstream runs use the same optimizer, warmup schedule, and mixed-precision settings as pretraining, with task-specific batch sizes and training budgets. For both YAM and Tianji–Sharpa, we fine-tune AW-0 for 20 epochs using 100 demonstrations per task. The YAM configurations use a per-GPU batch size of four, one gradient accumulation step, a learning rate of 10^{-4}, and weight decay of 10^{-2}. The RoboTwin configuration uses two 8-GPU nodes with a per-GPU batch size of 6 and one accumulation step, giving an effective batch size of 96, a learning rate of 2\times10^{-4}, weight decay of 10^{-2}, and a budget of 100,000 optimizer steps. These settings are distinct from the larger pretraining batch in Table 6.

We pretrain AW-0 on the union of two real-world manipulation datasets: YAM and Egocentric Human data. The YAM corpus contains 73,101 episodes across 279 tasks, totaling 1,919.3 hours of bimanual robot teleoperation. Demonstrations are recorded at 30 Hz with left-wrist, right-wrist, and top-view RGB cameras. The egocentric human corpus contains 175,541 episodes across 327 tasks, totaling 993.9 hours, with a single head-mounted RGB camera and tracked hand motion retargeted to 22-DoF dexterous-hand joint targets per hand. Together, the two sources comprise 248,642 episodes and 2,913.2 hours. The human camera view is replicated across the three input slots; each training window is paired with its episode-level instruction and aligned program slice. Table 7 reports detailed statistics.

A skill is a reusable motion that the model can invoke by name. Each skill is a Python function written over the primitive API described in Appendix K.1, taking the objects it acts on as arguments, so that a subtask recurring across the corpus becomes a single call rather than being spelled out primitive by primitive. Following the annotation pipeline of Section 2.3, we generate a skill library for each dataset: 23 skills for YAM and 27 skills for the egocentric human data. Tables 8 and 9 list them, each with a one-line description of the motion it implements.

We visualize four of the 23 YAM skills. For each, we select one representative slice and show a strip of key frames, the corresponding task and subgoal annotations, the program slice, and the skill function. In the listings, calls in bold are library skills; all others are primitive calls.

Skill I: fold_edge_over
Task: “wrap blanket roll with ribbon”
Sub-goal: “Fold the blanket at the center of the board”Skill definition:def fold_edge_over(robot, item_name, left_delta, right_delta):
# pinch the edge with both arms
robot.grasp(item_name, 'left')
robot.grasp(item_name, 'right')
# carry the pinched edge across the item
robot.servo_ee_delta('left', left_delta)
robot.servo_ee_delta('right', right_delta)
# let the fold lie
robot.release('left')
robot.release('right')Program slice for this episode:fold_edge_over(robot, 'blanket', [0.15, 0.0, 0.0], [0.15, 0.0, 0.0])

Skill II: alternating_smoothing_pats
Task: “roll towels”
Sub-goal: “Flatten the towel”Skill definition:def alternating_smoothing_pats(robot, fabric_name, first_side, second_side):
# pat 1 -- press the fabric flat, then come off
robot.grasp(fabric_name, first_side)
robot.release(first_side)
# pat 2 -- the other hand, on the other half of the sheet
robot.grasp(fabric_name, second_side)
robot.release(second_side)Program slice for this episode:# Smooth out the towel with alternating pats
alternating_smoothing_pats(robot, 'towel', 'left', 'right')

Skill III: flip_object_over
Task: “fold napkin into a case and place utensils in it”
Sub-goal: “Turn over the spoon at the center of the board”Skill definition:def flip_object_over(robot, object_name, side, lift_delta, turn_deg=180.0):
# lift just clear so the object can swing
robot.servo_ee_delta(side, lift_delta)
# roll the wrist -- this is the flip
robot.rotate(object_name, turn_deg, side)
# let it fall the last centimetre, landing inverted
robot.release(side)
# pin the flipped object down again for the next slice
robot.grasp(object_name, side)Program slice for this episode:# Lift, roll the wrist to flip the spoon, drop it, and pin it
flip_object_over(robot, 'spoon', 'left', [0.0, 0.0, 0.1])

Skill IV: adjust_in_place
Task: “place bookmark into book”
Sub-goal: “Adjust the bookmark in the book”Skill definition:def adjust_in_place(robot, object_name, side, nudge_delta):
# take hold where it sits -- no lift
robot.grasp(object_name, side)
# slide it by a small bounded amount
robot.servo_ee_delta(side, nudge_delta)
# let it settle
robot.release(side)Program slice for this episode:adjust_in_place(robot, 'bookmark', 'right', [0.05, 0.0, 0.0])

We visualize six of the 27 egocentric human skills. For each, we select one representative slice and show a strip of key frames, the corresponding task and sub-goal annotations, the program slice, and the skill function. In the listings, calls in bold are library skills; all others are primitive calls.

Skill I: open_closure
Task: “assemble photo album”
Sub-goal: “Open the photo album”Skill definition:def open_closure(robot, container_name, closure_name, side, brace_side, axis, angle_deg):
# Hold the container body down so the closure, not the container, moves.
robot.grasp(container_name, brace_side)
# Take the closure where it sits.
robot.grasp(closure_name, side)
# Swing it open about its hinge.
robot.rotate(closure_name, axis, angle_deg, side)
# Come off the closure; it stays open.
robot.release(side)Program slice for this episode:open_closure(robot, 'photo album', 'photo album cover', 'left', 'right', 'object_long', 180.0)

Skill II: pour_into_target
Task: “serve rice into a bowl”
Sub-goal: “Pour a spoonful of rice into the bowl”Skill definition:def pour_into_target(robot, vessel_name, target_name, side, axis, angle_deg):
# Bring the held vessel's mouth over the receiving target.
robot.goto_relative(side, target_name, 'above')
# Tip it until it pours.
robot.rotate(vessel_name, axis, angle_deg, side)
# Bring it back upright; the pour ends.
robot.rotate(vessel_name, axis, -angle_deg, side)Program slice for this episode:pour_into_target(robot, 'spoon', 'bowl', 'right', 'object_long', 110.0)

Skill III: stir_contents
Task: “mixing solutions with stirring rod”
Sub-goal: “Stir the solution in the beaker”Skill definition:def stir_contents(robot, tool_name, vessel_name, side):
# Bring the held tool down inside the vessel, among the contents.
robot.goto_relative(side, vessel_name, 'inside')
# Sweep it round the vessel; the grip never changes, nothing is let go.
robot.rotate(tool_name, 'vertical', 360.0, side)
robot.rotate(tool_name, 'vertical', 360.0, side)Program slice for this episode:brace_and_take_tool(robot, 'beaker', 'left', 'stirring rod', 'right')
stir_contents(robot, 'stirring rod', 'beaker', 'right')

Skill IV: wipe_surface_back_and_forth
Task: “use lint roller on pants”
Sub-goal: “Remove the lint from the pair of pants”Skill definition:def wipe_surface_back_and_forth(robot, surface_name, relation, stroke_direction, return_direction, side):
# Park the cloth on the surface.
robot.goto_relative(side, surface_name, relation)
# Out and back.
robot.move_arm(side, stroke_direction)
robot.move_arm(side, return_direction)
# And again.
robot.move_arm(side, stroke_direction)
robot.move_arm(side, return_direction)Program slice for this episode:# Roll the lint roller back and forth over the pants (4 pairs of strokes)
for _ in range(2):
wipe_surface_back_and_forth(robot, 'pants', 'against', 'right', 'left', 'left')

Skill V: fold_over_and_crease
Task: “fold long pants”
Sub-goal: “Fold a pair of pants”Skill definition:def fold_over_and_crease(robot, item_name, side, brace_side, direction, fold_direction):
# Hold the item down so the fold, not the whole item, travels.
robot.grasp(item_name, brace_side)
# Take the free edge with the working hand.
robot.grasp(item_name, side)
# Carry the edge across the item ...
robot.move_arm(side, direction)
# ... and land it on the far side.
robot.move_arm(side, fold_direction)
# Let the folded edge down.
robot.release(side)
# Press the crease flat.
robot.press(item_name, side)Program slice for this episode:fold_over_and_crease(robot, 'pants', 'left', 'right', 'up', 'right')

Skill VI: wrap_around_target
Task: “use a curling iron to curl hair”
Sub-goal: “Wrap a section of hair around the curling iron”Skill definition:def wrap_around_target(robot, target_name, side):
# Start over the top of the target.
robot.goto_relative(side, target_name, 'above')
# Over the far side.
robot.move_arm(side, 'away_from_body')
# Down its far face.
robot.move_arm(side, 'down')
# Back underneath, toward the body.
robot.move_arm(side, 'toward_body')
# And up to close the circuit.
robot.move_arm(side, 'up')Program slice for this episode:# Wrap the hair around the curling iron twice.
for _ in range(2):
wrap_around_target(robot, 'curling iron', 'left')

In Figure 18 and 19, we visualize the skill distribution of variant skills in the pretrained code expert's embedding space. Each point represents one episode slice, placed by a two-dimensional t-SNE projection of its embedding and colored by the skill assigned to it. Dashed outlines mark the region covered by each skill, and the frames along the edges show a representative short span from an episode for selected skills. Figure 18 covers the 18 most common skills in the YAM library and Figure 19 the 16 most common skills in the egocentric human library. In both, slices assigned to different skills occupy largely separate regions, suggesting that the skill label is predictive of the motion in a slice and can serve as a conditioning signal for action generation.

In Sections 3.1 and 3.2, we compare AW-0 with four baselines: \pi_{0.5}, DreamZero, EgoScale, and WLA. Here, we describe how each is reproduced and adapted. On RoboTwin 2.0, we train a single multi-task policy per baseline on 2,500 demo_clean demonstrations (50 tasks \times 50 demonstrations). For each of the 8 real-world tasks, we train a separate policy per baseline using task-specific demonstrations sampled at 30 Hz. Every baseline receives a language instruction, the robot state, and three RGB views (one head and two wrist cameras). All baselines follow the same evaluation protocol as AW-0. All real-world rollouts are scored using the task-specific rubrics in Appendix E.

\boldsymbol{\pi_{0.5}} (Black et al., 2025) is a vision-language-action model that combines a pretrained vision-language backbone with a flow-matching action expert. We follow the official OpenPI implementation (footnote: https://github.com/Physical-Intelligence/openpi) and initialize all policies from the released \pi_{0.5} pretrained checkpoint. For each real-world task, we fine-tune a separate policy for 50 epochs on the task-specific demonstrations. We use 14-dimensional joint-space actions on RoboTwin and YAM, and 62-dimensional actions on Tianji–Sharpa consisting of a 9-dimensional end-effector pose and 22 hand joints per side. Arm actions are relative to the current state, while gripper and hand actions are absolute targets. We reinitialize the action input and output projections to accommodate these dimensions and fine-tune all model parameters. Following the official implementation, we use an action horizon of 50 for RoboTwin and 32 for the real robots, together with quantile normalization and standard image augmentation. All models are trained with AdamW on eight H100 GPUs using FSDP and a global batch size of 256. We use a peak learning rate of 5\times10^{-5} with cosine decay to 5\times10^{-6} and 2,000 warmup steps.

DreamZero (Ye et al., 2026) is a world-action model that jointly generates future video and actions using a pretrained video diffusion backbone. We follow the official implementation (footnote: https://github.com/dreamzero0/dreamzero) and initialize from the DreamZero-AgiBot checkpoint. We reproduce DreamZero on RoboTwin and all 8 real-world tasks, training a separate policy for 10 epochs on each real-world task. We fully fine-tune the diffusion transformer and state/action modules, while keeping the text encoder, image encoder, and video VAE frozen. Following the original design, the three camera views are tiled into one frame, and each training sample contains up to four action chunks of 24 steps. Actions are predicted relative to the state at the beginning of each chunk, with absolute hand-joint targets on Tianji–Sharpa. The state and action projections are adapted to the target embodiment, and the video and action losses are weighted equally. All policies are trained with AdamW on 32 H100 GPUs using a global batch size of 32. We use the same training hyperparameters across all tasks, with a peak learning rate of 2\times10^{-5} and 5% linear warmup followed by cosine decay.

EgoScale (using the GR00T N1.7-3B checkpoint) (Zheng et al., 2026) studies how pretraining on egocentric human demonstrations scales to dexterous robot manipulation. We reproduce this baseline using the GR00T N1.7 implementation (footnote: https://github.com/NVIDIA/Isaac-GR00T) and initialize from the pretrained nvidia/GR00T-N1.7-3B checkpoint. For each real-world task, we fine-tune a separate policy for 50 epochs while keeping the vision-language backbone frozen and training the projection layers and diffusion action head. We use relative joint actions for the arms on RoboTwin and YAM, relative end-effector actions on Tianji–Sharpa, and absolute targets for the grippers and 22-DoF dexterous hands. Following the official fine-tuning recipe, we apply state dropout with probability 0.2, random image cropping, and color jitter. We use an action horizon of 20 steps and train with AdamW on eight H100 GPUs with a global batch size of 640. The peak learning rate is 10^{-4}, with 5% linear warmup followed by cosine decay and a weight decay of 10^{-5}.

WLA (Yang et al., 2026) combines a vision-language backbone, a world expert, and an action expert to jointly model visual observations, language, and robot actions. We follow the official implementation (footnote: https://github.com/SJTU-DENG-Lab/WLA) and initialize the backbone and world expert from pretrained RynnBrain-2B and Sana-600M weights, respectively. The connector and action expert are randomly initialized. We train a separate policy for 50 epochs on each real-world task, updating all modules except the image VAE. Following the original design, the policy observes the current head and wrist views together with the head view from 8 steps earlier. We use 64 meta-queries and a 32-step action chunk, with absolute action targets normalized to [-1,1]. The world expert predicts the head-camera image at the end of the chunk, and its flow-matching loss is weighted by 0.1 relative to the action loss. All models are trained with AdamW on eight H100 GPUs using a global batch size of 256. We use a peak learning rate of 5\times10^{-5} with cosine decay to 5\times10^{-6} and 1,000 warmup steps.

We conduct data collection and policy evaluation on two robot embodiments: the YAM station with parallel grippers and the Tianji–Sharpa station with dexterous hands. This section describes the robot hardware, visual observations, and teleoperation interfaces used in our experiments. Figure 20 gives an overview of the two stations.

Robot Hardware and Control. The YAM station is a fixed-base bimanual system with two 6-DoF arms, each equipped with a parallel gripper. The Tianji–Sharpa station consists of a Tianji humanoid torso with two arms and two 22-DoF Sharpa dexterous hands. For AW-0, both embodiments use the 64-dimensional state–action representation described in Appendix A. The policy predicts relative end-effector poses together with absolute gripper targets on YAM or absolute finger-joint targets on Tianji–Sharpa.

Perception System. Both stations provide three RGB views: a head or overhead camera observes the tabletop workspace, and a camera at each wrist provides a local view of the gripper or hand and the manipulated objects. Each policy receives these images together with the task instruction and robot state. For AW-0, images are resized to 224\times320 pixels, and the state contains the end-effector poses and the gripper positions or hand-joint angles. The same camera streams are provided to all baselines, with method-specific preprocessing described in Appendix C.

Teleoperation and Data Collection. We collect YAM demonstrations using the standard YAM station configuration, with two master arms and two puppet arms. The operator moves the master arms, whose measured joint angles are sent as joint-position targets to the corresponding puppet arms. The triggers on the teaching handles control the parallel gripper openings, yielding six arm-joint targets and one gripper target per side. For Tianji–Sharpa, we use Manus gloves with PICO motion trackers attached to the gloves, together with an Apple Vision Pro worn around the operator's neck. The tracked wrist positions and orientations define poses in \mathrm{SE}(3), which are mapped to end-effector targets for the two Tianji arms. In parallel, finger-motion measurements from the Manus gloves are retargeted to 22-dimensional joint-position targets for each Sharpa hand. At each teleoperation step, the arm-pose and hand-joint targets are passed to the corresponding robot controllers to coordinate arm and finger motion. During collection, we record the three RGB camera streams, measured robot states, and commanded action targets. We use their timestamps to pair visual observations with robot states and action commands, and associate each demonstration with a task instruction for post-training.

We evaluate 8 real-world tasks that span two robot platforms. Arrange Fruits, Pack Backpack, Replace Lightbulb, and Fold and Stack Towels run on the YAM bimanual setup (two 6-DoF arms with parallel grippers); Arrange Fruits with Tongs, Deal Poker, Flip Scoreboard, and Solve Rubik's Cube run on the Tianji humanoid torso with two Sharpa dexterous hands. For each task we show the progress with the key frames of one training episode together with its code label; every color-coded block of the program is linked to the range of frames it explains. Each task is graded with an additive rubric that awards independent partial credit for each completed sub-step.

Task I: Arrange Fruits. Grading rubric (additive):

• +0.3: (a) Moves each fruit not named in the instruction from the left bowl into the right basket (or none to remove).
• +0.3: (b) Moves each named fruit not yet in the left bowl from the right basket into the bowl (or none to fetch).
• +0.4: (c) Final state matches the instruction and no fruit is dropped.

Task II: Pack Backpack. Grading rubric (additive):

• +0.15: (a) Left gripper grasps the zipper pull at the left end and slides it to the middle.
• +0.15: (b) Right gripper re-grasps the pull at the middle and slides it to the right end.
• +0.2: (c) Open the backpack with the left gripper.
• +0.2: (d) Picks up the item from the table and places it inside the open backpack.
• +0.15: (e) Right gripper grasps the pull at the right end and slides it back to the middle.
• +0.15: (f) Left gripper slides the pull from the middle to the left end; the backpack is fully closed with the item inside.

Task III: Replace Lightbulb. Condition A – dead bulb, replacement required. Grading rubric (additive):

• +0.1: (a) Left gripper presses the inline switch and the bulb stays dark.
• +0.1: (b) Left gripper braces the lamp base.
• +0.2: (c) Right gripper unscrews the dead bulb and lifts it out of the socket.
• +0.1: (d) Dead bulb is set down in the bulb tray.
• +0.1: (e) Right gripper picks up the spare bulb.
• +0.3: (f) Spare bulb is seated in the socket and screwed in clockwise until lit.
• +0.1: (g) When the bulb is lit, release both arms to their home positions.

Condition B – working bulb, no replacement needed. Grading rubric (additive):

• +0.3: Left gripper presses the inline switch and the bulb lights up.
• +0.7: The lit bulb shows the lamp still works, so both arms move back to their initial pose and stay still until the episode ends.

Task IV: Fold and Stack Towels. Grading rubric (additive):

• +0.2: (a) Grasps a towel edge and completes one fold.
• +0.2: (b) Each of the three towels is folded flat.
• +0.1: (c) Picks a folded towel up and lays it on the pile.
• +0.2: (d) All three folded towels end up in a single pile, none dropped or left behind.
• +0.3: (e) The pile reads, bottom to top, in the colour order the instruction names.

Task I: Arrange Fruits with Tongs. Grading rubric (additive):

• +0.2: (a) Right hand takes the tongs off their stand and keeps hold of them.
• +0.25: (b) Moves each fruit not named in the instruction from the left bowl into the tray (or none to remove).
• +0.25: (c) Moves each named fruit not yet in the left bowl from the tray into the bowl (or none to fetch).
• +0.2: (d) Final state matches the instruction and no fruit is dropped.
• +0.1: (e) Tongs are put back on their stand.

Task II: Deal Poker. Grading rubric (additive):

• +0.1: (a) Right hand grasps the card stack from above.
• +0.2: (b) Right hand transfers the stack to the left hand (handover).
• +0.3: (c) Right thumb flicks and adjusts until exactly one card protrudes from the top.
• +0.3: (d) Right hand grasps the single protruding card.
• +0.1: (e) Right hand inserts the card vertically into the card slot.

Task III: Flip Scoreboard. Grading rubric (additive):

• +0.1: (a) Left hand holds the scoreboard steady.
• +0.2: (b) Right hand flips a digit card over (at least one complete flip).
• +0.4: (c) Flips exactly k times, so the board shows the initial score plus k.
• +0.1: (d) Left hand releases the board.
• +0.2: (e) Right hand presses the red button while the target score is showing.

Task IV: Solve Rubik's Cube. For each rollout, one randomly selected face is scrambled clockwise or counterclockwise; the visible face colors and placement angle are fully randomized, with no fixed layer or turn sequence.Grading rubric (additive):

• +0.2: (a) Right hand grasps the cube and lifts it off the table.
• +0.3: (b) Fingers turn the scrambled layer a full quarter turn, leaving no layer half-turned.
• +0.3: (c) The cube is solved: every face shows a single colour.
• +0.2: (d) The cube is put back on the table and both hands let go.

We evaluate the pretrained code expert on the validation split of the pretraining dataset, before downstream task-specific fine-tuning. Table 10 reports quantitative metrics on this split, while Figures 29 and 30 present generated programs for YAM and egocentric human validation examples. We examine visual grounding, program structure, and semantic agreement with the reference annotations.

Quantitative Assessment. The model achieves 89.7% teacher-forced token accuracy and an 87.5% parse rate under free generation. Shuffling visual observations increases the code loss by 1.774, indicating that code prediction depends on the visual context. Entity and primitive F1 are 0.534 and 0.365, respectively, measuring overlap of quoted strings and robot API names with the reference programs. These scores indicate partial semantic agreement despite the higher syntactic validity. In addition, 29.2% of generated programs contain skill-style helper calls, identified by calls that take robot as their first argument. Together, these results show that pretraining learns visually conditioned program generation and skill invocation, while precise semantic and argument prediction remain areas for improvement.

YAM Robot Demonstrations. Figure 29 shows a held-out YAM demonstration of “Clean keyboard with a cloth". The generated program expresses a cloth handover followed by repeated scrubbing through scrub_back_and_forth, preserving the object arguments, arm assignment, and loop structure of the reference subtask. This illustrates how a reusable skill can encode a repeated motion compactly. The generated displacement argument nevertheless contains two extra entries, illustrating that an appropriate skill-level plan can still contain parameter errors that prevent reliable direct execution.

Egocentric Human Demonstrations. Figure 30 illustrates subtask-level planning under the instruction “Squeeze toothpaste.” The generated program describes the preparatory subtask of removing the cap: stabilizing the tube, repeatedly twisting the cap, and lifting it away. It matches the annotated calls, arguments, and loop structure for this validation window. The two examples illustrate how the pretrained model represents manipulation through explicit object references, ordered calls, and repetition across robot and human observations, while the aggregate metrics distinguish this representational capability from uniformly correct or executable code.

We examine code generated online by the post-trained model during real-world deployment on the YAM and Tianji–Sharpa platforms. Figures 31 and 32 show examples from Pack Backpack and Solve Rubik's Cube, respectively. Each example pairs the observation at a replanning point with the generated program and subsequent recorded frames from the same deployment rollout. Reference programs are drawn from separate annotated demonstrations of the same task and subtask, providing qualitative comparisons of skill selection, object arguments, and call structure.

YAM Deployment Rollout. Figure 31 shows Pack Backpack at the stage where the right arm grasps the zipper pull at the middle of the seam. The generated program invokes approach_best_grasp for the zipper pull, followed by robot.close_gripper('right'). The generated program uses the same grasping skill, target object, arm assignment, and numerical arguments as the reference program. This example illustrates how the post-trained model expresses a task-specific manipulation step through a reusable grasping skill and an explicit gripper command.

Tianji–Sharpa Deployment Rollout. Figure 32 illustrates subtask-level planning for Solve Rubik's Cube while the right hand holds the cube steady. The generated program reproduces hold(robot, 'rubiks cube', 'right'), matching the annotated skill, object, and hand assignment. The generated skill call agrees with the reference program in skill choice, object, and hand assignment. The output nevertheless contains a malformed comment: the word cube is omitted, and the comment continues on a new line without a comment marker. This formatting error prevents direct execution of the raw output despite the correct skill call. Together, the two examples illustrate task-specific semantic agreement across parallel grippers and dexterous hands, while distinguishing appropriate skill selection from uniformly valid executable code.

We further examine whether code-based planning improves the alignment of predicted visual dynamics with task semantics. As shown in Figure 33, we compare the video generation quality of the right wrist camera of AW-0 with a natural-language subgoal variant (Subgoal) and a model without a planning stream (AW-0 w/o code), under the instruction “Arrange the fruits so that finally there is only mango in the left bowl.” This task requires identifying the intended object and grounding the desired outcome in an appropriate interaction. The three variants are evaluated under the same task and scene configuration, with similar but non-identical initial observations from separate rollouts.

The predicted futures reveal distinct differences in task understanding: AW-0 directs the interaction toward the mango, whereas Subgoal exhibits ambiguous targeting between the apple and mango, and AW-0 w/o code targets the apple. These observations suggest that structured skill calls and explicit object arguments provide a more specific planning condition for video prediction than a natural-language subgoal alone. By connecting instruction semantics to concrete interactions, the code stream helps align predicted dynamics with the intended task. Since the action expert also conditions on predictive video representations, this alignment provides a pathway for improving language following, complementing the execution results in our main experiments.

We further examine whether code-based planning improves the prediction of physical dynamics and skill robustness during precise manipulation. As shown in Figure 34, we compare the future frames predicted by AW-0, Subgoal, and AW-0 w/o code for the right wrist camera on Deal Poker, focusing on the final stage in which the right hand inserts a card into a slot. This interaction requires aligning the held card with the narrow opening and guiding it into the slot, making the relative motion of the hand, card, and receptacle critical to an accurate prediction. The three variants receive similar but non-identical initial observations sampled at nearby time steps from the same episode.

The predicted futures reveal distinct differences in the insertion motion: AW-0 depicts the card aligning with and entering the slot more accurately, whereas Subgoal and AW-0 w/o code produce less coherent insertion sequences, with blur and distortions around the hand and manipulated objects. These observations suggest that explicit skill calls and object arguments help the video expert better predict how the hand and card should move relative to the receptacle. The implementation of the code expert supports more accurate video generation, and therefore provides the action expert with more reliable predictive guidance for precise manipulation.

Figure 35 is a qualitative example of hybrid evaluation on the “Fold and Stack Towels" task, showing a single episode that continues from the upper to the lower row. We observe that dexterous folding motions rely primarily on direct actions from the action expert of AW-0, while code execution handles more structured motions, such as approaching and stacking the towels, which provide direct guidance on the stacking order and motion direction, improving the semantic reasoning capability for the whole system.

Figure 36 shows representative failure cases across various scenarios. The red boxes highlight the local interactions where execution fails. These examples illustrate some remaining challenges despite the overall performance gains of AW-0.

1) Container Grasp. In Arrange Fruits in the first row, the target fruit lies close to the basket rim. As the gripper approaches, its left jaw moves outside the basket, causing the closing motion to grasp the basket wall rather than only the fruit. This case highlights the need to account for surrounding geometry when selecting a grasp. Explicitly estimating the rim geometry and checking clearance along both the approach and closing paths could help reject obstructed grasps and select a more accurate approach direction.2) Object Dragging. In Pack Backpack in the second row, when the robot pulls the zipper, the backpack moves and tips over rather than remaining stable. The visible failure suggests insufficient stabilization of the deformable object during manipulation. A potential improvement is to coordinate the supporting and pulling motions through more frequent visual feedback: the supporting hand could adjust its grasp as the bag deforms, while the pulling hand could slow down or pause when the bag starts to shift or tip.

3) Imprecise Alignment. In Replace Lightbulb in the third row, the robot grasps the bulb and brings it above the socket, but the bulb base remains misaligned with the socket opening and is not screwed in. Completing this step requires both positional and angular alignment before the threads can engage. The case illustrates a gap between moving an object to the correct region and achieving the precise relative pose required for insertion and rotation.4) Tool Collision. In Arrange Fruits with Tongs in the fourth row, the tongs approach the pear with their tips closed and poke it rather than opening around it to establish a grasp. Successful tool use requires coordinating the tool's opening with its approach trajectory, in addition to holding and positioning the tool itself. Explicitly tracking the tong tips and opening width could help condition actions on the tool's state. A feedback check that the tips are open and positioned on opposite sides of the fruit before closing could further reduce unsuccessful contact.

5) Slipping Off. In Deal Poker in the fifth row, the left hand fails to secure the card stack during handover from the right hand, causing the cards to slip. A successful handover requires the receiving hand to establish a stable grasp before the giving hand releases the stack. This failure illustrates the difficulty of coordinating grasping and release across the two hands while maintaining support for the cards throughout the transfer.6) Contact Loss. In Flip Scoreboard in the sixth row, the fingertip reaches the edge of a score card but slides off before turning it over, leaving the displayed score unchanged. Unlike a brief touch, a successful flip requires sustained contact along the card's rotation about the binding. This case illustrates the difficulty of sliding the fingertip along the first card to lift its edge while maintaining continuous contact.

Algorithms 1 and 2 outline the workflow of the hybrid mode. From an observation o and instruction \ell, one forward pass of the policy \pi returns a direct action chunk a_{1:H} over the replan horizon H and a code slice c. Every slice executes in a shared namespace \mathcal{G} containing the environment's skill API, reusable helper definitions, and its setup block (episode-start perception), which persists throughout the episode. At each replanning point, Arbitrate decides which of the two channels drives the robot. The returned flag indicates whether code execution was attempted, not whether it succeeded. If execution is attempted, the action chunk from the same policy invocation is discarded, even if an exception occurs; the next iteration observes the current state and replans. Otherwise, the chunk is played for H steps as in the default mode.

The original generated slice c is checked without repair. It is executed only if it passes the following gates in order:

1. Stage Gate: its stage is not complete, is not below a stage already attempted, and has fewer than R failed execution attempts;
2. Dependency Gate: every variable it reads is defined in \mathcal{G}; and
3. Syntax, Whitelist, and Vacuity Gates: it compiles as generated, calls only whitelisted functions, and contains an executable statement.

Table 11 reports the per-task breakdown of the aggregate results in Table 1.

Perception and Control Primitives. During hybrid evaluation, generated code executes through perception and control primitives implemented using Molmo for visual grounding (Deitke et al., 2024), SAM 3 for segmentation (Carion et al., 2025), Contact-GraspNet for grasp generation (Sundermeyer et al., 2021), and RoboTwin's native motion-planning and gripper-control interfaces (Chen et al., 2025). Neither the generated programs nor the underlying primitives access privileged simulator information during evaluation. For demonstration collection, we preserve the same API interface but use privileged object states and annotated grasps to improve collection efficiency.

Scope of Comparison. The default and hybrid configurations differ in both their training demonstration collection pipelines and execution modes to support their training requirements. Unlike the default configuration, which uses demonstrations collected with RoboTwin's native motion planner, the hybrid configuration uses CaP-based data collection because it naturally pairs each demonstration with an aligned program executable in the RoboTwin simulator, supporting the code execution component of hybrid actions. Therefore, our resulting comparison evaluates the two configuration as a whole and does not isolate the effect of hybrid execution alone.

Code and Action Usage. Table 12 reports the fraction of simulation steps driven by each of the two control channels during the hybrid evaluation in Table 1: execution of code produced by the code expert and direct action chunks generated by the action expert. On successful episodes, code drives 14.5% of steps in the clean setting and 35.0% in the random setting. The gap follows from Table 1. In the clean setting, the action expert alone is already strong (80.8%), so there is little for code to add. Admitting code in fact lowers success to 63.1%, which we attribute to the code channel inheriting the failure modes of the low-level perception and control primitives it calls, and the arbiter admits a code slice on validity alone, without predicting whether it will outperform the action chunk it displaces. In the random setting, where direct action degrades to 20.6%, the code channel carries a third of the physical time on the episodes that succeed and lifts success to 29.2%. Executing the code expert's generation thus complements the action expert especially where learned actions are less reliable, here under visually out-of-distribution scenes. Pooled over all episodes, code execution accounts for only 8.5% of steps in the clean setting and 5.6% in the random setting. The pooled share is dominated by failed episodes, which run to the step cap and accumulate roughly five times as many steps as successful ones (52.7k vs. 11.5k in clean, 55.5k vs. 8.4k in random), almost all under direct action.

Qualitative Examples of Code–Action Interleaving. Figure 37 shows two qualitative examples of how code execution and direct action generation interleave within an episode. The top progress bar shows an episode of blocks_ranking_rgb, where the three blocks must be arranged in a row in red, green, blue order. We observe that the three pick-and-place cycles are almost entirely carried out by code execution, with direct action generation taking over around a stage transition. For tasks of this kind that involve regularly shaped objects and well-structured motions, the integration is able to utilize the stable and smooth behavior yields by the CaP actions. The bottom progress bar shows an episode of pick_dual_bottles, where the two bottles must be picked up and held together. Here, code execution correctly localized the bottle and moves the gripper to it but fails to secure the grasp due to its less regular shape. Direct action generated from the model then takes over and completes the grasp, after which code execution resumes to finish the episode.

Figure 38 shows the two API surfaces for the parallel-gripper (left) and the dexterous-hand (right) embodiments. They share ten semantic contact verbs but differ in their low-level primitives: end-effector poses for the gripper and relative motion for the dexterous hand. Each API call declares the contact events it produces, which is what allows a program to be grounded against a demonstration.

The framework has two agents: a VLM annotator that writes program lines for one subtask slice per call, and an agentic coordinator that audits annotator output between rounds and grows the skill library (Stage 1). After the library is frozen, the same annotator runs over the remaining corpus and over downstream demonstrations, reusing cached subtask code where the embodiment's evidence verifies it. The full API docstrings and skill documentation are appended to the system prompt and omitted here.

Shared annotator system instructions

• Role and output Read the frames of one subtask and express the observed actions in Python using only the robot API and documented skills. Output only new code for this subtask, continuing the preceding program.
• Grounding and coverage The subtask annotation states what happens; the frames ground which objects and how they move. Cover the whole frame span and invent nothing that neither source supports.
• Abstraction level Prefer the single highest-level call whose contact effects match; never wrap a semantic verb in the primitives it already contains. Use bare open/close primitives only when no verb fits.
• Repetition and literals Write repeated in-place twisting as one literal loop with the count read from the frames. Hand sides and loop counts must be literals. End on a complete statement.
• Static validation Code is checked without execution: syntax in program context, existence of called functions, argument binding, and vocabulary membership of literals. On failure, the annotator sees its output along with the identified problems and rewrites the slice.

System prompt for egocentric human demonstrations

• Setting Two hands, ‘left' and ‘right', seen from a head-mounted camera. No contact-event list is given; infer every contact from the frames and the annotation, judging object motion relative to the scene.
• Contact semantics pick/grasp close a hand (pick lifts, grasp holds in place); place/release/insert/attach open it; regrasp opens then closes; press closes then opens; handover closes the receiver before opening the giver.
• Motion and naming Motion is purely semantic: move_arm along a documented direction, goto_relative at a documented relation to a named object, rotate about a documented axis. No coordinates; objects are named in plain language.
• Conventions Calls return None. Distances only when visibly measurable; angles as semantic constants.

System prompt for YAM robot demonstrations

• Setting Two arms seen from a top-down camera. The sensor-derived gripper-event list is authoritative: emitted code must reproduce every open/close transition on the named arm, in order, with nothing added, dropped, or reordered.
• Event gaps If the annotation implies a contact the event list lacks, write the approach and transport in full but do not emit the missing open/close call.
• Contact semantics grasp/pick close; place/release open; regrasp opens then closes; insert closes then opens; handover closes, closes, then opens. A repeated same-arm cycle in the list is one literal loop.
• Perception and motion Every pose used by a motion call comes from a perception result earlier in the program; nothing is hard-coded. Absolute single-arm, synchronized two-arm, and small relative motion each have their own primitive.

Per-slice user message

• Overall task [task instruction]
• Current subtask [human subtask annotation]
• Frames [ordered frames with timestamps]. On YAM, the frame before each gripper event is captioned with the event index, kind, arm, and time.
• Program so far [code written for the preceding subtasks]
• Egocentric directive Decide from the frames and annotation where each hand closes on and opens off an object, and produce exactly those contacts with the visible motion around them. A comment-only slice is correct only when the hands neither move nor touch anything.
• YAM directive Observed gripper events: [ordered list, repeated cycles collapsed to a count]. Reproduce this sequence exactly, with the visible motion around it; a brief blip with nothing in hand gets the minimal call.
• Output Continue the program with this subtask's code only.

Only the directive for the current dataset is included.

Annotation under the frozen library

• Same prompt, frozen vocabulary The system prompt above is used unchanged, with the skill block fixed to the frozen library. Inherited skills are called as plain functions with robot as the first argument.
• No new vocabulary The annotator may not invent helper names; a maneuver no verb or skill expresses is written from primitives. Extending the library is the coordinator's decision.
• Verified reuse A per-task cache of subtask code is consulted before each call. A variant is reused only when the embodiment's evidence verifies it: on YAM, the contact events its code implies match the observed events (mirrored forms accepted, motion arguments re-derived); on egocentric data, its hand-motion profile is within a fixed distance of the new slice, and loop templates are never reused. A miss triggers the ordinary call, whose output is written back.

Coordinator protocol (Stage 1, skill discovery)

• Role A coding agent under a fixed written protocol that dispatches rounds, audits their output, and owns the skill library. It never writes program code and never edits the annotator, prompts, API, or verification code.
• Round lifecycle Dispatched, gated (all episodes annotated and scored), audited, done (admissions applied and logged). The library is frozen from dispatch to done, and the next round starts only then.
• Audit Read the round summary, then per-task results, then the stored trace of each questionable slice (frames seen, code written, verification outcome). Classify each as detector limit, annotator misread, abstraction-level break, vocabulary gap, or segmentation artifact. Only vocabulary gaps feed skill proposals.
• Skill admission A candidate is a straight-line composed function over the API, distilled from observed call sequences. It is admitted if generic (episode-specific values lifted to parameters), recurrent across at least two tasks, at the right level (a maneuver, not a task recipe), and not a duplicate. Its contact contract is derived by expansion; on YAM the evidence slices must also reconcile against observed events. Admitted skills enter the annotator prompt from the next round.

Program Grounding and Skill Coverage. The code stream learns from automatically annotated demonstrations and a curated skill library, making its planning quality dependent on annotation accuracy and skill coverage. Generated programs can select an appropriate skill yet contain incorrect arguments, as illustrated in Appendix F.1; syntactic validity therefore does not ensure correct execution. Future work could use execution feedback to improve program grounding and discover or refine reusable skills beyond the initial library.Predictive Robustness and Hybrid Execution. Structured planning improves WAM performance but does not eliminate sensitivity to visual distribution shifts. Hybrid execution improves robustness under RoboTwin randomization, yet reduces clean-setting performance (Appendix J.2). This trade-off reflects both the reliability of the underlying primitives and the limitation of selecting code based on validity without estimating its advantage over the action chunk. Learning to arbitrate using execution reliability and expected task progress could better exploit the complementary strengths of the two modes.

Toward Continual Agentic Learning. Our real-world hybrid evaluation is limited to YAM platform due to the lack of code-as-policy systems for dexterous hand, and the current system does not learn online from its own interactions. The unified design nevertheless offers a basis for coupling program generation and world prediction with execution feedback and model updates. Future work could explore iterative plan revision, skill-library expansion, and autonomous learning through sustained environmental interaction, with broader evaluation across tasks and embodiments.