Humans imitate at the level of intent: given a demonstration, we infer its goal and carry it out with whatever tools, objects, and layouts are at hand. Current robot policies instead learn observation-to-action mappings from visual inputs and language instructions, without explicitly inferring the demonstrated task. Learning from human video thus remains largely trajectory-level: models can replay motions in near-identical scenes, but still struggle to imitate what the demonstrator intends rather than merely what they do. We introduce The Imitator Game, a four-level benchmark (L0–L3) that progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice and task understanding becomes necessary.

We pair it with IG-10K, the largest environment-aligned paired human–robot dataset to date and the only one instantiated across all four levels in both real and simulated settings (20,000+ paired episodes, 50+ tasks, 6 domains), and Imitator Arena, an open platform for blind A/B human evaluation.

Across nine state-of-the-art models, performance is stable from L0 to L2 but collapses at L3, identifying functional substitution – achieving the same intent through a different object affordance – as the decisive barrier to intent-level imitation. Human-video-conditioned models outperform caption-conditioned ones, yet every model falls below 13\% zero-shot success on unseen tasks; fine-tuning IG-10K-pretrained models with only 10 paired human-robot demonstrations yields large gains that grow with pretraining scale.

When people watch someone stir soup, they do not memorize a sequence of motions. They infer a goal-directed behavior: the soup needs to be stirred, with whatever is available. If the pot is moved, the spoon replaced with chopsticks, or the bowl exchanged for a wok, people adapt immediately while preserving the underlying intent. This form of goal-directed imitation (Tomasello, 2009) – reproducing the purpose of an action rather than its exact motion – is a hallmark of human and great-ape cognition.

Robots do not yet share this ability. Most approaches for learning from human video rely on trajectory remapping: extracting hand keypoints or optical flow from a demonstration (Qin et al., 2022; Xu et al., 2023), retargeting the motion to robot actions, and replaying it in a closely matched scene (Sivakumar et al., 2022; Lepert et al., 2025). These methods are useful for bootstrapping robot behaviour, but they depend heavily on the demonstration and execution scenes remaining similar. Small changes in object placement, geometry, or available tools can already invalidate the demonstrated motion, even when the underlying task itself is unchanged.

The question we ask differs from that of prior work. Existing generalization benchmarks (Pumacay et al., 2024; Sedlacek et al., 2026) mainly evaluate robustness to perturbations of scenes encountered during training, while video-conditioned policies (Jain et al., 2024; Kim et al., 2025; Tang et al., 2025) are typically tested in-distribution, with only limited mismatch between the demonstration and execution environments. In contrast, we evaluate imitative ability: whether a policy can watch a human demonstration video and reproduce its intent in a previously unseen scene, even when the available objects differ substantially in placement, geometry, or function from those originally shown in the demonstration. The central challenge is not trajectory reproduction itself, but determining what aspects of the demonstrated behavior should still be preserved under a changed scene. This ability is ultimately what determines whether the large amount of human video already available online can serve as a practical supervision source for robot manipulation.

Drawing on work in cultural learning (Tomasello, 2009) and goal-directed imitation (Meltzoff, 1995), we formulate this ability as four levels of increasing mismatch between the demonstration and the robot's own scene:

• L0 Scene-identical execution. The imitator scene matches the demonstration scene; the policy is expected to reproduce the demonstrated trajectory.
• L1 Spatial adaptation. Object identities are preserved but their spatial configuration differs; the policy must achieve the same object-level outcome through a different trajectory.
• L2 Visual-physical generalization. Task-relevant objects differ in appearance or geometry while preserving semantic role; the policy must generalize across visually or physically different instances of the same object category.
• L3 Intent-level transfer. The demonstrated object is replaced by one of different semantics or function (e.g., teapot \to water tap for “fill the cup”); the policy must infer the underlying intent and realize it through a different affordance structure.

Contributions. We introduce the Imitator Benchmark, a four-level evaluation framework for robot video imitation that measures imitative capability under progressively increasing mismatch between demonstration and execution scenes. To support this benchmark, we construct IG-10K, the largest environment-aligned human–robot paired manipulation dataset to date, containing 20,000+ paired episodes, 200+ task variants, and 6 domains across both simulation and the real world. We further introduce Imitator Arena, an open platform for structured blind human evaluation of robot imitation behavior, and perform a systematic evaluation of 9 state-of-the-art models (15 trained variants spanning language-conditioned VLAs, cross-embodiment skill methods, and video-conditioned visuomotor policies) across multiple training scales and regimes.

Learning manipulation from human video. Prior work learns from human videos by retargeting extracted hand or object poses (Qin et al., 2022; Sivakumar et al., 2022), following optical-flow or object-centric motion cues (Tang et al., 2025; Dan et al., 2025), inpainting human hands as robot grippers (Lepert et al., 2025), or filtering cross-embodiment actions (Dan et al., 2025). Skill-representation methods such as XSkill (Xu et al., 2023), UniSkill (Kim et al., 2025), RHyME (Kedia et al., 2025), and ImMimic (Liu et al., 2025) learn human-robot correspondences, and Vid2Robot (Jain et al., 2024) conditions control on a video prompt. However, these methods assume the robot scene is largely congruent with the demonstration and are evaluated in-distribution, without graded mismatch between the two.

Benchmarks and datasets for manipulation. Existing benchmarks study robustness, long-horizon reasoning, and generalization in robot manipulation. Benchmarks such as COLOSSEUM (Pumacay et al., 2024), RoboTwin 2.0 (Chen et al., 2025), and REALM (Sedlacek et al., 2026) perturb visual, physical, embodiment, or environment factors, while CALVIN (Mees et al., 2022), LIBERO (Liu et al., 2023), and VLABench (Zhang et al., 2025) focus on compositional language-conditioned tasks. Large-scale datasets including Open X-Embodiment (O'Neill et al., 2024), DROID (Khazatsky et al., 2024), BridgeData V2 (Walke et al., 2023), and FMB (Luo et al., 2025) instead emphasise diversity across tasks, scenes, and embodiments. In contrast, IG-10K pairs human demonstrations with progressively mismatched robot scenes, allowing direct evaluation of transfer from human video and intent-level imitation.

Robot manipulation models. Robot manipulation policies range from specialised visuomotor methods – such as ACT (Zhao et al., 2023), Diffusion Policy (Chi et al., 2025), and VQ-BeT (Lee et al., 2024) – to large-scale VLA and diffusion-based foundation models including OpenVLA (Kim et al., 2024; Kim et al., 2025), \pi_{0.5} (Black et al., 2024; Intelligence et al., 2025), the GR00T series (Bjorck et al., 2025), RDT-1B (Liu et al., 2025), and H-RDT (Bi et al., 2026). These systems have advanced language conditioning, action generation, cross-robot pretraining, and few-shot adaptation, but are typically evaluated on executing a specified task within the robot's own scene. In this work, we instead study whether a policy can infer and reproduce the demonstrated behaviour from a human video under mismatched execution conditions, where the robot scene may differ substantially from the demonstration itself.

This section has two parts. Section 3.1 formalizes the imitator capability – the ability to infer and reproduce the behavior demonstrated in a human video – as a four-level hierarchy, where each level corresponds to a different degree of scene mismatch and the corresponding fidelity of imitation. Section 3.2 introduces the Imitator Benchmark: IG-10K, the largest environment-aligned human–robot paired dataset to date; Imitator Arena, a unified platform for simulation and real-world evaluation with both automated metrics and a human-evaluation interface; and the adaptation of three families of state-of-the-art baselines, each trained on IG-10K and evaluated in the Arena.

Problem setting. The imitator is given a human demonstration video V, recorded in some scene \mathcal{S}_{\text{demo}}. The task is defined implicitly by whatever is demonstrated in V – there is no separate task description, goal predicate, or symbolic specification. The imitator is then placed in its own scene \mathcal{S}_{\text{imit}}, where it receives observations o_t and must generate actions a_t that reproduce the demonstrated task as faithfully as \mathcal{S}_{\text{imit}} permits. Formally, an imitator policy is a mapping \pi\bigl(a_t \mid o_t, V\bigr); goal predicates are used only inside the evaluation platform for simulation scoring, never as policy input.

Scene decomposition. We represent a scene as an object collection and its spatial configuration, \mathcal{S} = \bigl(\{O_i=(A_i, G_i, S_i)\}, P\bigr), where A_i, G_i, and S_i denote the appearance, geometry, and semantic category of object O_i, respectively, and P denotes the spatial configuration of the object collection. In our specific definition context, semantic categories are distinguished mainly by the general purposes of objects, and spatial configuration primarily concerns the initial placement of task-relevant objects at the start of the episode.

The distinction between intrinsic object properties and extrinsic configuration is central to the hierarchy: changing P preserves object identity but changes where objects are placed, whereas changing (A,G,S) changes the behavior logic to accomplish the tasks shown in the demonstration scene. For the functional task objects in IG-10K, we use the operational assumption while the reverse directions do not generally hold. This assumption is not intended as a general ontology of objects; it is used to construct benchmark instances where each level transition is induced by one interpretable mismatch. It also implies that comparing P is meaningful only when demonstration and imitator scenes share a common object set. Edge cases are discussed in Appendix A.1.

How closely the imitator can copy the demonstration is decided by its own scene \mathcal{S}_{\mathrm{imit}}. As \mathcal{S}_{\mathrm{imit}} diverges away from \mathcal{S}_{\mathrm{demo}} along (P,A,G,S), what can still be matched becomes less and less specific: first the exact motion, then the exact final state of each object, then only the task in the ordinary sense of the word, and finally only the purpose behind it. Table 1 defines this hierarchy as preservation patterns over P, A\text{-or-}G, and S; Figure 2 illustrates the progression on a single demonstration. We treat P as preserved when the motion shown in V would still work if it were replayed directly in \mathcal{S}_{\mathrm{imit}}, and as changed otherwise. The A\text{-or-}G column is preserved only when the task-relevant object set is intrinsically identical, and S is preserved when each task-relevant object keeps its semantic category. Thus, each \mathrm{L}{\to}\mathrm{L}{+}1 transition permits greater differences between the demonstration and the imitator scene, while correspondingly broadening the criterion for successful imitation. This gives the hierarchy a clear progression from matching actions, to reproducing object outcomes, to preserving semantic function, and ultimately to fulfilling the underlying intent. Additional examples and edge cases are deferred to Appendix A.2.

The benchmark comprises three reusable artifacts. IG-10K (Section 3.2.1) is the largest environment-aligned human-robot paired manipulation dataset to date, instantiated across all four levels of Table 1. Imitator Arena (Section 3.2.2) supports automated rule-based scoring in simulation and structured human evaluation. Baseline adapters (Section 3.2.3) let three different families of methods share the same input with human demonstration videos and robot observation. Using these we train or fine-tune nine state-of-the-art models, yielding fifteen trained variants, evaluated uniformly within the Arena.

Paired-episode design. The atomic unit of IG-10K is a paired episode: a human demonstration video V in \mathcal{S}_{\text{demo}}, together with one or more robot episodes in scenes \mathcal{S}_{\text{imit}} whose relation to \mathcal{S}_{\text{demo}} instantiates one row of Table 1 – from a shared object set and configuration (L0), through rearrangement (L1) and same-category substitution (L2), intent-base functional substitution (L3). The levels are therefore decided by the demonstration-execution pairs, which is why the Arena supports fair level-agnostic comparison across anonymous policies for human evaluators (Section 3.2.2). Hence, level is not a property of a task: the same human clip can serve four levels of its task, and only the robot-side scene changes. At this stage, each task-level pair uses exactly one substitution pattern for equal supervision across levels. More variation is left to future versions of the benchmark.

Task suite and scale. IG-10K contains 50+ base tasks across six domains – Household (12), Supermarket (7), Restaurant (17), Logistics (4), Hospital (6), Laboratory (7), yielding 200+ task variants with four levels each suite, 50+ paired episodes each. Since the number of possible substitutions grows as the level rises with more mismatch, each task is paired with one substitution pattern per level at this stage for comparable comparison in levels. More variation is left to future versions of the benchmark. A few tasks omit L3 where substitution would be ill-defined or unsafe, e.g. hazardous-liquid handling in the laboratory. In total IG-10K provides 20,000+ paired episodes, released in lerobot-0.5.0 format.

Each episode is multi-view and richly annotated – 3D MANO hand pose, segmentation masks, and multi-abstraction language – placing IG-10K ahead of prior resources on the axes that matter for imitation (Table 2). Real data are teleoperated in VR on a dual-arm Realman platform, and simulated data on a dual-arm Franka in ManiSkill3 with OMPL-planned trajectories over hand-crafted waypoints; the two are kept as separate domains throughout, but share the data format, the human-to-robot pairing, and the task split. Simulated pairs reuse the real human clip of the same task and replace only the imitator scene; MANO hand pose is kept only for real-world human episodes. Collection platforms, assets, the camera rig, the annotation toolchain, and representative episodes (Figure 3 & Figure 5) are detailed in Appendix B.1.

Imitator Arena evaluates any policy following the imitator-game paradigm through two complementary score streams: automated goal-centric metrics in simulation, where the scene state is fully observable, and structured human preference judgments in both simulation and the real world.

Unified interface. Every policy receives the same specification – the demonstration video V and the observation stream \{o_t\} – an generates a_t = \pi(V, o_t). Video-conditioned policies take these videos and images as they are; language-conditioned VLAs reach the demonstration videos through a platform-side step in which the Arena applies a fixed, deterministic captioner T(V) and supplies the result as language conditioning. To make it a fair comparison between VLAs and video-conditioned models, we keep captions dense and as informative as the video itself – task intent, object layout, and the per-arm sub-task sequence with its motions – and verify them by hand; see details in Appendix B.4. T is fixed across all VLA baselines, so comparisons can reflect VLA capacities rather than how V is captioned. Crucially, the episode level is never communicated to the policy: like a person watching the video, the imitator is not told how far its own scene has been changed, and has to work out from V and its observations how much of the demonstration is still reproducible.

Human evaluation. Human evaluation provides the level-sensitive scoring channel for both simulation and real-world rollouts. For each episode, evaluators view three synchronized videos: the demonstration V and two anonymized, randomly ordered rollouts from different models on the same imitator scene. The episode level is withheld, so evaluators judge what fidelity the scene permits from the demonstration–rollout pair itself with subjective human preference in imitative capability rather than from an explicit difficulty label. They answer three questions, worded the same way at every level: per-rollout task completion (success/partial/fail), per-rollout imitation quality q\in[0,10] on a fixed five-anchor rubric, and pairwise overall preference. These judgments yield \mathrm{SR}_{\mathrm{human}} (counting only success), mean imitation score \overline{Q}, and per-model win rate \mathrm{WR}.

Two designs guarantee the fairness and quality of human evaluation. First, an A/B screen pairs two rollouts only when they are directly comparable: rollouts are bucketed by transfer setting (seen, zero-shot, transfer) and, within a bucket, paired only when they share the same domain, task, level, and episode, so the two videos differ only in the policy. Second, each screen is answered by a single evaluator, which keeps judgments independent. Reliability comes from repetition in random evaluation pair rollouts. The bucketing protocol and formal aggregation rules are given in Appendix B.3.

Automated metrics. In simulation, two hand-crafted metrics provide fast, reproducible proxies for absolute success: the final success rate \mathrm{SR}, the fraction of episodes in which all task-relevant goal-state predicates hold at termination, and the denser sub-goal success rate \mathrm{Sub\text{-}SR}, the mean fraction of a task's ordered sub-goal phases completed. Both are level-agnostic, purely reflecting the task completion at the specific task and level. The comparison between each level is therefore fair since we didn't impose any hierarchical human priors for any level, and leaves the level-by-level view to the cross-level analysis. Formal definitions and per-task predicate construction are in Appendix B.3.

Three families of existing methods are evaluated under the same interface of Section 3.2.2, all receiving the same inputs (V, o_t) and trained or fine-tuned exclusively on IG-10K (Figure 7). Across all families, the video encoder or vision-language backbone is frozen, and only the action-generation module is trained. This keeps the adaptation budget comparable across methods and isolates how different policy architectures use the same human-video specification. As a result, the benchmark evaluates differences in action-generation interfaces rather than differences in large-scale representation adaptation. Full backbone fine-tuning is outside the scope of this comparison, since it would require substantially different optimization recipes, memory budgets, and training strategies across VLA, skill-based, and visuomotor models. Freezing also trades some accuracy for comparability and cost. In the engineering implementation, we cache the backbone activations, achieving orders-of-magnitude faster training at our evaluation scale.

VLA models (GR00T-N1.6 (Bjorck et al., 2025), RDT-1B (Liu et al., 2025), \pi_{0.5} (Intelligence et al., 2025), OpenVLA (Kim et al., 2024)) are language-conditioned, so the fixed captioner T(V) of Section 3.2.2 supplies the prompt. Skill-based models (XSkill (Xu et al., 2023), UniSkill (Kim et al., 2025)) learn shared human-robot skill representations and act on the retrieved skill. Vision-action models (ACT (Zhao et al., 2023), Diffusion Policy (Chi et al., 2025), VQ-BeT (Lee et al., 2024)) gain video conditioning by concatenating a task embedding from one of three frozen encoders (DINOv2-ViT-L/14 (Oquab et al., 2023), SigLIP2-SO400M (Tschannen et al., 2025), VideoMAE-Large (Tong et al., 2022)) to the observation tokens. Per-family mechanisms, encoders, and recipes are in Appendix B.4.

The protocol does not favor any kind of model. By spanning language-conditioned generation (VLA), cross-embodiment skill retrieval, and video-conditioned action regression (VA) through the same interface, on identical data and identical episodes, the Arena is insensitive to the architectural paradigm under test. Whether any current paradigm closes the gap between trajectory-level imitation at L0 and intent-level affordance adaptation at L3 is the empirical question taken up in Section 4.

We evaluate human-video imitation along three axes: (Q1) the relative strength of VLAs, video-skill methods, and video-conditioned visuomotor agents, (Q2) whether paired human–robot pretraining improves unseen-task transfer, and (Q3) where the L0–L3 hierarchy becomes difficult. Across simulation, real-world rollouts, and Imitator Arena judgments, we find that video-based methods provide the strongest seen-task imitation, paired pretraining mainly improves few-shot adaptation rather than direct zero-shot transfer, and L3 affordance substitution is the clearest bottleneck.

Tasks and protocol. We evaluate ten manipulation tasks across the L0–L3 hierarchy: five seen tasks from the training set, and five disjoint unseen tasks for zero-shot or few-shot transfer. The two evaluation groups are chosen to probe different things. The five seen tasks are the long-horizon or dexterous manipulation tasks – stirring with a thin spoon, folding a towel, hanging a mug on a rack, pick up a paper, placing plates in a rack – and ask how faithfully a model reproduces demanding learned behavior given a human reference. The five unseen tasks are relatively simpler and closer to atomic skills (single-arm and bimanual placement, pouring, and an articulated folding task), but their demonstrations, objects, and layouts never appear training set, so they ask whether a new skill can be acquired from the reference alone. Both groups are run at all four levels, giving 40 task–level pairs per domain.

Episodes are balanced across levels; example clips are shown in Figure 4. Task evaluation perturbs object placements, so success requires robust imitation rather than fixed-layout replay. Simulation and the real-world robot scene are two separate domains in IG-10K, and each model is trained and evaluated inside one domain only. But the sim and real domains share the same human-to-robot pairing, the same pretraining task split, and the same model configurations.

IG-10K pretraining & transfer. When using IG-10K pretraining, we pretrain on paired human-robot corpora of 15, 30, or 45 tasks (15 \subset 30 \subset 45, and each budget was assembled to hold motion, object, and scene diversity as comparable as possible, so the corpus size is the main thing to be considered in scaling), 50 demonstrations per task are included in pretraining and excluding all unseen tasks. The task splits are given in Appendix B.2. We evaluate seen tasks with models pre-trained on the three corpora.

For unseen tasks we evaluate three settings: zero-shot transfer (ZS), in which pre-trained models directly execute on unseen tasks without any further parameter updates; from-scratch few-shot learning (Scr.), in which models are trained only on 10 robot demonstrations of the five unseen tasks; and pre-train and fine-tune (P+FT), in which IG-10K-pre-trained models are fine-tuned on unseen tasks and 10 demonstrations per task. Every level receives the same amount of supervision.

Baselines & evaluation. We compare: (1) language-conditioned VLAs (GR00T-N1.6, RDT-1B, \pi_{0.5}, OpenVLA), (2) video-skill methods (XSkill, UniSkill), and (3) video-conditioned visuomotor models (ACT, Diffusion Policy, VQ-BeT) paired with DINOv2, SigLIP2, or VideoMAE encoders. Simulation evaluates all policies and variants using success rate (SR) and sub-goal success rate (Sub-SR).

Four representative models (\pi_{0.5}, XSkill, ACT/DINOv2, and DP/DINOv2) on a dual-arm platform are evaluated in the real world, with 5 trials per task-level pair against 10 trials in simulation. Imitator Arena further provides human success judgments, imitation score Q, and pairwise win rate (WR). Appendix Figure 13 provides the alignment and validation of the two evaluation metrics.

Table 3 and 4 report the comprehensive performance in sim and real. The results suggest that video-based pipelines are stronger for seen-task imitation but pre-training matters in few-shot learning. In simulation, ACT/DINOv2 achieves the highest success rate (SR =0.81, Sub-SR =0.93), with XSkill and ACT/SigLIP2 close behind in seen-task evaluation; the strongest language-conditioned VLA, \pi_{0.5}, is lower (SR =0.73) in seen tasks but leads in few-shot adaptation (SR =0.80 in Scr. and SR =0.85 in P+FT). Appendix Figure 8 gives an intuitive view of comprehensive simulation model performance.

In the real-world Arena, XSkill gives the strongest human-judged imitation among the representative models across all settings (SR =0.63 in Seen, SR =0.29 in ZS, SR =0.28 in Scr., SR =0.49 in P+FT), and video-based models constantly beat language-based models in Zero-shot transfer. Within Video-VA, DINOv2 and SigLIP2 are consistently stronger than VideoMAE (Appendix Figure 9), indicating that the video representation is also an important design choice.

In simulation, paired pretraining mainly supports few-shot adaptation. Zero-shot performance remains near the floor across paradigms, with the best simulation result reaching only \mathrm{SR}=0.13. Current frameworks still struggle to generalize to unseen tasks without task-specific robot data. In contrast, P+FT improves few-shot learning for most models, with benefits growing with scale: from 15 to 45 pretraining tasks, P+FT success improves for 14 of 15 simulation variants (Appendix Figure 10), and P+FT wins over Scr. for 12 of the 15 simulation variants (Table 3).

In the real-world evaluation, all four representative models improve in the same way in few-shot adaptation, and the three video-conditioned ones also gain some zero-shot ability as the corpus grows, while the language-conditioned \pi_{0.5} does not (Appendix Figure 10). According to Appendix C.4, Table 9, P+FT success rises from 15 to 45 pre-training tasks at every one of L0–L3, in both domains. Robust zero-shot imitation from human video remains difficult, but more paired data does make a model easier to adapt.

We analyze the hierarchy in the real-world P+FT setting, where policies receive task-specific adaptation and are judged by humans. Table 5 shows the main difficulty appears at L3. Averaged over the four representative models, SR is nearly stable from L0 to L2 (around 0.4) but drops to 0.29 at L3; the human imitation score follows suit, falling from about 6.2 on L0–L2 to 5.62 on L3. This drop is clearest for the strongest L0–L2 policy: XSkill holds SR \approx 0.53–0.57 through L2 but falls to 0.29 at L3, while other policies show a flatter profile from a lower operating point. Current policies therefore cope with a rearranged layout, and with a replacement object of the same kind, but not with an object that has to be used in a different way to reach the same end.

The Imitator Game Benchmark studies robot video imitation under progressively increasing mismatch between the human demonstration and the robot's own scene. To support this setting, we introduce IG-10K, a large-scale environment-aligned human–robot paired dataset spanning four levels of imitation, together with Imitator Arena for structured human evaluation of imitation behavior. Our experiments across multiple policy families and training regimes show that current systems transfer poorly to unseen tasks without task-specific robot data, and struggle when imitation requires functional substitution and intent-level adaptation.

These suggest that intent-level imitation is unlikely to arrive as a by-product of scaling either the corpus or the action head only, and that the productive direction is an interface that keeps the demonstration available as evidence about purpose – affordance-aware object grounding, explicit goal inference, or intermediate representations that survive object substitution – rather than as a specification to be transcribed once and executed. IG-10K, Imitator Arena, and the L0–L3 protocol are released so that such interfaces can be measured on the rung where they would actually differ.

Limitations. IG-10K remains finite and manually designed, leaving broader scaling and more open-ended transfer substitutions to future work. Each task–level pair currently carries a single substitution pattern, which buys comparability across levels at the cost of not measuring variance over substitutions, and the hierarchy grades scene mismatch while holding the demonstrator–robot embodiment gap approximately fixed. Finite assets and rule-based planned motion further bias the simulated L3 substitution results. The evaluated policies are adapted to the Imitator Game rather than designed for intent-level human-video imitation, the best suitable framework for the Imitator Game remains an open question. Current models will fail in cases mainly in unseen zero-shot generalization tasks and functional substitution, we will discuss it in Appendix G.

This appendix expands the scene decomposition of Section 3.1.1. Recall that a demonstration is recorded in \mathcal{S}_{\text{demo}} and the robot acts in \mathcal{S}_{\text{imit}}. We write a scene as \mathcal{S}=(\{O_i\},P), where P denotes the spatial configuration, and each task-relevant object O_i=(A_i,G_i,S_i) has appearance A_i, geometry G_i, and semantic category S_i. The appendix clarifies when P is comparable across the two scenes and how the benchmark uses the construction rule S \!\Rightarrow\! G \!\Rightarrow\! A to define the level hierarchy.

When is P a meaningful comparison dimension? The spatial configuration P is only a meaningful comparison dimension between \mathcal{S}_{\text{demo}} and \mathcal{S}_{\text{imit}} when the two scenes share the same object set. Once the objects differ, there is nothing to line up one-to-one: a teapot's pose and a water tap's pose are not two values of the same quantity, so P is not “different” – the question simply does not arise. This is why, in the level table of Section 3.1.2, the P column is marked “n/a” and not “\times” at the levels where the object set has changed. For the same reason there is no level that changes P and the object set at once: such a level could not be described in these terms at all.

Edge cases of the implication chain. The assumption S \!\Rightarrow\! G \!\Rightarrow\! A has rare counterexamples in which semantic category and physical form come apart – a wax banana (the semantics of “banana” without its edibility or, depending on construction, its exact geometry), or a decorative object shaped like a cup that is in fact a candle holder (the geometry of a cup with the semantics of a candle holder). In such cases the conventional link between category and form is deliberately broken for ornamental or deceptive purposes. We treat these as out of scope for benchmark instances: every task-relevant object in IG-10K is a functional exemplar of its semantic category, so that the chain holds, and object-substitution levels can be defined by a single intrinsic change. We list these cases to state the assumption precisely; none of them occurs in IG-10K at this stage.

This appendix expands Section 3.1.2 with the full per-level scene relations and worked examples, the L2 appearance-only sub-case, the correspondence to developmental-psychology constructs, and the relationship of our scene-mismatch hierarchy to embodiment-gap hierarchies.

L0 – Trajectory imitation. Scene relation. The object set is intrinsically identical (A and G preserved on every task-relevant object) and P is preserved in the operational sense of Section 3.1.2–the trajectory implicit in V would still succeed in \mathcal{S}_{\text{imit}}. What the scene permits. The demonstrated end-effector motion and object motion remain feasible. What the imitator is graded on. The rollout should preserve the demonstrated motion pattern as closely as the shared scene permits, while the benchmark still scores the episode through the Arena judgments and task-completion metrics rather than a hand-designed trajectory-distance metric. Example. Same teapot and cup in the same positions; the imitator pours along essentially the demonstrated motion.

L1 – Object end-state imitation. Scene relation. The object set is intrinsically identical but P has changed – a literal replay of the trajectory implicit in V would no longer succeed. What the scene permits. The demonstrated trajectory is no longer correct, but exactly the same objects can be brought to exactly the same final states. What the imitator is graded on. Match of final object states with V; trajectory similarity is not rewarded. Example. Same teapot and cup, but the cup has moved to the opposite side of the table; the imitator should still leave the cup full of water.

L2 – Semantic-task imitation. Scene relation. At least one task-relevant object has changed in A (and possibly G); each substitution preserves the object's semantic category C. What the scene permits. Because the objects differ, the precise final states of V are not reproducible (a handled mug is not a handleless cup), but the semantic task – “make a cup of tea,” “set the table,” “stir the soup” – remains well-defined. What the imitator is graded on. Completion of the semantic task implied by V, judged at an abstraction that survives object substitution. Example. The demonstrator pours from a ceramic teapot into a handleless cup; the imitator's scene has a glass teapot and a handled mug, and the expected behavior is still “pour tea from the teapot into the mug.”

The L2 appearance-only sub-case. Within L2 it is sometimes useful to distinguish the sub-case in which only A changes while G is preserved (a ceramic cup replaced by a glass cup of identical shape) from the sub-case in which both A and G change (the cup replaced by a mug of different shape). The first isolates a pure perceptual invariance – the policy must recognize that two visually different objects play the same functional role – while the second additionally demands that the policy adapt its grasp and motion to a different shape. We report both sub-cases where a task admits them, but treat them as a single L2 level for headline metrics because both share the same fidelity ceiling: the semantic task rather than per-object final states.

L3 – Affordance-adapted imitation. Scene relation. At least one task-relevant object changes in semantic category C; under our benchmark construction this also changes geometry G and appearance A. What the scene permits. The object-specific execution shown in V is no longer directly reproducible, but the underlying purpose can still be served by selecting an object in \mathcal{S}_{\text{imit}} whose affordance can be adapted to the same end. What the imitator is graded on. Achievement of an action that satisfies the underlying intent of V via affordance adaptation, assessed by human evaluators in the Arena (Section 3.2.2); neither trajectory nor specific final states are expected to resemble V. Examples. (i) The demonstrator drives a nail with a hammer; the imitator's scene contains no hammer but a stone, and the expected behavior is to use the stone as a hammer. (ii) The demonstrator steeps a tea bag to make a hot drink; the imitator's scene contains no tea bag but an instant drink powder, so the expected behavior is to make a drink by mixing rather than steeping.

Correspondence to developmental-psychology constructs. The four fidelity levels take operational inspiration from a hierarchy of social-learning constructs: surface-motor copying without attribution of instrumental efficacy (“mimicking” in Tomasello's sense (Whiten et al., 2004; Tomasello, 2009)), end-state emulation (Wood, 1996; Tomasello, 2009; Whiten et al., 2009), goal-directed (intentional) imitation (Meltzoff, 1995; Tomasello, 2009; Gergely et al., 2002), and analogical transfer of an observed practice to functionally equivalent means (Brown, 1989; Chen et al., 2000; Byrne and Russon, 1998). We adopt these as labels for our hierarchy of replication fidelities rather than as claims about the cognitive mechanisms operating in our system, and in particular do not claim that our system attributes mental states. The four levels are our synthesis: to our knowledge, no single published taxonomy enumerates exactly these four, and the framing builds on Tomasello's mimicking/emulation/imitation distinction and Byrne and Russon's (Byrne and Russon, 1998) program-level imitation. What we actually measure is simpler than any of these constructs: as the robot's scene moves further from the demonstration, what an imitator can still reproduce becomes less and less specific.

Relationship to embodiment-gap hierarchies. Our hierarchy is orthogonal to embodiment-gap hierarchies such as RHyME's (Kedia et al., 2025), which grade the mismatch between the demonstrator's body and the robot's body while holding the scene fixed. The Imitator Game holds the demonstrator–robot embodiment gap approximately fixed and instead grades the scene-level mismatch between \mathcal{S}_{\text{demo}} and \mathcal{S}_{\text{imit}}. We likewise assume a visual front-end invariant to lighting and viewpoint variation between V and the imitator's observations, treating these as out of scope rather than as additional levels. The two axes – embodiment gap and scene-level mismatch – are complementary, and combining them is a natural direction for future benchmarks.

This appendix expands Section 3.2.1 with the full task suite, the data-collection platforms and acquisition parameters, the sensing rig, and the multi-modal annotation toolchain. Figure 3 shows representative paired episodes, the object assets, and the offline annotation layers; Figure 5 shows the collection rig.

Data collection. Real-robot data are collected on a dual-arm Realman platform, teleoperated via VR, with full joint angles and 6-DoF end-effector poses recorded at a control frequency of 30 Hz. Simulation data are collected on a dual-arm Franka in ManiSkill3, with assets drawn from YCB, PartNet-Mobility, RoboTwin, and Sketchfab; trajectories are planned by OMPL from hand-crafted waypoints. Of the 20{,}000+ paired episodes in total, 11.7K are real human-robot paired episodes and 10K are paired simulation episodes. Each real paired episode contains one human clip and one corresponding real-robot clip; the simulation episodes provide paired interactive environments with fully observable automated metrics. Per task–level pair we collect 50 paired episodes on average.

Sensing rig. All human, real-world robot, and simulation scenes are captured by a wide-angle ZED2i ego-centric camera together with three RealSense third-person cameras (left-frontal, top-down, right-frontal), all providing RGB-depth vision. For model input we standardize on the ZED2i RGB stream for compatibility with the diverse pre-training setups of the baseline families (Section 3.2.3); the remaining views and the depth channels are retained in the released data for use by future methods.

Task suite. Table 6 lists all 50+ base tasks by domain, together with the levels at which each is instantiated. A task omits L3 only where affordance substitution would be semantically ill-defined or physically unsafe; the omitted cases and their justification are noted in the table.

Multi-modal annotation. Each video carries three annotation layers. (i) Language: structured descriptions of the demonstrated task at three levels of abstraction–per-arm sub-task sequence, semantic task description, and underlying intent–authored by human annotators and augmented with Qwen3-VL. These three abstraction levels are precisely what the VLA captioner of Section 3.2.3 consumes. (ii) Hand pose: 3D MANO hand pose estimated by WiLoR (per-view; see Appendix E). (iii) Segmentation: semantic segmentation masks from Grounded-SAM-2, conditioned on per-video object inventories. The mask layer is an offline annotation produced for analysis and future training objectives rather than a component of the scoring pipeline; its generation procedure and quality control are described separately in Appendix D.

Constructing the L2 sub-cases. The appearance-only and shape-changing sub-cases of L2 (Appendix A.2) are built by controlled substitution. For the appearance-only sub-case we replace a task-relevant object with one of the same semantic category and the same geometry but different appearance (e.g. a ceramic cup and a glass cup sharing a mesh up to material), isolating perceptual invariance. For the shape-changing sub-case we additionally vary geometry within the category (e.g. a handleless cup and a handled mug), so the policy must also adapt grasp and motion.

L3-omitted tasks. A small number of tasks omit L3 where affordance substitution would be semantically ill-defined or unsafe; these are marked in Table 6, the laboratory hazardous-liquid cases being the canonical example.

This appendix expands the split description of Section 3.2.1 and states exactly which data are used to pre-train the models evaluated in Section 4.

Two domains, one experimental structure. Simulation and the real world are separate domains. The embodiments differ (dual-arm Franka in ManiSkill3 versus dual-arm Realman), and the visual sensor statistics differ, so we never mix the two corpora: simulation models are trained on simulation data and evaluated in simulation, real-world models are trained on real data and evaluated on hardware. Everything else is held common. Both domains use the same model configurations and hyper-parameters, the same input/output interface (the ZED2i RGB stream as input; dual-arm 7-DoF joint commands plus gripper as output), and the same lerobot data format, so a single training codebase produces both sets of variants. Both use the same human-to-robot pairing: one human demonstration video is paired with the robot episodes of the corresponding task, and the same human clip serves all four levels of that task. Both use the same task split, listed below. A small number of real tasks have no simulated counterpart, because the required objects or contact behaviors are not reproducible with our simulation assets; these are the only points at which the two domains' corpora differ, and the 15/30/45 nesting and the seen/unseen assignment are preserved in each domain. Simulated paired episodes do not use a separately recorded human demonstration. A simulated episode reuses the real human clip recorded for the same task, with only the imitator scene replaced by its simulated counterpart. MANO hand pose is provided only for real human clips.

Nested pre-training corpora. The three pre-training corpora are nested, \mathcal{C}_{15} \subset \mathcal{C}_{30} \subset \mathcal{C}_{45}, and were assembled so that motion types, manipulated objects, scene domains, and horizon lengths stay as diverse as each budget permits. The intent is that corpus scale is the variable that changes between them, rather than diversity or task character; scaling results (Section 4.3) should be read against this construction. The five unseen tasks are excluded from all three corpora. Table 7 lists the membership.

Seen and unseen evaluation tasks. Evaluation uses five seen tasks drawn from \mathcal{C}_{15} and five unseen tasks. The seen tasks are the long-horizon or dexterous members of the corpus – stirring with a thin spoon, folding a towel, hanging a mug on a rack, filing a folder, and placing plates in a rack – and test how faithfully a model reproduces demanding learned behavior given a human reference. The unseen tasks are individually simpler and closer to atomic skills (single-arm placement, bimanual scanning, bimanual placement, pouring, and an articulated folding task), but their demonstrations, objects, and layouts are absent from every corpus, so they test acquisition of a new skill from the reference alone rather than recall. Both groups are evaluated at all four levels, giving 40 task–level pairs per domain, with 10 trials per pair in simulation and 5 on hardware.

Substitution patterns and supervision per task–level pair. Each task–level pair uses exactly one substitution pattern. The number of possible substitutions grows as the level rises, so allowing more of them at L3 than at L0 would mean comparing how varied the substitutions are instead of how well the policies handle them. Each pair receives 50 pre-training demonstrations, collected with small position variations for robustness. Supervision is therefore equal across levels, and the 10 few-shot demonstrations are not a budget divided among the four levels. Broadening the substitution patterns per level is left to future versions of IG-10K.

This appendix expands Section 3.2.2 with the platform and annotation protocol, the imitation-quality rubric, the pairwise-comparison protocol, the full execution-and-scoring procedure, the formal metric definitions with a worked per-task example, and annotation reliability.

Platform and annotation protocol. Imitator Arena is a web platform on which each annotation screen presents three time-synchronized videos–the demonstration V and two anonymized, randomly ordered rollouts on the same imitator scene–and collects the three judgments of Section 3.2.2. Annotation is carried out by a pool of 10 human evaluators who divide the workload roughly evenly, so that each annotation screen is judged by one human evaluator and each human evaluator contributes approximately one tenth of the total. The evaluators are volunteers with no involvement in the project; no crowdsourcing platform was used, participation was voluntary and uncompensated by task, and no personal data were collected, so the study raises no additional ethical concerns. Restricting a screen to a single evaluator is what keeps judgments independent and lets the platform double as an anonymous public leaderboard, and reliability is obtained by repetition rather than by co-annotation of a screen: screens are drawn at random from the eligible pool, so at the study's volume each demonstration–rollout pair is judged 3–5 times, by different evaluators, on independently sampled screens. Supporting multiple annotators per screen, and reporting an explicit inter-annotator agreement statistic alongside the qualification pass, is a planned extension of the platform. In total the human study comprises 15{,}000 pairwise comparisons in simulation and 5{,}000 on real hardware, drawn from 36{,}000 distinct simulation rollouts and 4{,}800 distinct real-world rollouts (the per-bucket breakdown is given below). Model identities and imitation levels are never shown, and the left/right order of the two rollouts is randomized to control position bias.

Quality rubric. The per-rollout imitation-quality score q \in [0,10] (question (2) of Section 3.2.2) is collected on a fixed anchored scale displayed on every annotation screen, so that scores are comparable across human evaluators, models, and levels. Human evaluators are shown the five anchors of Table 8 and may assign any integer in [0,10], interpolating between adjacent anchors. The same rubric is applied identically at all levels: it is the demonstration V, not a level label, that defines what a faithful imitation is, consistent with the level being withheld from the human evaluator.

Pairwise-comparison protocol. An A/B screen pairs two rollouts only when they are directly comparable, which we enforce by bucketing every evaluation rollout by its transfer setting and pairing only within a bucket. The three buckets are seen (the five seen evaluation tasks), zero-shot (the five unseen tasks executed directly from a pre-trained checkpoint), and transfer (the same five unseen tasks after few-shot adaptation, comprising both the pre-train+fine-tune and the from-scratch conditions, which are therefore mutually comparable); the three settings are defined in Section 4.1. Within a bucket, two rollouts are paired only when they share the same task, level, and episode, so the two videos differ only in the policy that produced them; rollouts from the same model at different pre-training scales (15/30/45) are comparable and are paired within the bucket. Simulation and real-world rollouts are never paired with each other, as they originate from distinct visual and dynamical domains; pairing is therefore confined to a single (domain, setting, task, level, episode) cell, and because the Arena randomizes which rollout is shown as A versus B, a pair is unordered. In simulation, where all baselines are evaluated, this yields 15{,}000 annotated screens–6{,}000 seen, 6{,}000 transfer, and 3{,}000 zero-shot. On real hardware four models are evaluated; the 5{,}000 real-world screens are therefore split 2{,}500 seen and 2{,}500 transfer.

Execution and scoring. For each evaluation episode the platform rolls out the policy until the task terminates or a fixed horizon is reached, and records the full trajectory and the final scene state. Two scoring streams are then produced. The first is the per-rollout absolute-success judgment: in simulation, where the scene state is fully observable, it is scored by the final and sub-goal success rates defined below (fast, reproducible, and suited to large-scale evaluation) and additionally by Arena human evaluation, so that the two can be cross-validated; on real hardware no automated metric is available and the Arena human judgment is the sole source. This judgment is the per-rollout task-completion question (question (1) of Section 3.2.2), recorded on a three-way scale (success / partial / fail), together with the quality score q. The second stream is pairwise human preference, collected in both regimes as described above. The metrics built from these two streams are defined next.

Human success rate, mean quality, and win rate. The human success rate is the fraction of rollouts judged success; a partial verdict does not count toward success, matching the hard criterion of the automated \mathrm{SR} below. Over a set of judged rollouts \mathcal{R}, both computed over distinct rollouts (36{,}000 in simulation, 4{,}800 on real hardware), so \overline{Q} admits partial credit and separates failures that the binary success question collapses. For the pairwise preference question, each annotated screen awards 1 to the preferred rollout, 0.5 to each rollout on a tie, and 0 to the other; a model's win rate \mathrm{WR} is its mean awarded score over all screens in which it participates. Because pairing respects the (domain, setting, task, level) structure above, \mathrm{SR}_{\text{human}}, \overline{Q}, and \mathrm{WR} can all be resolved by level post hoc even though the level is hidden from human evaluators at judgment time.

Final success rate. For each episode a binary success predicate checks whether all task-relevant goal states are satisfied at termination. Aggregated over the evaluation set \mathcal{E}, \mathrm{SR} is a hard, binary criterion: an episode either satisfies all goal predicates or it does not. The phase definitions and success predicates are hand-crafted per task to reflect that task's goal states and intermediate milestones.

Sub-goal success rate. Each task is decomposed into M ordered sub-goal phases. For each phase j the tracker records the peak value of the corresponding shaped sub-reward over the episode horizon, and the phase is counted as completed when that peak exceeds a fixed threshold. Letting m_e \le M denote the number of completed phases in episode e, \mathrm{Sub\text{-}SR} measures the fraction of the task pipeline the policy executes and serves as a continuous proxy for imitation depth when \mathrm{SR} is near zero.

Scope of the automated metrics. Both metrics apply uniformly across all levels as general task-completion indicators, which is why the simulation experiments report them as a single overall score rather than per level; the per-level fidelity view is the responsibility of the Arena human evaluation. We do not define a trajectory-similarity metric because the human–robot embodiment gap leaves a trajectory-level predicate without an unambiguous reference, so \mathrm{SR} and \mathrm{Sub\text{-}SR} are the sole automated proxies. On real hardware neither automated metric is used, and all absolute success judgments come from Arena human evaluation.

Worked example: predicate and phases for PourKettle. To make the hand-crafted predicates concrete, we give the full specification for one task. The PourKettle goal predicate is the conjunction of a fill condition and a spill condition, with thresholds \tau_{\text{fill}}=20 and \tau_{\text{spill}}=20 on the simulated liquid volume. The task decomposes into M=4 ordered sub-goal phases–reach (gripper within \epsilon of the kettle handle), grasp (stable contact, kettle lifted), transport (kettle spout positioned above the cup), and pour (kettle tilted past the pour angle while above the cup)–each with a shaped sub-reward whose peak is thresholded as in the \mathrm{Sub\text{-}SR} definition.

This appendix expands Section 3.2.3 with the per-family mechanisms, encoders, and training recipes. The video encoder or vision–language backbone is frozen and only the action-generation module is trained on IG-10K paired data, following each method's original recipe except where the imitator-game interface requires adaptation. Figure 7 shows how each family is wired up to be called with the reference videos and robot observation; the full per-model hyperparameters are in Appendix F.

VLA family. The four VLA backbones (Bjorck et al., 2025; Liu et al., 2025; Intelligence et al., 2025; Kim et al., 2024) are language-conditioned, so the demonstration is first passed through the fixed captioner T(V) \to \ell, implemented via human annotation augmented with Qwen3-VL, which emits the per-arm sub-task sequence, the semantic task description, and the underlying intent–the same three abstraction levels carried as language annotation in IG-10K (Appendix B.1). The description \ell is supplied as the language prompt and the action head is trained while the VLM backbone remains frozen. T is held fixed across all VLA baselines so that performance differences reflect the action heads rather than the captioner; concretely, T applies a single frozen prompt template to Qwen3-VL and emits the three abstraction levels in a fixed format, with no per-model tuning. A caption that was uniformly poor would pull every VLA down together, so we checked the caption channel itself in two ways. First, the captions carry the same information as the video: they state the task intent, the object layout, and the per-arm sub-task sequence together with its motions, and every caption is checked by a human annotator. Second, we test T directly by replacing the VLM-augmented captions with the purely human-written ground-truth annotations of IG-10K and re-evaluating \pi_{0.5}: seen-task \mathrm{SR} changes by 0.01 \pm 0.05 across the three corpus scales, i.e. within noise. We did not search over caption styles, so it is still possible that shorter prompts suit small corpora better; a fuller ablation is future work.

Skill-based family. XSkill (Xu et al., 2023) discovers a set of cross-embodiment skill prototypes from unlabeled human and robot videos via self-supervised contrastive learning with Sinkhorn–Knopp clustering, and conditions a diffusion policy on the prototype index retrieved for the current observation. UniSkill (Kim et al., 2025) instead learns embodiment-agnostic skill representations through future-frame prediction (Inverse and Forward Skill Dynamics), which removes the need for domain alignment and allows skill transfer from human videos to robot policies trained solely on robot data. Both are trained from scratch on IG-10K, with the skill-conditioned policy stage trained with a frozen video extractor.

VA family. The action-generation backbones (Zhao et al., 2023; Chi et al., 2025; Lee et al., 2024) are adapted by extracting a task representation from V with one of three frozen video encoders–DINOv2-ViT-L/14 (Oquab et al., 2023), SigLIP2-SO400M (Tschannen et al., 2025), or VideoMAE-Large (Tong et al., 2022)–and concatenating the video embedding to the robot observation embedding at every inference step. The encoder is frozen and shared across all three backbones, and only the action head is trained from scratch, giving the three-encoder \times three-head grid evaluated in Section 4.

Why the task encoder is frozen. Freezing the video encoder or vision-language backbone is a comparability-versus-cost trade-off, and we tested the alternative before adopting it. In early runs we unfroze the task-related encoders – the video encoder for the skill and VA families, the VLM backbone for the VLA family – and observed comparable gains on both sides, giving no evidence that freezing penalizes the VLA family in particular. Unfreezing, however, forecloses caching: with a frozen task encoder the encoding of V can be computed once and reused across epochs, which at the scale of our grid (15 trained variants \times three corpus scales, plus few-shot regimes) is an orders-of-magnitude difference in training cost and is what makes the full comparison feasible. We therefore freeze the task encoder in every family and train only the action head, with comparable trainable-parameter counts, and treat a fully unfrozen comparison – and a quantitative estimate of how much end-to-end adaptation adds per family – as future work.

Training configuration. All variants are trained on IG-10K under matched compute and a common optimizer schedule, with only the action-generation module updated.

This appendix is organized around the three questions of Section 4, and every figure answers exactly one of them. Q1 – which imitation interface is strongest? Figure 8 compares the three paradigms in simulation and under blind human preference, and Figure 9 isolates the frozen video encoder inside the Video-VA family. Q2 – does scale help zero-shot transfer or few-shot adaptation? Figure 10 reports P+FT scaling and the per-model gain, and Figure 11 resolves the same experiment by hierarchy level. Q3 – where does the hierarchy become hard? Figure 12 contrasts the clean real-world coarse-to-fine decline with the non-monotone simulation profile, and Table 10 checks that the demonstration is what specifies the task in the first place. Figure 13 then asks how far the automated and human scores, and the simulated and physical domains, can stand in for one another; the per-task numbers behind all of these aggregates are in Table 11.

Two scoring channels appear throughout and are always named. The automated channel is the simulated success predicate, reported as \mathrm{SR} and \mathrm{Sub\text{-}SR}; the Arena channel is human judgment, reported as \mathrm{SR}_{\mathrm{human}}, the imitation score \overline{Q} on the 0–10 rubric of Appendix B.3, and the blind A/B win rate \mathrm{WR}. The Arena runs on both domains: on the simulated rollouts, where it can be compared against the automated channel directly, and on hardware, where there is no automated predicate and every reported \mathrm{SR} is therefore a human verdict. Throughout, simulation numbers are averaged over the \{15,30,45\} corpus scales as in the body, and the Video-VA family is shown at its DINOv2 backbone unless noted.

Figure 8 unpacks the headline ranking of Section 4.2 along three axes. Panel (a) places all fifteen trained variants on the (\mathrm{seen\text{-}SR},\,\mathrm{P{+}FT\text{-}SR}) plane, both axes automated. The two video-conditioned families cluster in the upper right, while the VLA family stretches along the diagonal from OpenVLA (0.29) to \pi_{0.5} (0.73); seen-task strength and few-shot adaptability are correlated but not interchangeable, and the variants furthest above the diagonal (GR00T, \pi_{0.5}) adapt better than their seen-task score would predict.

Panel (b) shows why a family-level bar chart would mislead. Inside the VLA family the variants span 0.29–0.73 and inside Video-VA they span 0.14–0.81, so for two of the three families the spread within the family is wider than the widest gap between any two family means (family means 0.49, 0.77, 0.57; widest gap 0.28). Video-Skill is by far the tightest: its two members behave alike, which is what makes it predictable, not necessarily what makes it best.

Panel (c) shows that the ordering is not an accident of how the automated predicate is written. Under blind A/B judgment in the simulation Arena – the same rollouts, scored by people instead of by the predicate – win rate gives the same family ranking (Video-Skill 0.64, Video-VA 0.50, VLA 0.42), with ACT/DINOv2 the single most preferred variant (\mathrm{WR}=0.82, \overline{Q}=8.1).

Because every Video-VA variant receives its task specification through a frozen encoder, the encoder is a design choice of for its action head. Figure 9 separates the two. On seen tasks (panel a, automated) the encoder ordering for ACT is DINOv2 \ge SigLIP2 > VideoMAE (0.81/0.79/0.72), but the size of the effect depends on which action head it is paired with: VideoMAE costs ACT 0.09 SR and costs Diffusion Policy 0.53 (0.67 \to 0.14), which is close to total failure. Encoder and action head therefore cannot be chosen independently.

After few-shot adaptation (panel b, automated), the gap almost closes for ACT (0.84/0.80/0.82) but not for DP or VQ-BeT: ten demonstrations can make up for a poor task embedding only when the action head is already strong.

Panel (c) replicates the ablation on hardware, where the scores are Arena human verdicts. DINOv2, SigLIP2, and VideoMAE were run on hardware with ACT only at the 45-task pre-training scale. At this matched scale the hardware ordering reproduces the simulation ordering in every regime: VideoMAE is last everywhere, and the two image-level encoders are close to each other, align with the simulation results.

Figure 10 is the evidence behind the scaling claim of Section 4.3. Panel (a) plots P+FT success against corpus size in both domains – automated in simulation, Arena on hardware. All three paradigms improve, in simulation and on hardware alike, and by 45 tasks every family's P+FT curve lies above the scratch baseline for its own domain (sim 0.35, averaged over all 15 variants; real 0.25, averaged over the four representative models). The gain therefore comes from the paired corpus, not from the ten few-shot demonstrations, which both conditions receive.

Panel (b) resolves this per model. Going from 15 to 45 pre-training tasks raises \Delta\mathrm{SR} for 18 of the 19 trained variants across the two domains (14/15 in simulation, 4/4 on hardware). We read this as a limit of its skill-dynamics stage, though with one data point we cannot separate that from noise.

Panel (c) reports what the aggregate hides. On hardware, zero-shot success on unseen tasks rises steadily with corpus size for all three video-conditioned models (XSkill 0.24 \to 0.33, ACT/DINOv2 0.22 \to 0.30, DP/DINOv2 0.19 \to 0.25), but stays at about 0.04 for \pi_{0.5} at every scale. More paired human-robot video therefore buys zero-shot ability only for policies that watch the video itself; passing the same demonstration through a caption first does not turn extra pre-training into transfer.

The body averages over corpus scales for compactness. Figure 11 resolves the zero-shot and P+FT settings by level and scale in both domains, and Table 9 gives the same numbers. Three readings follow. First, in every panel except simulation zero-shot, success improves monotonically or near-monotonically from 15 to 45 pre-training tasks at every level: the benefit is not confined to the easy end of the hierarchy.

Second, simulation zero-shot (panel a) sits at the floor throughout (\mathrm{SR} \le 0.14) and its level ordering moves around without a readable pattern. We draw no conclusion from it, and do not use it to rank encoders or paradigms.

Third, the L3 curve behaves differently in the two domains. In the real world (panel d) it improves with scale (0.23 \to 0.36) while remaining a clear band below L0–L2 at every scale, i.e. more paired pre-training helps at the intent level but does not close the intent-level gap. In simulation (panel b) L2 rather than L3 is the depressed level, for the construction reason analyzed in Section C.5.

Figure 12 lays out four panels in a single row – real world (Seen, P+FT) then simulation (Seen, P+FT) – success rate on the primary axis and a completion-quality score on the secondary axis in every panel. In panel (a), success is flat from L0 to L2 (0.54/0.55/0.54) and drops at L3 (0.39); \overline{Q} follows the same shape (6.97/7.12/ 7.07 \to 6.09). XSkill is strongest at every level (0.64/0.69/0.63/0.57) and still loses ground at L3.

In panel (b), success is flat from L0 to L2 (0.42/0.42/0.39) and drops at L3 (0.29). XSkill carries the drop (\mathrm{SR}\approx0.53–0.57 through L2, falling to 0.29 at L3; \overline{Q} 7.2\to5.6), while the other three models sit in a comparatively narrow \overline{Q}\approx5.4–6.3 band at every level. Current policies cope with a rearranged layout, or a replacement object of the same kind, but not with an object that has to be handled differently.

In panel (c), the all-model curve is non-monotone: it dips at L2 and rebounds at L3 (0.62/0.57/0.48/0.63), and the VLA family peaks at L3 above its own L0 value (0.55/0.42/0.36/0.62). The per-family Sub-SR curves show the identical dip-then-rebound shape (VLA 0.81/0.70/0.68/0.83, Video-Skill 0.92/0.89/0.89/0.91, Video-VA 0.79/0.77/0.72/0.78), so the effect is not confined to one paradigm or to the binary predicate. Figure 13(b) reproduces this shape on the same rollouts under Arena scoring, so it is not an artifact of the automated channel.

In panel (d), the L2 dip recurs (0.58/0.55\to0.37) with a partial rebound at L3 (0.54) that stays below L0, unlike panel (c). The pattern holds by family (VLA 0.52/0.55/0.33/0.53, Video-Skill 0.77/0.68/0.42/0.75, Video-VA 0.56/0.53/0.38/0.49) and in Sub-SR (VLA 0.72/0.74/0.41/0.70, Video-Skill 0.89/0.84/0.55/0.89, Video-VA 0.73/0.71/0.46/0.65), so the L2 dip is a property of the simulated L2 condition itself rather than of the seen/unseen split.

Why L2 dips in simulation. Two construction choices explain it. First, the simulated L2 condition also swaps which arm performs the manipulation – deliberately, to remove any motion-level shortcut, since L1 already moves the objects and L3 already changes what has to be done. Second, the simulation asset pool is finite and its motions are rule-based, so on simple pick-and-place tasks a substituted L3 object can often still be handled with roughly the strategy seen in training. Hardware offers no such shortcut: a functionally different object differs in mass, shape and how it must be gripped, and the robot has to adapt for real. This is why the body reports the simulation aggregate without splitting it by level, and reads the per-level picture from the Arena instead.

A policy that had simply memorized its training tasks could reproduce much of Section 4.2 without reading the conditioning video at all, since the imitator scene alone often identifies the task. We test this on the five seen tasks with the 45-task checkpoints: at inference we replace the conditioning demonstration V with a video of a different IG-10K task, leaving the simulated imitator scene, the robot and every other input untouched, and re-run the evaluation at all four levels.

In the similar regime the replacement shares broad manipulation structure with the original (StirSpoon\toGrindFood, FoldTowel\toPlaceClothBasket, PlaceMugRack\toPlaceCommodityRack, PlaceFileFolder\toPlaceMagazineFolder, PlacePlateRack\toPlaceBookBookcase); in the unrelated regime it shares nothing (OpenBox, PickWash, PlaceBrushRest, PlaceScrewdriver, ScanPillBottle, assigned in that order).

The benchmark reports two score streams and two domains, and Figure 13 quantifies how far each can substitute for the other. Panel (a) plots the automated predicate against the Arena judgment on the identical simulation rollouts. The two agree closely: r = 0.858 for \mathrm{SR} vs. \mathrm{SR}_{\mathrm{human}} and r = 0.861 for \mathrm{Sub\text{-}SR} vs. \overline{Q} when the same cells are resolved by level. The automated metric is therefore a faithful, cheap proxy for the human judgment in simulation, which is what licenses running the full 15-variant grid automatically.

Panel (b) resolves the same agreement by hierarchy level, on the seen-task rollouts: the automated \mathrm{SR} and Arena \mathrm{SR}_{\mathrm{human}} curves, and separately the automated Sub-SR and Arena \overline{Q} curves, both dip at L2 and both reproduce the L3 rebound (Section C.5).

Beyond the ten seen/unseen tasks used for cross-model comparison in the main text, we also report a full-coverage sanity check: DP and ACT trained on the complete IG-10K simulation corpus (all 50 tasks, 50 demonstrations each, all 200 task–level variants) and evaluated on every task. The purpose is not to compare these two models against the others, but to confirm that the ten selected tasks are representative of the wider pool – i.e., that the benchmark's difficulty comes from the L0–L3 hierarchy itself rather than from an unrepresentative or favorably chosen task subset – and to leave future work a reference that is not limited to the ten tasks used elsewhere in the paper. The full per-task reference is Table 11 in Appendix H.

The segmentation masks in IG-10K are produced as an offline preprocessing step rather than as part of the policy input or the benchmark scoring pipeline. They serve as an additional object-centric annotation layer of the dataset for analysis, dataset inspection, and future training objectives.

Text-conditioned mask proposal. We use a Grounded-SAM-2 based pipeline to generate the masks. For each task, annotators manually inspect episodes and summarize the relevant objects for human, robot, and simulation. The object list is converted into text prompts for GroundingDINO, which proposes object boxes for each camera episode using task-specific box and text confidence thresholds. These boxes are then passed to SAM-2 to produce the corresponding video masks for that episode. The procedure therefore combines human-curated object prompts with open-vocabulary detection and video segmentation.

Filtering and quality control. GroundingDINO detections are filtered by box and text confidence thresholds. In our runs we use conservative defaults of box_threshold=0.35 and text_threshold=0.25. We additionally render mask overlays on sampled camera episodes for visual inspection, run audit scripts to detect empty or missing episode-camera masks, and manually filter problematic results before treating the masks as part of the released annotation set.

MANO hand pose annotations in IG-10K are auto-generated and serve as auxiliary hand annotations.

Per-view WiLoR estimation. We estimate MANO-style hand poses with WiLoR on multi-view RGB videos. The camera streams are synchronized at the episode timestep level. Each view is processed independently, and predictions are stored separately for the left and right hands.

For frame t, camera c, and hand side h \in \{\mathrm{left}, \mathrm{right}\}, we store where \theta^{g} is the MANO global orientation, \theta^{p} contains the 15 articulated MANO joint rotations, and \beta denotes MANO shape coefficients. Rotations are stored in axis-angle form. We additionally store where \tau is the full-image camera translation converted from WiLoR's crop-level camera prediction, f is the focal length used for projection, and J denotes the 21 predicted 3D hand keypoints. Missing detections are stored as zero-valued placeholders with an invalid flag. Since calibrated camera extrinsics are unavailable, we do not transform estimates into a shared world coordinate frame; the annotations should be interpreted as per-view MANO estimates.

Shape-stabilized keypoints. To reduce temporal jitter caused by frame-wise MANO shape estimation, we compute an episode-level mean MANO shape \bar{\beta}_{c,h} from valid detections and use it only to recompute the stored 3D keypoints, while keeping other MANO parameters unchanged.

All baselines use the same IG-10K task splits, ZED2i RGB stream, and 16-dimensional dual-arm joint-position/gripper action space. For unseen-task transfer, both scratch and pre-train+fine-tune use the same 10 robot demonstrations per task.

We keep the two-stage UniSkill pipeline: skill-dynamics learning followed by a skill-conditioned diffusion policy. To align different human and robot execution speeds, the robot-side IDM transition uses robot_frame_gap=35. Stage 1 trains the UniSkill dynamics modules with 10-frame, 30 Hz clips, batch size 192, AdamW, learning rate 1{\times}10^{-5}, cosine decay, 500 warmup steps, and bfloat16 precision, while the VAE, text encoder, and depth estimator remain frozen. Stage 2 trains the policy with observation horizon 2, prediction horizon 16, action horizon 8, batch size 128, AdamW, learning rate 1{\times}10^{-4}, DDPM diffusion with 100 steps, and alignment loss weight 1.0.

We preserve XSkill's two-stage prototype-learning and prototype-conditioned policy structure, replacing offline prototype labels with prototypes extracted from paired IG-10K clips. Stage 1 uses 30-frame clips split by slide=3 into 4-frame windows, 128 prototypes, batch size 128, Adam, initial learning rate 2{\times}10^{-4}, OneCycleLR with maximum learning rate 4{\times}10^{-4}, temperature 0.1, and 3 Sinkhorn iterations. Stage 2 snaps each human video to 30 prototype tokens and trains a DDPM policy with observation horizon 4, prediction horizon 16, action horizon 8, batch size 128, AdamW, learning rate 1{\times}10^{-4}, cosine decay, 500 warmup steps, and 60 diffusion steps. At evaluation time, the robot-side prototype is computed from 4 frames sampled from the recent observation history.

We initialize from openvla/openvla-7b and keep the 7B vision-language backbone frozen. The human video is converted into the fixed caption T(V), and only the OFT-style (Kim et al., 2025) continuous action head and proprioceptive projector are trained for our 16-DoF dual-arm action space. To reduce training cost, we precompute action-token hidden states with shape (256,4096) per sample and add proprioception through an output-side projector before the action head. The action horizon is 16; training uses AdamW, learning rate 1{\times}10^{-4}, bfloat16 autocast, batch size 256, and 10 epochs across all three pre-training scales (45/30/15 tasks). Transfer runs also use 10 epochs with batch size 512.

We adapt the public \pi_{0.5}-DROID checkpoint. The PaliGemma vision-language prefix remains frozen, and the trainable modules are the action-expert branch, action projections, and time-conditioning MLPs. Inputs are the ZED2i RGB frame resized to 224{\times}224, normalized dual-arm state, and the fixed caption T(V). The model predicts 50-step continuous action chunks and is trained with the standard flow-matching objective. All pretraining runs use 10 epochs, AdamW, peak learning rate 3{\times}10^{-5} with cosine decay to 3{\times}10^{-6}, 1000 warmup steps, batch size 16, bfloat16 precision, and 10 denoising steps at inference.

We initialize from the public RDT-1B checkpoint and use the fixed caption T(V) as language input. SigLIP-SO400M/14 and T5-v1.1-XXL are frozen, with text features cached when possible. We train LoRA adapters in the RDT Transformer and condition adaptors for language, image, state, and action tokens; the base RDT weights remain frozen. Actions are mapped into the original RDT unified action space with a validity mask, using a 16-step horizon and 30 Hz control-frequency token. Training uses LoRA rank 576, alpha 1152, dropout 0.05, batch size 128, gradient accumulation 4, AdamW, learning rate 1{\times}10^{-4}, cosine schedule with 500 warmup steps, and bfloat16 precision. All pretraining runs use 10 epochs.

We adapt GR00T-N1.6-3B with the Eagle vision-language backbone frozen. The fixed caption T(V), current ZED2i RGB frame, and dual-arm proprioception condition a 16-step action chunk. We train the flow-matching DiT action expert together with embodiment-specific state/action encoders, decoder, action position embedding, and action-side normalization layers. For all pretraining runs, GR00T is trained for 10 epochs with AdamW, learning rate 1{\times}10^{-4}, weight decay 1{\times}10^{-5}, cosine schedule with 0.05 warmup ratio, batch size 256, bfloat16 precision, and 4 denoising steps at inference. Dataset statistics are recomputed for the NEW_EMBODIMENT normalization before each run.

ACT, Diffusion Policy, and VQ-BeT are conditioned on a cached task embedding z \in \mathbb{R}^{256} extracted from four uniformly sampled frames of the human video. We evaluate DINOv2-ViT-L/14, SigLIP2-SO400M, and VideoMAE-Large as frozen encoders, with a one-layer adapter trained with the policy. ACT injects z as an additional Transformer decoder token and uses L1 reconstruction plus KL loss (\lambda_{\mathrm{KL}}=10), action horizon 24, batch size 256, AdamW, learning rate 1{\times}10^{-4}, and 100 epochs. Diffusion Policy concatenates z into the global FiLM conditioning vector and uses a DDPM 1D UNet with action horizon 16, 100 train/inference diffusion steps, batch size 256, AdamW, learning rate 1{\times}10^{-4}, and 100 epochs. VQ-BeT prepends a projected z token to the observation sequence; its VQ-VAE is trained for 100 epochs on robot action chunks, then the GPT policy is trained for 100 epochs with batch size 256 and learning rate 1{\times}10^{-4}.

Evaluation. At inference, each video-conditioned policy encodes V once at episode start, while VLA policies reuse the fixed caption T(V). Simulation uses the rollout horizons and predicates in Appendix B.3, with 10 trials per task–level pair across all 15 variants; real-world evaluation uses \pi_{0.5}, XSkill, ACT/DINOv2, and DP/DINOv2 with 5 trials per task–level pair. Both domains evaluate the same 40 task–level pairs (Appendix B.2).

Real-world rollouts expose failure modes that are partly hidden by binary success rates.

Model-level patterns. XSkill produces the most consistent real-world behavior among the evaluated models. On seen tasks, its rollouts usually preserve the demonstrated action sequence and tolerate moderate object-placement changes; on unseen tasks it also shows useful partial transfer, but the records still contain failures from insufficient lift, premature gripper closure, or failure to release at the target. ACT/DINOv2 and DP/DINOv2 are less uniform. Their successful rollouts often follow the correct high-level subtask order, but many failures come from contact timing: the gripper closes before reaching the object, does not descend far enough, or releases too early/too late. \pi_{0.5} is the least stable in zero-shot real-world transfer. Its seen-task and fine-tuned runs can complete some tasks, but unseen zero-shot records repeatedly show arm jitter, early closure, and left-gripper non-closure, so its language-conditioned interface does not by itself give reliable physical transfer.

Task-level patterns. The clearest real-world failure cases are tasks where a small contact error destroys the rest of the rollout. Stir Soup in Bowl fails frequently because the spoon is thin and slips or is never fully captured. Discard Food Waste is difficult across models because it combines small/deformable food items, bimanual coordination, and precise release; many failures are left-gripper non-closure or no release after grasp. Scan Beverage Barcode similarly stresses grasp stability on the scanner or beverage, and several models reach the object but fail to close or hold the gripper. Hang Cup on Rack is a height-sensitive placement task: policies often grasp the mug but fail to lift high enough or align with the rack. By contrast, Place Plates in Rack, File Medical Documents, Fold Towel into Basket, and Seal and Pack Box show more coherent behavior, because the required grasps and placements are wider-tolerance and less dependent on exact gripper contact.

Unseen transfer. The unseen tasks should be read as evidence of partial generalization, not solved real-world imitation. Return Remote to Box is representative: several policies can localize and grasp the remote, and performance is often reasonable at L0–L2, but L3 exposes the missing affordance adaptation–the object is grasped with a plausible motion but placed at the wrong location, not lifted high enough, or released before reaching the substitute container. Pour Water into Cup also improves with adaptation, but failures still arise from missing the handle or closing before contact. The released rollout metadata stores task, level, model, corpus scale, setting, episode identifier, and video path, so each qualitative failure mode can be traced to representative rollouts in the benchmark release. These cases suggest that the models learn task-relevant visual and motion priors from IG-10K, while the few-shot real-world data are still too small to calibrate precise grasp points, release timing, and level-specific affordance substitutions.

Simulation per-task view. Table 11 gives the same diagnosis at larger scale. Tasks such as FoldBox, GrindFood, PlaceMugRack, and PickRemoteControl have high mean success, indicating that the policies can often infer the object and intended task. Yet the per-level breakdown is more informative than the mean: for PickRemoteControl, both DP and ACT remain strong through L0–L2, while L3 drops, matching rollouts where the remote can be found or grasped but the changed target relation is not executed correctly. Lower-scoring tasks such as PlaceCupPlate, PlaceFoodScale, and PlaceCommodityRack require tighter spatial alignment and contact geometry, so failures are better interpreted as insufficient manipulation coverage and calibration rather than a complete absence of task understanding.

Per-task reference results for simulation success rate. Each cell reports success rate (%) over 10 evaluation episodes for the corresponding task and imitation level.