Vision-Language-Action (VLA) models can follow instructions and manipulate objects, but their performance often collapses out of distribution (OOD), when the scene, viewpoint, or object differs from training. Adapting to each new situation typically requires collecting more data and fine-tuning. We present StellaVLA, a framework that instead adapts at test time by conditioning on a single retrieved demonstration. The key idea is to move beyond imitating what an expert did and instead convey why: an automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost. Provided as in-context guidance, this structured demonstration lets the policy reason about the task rather than mimic a pixel trajectory, which also makes it transferable across embodiments (real-robot, human-hand, or XR demonstrations). A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone, preserving real-time, high-frequency control with no added latency. On the VLA-Arena leaderboard（注：https://vla-arena.github.io/#leaderboard） (Aug 1, 2026), StellaVLA ranks first with an overall score of 0.63, versus 0.44 and 0.22 for the strong prior models (\pi_{0.5} and LingBot-VLA), and it further leads on LIBERO with 98.8% average success rate and LIBERO-Plus with 85.1% success rate. Our real-robot benchmark demonstrates that StellaVLA can use both human/robot demos and human-to-robot (XR) demos as in-context structured demonstration to help VLA model adapt to OOD tasks.

Vision-Language-Action (VLA) models have become a prominent paradigm for robotic manipulation [4, 23]. Built on pretrained Vision-Language Models (VLMs), they translate visual and textual inputs into physical actions. In practice, however, their performance degrades sharply out of distribution (OOD), when the scene, viewpoint, or object differs from training [9, 49, 61], and recovering typically requires collecting new data and fine-tuning. In-Context Imitation Learning (ICIL) offers a way to adapt at test time without weight updates [13, 57]. By appending retrieved expert trajectories, e.g., sequences of observations and low-level actions, as a contextual prefix, ICIL gives the policy a non-parametric memory it can imitate on the fly. But existing frameworks use these trajectories only as raw observations and continuous actions, which encourages surface-level imitation: the policy sees what the expert did without the why. Lacking that structure, it often treats the demonstration as noise and falls back on its pretrained priors [49], which is a behavioral inertia that keeps it from generalizing to out-of-distribution tasks.

Reasoning provides the missing structure. Foundation models generalize better when the intermediate reasoning is made explicit rather than left implicit [17, 51, 59, 60]: decomposing a task into logical steps exposes the why behind each action. The question we address is how to surface this reasoning inside retrieved demonstrations, so that a VLA policy imitates the expert's reasoning rather than only its motions. Motivated by this idea, we propose StellaVLA (Figure 1), which advances ICIL by turning demonstrations into structured, reasoning-augmented context. An automated offline pipeline converts raw expert trajectories into this form at zero human-annotation cost: following the “language-as-action” paradigm [8, 29], a VLM segments long-horizon trajectories and generates hierarchical rationales, high-level semantic rationales (sub-goals) and fine-grained kinematic rationales (movements), which augment each demonstration alongside its original observations and actions. Applied across sources, this yields a unified cross-embodiment demonstration pool.

Unlike prior ICIL methods that rely on prompting alone [21, 58], StellaVLA couples in-context prompting with explicit supervision through a parallel dual-training design. During training, the model conditions on the retrieved rationale-augmented demonstrations and is jointly optimized to predict low-level actions and to articulate the corresponding rationales, internalizing both what to do and why. The language supervision grounds actions in their semantic intent, while the retrieved rationales act as a reusable template for how to plan in similar situations. At inference, the spatial-language expert is removed entirely: the policy uses only the action expert, guided by the KV-cached demonstration prefix, giving real-time high-frequency control with no added latency while still benefiting from the task structure learned during training.

Our main contributions are:

• We propose StellaVLA, a retrieval-augmented VLA framework that shifts ICIL from imitating actions to imitating the reasoning behind them, reducing behavioral inertia under OOD conditions.
• A parallel dual-training design, fed by a zero-human-cost offline extraction pipeline, that internalizes expert reasoning during training and strips the spatial-language expert at inference, leaving the control loop free of autoregressive decoding overhead.
• Extensive simulation and real-robot manipulation experiments show that StellaVLA ranks first on the VLA-Arena leaderboard (Aug 1, 2026) with an overall score of 0.63, while achieving 98.8% average success in LIBERO and leading in LIBERO-Plus and our real-robot benchmark.

Vision-language-action (VLA) models adapt pretrained vision-language backbones for robotic control. Actions can be generated as discretized tokens [23], or predicted through regression, flow-matching, and frequency-domain action heads [4, 16, 24, 35, 37]. Recent work has further improved computational efficiency [11, 34, 47, 48] or augmented policy representations with geometric, depth, and world-model priors [19, 26, 27, 32, 40, 42, 56]. In contrast, StellaVLA retains a standard VLM backbone with a continuous-action expert and focuses on a complementary question: how expert demonstrations should be represented and exploited as in-context supervision.

Language-as-action methods represent robot behavior using the vocabulary of pretrained VLMs, providing a semantic interface between high-level reasoning and continuous control. Prior work verbalizes low-level actions, jointly models language and action tokens, or represents action hierarchies and subtasks in language [2, 15, 28, 52]. By expressing behavior in the VLM's semantic space, these approaches retain an interpretable correspondence between task intent and robot motion.

StellaVLA uses language as a shared representation for both retrieved context and training supervision. Each demonstration is organized into subgoals that pair semantic descriptions and keyframes with language-rendered robot states and 2D/3D motions, yielding a structured semantic state-action sequence. During training, a language expert predicts the current subtask and a language rendering of the same action chunk predicted by the continuous action expert. This shared interface connects demonstration-level procedural structure with step-level control, while the language expert is removed at inference. This differs from latent-action approaches, which encode behavior into learned latent variables outside the VLM's native semantic space [5, 50].

Robot policies can adapt at deployment either by updating model parameters or by conditioning a fixed policy on additional experience. Test-time training methods update fast weights, latent prompts, or memory using deployment data [12, 22, 43, 55]. In contrast, context-conditioned approaches leave the policy parameters unchanged and adapt behavior through demonstrations, retrieved experience, or execution history [31, 33, 39, 57]. Recent work has further explored in-context imitation and retrieval-based adaptation for VLAs [13, 20, 41].

The supplied context, however, can play substantially different roles. \pi_{0.7} conditions on subtask instructions, subgoal images, episode metadata, and control modes to specify an execution strategy [36], while Qwen-RobotManip uses recent observation–state–action chunks to adapt to the dynamics of the current episode [38]. StellaVLA instead retrieves a prior demonstration and converts it into a structured procedural context composed of segment-level subtasks and grounded 2D/3D motion. The retrieved episode therefore serves as an explicit task demonstration rather than merely additional execution history, enabling adaptation without parameter updates.

Human videos and heterogeneous robot data provide broader behavioral coverage than single-platform demonstrations, but differ in both embodiment and action space. Prior work addresses this mismatch through shared latent actions, cross-embodiment motion representations, world models, video conditioning, retargeting, or canonicalized action spaces [3, 6, 7, 14, 18, 38, 44].

StellaVLA instead addresses cross-embodiment demonstrations at the representation level. Real-robot, human-hand, and XR-retargeted demonstrations are converted into a common structured representation of semantic subtasks and grounded motion before retrieval. Crucially, off-embodiment demonstrations are used only as context, and executable actions are always predicted and supervised in the target robot's native control space. This separation allows heterogeneous demonstrations to provide procedural guidance without requiring their source action spaces to be directly aligned with the target policy.

In this section, we detail the StellaVLA framework, illustrated in Figure 2, which advances ICIL with structured contextual rationales to improve VLA generalization. We first present an automated offline pipeline that converts raw trajectories into rationale-augmented segments (Sec. 3.1), then introduce a parallel dual-training paradigm that uses these trajectories as both explicit supervision and in-context prompts (Sec. 3.2), and finally describe an asymmetric inference strategy that leaves the control loop free of autoregressive decoding overhead (Sec. 3.3).

Raw trajectory formulation. An expert demonstration is recorded as \tau = \{(o_t, a_t)\}_{t=1}^T, where o_t denotes sensory input (e.g., multi-view RGB and proprioception) and a_t \in \mathbb{R}^{d_a} is the continuous control command (e.g., end-effector pose and gripper state). Such trajectories may come from diverse embodiments and collection paradigms, including real-robot teleoperation, human hand tracking, or XR-retargeted demonstrations [44]. Although they encapsulate successful executions, the raw numerical actions lack explicit semantic rationale, which hinders cross-embodiment reasoning.

Semantic segmentation via causal deduction. To bridge the semantic gap without incurring prohibitive human annotation costs, we introduce an automated offline pipeline (Figure 2, right) that deduces the expert's underlying thought process. The core motivation is to infer the “cause” (the expert's underlying structured rationale) from the “effect” (the observed physical execution). Given a raw trajectory \tau and its corresponding high-level task instruction \mathcal{I}, we utilize a powerful off-the-shelf Vision-Language Model (e.g., Qwen3-VL) to explicitly decompose the continuous trajectory into K discrete, semantically meaningful segments. For each segment k \in \{1, \dots, K\} spanning from time step t_{\mathrm{start}}^{(k)} to t_{\mathrm{end}}^{(k)}, the VLM analyzes the visual changes to identify the high-level sub-goal achieved. This process captures the true decision-making process that governs the expert's behavior.

Kinematic verbalization via language-as-action. Once the trajectory is segmented, we adopt a “language-as-action” paradigm that deterministically translates physical actions within each segment into structured text. Because VLAs are built upon VLMs with a strong affinity for semantic language[15, 54], these rationales can be processed with the original text vocabulary, without introducing specialized action tokens.

Ultimately, this two-tier process generates a Structured Context l_k for each segment that explicitly articulates the underlying logic through a hierarchical representation:

• Semantic Rationale (Sub-goal Description): Captures the high-level semantic objective achieved in this segment (e.g., “Reach for the handle of the blue mug”), automatically identified by the VLM based on visual observations.
• Kinematic Rationale (Movement Description): Verbalizes fine-grained motion as a 3D movement in the workspace (e.g., “Move the end-effector by \Delta x=+0.05, \Delta y=-0.02, \Delta z=+0.10 and close gripper”) and a 2D movement obtained by projecting the 3D trajectory onto the camera plane with known intrinsics/extrinsics. This dual form encourages the VLM to reason about spatial logic and visual grounding via text tokens, rather than emitting raw floating-point actions.

Both kinematic fields are produced by a single deterministic verbaliser \Phi, which maps any contiguous span of actions to its 3D displacement in the workspace and to the 2D projection of that displacement onto the image plane. Applying \Phi over a whole segment yields the movement description carried inside l_k; we reuse the same operator at the much shorter scale of an action chunk in Sec. 3.2, which is what places the retrieved demonstration and the policy's own prediction in one common vocabulary. The sub-goal description, by contrast, is attached to every frame: each step t carries the subtask s_t that the expert is executing in the observation o_t.

Rationale-augmented demonstration pool. Through this process, the original raw trajectory \tau is transformed into a rationale-augmented trajectory \tau_{\mathrm{rat}} = \{(o_t, a_t, s_t, l_k)\}_{t=1}^T, where each time step t is now grounded not only by its corresponding observation and action, but also by its own subtask label s_t and by the explicit segment-level structured rationale l_k of the segment it belongs to. By processing all available expert demonstrations through this pipeline, we construct a comprehensive demonstration pool \mathcal{D}_{\mathrm{pool}} = \{\tau_{\mathrm{rat}}^{(i)}\}_{i=1}^N.

Retrieval-augmented prompt formulation. During training, the learning process is formulated as a leave-one-out retrieval task. To train the policy to imitate a specific target trajectory \tau_{\mathrm{tgt}} sampled from \mathcal{D}_{\mathrm{pool}}, the system queries the remaining pool \mathcal{D}_{\mathrm{pool}} \setminus \{\tau_{\mathrm{tgt}}\} and retrieves the top-1 rationale-augmented expert trajectory, ranked by cosine similarity between the language embeddings of the task instructions. Retrieval is therefore purely linguistic, which is what lets a demonstration recorded on another embodiment be retrieved for a robot episode. The retrieved trajectory is formatted into a contextual prefix prompt \mathcal{P}_{\mathrm{demo}}. The final input to the VLA model at time step t is constructed by concatenating the retrieved demonstration, the current task instruction \mathcal{I}, and the current observation o_t: Crucially, because \mathcal{P}_{\mathrm{demo}} contains the rich l_k descriptions from the experts, it acts as an implicit in-context learning (ICL) signal, providing the model with a cognitive template of how to “think” and plan in similar scenarios.

Parallel experts architecture. The core of our VLA policy is a unified Vision-Language Model backbone f_\theta equipped with two parallel experts (Figure 2, bottom left). Given the input x_t, the backbone processes the multimodal sequence into a shared latent representation h_t = f_\theta(x_t). This single latent vector is then simultaneously fed into:

1. An action expert, a lightweight MLP head that regresses the continuous action chunk \hat{A}_t = (\hat{a}_t, \dots, \hat{a}_{t+H-1}) spanning the next H control steps.
2. A spatial-language expert, the native autoregressive LM head, that emits the rationale \hat{c}_t = (\hat{s}_t, \hat{m}_t) of the current step: the subtask \hat{s}_t being executed in o_t, and the verbalised 3D and 2D movement \hat{m}_t of that very same action chunk.

It is vital to note that these two experts operate strictly in parallel. The action prediction does not causally depend on the generated text tokens at time step t. They share the representation h_t and nothing else.

Spatial language supervision. While the retrieved demonstration provides segment-level context, spatial-language supervision is defined at each time step as where s_t is the subtask label for o_t, A_t = (a_t, \dots, a_{t+H-1}) is the ground-truth action chunk, and \Phi(A_t) verbalises its 3D and 2D movement. The subtask label provides semantic supervision for the current task stage; the 2D movement grounds the prediction in o_t, while the 3D movement provides kinematic supervision in the workspace.

Dual rationale objective. During training, the model is optimized to minimize a joint objective: where \mathcal{L}_{\mathrm{act}} is the regression loss for the continuous control commands (e.g., L_1 loss), \mathcal{L}_{\mathrm{lang}} is the standard autoregressive cross-entropy loss for the structured rationale text generation, and \lambda is a balancing coefficient.

The two experts provide complementary supervision for the shared representation h_t. Since \Phi is deterministic, A_t and \Phi(A_t) describe the same motion in continuous and linguistic forms. Together, the explicit semantic, grounding, and kinematic supervision in c_t and the contextual guidance from \mathcal{P}_{\mathrm{demo}} encourage h_t to connect the demonstrated task structure and current observation with the robot motion required at time t.

Action-only execution. A fundamental limitation of standard Embodied CoT paradigms is the severe latency introduced by autoregressive text generation during inference[17, 25, 51, 59], which often breaks the strict timing requirements of high-frequency robotic control. Our parallel architecture elegantly resolves this bottleneck. Because the action expert and the spatial-language expert read from h_t independently, and the profound physical “mental model” has already been forged into the backbone weights during training via \mathcal{L}_{\mathrm{lang}}, the language read-out becomes strictly optional at deployment.

During inference, the autoregressive spatial-language expert is entirely stripped away. The policy executes a single forward pass through the backbone and the lightweight MLP action expert to output the action chunk \hat{A}_t. This asymmetric strategy decouples the acquisition of reasoning (paid for during training) from its execution, so continuous control incurs no autoregressive decoding overhead.

Demonstration prefix caching. To further optimize inference efficiency, we exploit the static nature of the retrieved context[48]. For a given novel task, the retrieved rationale-rich demonstration prefix \mathcal{P}_{\mathrm{demo}} is pinned at the beginning of the rollout and remains immutable for the duration of the episode. Consequently, the key-value (KV) cache for the entire prefix is computed exactly once at t=1. For all subsequent time steps, the policy only needs to forward the live suffix (the current observation o_t), drastically reducing the computational footprint. This caching mechanism ensures that carrying a rich, multi-step rationale demonstration incurs virtually zero marginal cost during the high-frequency control loop.

We evaluate four questions: (i) whether structured demonstrations improve in-distribution performance and generalization under task and visual shifts, (ii) whether the policy uses the retrieved demonstration and which parts of it carry transferable information, (iii) how spatial-language supervision affects the learned representation, and (iv) whether the resulting policy can use cross-embodiment context without sacrificing deployment efficiency.

Model and Training. StellaVLA couples a Qwen3-VL-4B backbone [1] with an OpenVLA-OFT-style MLP action expert [24]. At each step, the model receives third-person and wrist RGB observations, a language rendering of the robot state, the task instruction, and one retrieved same-task structured demonstration. The action expert regresses an action chunk under an L_1 objective. In parallel, the spatial-language expert is trained by cross-entropy (weight \lambda=0.3) to predict the current subtask and the 2D/3D description of the same action chunk. The subtask, 2D movement, and 3D movement provide semantic, visual-grounding, and kinematic supervision, respectively. The spatial-language expert is not decoded at inference.

All models are fully fine-tuned from Qwen3-VL-4B-Instruct for 30k steps with a global batch size of 128. Context-demonstration dropout is 0.0 when a same-task demonstration is always available and 0.5 otherwise. Appendix A gives optimization, retrieval, and platform-specific details.

Protocol and Matched Control. We report task success rate under each benchmark's official protocol. StarVLA-OFT [42] is trained with the same backbone, action expert, and data as StellaVLA, but receives neither a retrieved demonstration nor spatial-language supervision. It therefore measures the gain from the complete StellaVLA design rather than either component alone. To isolate the role of context, Table 5 instead intervenes on a fixed StellaVLA checkpoint by supplying the correct, no, or a wrong-task demonstration at evaluation.

We first evaluate StellaVLA on three simulation benchmarks, as shown in Figure 3, probing in-distribution competence, task-level generalization, and zero-shot robustness.

StellaVLA reaches 98.8\% average success on LIBERO (Table 1), showing that structured context does not compromise in-distribution performance. Relative to StarVLA-OFT, the gain is small on Object (+0.4) but larger on Goal (+3.4) and Long (+3.0), where the current observation alone may not determine the intended outcome or subgoal order. The same pattern becomes more pronounced when the demonstration is removed or replaced at evaluation (Table 5).

VLA-Arena [53] evaluates task-level generalization across 11 suites in four categories: Safety, Distractor, Extrapolation, and Long Horizon. Each suite contains three difficulty levels, from L0 (in-distribution) to L2 (hardest), while training uses L0 data only. We follow the official protocol and compare against OpenVLA-OFT [24], LingBot-VLA [45], Motus [3], GR00T-N1.6 [32], Evo-Depth [27], and \pi_{0.5} [37]. At test time, StellaVLA receives one structured demonstration of the target task without any parameter update. As shown in Table 2, StellaVLA achieves the best mean success rate at all three levels: 0.84, 0.62, and 0.43 on L0, L1, and L2, respectively. Its overall score is 0.63, compared with 0.44 for the strongest baseline, \pi_{0.5}. The margin over \pi_{0.5} grows from 0.15 at L0 to 0.24 at L1 and remains 0.17 at L2, where no parameter update is performed and only the target-task demonstration is supplied.

The per-suite results further delimit this gain. StellaVLA maintains 0.80 on Unseen Objects across all three levels, consistent with transferring a procedure when the object referent changes. Task Workflows rises from 0.64 at L0 to 0.84 at L2, suggesting that context can become more useful as the test workflow departs from training. Long Horizon remains near zero at L1/L2 for every method, including ours: a fixed prefix specifies the procedure but cannot re-plan after execution drift.

LIBERO-Plus [10] perturbs LIBERO tasks along seven axes: viewpoint, robot state, sensor noise, object layout, background, lighting, and language. We evaluate the LIBERO-trained checkpoint without retraining and compare with the zero-shot baselines reported in [10]. StellaVLA reaches 85.1\% on average (Table 3), outperforming StarVLA-OFT by 10.1 points. The largest gains occur under camera viewpoint (+23.5), sensor noise (+19.7), robot initial state (+14.7), and language perturbations (+8.3), where the observation or instruction changes while the task procedure remains intact.

The smaller gains are also informative. Background and lighting are already near saturation for both models, leaving little room for improvement. Under object-layout changes, StellaVLA gains only 0.1 point because the demonstration's scene-specific spatial relation no longer matches the current scene. Structured context therefore transfers task procedure more reliably than a fixed spatial trajectory.

Platform, Tasks, and Data. We evaluate on a 6-DOF AgileX Piper arm with third-person and wrist-mounted RGB cameras. The benchmark contains four tabletop tasks: precision pen-to-cup placement, carrot-to-bowl pick-and-place, placing blocks in a drawer and closing it, and stacking three bowls. We collect 125 teleoperated robot episodes (71{,}702 frames), covering precision, articulated-object, and multi-stage manipulation.

Cross-Source Demonstration Pool. The retrieval pool combines the robot episodes with 26 human-hand takes recorded in XR and 26 frame-aligned trajectories retargeted to the Piper. All three sources are converted into the same structured representation. The 2D grounding point is aligned across the human wrist and robot flange, and off-embodiment frames are used only as retrieved context: all executable action targets remain from the real robot.

Protocol and Baselines. We evaluate four in-distribution tasks, two OOD-L1 object-attribute perturbations, and one unseen OOD-L2 drawer task, with 10 rollouts per cell. Object placements are re-randomized and matched across methods. StarVLA-OFT is the matched control for the complete StellaVLA design, while the separately pretrained \pi_{0.5} [37], fine-tuned on the same episodes, provides an external reference. Appendix B specifies the control stack, perturbations, success criteria, and progress metric.

Results. StellaVLA reaches 85.0\% success in distribution and 75.0\% on OOD-L1 (Figure 4). On the two tasks with an L1 variant, its average decreases from 80.0\% in distribution to 75.0\%, a 5-point drop; StarVLA-OFT and \pi_{0.5} drop by 25 and 20 points on the same tasks. This smaller paired degradation indicates greater robustness when the object attributes change, although the StarVLA-OFT comparison measures the demonstration and spatial-language supervision jointly.

No method completes the unseen OOD-L2 task. StellaVLA nevertheless reaches an average progress score of 1.9 out of four, compared with 1.5 for \pi_{0.5} and 1.1 for StarVLA-OFT. This partial progress suggests that the retrieved procedure remains useful, but does not constitute zero-shot task completion.

Cross-Source Consistency. The mixed-source rollouts above cannot isolate the effect of demonstration provenance, because each cell samples from all three sources. We therefore evaluate a fixed checkpoint on held-out teleoperated episodes while holding the current observation fixed and changing only the retrieved source. We measure the mean L_1 disagreement D(S_1,S_2) between predicted action chunks, normalized by the per-dimension standard deviation \sigma of the ground-truth actions. The paired human and XR-retargeted demonstrations share the same trajectory and timing, so their comparison changes embodiment appearance without changing the demonstrated behavior.

Across 1{,}780 ID and 1{,}719 OOD frames, Table 4 shows that real-robot, human-hand, and XR-retargeted context yield closely matched predictions. Source-to-source disagreement is 0.0014–0.0016\sigma, corresponding to 0.02^\circ–0.03^\circ of joint angle and less than 0.05 mm of gripper width. This supports consistency across the three structured demonstration sources.

This test is limited to single-step predictions. Removing the demonstration changes the action by only 0.0041–0.0045\sigma, and small differences may still accumulate in closed-loop execution. The result therefore establishes source consistency under matched observations, not closed-loop invariance or the standalone contribution of context on hardware.

Unless stated otherwise, these analyses use the LIBERO checkpoint and report the four-suite average (AVG) and, where available, the LIBERO-Plus average (LP). Evaluation-time interventions change only the demonstration supplied to a fixed checkpoint.

Causal Role of the Demonstration. Table 5 evaluates the same trained checkpoint with the correct demonstration, no demonstration, and a demonstration from a different task. Removing the demonstration reduces the average success rate from 98.8 to 62.4, while providing a wrong-task demonstration further lowers it to 44.9. The latter result is particularly informative: if the policy largely ignored the retrieved context, missing and mismatched demonstrations would produce similar outcomes. Instead, the additional degradation shows that StellaVLA actively uses the demonstration to determine the intended task and adjusts its behavior accordingly, even when the supplied context is misleading.

The per-suite results further clarify when this contextual specification matters. On Goal, performance falls to 24.8 without a demonstration and to 0.0 with a wrong one, since tasks may involve similar objects and scenes but require different outcomes. The demonstration is therefore critical for disambiguating the intended goal. By contrast, Spatial performs similarly with no and wrong demonstrations (72.2 and 72.6), suggesting that the current observation still provides useful cues about where the interaction should occur. These results provide direct evidence that the retrieved demonstration serves as a task specification rather than being ignored as auxiliary visual context.

Spatial-Language Components. We next isolate the two movement fields in the structured demonstration. Removing the 3D movement lowers AVG from 98.8 to 97.8, whereas removing the 2D gripper path lowers it to 97.3. Although the 2D path is a projection of the same physical motion, it directly anchors that motion in the current image. Its larger effect is consistent with the 2D component providing visual grounding and the 3D component providing workspace-level kinematic information. This experiment does not isolate the semantic subtask label, for which no removal result is available.

Demonstration Modality. Table 7 examines which part of the demonstration provides the useful context. At evaluation time, text-only demonstrations nearly match the full image+text input, achieving 98.8/84.4 compared with 98.8/85.1 on LIBERO AVG and LIBERO-Plus. In contrast, image-only demonstrations decrease performance to 92.9/75.7, with the largest loss appearing on Long. These results indicate that most of the transferable task information is captured by the structured language representation, including the subgoal sequence and spatial movement, rather than by raw demonstration frames alone. This also supports the use of demonstrations across embodiments, since the structured description abstracts away much of the source-specific visual appearance.

The training-time block of Table 7 reveals a complementary trade-off. Image-only demonstrations provide slightly higher in-distribution performance than text-only demonstrations (98.4 vs. 97.3), but generalize substantially worse on LIBERO-Plus (78.7 vs. 85.0). A plausible explanation is that visual demonstrations allow the policy to exploit appearance correspondence between the demonstration and the current observation, which is effective in distribution but becomes unreliable under visual perturbations. Structured language removes much of this shortcut and instead encourages the policy to rely on task-level and spatial structure that remains more stable across distribution shifts.

Context Granularity. Performance changes by only 0.7 point when the demonstration is reduced from ten to three subgoal keyframes (Table 9 in Appendix C). This saturation suggests that the subgoal sequence, rather than dense trajectory replay, carries the useful context.

Effect of Language-Loss Weight. Table 6 varies the weight \lambda of the spatial-language objective. The in-distribution average follows a shallow inverted-U and peaks at \lambda=0.3 (98.8), whereas LIBERO-Plus robustness decreases from 86.9 at \lambda=0 to 81.9 at \lambda=1.0. Spatial-language supervision therefore improves in-distribution precision at the selected weight but does not monotonically improve OOD robustness; context conditioning alone already provides a strong robustness signal. One possible explanation is that a large language weight overemphasizes the offline annotation schema. We use \lambda=0.3 as the best in-distribution setting.

Inference Efficiency. Table 8 separates the cost of context encoding from language decoding. A forward pass without a demonstration takes 64 ms. Adding an image+text demonstration raises this to 183 ms without a cache, while caching the immutable prefix reduces the steady-state cost to 91 ms. In a paired measurement, the action-only path takes 88 ms, whereas decoding the 83-token spatial-language output increases latency to 3177 ms, about 36\times slower.

The 88–91 ms values measure the model-side path; the real-robot pipeline, including observation and control overhead, runs at approximately 205 ms per action chunk (Appendix B). Thus, removing the spatial-language expert avoids autoregressive decoding at deployment, while prefix caching amortizes demonstration encoding across the rollout.

We presented StellaVLA, a framework that improves VLA generalization by converting retrieved expert demonstrations into structured in-context guidance. An automated offline pipeline extracts hierarchical semantic and kinematic rationales from heterogeneous demonstrations, while a parallel dual-training objective jointly learns continuous control and spatial-language reasoning. At inference, the spatial-language expert is removed and the fixed demonstration prefix is KV-cached, allowing the policy to benefit from structured reasoning without autoregressive language-decoding overhead. Experiments on LIBERO, VLA-Arena, LIBERO-Plus, and real-world manipulation show that StellaVLA preserves strong in-distribution performance while substantially improving robustness to task and environment shifts, including when demonstrations originate from different embodiments.

Contributor: Siyu Xu, Yunke Wang, Zijian Wang, Dihao Zhu, Chenghao Xia, Chengbin Du, Daochang Liu, Tao Huang, Chang Xu.

Optimization. All models are fully fine-tuned from Qwen3-VL-4B-Instruct with AdamW (\beta = 0.9/0.95, weight decay 10^{-8}), a backbone learning rate of 5\times10^{-6} and an action-expert learning rate of 10^{-4} under a cosine schedule (500 warmup steps), gradient clipping 1.0, bf16 mixed precision, and DeepSpeed ZeRO-2, for 30k steps at a global batch size of 128.

Regularization. Two per-sample dropouts regularize training. A 2D gripper-path dropout (0.5) keeps the 3D steering command usable when the pixel trace is missing. A context-demonstration dropout matches demonstration availability at deployment: 0.0 when a same-task demonstration is always retrievable (LIBERO, LIBERO-Plus) and 0.5 otherwise (VLA-Arena, the real robot).

Retrieval. Demonstrations are retrieved by the language embedding of the task instruction alone; no visual features enter the query. In simulation the instruction set is closed, so this reduces to an exact task match, and we condition on the single nearest demonstration (M=1) throughout. Training uses leave-one-out retrieval, which excludes the target episode from the pool.

Action Expert and Spatial-Language Expert. A set of action-placeholder tokens is appended to the user turn, causally preceding the language-action chain-of-thought. After a single backbone forward, their final-layer hidden states pass through a two-block residual MLP that regresses the action chunk under an L_1 objective. The compact “subtask + 3D/2D steering command” chain-of-thought is retained from a language-action formulation [52] and supervised against offline-generated annotations by cross-entropy; it is not decoded at action inference. The steering command is the target \Phi(A_t) of Eq. 2, that is, the same action chunk the MLP regresses, rendered by the verbaliser \Phi over the chunk span rather than over a whole segment; the subtask is the per-frame label s_t.

Per-Platform Instantiation. In simulation the policy reads 256\times256 images, predicts end-effector deltas over an action chunk of length 8, and the retrieved demonstration carries up to ten subgoal keyframes. On the real robot it reads 640\times480 images, predicts a 7-dimensional chunk of length 16 (six joint-position deltas relative to the episode start plus an absolute gripper width, all q_{99}-normalized), and the demonstration carries up to eight keyframes. The real-robot state is rendered in language as the end-effector pose together with the joint angles.

Control Stack. The AgileX Piper is driven over CAN at 30 Hz without a ROS stack. Each 16-step chunk spans \approx\!0.53 s and is executed in a synchronous receding-horizon loop with a horizon of 8. Joint targets pass through a One Euro filter and a 25^\circ per-step jump guard before reaching the arm. Because the demonstration prefix is fixed for the duration of a rollout, its key–value cache is computed once and reused, and steady-state inference settles at \approx\!205 ms per chunk, comfortably inside the control budget.

Data Collection. The 125 teleoperated episodes (71{,}702 frames) were collected by master–slave teleoperation across the four tasks: pick the pen and put it in the cup, pick the carrot and put it in the bowl, put the blocks into the second drawer and close it, and stack the three bowls on top of each other. They cover precision alignment, free-space pick-and-place, a long-horizon sequence involving an articulated object, and multi-stage stacking respectively.

Cross-Source Pool Construction. Twenty-six human-hand takes recorded in XR yield two aligned datasets of 26 episodes and 10{,}996 frames each: one preserving the operator's bare hand, one showing a retargeted Piper arm executing the identical trajectory. The two share episode structure and timing frame for frame, so the appearance of the embodiment is the only variable separating them. The grounding point of the 2D gripper path uses the operator's wrist for the human take and the flange for the retargeted arm, both matched to the real robot's convention, so that a rendered coordinate denotes the same physical quantity regardless of provenance. The three sources are merged by exact task-string match, retrieval excludes the current episode, and the off-embodiment data enters only the retrieval pool: no human or retargeted frame ever supplies an action target, so the policy's motor supervision remains entirely real-robot.

Evaluation Conditions. Every reported cell is 10 rollouts scored by a human operator against a fixed success criterion, with object placements re-randomized per rollout and held identical across methods. OOD-L1 severity is not calibrated across tasks: it compounds a colour and an object substitution for pen\tocup but changes only colour for carrot\tobowl, so L1 is comparable within a task rather than across tasks. On OOD-L2 the progress score awards one point for each of the three blocks placed and one for closing the drawer, out of four. Single cells carry roughly 10–15 points of noise at 10 rollouts, so we base conclusions on the per-condition averages, which aggregate 40 rollouts in distribution and 20 under L1, and on paired degradations. Because only pen\tocup and carrot\tobowl have an L1 variant, the degradation quoted in the main text compares each policy against its own score on those two tasks (80.0\% for StellaVLA, 55.0\% for StarVLA-OFT, 80.0\% for \pi_{0.5}) rather than against the four-task in-distribution average. On OOD-L2 the progress scores are 1.9 for StellaVLA, 1.5 for \pi_{0.5} and 1.1 for StarVLA-OFT; the task is absent from real-robot training, so the human take is the only entry in the pool holding a demonstration of it.

Baselines. The matched control is StarVLA-OFT [42], trained on the same real-robot episodes with the same backbone, action expert and action space, and differing only in receiving no demonstration and no auxiliary spatial-language expert. We additionally fine-tune the openpi flow-matching policy \pi_{0.5} [37] on the same episodes; as a separately pretrained model family it places our absolute numbers on an external scale rather than serving as a matched control.

Subgoal granularity (Table 9) is nearly flat: three keyframes already reach 98.1 AVG, within 0.7 of the full ten (98.8). Denser sampling adds frames but no new sub-goals, and it is the sub-goal decomposition the policy consumes, so returns saturate as soon as the plan is complete rather than as soon as the trajectory is densely covered. This is also what keeps the prefix short enough to cache.

Language as a Cross-Embodiment Bridge. Under matched observations, real-robot, human-hand, and XR-retargeted demonstrations produce closely matched action predictions (Table 4). The hardware results also use demonstrations sampled from the merged three-source pool rather than from the robot source alone. Together, these results indicate that the structured representation reduces sensitivity to source-specific appearance and coordinates. They do not establish closed-loop invariance across pinned sources, which requires a larger source-controlled rollout study.

Decoupling Reasoning from Latency. Autoregressive language generation is incompatible with the latency budget of real-time control. StellaVLA instead trains the backbone under joint action and spatial-language supervision, then removes the spatial-language expert and caches the demonstration prefix at deployment. Control therefore incurs no autoregressive language-decoding overhead (Table 8).

Limitations and Future Work. Despite its strong performance, our framework has several limitations. Long-horizon compositionality: while StellaVLA substantially improves task-level generalization and robustness, success rates on extremely long-horizon, multi-stage tasks (such as the Long Horizon suite in VLA-Arena) remain low for all evaluated models. The demonstration prefix is fixed for the duration of an episode, so it cannot re-plan once execution has drifted, and in-context conditioning alone therefore does not resolve multi-step error accumulation. Dependence on retrieval quality: the effectiveness of our in-context adaptation relies on retrieving a relevant expert demonstration. As the wrong-demonstration ablation shows, conditioning on a mismatched demonstration degrades performance below the no-demonstration baseline, so more robust retrieval under visual and semantic domain shift is a critical next step. Finally, Coarse kinematic discretization: our textualized kinematic rationale relies on a quantized representation of spatial movements to fit within the VLM's vocabulary. While this quantization suffices to shape the shared representation, it may introduce discretization errors that limit the precision of fine-grained manipulation. Continuous tokenization or hybrid representation schemes could mitigate this.