# Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics
**Authors**: Yibo Qiu, Shuâang Sun, Haoliang Ye, Ronald X Xu, Mingzhai Sun
> â Equal contribution. â Corresponding authors.All authors are with the Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, and also with the School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, (H. Ye), {xux, work was supported by the Gusu Leading Talent Entrepreneurship and Innovation Program (Grant No. ZL2024349).
## Abstract
Long-horizon precision manipulation in laboratory automation, such as pipette tip attachment and liquid transfer, requires policies that respect strict procedural logic while operating in continuous, high-dimensional state spaces. However, existing approaches struggle with reward sparsity, multi-stage structural constraints, and noisy or imperfect demonstrations, leading to inefficient exploration and unstable convergence. We propose a Keyframe-Guided Reward Generation Framework that automatically extracts kinematics-aware keyframes from demonstrations, generates stage-wise targets via a diffusion-based predictor in latent space, and constructs a geometric progress-based reward to guide online reinforcement learning. The framework integrates multi-view visual encoding, latent similarity-based progress tracking, and human-in-the-loop reinforcement fine-tuning on a Vision-Language-Action backbone to align policy optimization with the intrinsic stepwise logic of biological protocols. Across four real-world laboratory tasks, including high-precision pipette attachment and dynamic liquid transfer, our method achieves an average success rate of 82% after 40â60 minutes of online fine-tuning. Compared with HG-DAgger (42%) and Hil-ConRFT (47%), our approach demonstrates the effectiveness of structured keyframe-guided rewards in overcoming exploration bottlenecks and providing a scalable solution for high-precision, long-horizon robotic laboratory automation.
## I INTRODUCTION
*[Error downloading image: 2603.00719v1/fig/Illustration_motivation.png]*
Figure 1: Illustration of our Keyframe-guided Online RL framework. Given an initial observation, the framework generates a sequence of multi-view keyframes (e.g., Position $\rightarrow$ Aspirate $\rightarrow$ Lift) to represent the target task. These keyframes act as intermediate goals to provide stage-wise rewards via state-similarity calculation. Compared to state-of-the-art baselines, our approach significantly accelerates convergence and achieves a superior average success rate of 82%, outperforming the baseline by 47%.
For decades, researchers have pursued intelligent laboratory automation to enable robots to autonomously perform delicate operations such as pipetting and sample loading [25, 1]. Reliable execution of these procedures is essential for long-term biological experiments and for transitioning laboratory systems from rigid preprogrammed pipelines to flexible autonomous intelligence. Despite recent advances in robotic reinforcement learning (RL) and imitation learning (IL), achieving robust, high-precision manipulation under strict stepwise procedural constraints remains challenging.
A central difficulty arises from the long-horizon and multi-stage nature of laboratory tasks. For example, a complete pipetting procedure involves tightly coupled stepsâalignment, insertion, aspiration, transfer, and resetâeach governed by strict physical and logical constraints. However, standard RL formulations typically provide only sparse terminal rewards upon task completion. This creates a prolonged feedback vacuum across hundreds of intermediate transitions, resulting in inefficient exploration, unstable convergence in high-dimensional continuous action spaces, and insufficient guidance for precision-critical operations.
To mitigate these issues, prior work has explored imitation learning and human-in-the-loop reinforcement learning. HG-DAgger relies on behavior cloning from expert demonstrations [19], while Hil-SERL and Hil-ConRFT incorporate human intervention to improve safety and sample efficiency [23, 6, 22]. Nevertheless, behavior cloning depends heavily on high-quality demonstrations and is sensitive to suboptimal data. Human-in-the-loop approaches improve safety but still rely on sparse or manually designed reward signals and do not explicitly exploit the inherent stage-wise structure of laboratory procedures. Furthermore, single-view perception introduces depth ambiguity and occlusion, increasing uncertainty in state estimation and reward computation.
This paper addresses the following question: How can we leverage the intrinsic stepwise structure of biological protocols to alleviate reward sparsity and improve learning efficiency without requiring carefully engineered rewards or expert-quality demonstrations?
Inspired by the multi-stage nature of laboratory procedures, we propose a Keyframe-Guided Reward Generation Framework for long-horizon precision manipulation. The core idea is to automatically extract structurally meaningful keyframes from demonstrations using kinematics-aware latent dynamics analysis, and to transform these keyframes into stage-wise supervision signals for reinforcement learning.
Specifically, we introduce a coarse-to-fine keyframe extraction mechanism that detects critical operational nodes (e.g., positioning $\rightarrow$ aspiration $\rightarrow$ lifting) from non-expert demonstrations by analyzing latent feature dynamics. We further employ multi-view visual encoding and latent similarity metrics to construct a compact state representation, from which a geometric progress-based reward is generated to provide continuous intermediate guidance. This structured reward design aligns policy optimization with the procedural logic of laboratory workflows.
We validate the proposed framework on four real-world laboratory manipulation tasks involving long-horizon operations and sub-millimeter precision. Experimental results demonstrate substantial improvements in convergence speed and task success rate compared to state-of-the-art hybrid learning baselines. These findings suggest that structurally grounded keyframe rewards offer an effective and scalable solution for deploying reinforcement learning in precision laboratory automation.
The main contributions of this work are:
- A kinematics-aware keyframe extraction mechanism that automatically identifies structurally critical states from non-expert demonstrations.
- A latent similarity-based stage-wise reward formulation that transforms extracted keyframes into progress-aware dense rewards for long-horizon RL.
- Integration of multi-view perception, keyframe-guided rewards, and human-in-the-loop fine-tuning on a Vision-Language-Action backbone for real-world laboratory manipulation.
- Extensive real-world validation across four laboratory automation tasks demonstrating significant improvements in convergence speed and success rate.
## II RELATED WORK
### II-A Laboratory Automation and Robotic Manipulation
Laboratory automation has become a cornerstone of modern life sciences and drug discovery, enabling high-throughput and high-precision experimental workflows [26, 32]. Recent systems have evolved from task-specific liquid-handling platforms [33] to modular cell culture systems and autonomous robots for synthetic chemistry [12, 7, 21]. Despite these advances, most platforms remain highly specialized and rely on predefined procedural scripts, limiting their adaptability to diverse and evolving laboratory requirements [35].
Autonomous execution of complex laboratory protocols requires reliable coordination across multiple tightly coupled stages, often under sub-millimeter precision constraints [40, 29, 28]. While deep reinforcement learning (DRL) provides a promising framework for adaptive control, laboratory tasks are inherently long-horizon and multi-stage, making naive RL formulations vulnerable to sparse rewards and inefficient exploration in high-dimensional state spaces. Existing approaches typically do not explicitly leverage the intrinsic procedural structure of laboratory workflows when designing learning objectives.
### II-B VLA Models and Reinforcement Fine-Tuning
Vision-Language-Action (VLA) models have recently emerged as a scalable paradigm for generalizable robotic manipulation [38, 3, 20, 2, 16]. To adapt large-scale models to specific tasks, recent work integrates reinforcement learning and fine-tuning strategies, including offline RL [14, 39] and online RL refinement [11, 36, 17]. These approaches demonstrate improved performance in multi-step manipulation scenarios.
However, the effectiveness of RL-based fine-tuning critically depends on reward design, particularly in sparse-reward environments [15, 13]. Physics-informed reward shaping [10] and goal-conditioned RL [13] have been proposed to guide exploration, yet they typically require manually engineered reward terms or predefined subgoals. In laboratory automation settings, where procedural steps follow strict physical and logical constraints, designing such reward functions becomes increasingly challenging and task-specific [8]. Recent advancements in large models offer a promising alternative for automating reward engineering, allowing for the dynamic generation of task-specific rewards without requiring manual intervention or predefined subgoals [37, 34, 24, 27, 4].
In contrast to prior reward engineering approaches, we propose to automatically extract structurally meaningful keyframes from demonstrations and transform them into stage-wise progress-aware rewards. This allows reinforcement learning to align with the intrinsic stepwise structure of laboratory protocols without manual reward specification.
## III METHOD
*[Error downloading image: 2603.00719v1/fig/Overview.png]*
Figure 2: Overview of our Keyframe-guided RL framework. First, the system extracts keyframes by filtering feature dynamics from demonstrations. A diffusion-based generator is then trained to predict these keyframe sequences from initial observations. During online RL, the reward module calculates latent similarity to provide stage-wise guidance. These rewards drive the Hil-ConRFT update for policy learning.
This paper presents a keyframe-guided online reinforcement learning (RL) framework designed for laboratory automation. By integrating online interaction with multi-stage reward signals, the proposed system accelerates policy convergence and enables high-precision manipulation. As illustrated in Fig. 2, the framework consists of three primary modules: Latent Dynamics-Based Keyframe Extraction, Generative Diffusion-Based Reward Design, and Human-in-the-Loop (HiL) Policy Optimization.
### III-A Latent Dynamics-Based Keyframe Extraction
Keyframes are extracted from expert demonstration sequences. Given multi-view observations $\mathbf{o}_{t}=\{\mathbf{o}_{t}^{\text{front}},\mathbf{o}_{t}^{\text{wrist}}\}$ , we use a discrete vector-quantized encoder $E$ pretrained on offline demonstrations to produce compact latents [9]. We define a composite feature
$$
\mathbf{f}_{t}=\text{concat}\!\left(E(\mathbf{o}_{t}^{\text{front}}),\ \lambda E(\mathbf{o}_{t}^{\text{wrist}})\right), \tag{1}
$$
with view weight $\lambda=1.5$ .
We apply 1D Gaussian smoothing along the temporal axis to obtain $\tilde{\mathbf{f}}_{t}$ , and compute latent velocity and acceleration as:
$$
v_{t}=\|\tilde{\mathbf{f}}_{t}-\tilde{\mathbf{f}}_{t-1}\|_{2},\qquad a_{t}=v_{t}-v_{t-1}. \tag{2}
$$
Stage 1: Turning-Point Mining.
Turning points are identified by detecting peaks in both $v_{t}$ and $|a_{t}|$ ,which results in a candidate set $\mathcal{K}$ of turning points.
Stage 2: Fixed-Budget Selection.
From the candidate set $\mathcal{K}$ , a fixed number of keyframes ( $K=6$ ) are selected based on the transition score $s(k)=\max{v_{k},|a_{k}|}$ . Each selected keyframe produces view-specific latents:
$$
\bar{\mathbf{z}}_{h}^{v}=E\!\left(\mathbf{o}_{k_{h}}^{v}\right). \tag{3}
$$
### III-B Generative Diffusion-Based Online Reward System
During online execution, a diffusion-based model [4] is used to generate a stage-wise target sequence:
$$
\{\hat{\mathbf{z}}_{h}^{v}\}_{h=1}^{H}, \tag{4}
$$
conditioned on the task instruction and initial observation. These generated targets align with the task stages, providing a generalized reference for the reward module.
#### III-B 1 Anchor-Centric Sequence Construction
To bridge the gap between extracted keyframes and the required training sequence length ( $H=8$ ), we adopt an anchor-centric sampling strategy. The detected keyframes $\{k_{h}\}_{h=1}^{K}$ serve as fixed temporal anchors. The remaining slots are filled by uniformly sampling from the non-keyframe pool $\mathcal{P}=\{0,\dots,T-1\}\setminus\{k_{h}\}_{h=1}^{K}$ , ensuring the final task state is included:
$$
\begin{split}S=&\{k_{h}\}_{h=1}^{K}\cup\{\text{uniformly sampled}\\
&\text{non-keyframe indices from }\mathcal{P}\}.\end{split} \tag{5}
$$
This subset $\{\mathbf{o}_{t}\}_{t\in S}$ prioritizes key interaction nodes for reward calculation.
#### III-B 2 Multi-View Fusion Similarity
The alignment between the current state and the target sequence is evaluated via cosine similarity in the latent space:
$$
S_{t}(h)=\sum_{v\in\mathcal{V}}w_{v}\cdot\frac{\mathbf{z}_{t}^{v}\cdot\hat{\mathbf{z}}_{h}^{v}}{\|\mathbf{z}_{t}^{v}\|_{2}\,\|\hat{\mathbf{z}}_{h}^{v}\|_{2}}, \tag{6}
$$
where $\mathcal{V}=\{\text{front},\text{wrist}\}$ and $w_{v}$ are view weights (detailed in Appendix V-A).
#### III-B 3 Progress Tracking and Reward Calculation
We maintain a progress state $M_{t}\in\{1,\dots,H\}$ and advance it when $S_{t}(M_{t})\geq\theta$ :
$$
M_{t+1}=\begin{cases}M_{t}+1,&\text{if }S_{t}(M_{t})\geq\theta\text{ and }M_{t}\leq H\\
M_{t},&\text{otherwise}.\end{cases} \tag{7}
$$
The guided reward is formulated as
$$
R_{\text{total}}=\epsilon_{\text{step}}+\mathbb{I}(M_{t+1}>M_{t})\cdot G_{\text{stage}}(M_{t+1}), \tag{8}
$$
where $\epsilon_{\text{step}}=-0.05$ is a small step penalty. We use a geometric progression
$$
G_{\text{stage}}(n)=r_{1}q^{\,n-1},\quad n=1,\dots,H, \tag{9}
$$
with a scaling factor $q=1.3$ , and a fixed total reward budget
$$
\sum_{n=1}^{H}G_{\text{stage}}(n)=10. \tag{10}
$$
### III-C Online Policy Evolution and Q-Learning Update
Our policy backbone is based on Octo [31], a pre-trained Vision-Language-Action (VLA) model. The action head uses a diffusion policy to generate control signals through iterative denoising. For real-time online learning, we apply a consistency policy [5, 30, 18] to distill the denoising process into a single-step inference. Stochastic exploration is maintained via noise sampling:
$$
w\sim\mathcal{N}(0,\sigma^{2}\mathbf{I}), \tag{11}
$$
where $\sigma$ controls the initial noise scale. Different samples of $w$ yield diverse actions, facilitating better exploration.
The policy is updated using a mixed-buffer strategy, sampling equally from the online replay buffer $\mathcal{D}_{\text{online}}$ and the demonstration buffer $\mathcal{D}_{\text{demo}}$ . We integrate a Human-in-the-Loop (HiL) mechanism where operators provide corrective interventions during unsafe or suboptimal behaviors; these corrections are stored in $\mathcal{D}_{\text{demo}}$ to guide the update. Our optimization follows the HiL-ConRFT framework [6].
#### III-C 1 Critic Network Update
The critic is optimized via Bellman regression with a target ensemble to reduce overestimation:
$$
a^{\prime}\sim\pi(\cdot|s^{\prime}),y=r+\gamma\cdot\min_{i}Q_{\bar{\theta}_{i}}(s^{\prime},a^{\prime}). \tag{12}
$$
where $\gamma$ is the discount factor The critic loss is
$$
\mathcal{L}_{\text{critic}}=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\left[\left(Q_{\theta}(s,a)-y\right)^{2}\right], \tag{13}
$$
where $\mathcal{D}=\mathcal{D}_{\text{demo}}\cup\mathcal{D}_{\text{online}}$ .
#### III-C 2 Actor Network Update
The actor is optimized by a weighted combination of behavior cloning and Q-maximization:
$$
\mathcal{L}_{\text{actor}}=\lambda_{\text{bc}}\mathcal{L}_{\text{BC}}+\lambda_{\text{q}}\mathcal{L}_{\text{Q}}, \tag{14}
$$
where $\lambda_{\text{bc}}=0.5$ and $\lambda_{\text{q}}=1$ . The behavior cloning loss is
$$
\mathcal{L}_{\text{BC}}=\mathbb{E}_{(s,a)\sim\mathcal{D}}\!\left[\|a-\pi_{\phi}(s)\|_{2}^{2}\right], \tag{15}
$$
and the Q-loss maximizes the expected critic value
$$
\mathcal{L}_{\text{Q}}=-\mathbb{E}_{s\sim\mathcal{D}}\!\left[Q_{\theta}(s,\pi_{\phi}(s))\right]. \tag{16}
$$ *[Error downloading image: 2603.00719v1/fig/Task_illustrations_and_failure_modes.png]*
Figure 3: Biological Laboratory task illustrations and common failure modes. (A) Petri Dish De-lidding. This task involves using a gripper to lift and relocate the lid of a petri dish to the side. Common failures include premature release, or colliding with the dish base due to insufficient height. (B) Centrifuge Tube Loading. This task involves picking a tube from a rack and placing it into a centrifuge slot. Common failures include missing the tube, dropping it early, or colliding with the centrifuge rim. (C) Precision Liquid Transfer. This task involves absorbing liquid from a tube and spraying it into a petri dish. Common failures include misalignment with the tube opening, colliding with the tube wall, or spraying liquid before reaching the target. (D) Pipette Tip Attachment. This task involves inserting the pipette into a tip box to secure a new tip. Common failures include misalignment, insertion errors, or the tip slipping off during raising.
## IV EXPERIMENTS
### IV-A Experimental Platform and Task Parameter Settings
To evaluate the effectiveness of the proposed keyframe-guided reward framework in real-world laboratory automation, we design four representative manipulation tasks (Fig. 2, Fig. 3). These tasks span object relocation (centrifuge tube loading and petri dish de-lidding), high-precision contact-rich manipulation (pipette tip attachment), and dynamic liquid handling (precision liquid transfer). All training and inference were conducted on a workstation equipped with an NVIDIA RTX 5880 Ada GPU.
Observation and Action Spaces. All tasks are operated at a control frequency of 10 Hz. The observation space $\mathbf{s}_{t}$ consists of dual-view RGB images (a $128\times 128$ wrist camera and a $256\times 256$ side camera) and proprioceptive states, including end-effector (EE) poses and gripper/pipette status. The action space $\mathbf{a}_{t}$ is defined as a 6-DoF incremental EE pose in Euclidean space, augmented by task-specific dimensions: a 1D binary gripper state for Tasks 1 & 2, and coordinated dexterous hand/pipette triggers for Tasks 3 & 4.
Task Specifications and Randomization. We evaluate our framework on four representative laboratory automation tasks using a Jaka Minicobo robot: Tasks 1 & 2 (Object Relocation): These involve Petri Dish De-lidding and Centrifuge Tube Loading. A parallel gripper is used to execute multi-stage pick-and-place workflows. Task 3 (Pipette Tip Attachment): A high-precision, contact-rich task requiring sub-millimeter accuracy, utilizing a Linkerhand O7 dexterous hand to interface with pipette tips. Task 4 (Precision Liquid Transfer): A complex dynamic task requiring synchronized control of the dexterous hand and pipette triggers for aspiration and dispensing. To ensure policy robustness, initial object positions are randomized within a $3\times 3\text{ cm}^{2}$ ( $2\times 2\text{ cm}^{2}$ for Task 3) horizontal area across all episodes.
### IV-B Comparison with Baseline Methods
Selection of baseline methods. To comprehensively evaluate the effectiveness of the proposed method in real-world laboratory scenarios, we compare our method (Ours) with mainstream policy fine-tuning and reinforcement learning baselines across four representative manipulation tasks shown in Fig. 3. The selected baseline methods primarily include: HG-DAgger, a classic method that achieves policy fine-tuning through supervised learning combined with manual online corrections; Hil-ConRFT, an advanced learning framework combining human interventions with reinforcement fine-tuning; and Hil-SERL , a reinforcement learning method that integrates human interventions and utilizes offline data to accelerate online exploration.
Experimental Settings and Fairness. To ensure a rigorous and unbiased comparison, we unified the underlying configurations across all evaluated methods, despite the inherent differences between laboratory manipulation scenarios and the original validation environments of certain baselines. For methods involving human-in-the-loop interventions, the frequency and criteria for human takeover and corrective data were strictly standardized during training. Regarding the reward structure, all baseline methods were implemented with a penalty-based dense reward: a step-wise penalty of $-0.05$ was assigned for each incomplete frame, followed by a terminal reward of $+10$ upon task completion. This configuration was specifically designed to align the total potential reward magnitude with our proposed framework. Furthermore, the replay buffer initialization and mixed sampling mechanisms were synchronized across all baselines to eliminate performance discrepancies stemming from engineering-level discrepancies.
Comprehensive performance comparison. Table I details the final evaluation results of each method across the four tasks, and Fig. 5 displays the corresponding online learning curves. Experimental results demonstrate that after approximately 40-60 minutes of real-world online fine-tuning, our keyframe-guided method achieves an overwhelming advantage, reaching an average success rate of 82% across the four tasks. In contrast, the supervised fine-tuning method HG-DAgger, which relies heavily on data quality, achieves an average success rate of only 42%; the reinforcement fine-tuning method Hil-ConRFT, lacking intermediate state guidance, yields only 47%; and the reinforcement learning baseline Hil-SERL , relying on conventional dense or sparse rewards, falls into a severe exploration dilemma when facing these high-precision, long-horizon laboratory tasks, with a success rate of nearly 0%. *[Error downloading image: 2603.00719v1/fig/Impact_of_keyframe.png]*
Figure 4: Impact of keyframe introduction. Precision liquid transfer requires a strict sequence from insertion to dispensing. Unlike our kinematics-heuristic extraction, the baseline uses uniform sampling and misses critical bottleneck states like insertion and aspiration. This leads to erroneous spatial guidance and execution failure. In contrast, our method anchors these intermediate states, ensuring correct step-wise logic.
TABLE I: Performance Comparison of Different Methods on Robotic Manipulation Tasks
| Task | Ours | HG- DAgger | HIL- ConRFT | HIL- SERL |
| --- | --- | --- | --- | --- |
| Petri Dish De-lidding | 1.00 | 0.70 | 0.50 | 0.00 |
| Centrifuge Tube Loading | 0.50 | 0.50 | 0.40 | 0.00 |
| Precision Liquid Transfer | 1.00 | 0.00 | 0.10 | 0.00 |
| Pipette Tip Attachment | 0.80 | 0.50 | 0.90 | 0.00 |
| Mean | 0.82 | 0.42 | 0.47 | 0.00 |
*[Error downloading image: 2603.00719v1/fig/Learning_curves.png]*
Figure 5: Learning curves during online training. This figure presents the success rates, intervention rates, and episode lengths for HIL-ConRFT, HG-DAgger, HIL-SERL, and our method across four representative laboratory tasks. The metrics are displayed as a running average over 10 episodes against training time in minutes.
### IV-C Results Analysis
In this section, we provide an in-depth analysis of the performance of the proposed method (Ours) and baseline models in laboratory automation tasks, combining the success rate data in Table I, the key step visualization in Fig. 4, and the online learning curves in Fig. 5.
Keyframe-guided dense rewards significantly improve multi-task success rates and generalization capabilities. As shown in Table I, the proposed keyframe-guided reinforcement learning framework achieves the best performance across four laboratory manipulation tasks, with an average success rate of 82%. This is a substantial improvement compared to the supervised fine-tuning method HG-DAgger (42%), which relies on manual corrections, and the reinforcement fine-tuning method Hil-ConRFT (47%). Meanwhile, the reinforcement learning baseline Hil-SERL fails in these tasks, achieving a success rate of nearly 0%.
The proposed method effectively overcomes structural defects in long-horizon and high-precision contact tasks. Taking the âPrecision Liquid Transferâ task as an example, the success rates of HG-DAgger and Hil-SERL are both 0%, and Hil-ConRFT is only 10%, whereas our method achieves a 100% success rate. By analyzing the step-wise visualization in Fig. 4, the root cause becomes evident: the pipetting task involves strict physical logic constraints, including sequential steps such as inserting into the centrifuge tube to aspirate liquid, moving out of the tube, and moving above the petri dish to spray. Due to the lack of intermediate state guidance, baseline models often ignore intermediate key steps, resulting in severe collisions between the pipette tip and the centrifuge tube wall. In contrast, our method successfully captures these crucial intermediate steps by automatically extracting and generating a sequence of keyframes. This mechanism not only corrects the âskipping stepsâ defect of the baseline methods but also effectively ensures the safety and compliance of precision instrument operations.
Extremely high sample efficiency and rapidly decreasing human intervention rate. Observing the training curves in Fig. 5, within the 40-60 minute training window, the success rate curve of Ours exhibits the steepest upward trend across all tasks. At the same time, while all methods require high human intervention in the early stages, the intervention rate of Ours drops rapidly to near zero in a short period. This demonstrates that the agent can swiftly absorb the intermediate guidance signals provided by the multi-view keyframes to establish correct state-action mappings.
TABLE II: Quantitative Analysis of Multi-Level Keyframe Identification Accuracy
| Task | Step-wise Coverage (All) $\uparrow$ | Core Milestone Recall (Core) $\uparrow$ | | |
| --- | --- | --- | --- | --- |
| w/o KE | w/ KE | w/o KE | w/ KE | |
| Petri Dish De-lidding | 0.817 | 0.917 | 0.550 | 0.900 |
| Centrifuge Tube Loading | 0.667 | 0.933 | 0.350 | 0.850 |
| Precision Liquid Transfer | 0.550 | 0.900 | 0.400 | 1.000 |
| Pipette Tip Attachment | 0.933 | 0.983 | 0.800 | 1.000 |
| Mean | 0.742 | 0.933 | 0.525 | 0.938 |
### IV-D Ablation Studies
To systematically evaluate the contribution of our proposed modules, we conduct ablation studies focusing on the keyframe extraction mechanism and the keyframe-guided dense reward.
Effectiveness of Keyframe Extraction Mechanism To isolate the effect of our kinematics-heuristic keyframe extraction (KE), we conduct an ablation study by replacing this module with a standard time-based uniform sampling strategy (denoted as w/o KE). As reported in Table II, we evaluate the performance using two metrics: Step-wise Coverage (All) for overall sequence alignment and Core Milestone Recall (Core) for the identification of critical operational nodes.
Quantitatively, our KE method significantly outperforms the baseline, achieving an average Step-wise Coverage of $0.933$ compared to $0.742$ . More importantly, the advantage is even more pronounced in the Core Milestone Recall, where our method maintains a high recall of $0.938$ , while the baseline drops to $0.525$ . This gap is most evident in the âPrecision Liquid Transferâ task, where KE achieves a perfect recall ( $1.000$ ) of core milestones, whereas uniform sampling frequently misses these transient but essential states ( $0.400$ ). Table III provides the semantic grounding for these core milestones, detailing the specific physical transitionsâsuch as tip-to-liquid contact and aspirationâthat our mechanism successfully prioritizes. These results validate that capturing kinematics-aware keyframes is superior to uniform sampling for preserving the underlying logic of complex laboratory procedures.
Qualitatively, as visually compared in Fig. 4, the uniform sampling approach fails to capture the physical bottleneck states characterized by sudden kinematic changes. Critical intermediate steps, such as inserting the pipette tip into the centrifuge tube and aspirating the liquid, are completely omitted. Without these bottleneck states anchoring the trajectory, the generated sequence provides erroneous spatial guidance. This structural defect directly leads to critical failure modes during execution, such as colliding with the tube wall or misaligning the tip (as illustrated in Fig. 3 C). This ablation robustly demonstrates that our keyframe extraction mechanism is indispensable for capturing the underlying physical logic of long-horizon tasks.
Impact of Keyframe-Guided Reward. To isolate the contribution of our keyframe-guided reward generation, we compare our full pipeline with HIL-ConRFT. It is crucial to note that HIL-ConRFT shares the exact same VLA backbone (Octo), consistency policy action head, and human-in-the-loop training paradigm as our method. The fundamental difference is that HIL-ConRFT relies solely on standard sparse terminal rewards, completely ablating our stage-wise keyframe guidance. As quantitatively shown in Table I and the learning curves in Fig. 5, removing the keyframe-guided dense reward causes a severe performance drop. Most notably, in the complex âPrecision Liquid Transferâ task, the success rate plummets from 100% to a mere 10%. Furthermore, without the intermediate state guidance provided by our generated keyframes, the baseline policy frequently falls into local optima, exhibiting erroneous behaviors such as premature spraying or insufficient lifting height (Fig. 3).
## V CONCLUSIONS
This paper presented a keyframe-guided reinforcement learning framework for long-horizon precision manipulation in laboratory automation. By leveraging kinematics-aware keyframe extraction and diffusion-based stage target generation, the proposed method transforms demonstrations into structured, progress-aware reward signals without requiring manually engineered reward functions or expert-quality annotations. The resulting stage-wise reward mitigates temporal credit assignment challenges and aligns policy optimization with the intrinsic procedural logic of laboratory workflows.
Extensive real-world experiments across four multi-stage tasks demonstrate improved convergence speed, reduced reliance on human intervention, and an average success rate of 82% under online fine-tuning. The framework effectively addresses common failure modes in sequential precision tasks, particularly in high-accuracy liquid handling scenarios.
While the current implementation relies on RGB-based latent similarity, which can be sensitive to visual artifacts such as reflections or transparency, future work will explore multimodal representations incorporating tactile sensing, depth perception, and 3D geometric features. We also aim to integrate automated environment reset mechanisms to further reduce human involvement and enhance scalability. Overall, this work provides a structured and scalable pathway for deploying reinforcement learning in real-world laboratory robotics.
## APPENDIX
### V-A Task-Specific Configurations
To balance global coordination and local precision, task-specific view weights $w_{v}$ and keyframe sequences are defined as follows:
TABLE III: Task-Specific View Weights and Keyframe Semantic Sequences
| Task | $w_{\text{wrist}}$ | $w_{\text{front}}$ | $\theta$ | Keyframe Sequence |
| --- | --- | --- | --- | --- |
| Petri Dish De-lidding | 0.5 | 0.5 | 0.7 | Reach $\rightarrow$ Grasp $\rightarrow$ Lift $\rightarrow$ Relocate |
| Centrifuge Tube Loading | 0.6 | 0.4 | 0.6 | Approach $\rightarrow$ Pick $\rightarrow$ Align $\rightarrow$ Insert |
| Precision Liquid Transfer | 0.7 | 0.3 | 0.6 | Insert $\rightarrow$ Aspirate $\rightarrow$ Lift $\rightarrow$ Spray |
| Pipette Tip Attachment | 0.7 | 0.3 | 0.8 | Approach $\rightarrow$ Insert $\rightarrow$ Lift |
Note: Keyframes highlighted in red correspond to the core keyframes used for high-level semantic evaluation in Table II.
### V-B Algorithm
0: $\mathcal{D}_{demo},\mathcal{D}_{online}$ : Demonstration and replay buffers;
$\pi_{\phi}$ : VLA policy with parameters $\phi$ ;
$Q_{\theta}$ : Critic network with parameters $\theta$ ;
$E$ : Visual encoder; $G$ : Keyframe generator;
$\theta_{sim}$ : Similarity threshold; $H$ : Total stages;
$\lambda_{bc},\lambda_{q}$ : Balancing weights for BC and Q-learning.
1: Initialize stage index $M\leftarrow 0$ and $success\leftarrow\text{False}$ .
2: for each episode do
3: Obtain initial state $s_{0}$ and latent targets $\{\hat{z}_{h}^{v}\}\leftarrow E(G(s_{0}))$ .
4: while NOT $success$ and NOT $human\_exit$ do
5: Sample action $a_{t}\sim\pi_{\phi}(\cdot|s_{t})$ .
6: if human intervention triggered then
7: Execute expert $a_{t}^{*}$ , store $(s_{t},a_{t}^{*})$ to $\mathcal{D}_{demo}$ .
8: else
9: Execute $a_{t}$ , observe $s_{t+1}$ .
10: end if
11: // Latent Progress and Reward Calculation
12: Extract $z_{t+1}^{v}=E(s_{t+1}^{v})$ , compute similarity $S_{t}(M)$ via Eq. (6).
13: if $S_{t}(M)\geq\theta_{sim}$ and $M<H$ then
14: $r_{guided}\leftarrow G_{stage}(M)$ , $M\leftarrow M+1$ .
15: else
16: $r_{guided}\leftarrow 0$ .
17: end if
18: $r_{t}=\epsilon_{step}+r_{guided}$ , store $(s_{t},a_{t},r_{t},s_{t+1})$ to $\mathcal{D}_{online}$ .
19: // Policy ( $\pi$ ) and Critic ( $Q$ ) Update
20: Sample batch $\mathcal{B}\sim\{\mathcal{D}_{demo}\cup\mathcal{D}_{online}\}$ .
21: $y=r_{t}+\gamma\min_{i=1,2}Q_{\bar{\theta}_{i}}(s_{t+1},\pi_{\phi}(s_{t+1}))$ .
22: Update $Q_{\theta}$ by minimizing Bellman error $\mathcal{L}_{critic}(\mathcal{B},y)$ .
23: Update $\pi_{\phi}$ by minimizing $\lambda_{bc}\mathcal{L}_{BC}(\mathcal{B})+\lambda_{q}\mathcal{L}_{Q}(\mathcal{B})$ .
24: $s_{t}\leftarrow s_{t+1}$ , $t\leftarrow t+1$ .
25: if $M=H+1$ or $human\_verified$ then $success\leftarrow\text{True}$ .
26: end while
27: end for
Algorithm 1 Keyframe-Guided Online Policy Evolution
## References
- [1] K. Angers, K. Darvish, N. Yoshikawa, S. Okhovatian, D. Bannerman, I. Yakavets, F. Shkurti, A. Aspuru-Guzik, and M. Radisic (2025) RoboCulture: a robotics platform for automated biological experimentation. arXiv preprint arXiv:2505.14941. Cited by: §I.
- [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) $\pi_{0}$ : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: § II-B.
- [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. (2022) Rt-1: robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817. Cited by: § II-B.
- [4] Y. Chen, H. Li, Z. Jiang, H. Wen, and D. Zhao (2025) TeViR: text-to-video reward with diffusion models for efficient reinforcement learning. arXiv preprint arXiv:2505.19769v2. Note: Submitted to IEEE Transactions on Robotics Cited by: § II-B, § III-B.
- [5] Y. Chen, H. Li, and D. Zhao (2023) Boosting continuous control with consistency policy. arXiv preprint arXiv:2310.06343. Cited by: § III-C.
- [6] Y. Chen, S. Tian, S. Liu, Y. Zhou, H. Li, and D. Zhao (2025) Conrft: a reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450. Cited by: §I, § III-C.
- [7] T. Dai, S. Vijayakrishnan, F. T. SzczypiĆski, J. Ayme, E. Simaei, T. Fellowes, R. Clowes, L. Kotopanov, C. E. Shields, Z. Zhou, et al. (2024) Autonomous mobile robots for exploratory synthetic chemistry. Nature 635 (8040), pp. 890â897. Cited by: § II-A.
- [8] Z. Deng, Y. Dong, and X. Liu (2025) Reward shaping in reinforcement learning for robotic hand manipulation. Neurocomputing 638, pp. 130204. Cited by: § II-B.
- [9] P. Esser, R. Rombach, and B. Ommer (2021) Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873â12883. Cited by: § III-A.
- [10] R. Fareh, T. Siddique, K. Choutri, and D. V. Dylov (2025) Physics-informed reward shaped reinforcement learning control of a robot manipulator. Ain Shams Engineering Journal 16 (10), pp. 103595. Cited by: § II-B.
- [11] Y. Guo, J. Zhang, X. Chen, X. Ji, Y. Wang, Y. Hu, and J. Chen (2025) Improving vision-language-action model with online reinforcement learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15665â15672. Cited by: § II-B.
- [12] J. Hamm, S. Lim, J. Park, J. Kang, I. Lee, Y. Lee, J. Kang, Y. Jo, J. Lee, S. Lee, et al. (2024) A modular robotic platform for biological research: cell culture automation and remote experimentation. Advanced Intelligent Systems 6 (5), pp. 2300566. Cited by: § II-A.
- [13] X. He and C. Lv (2023) Robotic control in adversarial and sparse reward environments: a robust goal-conditioned reinforcement learning approach. IEEE Transactions on Artificial Intelligence 5 (1), pp. 244â253. Cited by: § II-B.
- [14] D. Huang, Z. Fang, T. Zhang, Y. Li, L. Zhao, and C. Xia (2025) Co-rft: efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning. arXiv preprint arXiv:2508.02219. Cited by: § II-B.
- [15] S. Ibrahim, M. Mostafa, A. Jnadi, H. Salloum, and P. Osinenko (2024) Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEEE Access 12, pp. 175473â175500. Cited by: § II-B.
- [16] P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al. (2025) $\pi^{*}_{0.6}$ : A vla that learns from experience. arXiv preprint arXiv:2511.14759. Cited by: § II-B.
- [17] P. Jin, Q. Wang, G. Sun, Z. Cai, P. He, and Y. You (2025) Dual-actor fine-tuning of vla models: a talk-and-tweak human-in-the-loop approach. arXiv preprint arXiv:2509.13774. Cited by: § II-B.
- [18] T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems 35, pp. 26565â26577. Cited by: § III-C.
- [19] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer (2019) Hg-dagger: interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8077â8083. Cited by: §I.
- [20] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. (2024) Openvla: an open-source vision-language-action model. arXiv preprint arXiv:2406.09246. Cited by: § II-B.
- [21] J. Lu, J. Pan, Y. Mo, and Q. Fang (2024) Automated intelligent platforms for high-throughput chemical synthesis. Artificial Intelligence Chemistry 2 (1), pp. 100057. Cited by: § II-A.
- [22] J. Luo, Z. Hu, C. Xu, Y. L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine (2024) Serl: a software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 16961â16969. Cited by: §I.
- [23] J. Luo, C. Xu, J. Wu, and S. Levine (2025) Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics 10 (105), pp. eads5033. Cited by: §I.
- [24] Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar (2023) Eureka: human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931. Cited by: § II-B.
- [25] A. O. M. Omair, A. M. A. Jabbar, and M. O. Albulushi (2023) Recent advancements in laboratory automation technology and their impact on scientific research and laboratory procedures. International journal of health sciences 7 (S1), pp. 3043â3052. Cited by: §I.
- [26] Y. Qiu, Z. Huang, Z. Wang, H. Liu, Y. Qiao, Y. Hu, S. Sun, H. Peng, R. X. Xu, and M. Sun (2025) Biomars: a multi-agent robotic system for autonomous biological experiments. arXiv preprint arXiv:2507.01485. Cited by: § II-A.
- [27] J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner (2023) Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310.12921. Cited by: § II-B.
- [28] P. Salazar-Villacis and B. Benyahia (2026) The adept framework for assessing autonomous laboratory robotics. Communications Chemistry 9 (1), pp. 99. Cited by: § II-A.
- [29] M. Seifrid, R. Pollice, A. Aguilar-Granda, Z. Morgan Chan, K. Hotta, C. T. Ser, J. Vestfrid, T. C. Wu, and A. Aspuru-Guzik (2022) Autonomous chemical experiments: challenges and perspectives on establishing a self-driving lab. Accounts of Chemical Research 55 (17), pp. 2454â2466. Cited by: § II-A.
- [30] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. Cited by: § III-C.
- [31] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: § III-C.
- [32] K. Thurow (2023) Strategies for automating analytical and bioanalytical laboratories. Analytical and Bioanalytical Chemistry 415 (21), pp. 5057â5066. Cited by: § II-A.
- [33] M. A. Torres-Acosta, G. J. Lye, and D. Dikicioglu (2022) Automated liquid-handling operations for robust, resilient, and efficient bio-based laboratory practices. Biochemical Engineering Journal 188, pp. 108713. Cited by: § II-A.
- [34] Y. Wang, Z. Xian, F. Chen, T. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan (2023) Robogen: towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455. Cited by: § II-B.
- [35] Ă. Wolf (2025) Integration framework for robotics in life science laboratories laboratory automation plug and play. Cited by: § II-A.
- [36] C. Xu, Q. Li, J. Luo, and S. Levine (2024) Rldg: robotic generalist policy distillation via reinforcement learning. arXiv preprint arXiv:2412.09858. Cited by: § II-B.
- [37] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K. Lee, M. G. Arenas, H. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. (2023) Language to rewards for robotic skill synthesis. arXiv preprint arXiv:2306.08647. Cited by: § II-B.
- [38] D. Zhang, J. Sun, C. Hu, X. Wu, Z. Yuan, R. Zhou, F. Shen, and Q. Zhou (2025) Pure vision language action (vla) models: a comprehensive survey. arXiv preprint arXiv:2509.19012. Cited by: § II-B.
- [39] H. Zhang, Z. Zhuang, H. Zhao, P. Ding, H. Lu, and D. Wang (2025) Reinbot: amplifying robot visual-language manipulation with reinforcement learning. arXiv preprint arXiv:2505.07395. Cited by: § II-B.
- [40] H. Zhao, F. Meng, C. Li, J. Huan, X. Wang, and X. Duan (2025) An autonomous intelligent robot for reagent handling in biological laboratories. In 2025 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 1â6. Cited by: § II-A.