Current research · Industrial robot learning
What infrastructure makes a learned robot policy reproducible, diagnosable, and safe enough to test on physical hardware?
I architected and implemented a shared ROS 2 platform that connects multimodal teleoperation, synchronized recording, dataset governance, policy training, offline diagnosis, calibrated simulation, staged evaluation, controlled deployment, and corrective-data capture. The contribution is not a single model: it is the evidence path that lets models change without losing experimental control.
Abstract
Robot learning becomes scientifically useful only when the demonstrations, model, simulator, controller, rollout, and human judgement can be traced to one another. This platform makes those handoffs explicit and treats every success, refusal, and intervention as evidence.
Select a stage to inspect its contract, evidence, and current boundary.
Device choice and controller mode remain independent, while readiness, workspace, discontinuity, and freshness checks fail closed before motion.
Vision, state, targets, gripper commands, timing, annotations, and intervention events are recorded together rather than reconstructed after training.
Technical QC validates streams and timing; read-only playback supports human judgement; accepted membership and splits are frozen and hashed before conversion.
ACT and a compact language-conditioned policy share data membership, held-out observations, sampled-frame exposure, seed, and deployment interface.
Software qualification precedes offline analysis, closed-loop simulation, shadow checks, bounded hardware trials, and broader physical rollout.
Takeovers reuse the same observation–action contract and pass through review, frozen membership, retraining, and re-evaluation rather than entering an informal correction pool.
System design
The platform is organized around stable observation, action, episode, policy-provider, and deployment contracts. Teleoperation devices, controllers, learning methods, robots, and simulators can change behind those interfaces without forking the complete workflow.
Controller mode and input device are orthogonal. VR, SpaceMouse, keyboard, gloves, kinematic leader devices, joint mirroring, and idle are admitted only through compatible control contracts.
Freshness, discontinuity, speed, workspace, action-shape, gripper-range, and controller-readiness checks reject invalid transitions before motion rather than repairing outputs silently.
Vision, robot state, target frames, gripper commands, operator marks, task text, and timing are recorded as one episode so action–observation alignment remains inspectable.
Automated technical QC checks whether data are complete and aligned. A separate semantic review decides whether the intended task actually succeeded.
Episode sidecars, QC, reviews, membership, splits, checkpoints, and evaluation reports are versioned and hashed before they may support a comparison.
Software qualification precedes offline evaluation, then closed-loop simulation, shadow checks, bounded physical trials, and only afterwards broader rollout.
Demonstration and data
A demonstration is not training data by default. It becomes eligible only after the system knows who collected it, under which task contract, whether the streams are sound, what happened semantically, and exactly which frozen split contains it.
Bind the session to a task contract, expected topics, controller topology, operator, recording root, and an explicit dry run.
Run exclusion, network, device, controller, recorder, and freshness checks before the first episode can begin.
Reset the scene, apply a controlled task label, record the demonstration, then explicitly save or discard it.
Rate, gap, timing, decode, topic, parity, and annotation checks do not masquerade as task success. Playback remains read-only during semantic review.
Accepted membership and splits are locked before conversion to the learning framework. Counts and hashes are checked at every handoff.
| Quality check | Question | Failure consequence |
|---|---|---|
| Required streams | Are all contracted observations and actions present? | A missing modality changes the learning problem. |
| Sampling rate | Is each topic inside its modality-specific rate contract? | A slow stream changes the effective control period. |
| Continuity | Are there gaps inside the episode? | A gap can break action–observation causality mid-contact. |
| Sensor timing | Do camera and robot timestamps agree? | Misalignment teaches the policy the wrong temporal relationship. |
| Bimanual parity | Do both arms have matching coverage? | A plausible but one-sided episode may enter a two-arm dataset. |
| Human marks | Are intervention and annotation events parseable? | Human knowledge that automation cannot infer is lost. |
Collection evidence
One controlled simulation-cell session tested the complete acquisition, review, freeze, and conversion path. The purpose was system qualification—not a claim about policy competence.
The initial QC produced 47 pass, 58 warn, and 9 fail verdicts. Investigation traced all nine failures to an upper-rate rule applied incorrectly to an event-driven gripper topic. The episodes were not silently relabeled: the checker contract was corrected and the evidence trail retained.
This is a useful systems result—a quality gate is a testable hypothesis about the data, not an oracle.
Policy learning and diagnosis
The policy program uses short, attributable manipulation skills: one prompt, one grasp, one graded outcome. This makes failure analysis practical and creates measurable seams for later skill composition.
A short ACT policy approaches, aligns, grasps, and retains one deformable component on a physical UR5e cell.
A language instruction selects one of three visually similar objects across nine layouts, adding grounding without lengthening the horizon.
Human takeover segments are recorded with the same observation–action contract and can be reviewed and mixed into later training sets.
| Dimension | ACT | Compact VLA | Interpretation |
|---|---|---|---|
| Parameters | 51.6M trainable | 450.0M total; 99.9M trainable | Most VLA vision-language parameters remained frozen. |
| Matched exposure | 320,000 sampled frames | 320,000 sampled frames | Same corpus, split, seed, held-out observations, and sampled-frame exposure. |
| Training wall clock | 2:02:37 | 2:41:07 | The VLA was chosen for native language conditioning, not lower cost. |
| Warm inference median | 18.5 ms | 142.7 ms | Loopback on one RTX 6000 Ada; not end-to-end control latency. |
| Diagnostic failure | Position-bound: success concentrated at center | Object-selective: all vent-hose trials failed | Opposite failure modes appeared under the same 18-trial matrix. |
| Action streaming variant | Boundary jump | Observed behavior |
|---|---|---|
| Raw streaming | 0.048 rad | Visible stutter |
| Deadband | 0.029 rad | Improved, but discontinuities remained |
| Exponential smoothing | 0.010 rad | Smooth while retaining decisive changes |
| Ramp + crossfade | 0.003 rad | Best smoothness metric, but hesitation at decision points |
Real → simulation → real
Simulation is used as an experimental instrument: reproduce a physical failure, isolate a cause cheaply, and require quantified parity before interpreting the result as relevant to the real system.
Robot descriptions, tool transforms, camera frames, scene assets, and controller interfaces are checked independently before policy testing.
Legacy, namespaced, larger-arm, and independent bimanual configurations are selected through configuration rather than code forks.
Recording, topic ownership, conversion, and training views are checked so a simulated experiment uses the same contracts as its physical counterpart.
A 42 mm grasp-transform correction was tested as a falsifiable explanation for a physical contact failure; closed-loop simulation then produced 96–105 mm retained lifts above an 80 mm criterion.
Kinematic and scene parity do not establish visual, contact, latency, or material parity. Return-to-real trials remain a separate evidence tier.
Bimanual and compliant extension
The same infrastructure was extended to compliance-controlled bimanual collection and rollout. A leader–follower arrangement lets one arm specify task motion while the other yields to shared-object tension.
F = K(xd − x) − Dẋ
Passive Cartesian compliance turns the arms into tunable mechanical impedances rather than independent position sources. This bounds internal interaction more naturally during contact-rich handling.
I built the platform and supervised an intern who used it to explore admittance-based data collection and rollout on a bimanual setup. The extension is presented as supervised collaborative work, not as my sole implementation.
Repeated-trial compliance comparisons and force/interaction traces remain the next evidence required.
Evidence hierarchy
A higher evidence tier can overturn a lower one. The reverse is not allowed.
Strongest · scarce
No robot motion
Not a transfer claim
Code evidence
Reproducibility and open questions
The public account keeps current employer IP abstract while exposing the experimental logic, negative results, comparison controls, and maturity boundaries.
| Status | Claim | Evidence | Boundary |
|---|---|---|---|
| Established | One reusable teleoperation-to-deployment architecture | Shared message/service contracts, recorder, QC, manifests and provider interface | Task-specific implementation remains private |
| Measured | Collection, conversion and simulator qualification | Throughput run, frozen splits, parity and clean-build tests | Mostly one controlled cell configuration |
| Diagnosed | Distinct grounding failures across two policy families | Matched held-out observations and exploratory trial matrices | Single-seed and task-bounded |
| Ongoing | Automated comparative rollout evaluation | Manual semantic evaluation exists today | Not yet end-to-end automated |
| Open | Robust real-world continual improvement | Intervention capture and corrective-data path implemented | Long-term retention and transfer not established |
The portfolio repository contains dependency-light evaluators for classification exports and synchronized trajectories, plus scripts that regenerate every reconstructed evidence graphic.