Dual-Arm PiPER Learning Stack
Built a full real-robot learning stack for a bimanual PiPER platform; ACT stacks five cups autonomously.
Summary
From CAN frames to closed-loop policies on a bimanual robot.
I built the complete real-robot learning pipeline for a bimanual AgileX PiPER platform: hardware bring-up, Lighthouse-tracked handheld teleoperation, a versioned 30 Hz data contract, policy training, and a fail-closed deployment gateway. With it I collected 100 teleoperated demonstrations and deployed learned policies on the real robot. The video shows ACT stacking five cups with both arms, with no teleoperation input.
Role: Sole engineer: hardware integration, software, data collection, training
Timeline: Aug–Oct 2026
Platform: 2× AgileX PiPER (6-DoF), 2× Pika Gripper, 2× Pika Sense, 4× RealSense D405 + D435i
Software: ROS 2 Jazzy · Python · LeRobot · PyTorch · Pinocchio · gRPC
(57.45 min)
contract
ACT, DP, SmolVLA
What I built
Infrastructure
Brought up two PiPER arms over CAN and four Pika devices over USB. Device identity is bound to hardware serials, both arms, grippers and all camera frames live in one URDF/TF tree, and one-command backends refuse to report READY until every stream and status flag checks out.
Pipeline
Teleoperation, recording, conversion, training and deployment share one versioned interface: 30 Hz, 14-D absolute joint state and action, three 640×480 RGB views, locked by a schema hash and a per-checkpoint manifest.
Teleoperation data
Mapped Lighthouse-tracked Pika Sense handhelds to both arms through relative-pose numerical IK with 1:1 gripper control, and collected 100 successful bimanual demonstrations (57.45 min) through a fail-closed raw-to-LeRobot export.
Policy training
Trained ACT and Diffusion Policy and fine-tuned SmolVLA on the same data. ACT runs live through an async PolicyServer/RobotClient with per-policy execution contracts, behind a gateway that owns the command topics and is designed for policy swap.
The stack is organized as one shared ROS 2 hardware layer with three loops on top: teleoperation, data/training, and policy deployment. Teleoperation and the learned policy are interchangeable sources of joint targets; the recorder only subscribes. Upstream drivers and SDKs (pyAgxArm, agx_arm_ros, pika_ros, LeRobot) stay pinned; the integration, contracts, gateway, and training/deployment path are what I built.
1 · Hardware & software stack
Two 6-DoF PiPER arms run on CAN; two mounted Pika grippers, two handheld Pika Sense trackers and a third-view RealSense run on USB. I kept the vendor SDK and ROS 2 drivers at pinned commits and built the integration layer on top: hardware identity by serial number, a combined URDF/TF tree for both arms, grippers and every camera frame, and backends with built-in health checks. Real-time servo control stays in the arm firmware; everything above it only ever sends six target joint angles per arm.
2 · Teleoperation & data
Each Pika Sense is tracked by two Lighthouse base stations at about 164 Hz. Hand motion is applied to the gripper as a relative pose, so the tracking frame never has to be registered to the robot, then solved with numerical IK and executed with the same smoothed joint mode used at deployment. The recorder only subscribes. It writes immutable raw episodes, which are reviewed by hand and exported to LeRobot v3 by a converter that fails closed on timestamp gaps, rate errors, camera skew above 30 ms, frozen or black frames, and out-of-bound joints.
3 · Training
State and action are both absolute 14-D vectors: six joints and a gripper opening per arm. Action is the commanded target rather than the next measured state, so the policy learns intent and is tracked by the same controller at run time. On one dataset I trained ACT and Diffusion Policy and fine-tuned SmolVLA from its pretrained base. Every checkpoint climbs the same ladder: data visualization, 100-step smoke test, single-episode overfit, full training, held-out offline evaluation, shadow mode, and only then live.
4 · Deployment & safety
A LeRobot PolicyServer runs inference; a RobotClient with a custom robot plugin talks to a ROS-side gateway over gRPC on a Unix socket. The gateway owns the arm and gripper command topics. It checks schema and checkpoint hashes on connect, health and freshness before enable, bounds and collisions for every action, and a 0.3 s watchdog throughout. On any failure it holds, closes both gates and does not re-enable itself. Swapping policies only means writing a new execution contract: how long an action chunk is and how overlapping chunks are merged.
Results
ACT, trained on my teleoperated demonstrations, stacking five cups with both arms on the real robot. Single rollout, shown at 1× speed, one take. This is a qualitative demonstration; a held-out evaluation over 20 unseen cup layouts is planned / in progress.
Engineering notes
Note 1 — A safety limiter that rewrote good actions
I first rate-limited the gripper at 0.05 m/s in the gateway. Comparing predicted and sent actions from live ACT runs showed that 43–56% of demonstrated gripper frames moved faster than that, and one closure stretched from 1.9 s to 5.3 s. The gateway now only rejects anomalies (NaN, out of bounds, collision, stale data, timeouts) and passes in-bound actions unchanged.
Note 2 — An inference queue that starved when the robot slowed down
LeRobot's async server skips observations whose 14-D state is within distance 1 of the previous one. When the arms moved slowly or paused, valid requests were dropped and the 10-step action queue ran dry before the next inference. My PolicyServer accepts one observation per control timestep and rejects only duplicates and stale timesteps.
Note 3 — Left and right are serial numbers, not USB ports
Both CAN adapters are renamed to can_left and can_right by their USB serial; cameras are bound by RealSense serial; devices without unique serials are bound by a verified hub topology. Moving a USB plug is harmless. Swapping the two CAN cables silently mirrors the robot, and the runbook says so.
Note 4 — A zero position just outside the joint limit
At rest, joint 2 can read a few hundredths of a radian below its URDF lower limit of 0, so a small regression move from there produced a −0.0322 rad target and was rejected. The fix tolerates the measured residual at the start and only allows motion back into the valid range.
Note 5 — Porting to ROS 2 Jazzy
The vendor's tracker node shipped as a Humble binary and would not load on Jazzy because of an ABI mismatch. I reimplemented it as a Jazzy node that publishes the same pose convention, so the teleoperation mapping did not change.
By the numbers
~21k lines of Python across 101 files · 185 test functions in 32 test files · 36 operational scripts · 18 ROS 2 launch files · 24 runbook documents · 5 upstream repositories pinned to exact commits
What’s next
Next: plug π0.5 and world-action models into the same interface (a new execution contract; data, robot plugin and gateway unchanged), move the PolicyServer to a remote GPU, and use the teleoperation rig for harder coordinated bimanual tasks.
Tech stack
ROS 2 Jazzy Python C++ pyAgxArm LeRobot PyTorch ACT Diffusion Policy SmolVLA Pinocchio gRPC libsurvive Intel RealSense SocketCAN