Quest VR teleop

Quest VR teleop drives both SO-101 arms from a Meta Quest. You see the arms through passthrough (or, in sim, in a floating camera tile), each controller grip is that arm's dead-man, your hand's movement drives the gripper tip through IK on the robot's real kinematics, and a capsule model of both arms runs inside the 60 Hz commit loop so you cannot command them into each other or through the bench.

It is mutually exclusive with human-pose teleop and leader↔follower teleop — one teleop kind holds the arms at a time.

Bring it up

Rehearse in sim first. It is the same page, the same HTTPS origin and the same backend code — only the arms are MuJoCo instead of Feetech, so nothing physical can move:

# desktop, repo root
scripts/quest-teleop/up.sh --sim     # MuJoCo bimanual, config.bimanual-sim.yaml
scripts/quest-teleop/up.sh --local   # real arms, backend on THIS desktop, no Jetson
scripts/quest-teleop/up.sh           # real arms, backend on the Jetson over ssh
scripts/quest-teleop/down.sh         # stop the desktop half

up.sh prints the URL to open in the Quest browser:

https://192.168.0.191:8444/teleop/vr

Accept the self-signed cert once, then press Enter Passthrough.

NEXT_PUBLIC_BACKEND_URL is inlined when the dev server starts, so changing the backend URL needs a frontend restart, not a reload.

Controls

The menu bindings had to land on the thumbstick clicks: trigger, grip, A/X and B/Y are all spoken for, and every one of them is either a dead-man or a safety action, so none can be shared with a menu. The left stick doubles up short-click/hold, so its click action fires on release (still instant to a human) — a hold must not cycle the view on its way to the reset. The HUD grab borrows the trigger only while that hand's arm is not driving: while driving, the trigger is the gripper, and the HUD refuses to move.

The reset goes through POST /teleop/human/home, inside the session — the discrete /arm/{id}/home move is (correctly) refused while a session owns the arms, and unlike that path the in-session slew rides the LPF, the rate ramps and the bimanual collision guard. A driving side is skipped, and a pending reset self-cancels the moment its side starts driving: the operator's hand outranks a parked reset.

Discarding a take is deliberately cockpit-only: a thumb brush must never be able to throw away an episode you just drove.

Architecture

VRPoseMode only ever produces the target. Everything downstream — the acquisition countdown, staleness, rate ramps, per-side authority, the collision guard, the recorder — is the same code the camera-based path uses. Position mode is a retargeter swap, not a parallel control stack.

Position mode

The default, and the one to use. The legacy body-angle mode is still in the hand mapping selector, and it will fight you:

Anchoring: the robot comes to you

Squeezing a grip anchors. At that instant the arm's current pose and your hand's current pose become a shared origin:

• The open-grip target is wherever the arm already is, so gate error is zero. The moment you squeeze, matching is already done and the countdown is the only wait — hold still, feel the buzz, drive. For vr_grip sessions that countdown is 1 s, not the camera path's 3 s: the anchor makes handover zero-error by construction, so the countdown only filters accidental grips. The same reasoning stretches the per-side tracking-loss grace to 700 ms (vs 300) — Quest controller tracking flickers, and in position mode a longer grace is safe: no new targets arrive during the gap and the next frame re-anchors at the hand, so there is nothing stale to jump to.
• While the grip is held, hand deltas move the arm. Release freezes the arm where it is and drops the anchor.
• The next squeeze re-anchors from wherever your hand is then. Big workspaces are covered by ratcheting: drag, release, move your hand somewhere comfortable, squeeze, drag again.

No limb-length calibration is involved anywhere. That is the other half of the point — the angle-copying path needs a body model; this one needs nothing about your body at all.

From hand delta to joint goal

The delta is taken in the operator's anchored heading frame — the direction you were facing when you squeezed, frozen for the life of the anchor, so turning your body mid-drag does not swing the arm. It then maps onto robot axes through the operator stance, a panel selector shipped on every frame:

Both arms use the same mapping, because the mounts are identical and side-by-side rather than mirrored. The stance also decides the hand↔arm pairing, applied when the session starts: in behind, the operator faces the same way the arms reach, so the arm under their right hand — frame-right in the overshoulder view — is the one the config names left (robot −x), and the frontend assigns the arms accordingly. Without this, hands-apart made the arms cross on screen, which reads as "the controls are inverted".

The target wrist point goes through damped least-squares IK (solve_wrist_point) which owns shoulder_pan, shoulder_lift and elbow_flex:

• Numeric on purpose, run against the same fk_points the collision guard trusts, so there is exactly one definition of the kinematics in the codebase.
• 24 iterations max, finite-difference Jacobian (0.25° probe), a 20°-per-iteration step cap, and joint limits clamped on every iterate — an unreachable target converges to the closest reachable pose instead of diverging.
• Damping is relative to the Jacobian's own scale: 1% of the mean diagonal of JᵀJ. With links in metres and joints in degrees those entries sit around 1e-5, so an absolute lambda would either swamp them and freeze the solve or vanish entirely.
• ~30 FK evaluations per solve, per driven side, at 30 Hz: microseconds of numpy.

The wrist joints do not go through IK. They ride the controller's attitude relative to its anchored attitude, and they are deliberately exaggerated:

The gripper is absolute, straight off the trigger — no anchoring, no gain.

Keeping the arm off the bench

This is the subtle one, and it is worth understanding because the symptom looks nothing like the cause.

The collision guard scales back the whole commanded step. So if the IK target sits in a region where every direction makes the floor slack worse, the guard clamps everything — including the perfectly safe lateral motion in the same step. The arm goes solid: not "stops descending", but stops moving at all, which reads as a seized joint or a too-tight limit.

Pushing your hand below the bench does exactly that. Two clamps prevent it, both applied to the target before the guard ever sees it:

Clamping the target turns "pushed too low" into "slides along the bench", and leaves the guard's own floors as the precise backstop rather than the thing doing the work.

The fingertip clamp has to iterate: raising the wrist target re-pitches the forearm, which moves the tip slightly less than the correction. Up to 12 passes, then it takes what it has — the guard is still behind it, so an unconverged last millimetre is safe, just not ideal.

server.py derives both floors from cfg.collision rather than hardcoding them, so a config whose bench sits at a different height gets the right clamp for free.

The collision guard

collision.py sweeps four capsules per arm along an analytic FK of the vendored MJCF, and filters every commanded step.

Coarse is the point: capsule distance is exact, cheap enough for the 60 Hz loop with room for a bisection search, and sound as long as the capsules contain the meshes. tests/sim/test_collision_sim.py holds both properties honest against MuJoCo — the FK against body kinematics, and soundness by asserting that whenever MuJoCo finds an inter-arm contact, this model's gap is already ≤ 0.

The filtering rules:

• A step that keeps ≥ margin_m clearance passes.
• A step that improves a bad pose always passes — escape is never blocked.
• Anything else is scaled back along its own direction and stops at the margin.
• Both arms scale together. The commanded step is one 10-DOF motion; scaling its components independently could turn a safe diagonal into an unsafe slide.
• The gripper never participates. Opening the jaw moves nothing this model tracks, and freezing it because the arms are close would cost you a grasp for no safety gain.

With 45 mm hand capsules and the sim's 15 mm margin, the closest two fingertips get is 105 mm. A direct cube-to-cube hand-off is therefore not possible; hand-offs go set-down-then-regrasp. The binding constraint there is the hand capsule radius, not the margin — dropping the margin further would not buy a hand-off.

When the guard bites you feel a light buzz and the HUD shows ◉ COLLISION HOLD. enabled: false turns it off entirely, and the HUD says so.

The sim arena

The bimanual sim config (config.bimanual-sim.yaml) composes a scene aimed squarely at pick-and-place data collection. See MuJoCo simulation for the preset machinery; the arena itself:

Three details are load-bearing rather than decorative:

• The backdrop is non-physical on purpose. If it ever gained collision geometry it would become a surface an arm could lean on — and one the guard, which knows only the floors in its config, has no idea about. A test pins contype/conaffinity at 0.
• The arena floor catches escapees. A cube swept off the bench used to accelerate away forever: a bad frame to record and a bad state to reset from.
• The place zone sits at y=-0.21, not -0.25. Landing a fingertip at bench level in the -0.24…-0.27 band needs wrist_flex at or past its +95° hard limit — swept, both arms, the worst depth on the entire midline, and precisely where the task ends. At -0.21 (~83% of reach) it needs ~75°, leaving the wrist somewhere to go.

The finger pads also run slide friction 2.0, where the vendored MJCF ships 1.0. Upstream tuned for bare PLA; the real pads are rubber TPU, and at 1.0 a pinched 4 cm cube slips out mid-transport. MuJoCo combines a contact pair by max, so this one value governs pad-cube friction on its own.

Cameras

Both operator cameras were pulled in tight after actually rendering the old framing: the arms were ~15% of frame height, useless for fine work. The wrist-cam constants were solved, not hand-placed — a desired world pose at a canonical grasp (0.14 m behind the fingertip, 0.11 m above, world-up in frame) transformed into the jaw's local frame. The bench itself gained a low-contrast 5 cm checker (a featureless grey slab gives no lateral or depth reference at all) and the shadow map went 1024 → 4096, because the gripper's shadow is the operator's main height cue.

overshoulder is listed first in the config, so it is what the cockpit's BASE tile and the headset HUD show by default — the view that matches the default behind stance. The operator views render at 960×720 (they are what you teleop from); overhead and the wrist close-ups stay 640×480. It is framed on the workspace, not on the arms' extremes: coverage is |x| ≤ ~0.44 m across the working depth band against cubes at ±0.28 m. A wide lateral swing past that clips at the frame edge — a deliberate trade, because the alternative (the near-eye-level framing it replaced) kept a 45° outward pan in frame, a pose that holds the gripper off the bench entirely, and spent ~60% of every useful frame on empty space.

Tests pin the framing for both operator cameras: every cube slot, the place zone read from the model rather than restated, and both arm mounts inside the frustum, plus the downward aim and each camera's lateral agreement with its stance.

Recording datasets

The sim chain records the same LeRobot datasets the real rig will — same recorder, same schema — so headset hours produce training data with no arms powered.

1. Draft the take on the desktop cockpit (/ → Dataset tab): the task string (one instruction per dataset, e.g. "pick the red cube, hand it to the other arm, place it in the box") and your HF username. The draft persists in the browser, so the headset can start takes from it.
2. Bring up sim, open the URL in the Quest browser, Enter Passthrough.
3. Hold A or X (~0.5 s) to start — both controllers buzz, the HUD shows ● REC <frames>. Drive the task. Hold A/X again to stop and save. Repeat per episode; the draft stays put.

Takes land in ~/.cache/huggingface/lerobot/<hf_user>/<task-slug> at 30 fps, h264:

The headset renders five cameras but the dataset carries three. overshoulder_sim (your default teleop view) and overhead_sim are operator aids with record: false — they never reach the dataset. The three that are recorded are keyed top / left_wrist / right_wrist to match π0.5's pretraining slots and the public bimanual SO-101 datasets. See dataset collection for the full schema and the reasoning.

The skipped counter beside the frame count tells you a take had gaps; the wall-clock deltas tell you where. An E-STOP mid-take is fine — the recorder notices the session die and saves up to that frame instead of appending a torque-off tail.

telemetry.hz is 30 in the sim config, matched to the Quest's ~30 Hz publish rate and LeRobot's SO-101 convention, and it is therefore also the recorded fps. The real rig stays at 20 Hz: the half-duplex Feetech bus already interleaves 60 Hz writes with those reads.

Replay a take onto the sim arms to eyeball what you actually drove (loops until stopped):

curl -X POST http://localhost:8000/teleop/sim/start \
  -H 'Content-Type: application/json' \
  -d '{"follower": "left", "leader": {"source": "replay",
       "dataset_path": "~/.cache/huggingface/lerobot/<hf_user>/<task-slug>"}}'
# ...and "follower": "right" for the other arm; /teleop/sim/stop to end.

HUD and diagnostics

The HUD floats over passthrough: per-side authority and countdown, grip state, live collision clearance, a red ● REC <frames> while a take rolls, an E-STOP button you can hit with the controller ray, and any backend error.

In sim the camera tile floats in the HUD and defaults on whenever the backend's base camera is a sim render — passthrough shows your room, and the sim arms live only in that tile. Cam off hides it. The desktop cockpit (/) shows the same view.

The HUD: two panels, movable

The HUD is a cluster of two separate world quads: the camera tile, and the status/menu panel hanging below it — the instructions never sit on top of the view they annotate. The tile is textured straight from the MJPEG <img> at display rate, at native resolution; the text panel repaints at ~10 Hz on its own canvas. (They used to share one 1024×768 canvas, which put the panel over the camera view, capped the tile at the panel's 10 Hz cadence, and resampled the camera twice.)

The whole cluster is grabbable, Quest-window style: point either quad with the controller ray and hold the trigger (with that hand's grip open — while driving, the trigger is the gripper). It follows the ray at its grab distance, keeps turning to face you, and the placement persists in localStorage. The grab geometry (rayQuadHit, clusterLayout, yawTowardHead) is pure and pinned by tests, not by strapping on a headset.

The view menu

The HUD carries a menu block listing every camera the backend advertises, with the active one marked, the tile size, the record command beside its live frame count, and the current operator stance (display-only — it is chosen on the panel before entering VR, but a wrong stance reads as "the arm fights me", so it has to be visible from inside the headset). It is generic over /cameras: in sim that is the MuJoCo views, and on the real rig it is the mast plus the egocentric gripper cams, with no further work.

A camera whose config says facing: operator (the tower mast cam, which shoots back at whoever drives) gets its tile mirrored horizontally for display — a facing-you feed reads left/right-flipped against your own hands unless it behaves like a mirror. Recorded pixels are never touched.

Both the chosen view and the tile size persist in localStorage, so a session resumes where the last one left off.

Tile size is the real lever on "the view is too small". The tile is a quad in the world, so apparent size is width ÷ distance, not pixels:

Changing it rewrites the quad's model matrix in place — no pipeline rebuild, no reallocation.

Per-side is the whole design. One controller leaving the Quest's tracking volume releases only that arm; the other keeps driving, and re-tracking re-acquires through the normal countdown.

REST + WebSocket surface

A VR frame looks like:

{
  "ts_ms": 1786134696657,
  "vr_mode": "pose",
  "dead_man": true,
  "head": { "position": [0, 1.6, 0], "orientation": [0, 0, 0, 1] },
  "left":  { "position": [-0.35, 1.40, 0.30], "orientation": [0, 0, 0, 1],
             "trigger": 0.0, "squeeze": false, "tracked": true },
  "right": { "position": [0.25, 1.15, -0.30], "orientation": [0, 0, 0, 1],
             "trigger": 0.85, "squeeze": true, "tracked": true }
}

Positions are WebXR-space metres (+Y up); orientations are the target-ray quaternion as (x, y, z, w) — not MJCF's (w, x, y, z), which is the ordering collision.py uses internally. squeeze per side is the dead-man; dead_man at the top level is the OR of the two.

First hardware run

Ten minutes, in this order. Do not skip to step 7.

1. Bench clear, supply on, a second person or the desktop cockpit at the E-STOP.
2. Mount geometry. Measure base-plate bolt to bolt; set collision.mounts x to ±half that, and table_z_m if the bench surface is not the mount plane. Restart the backend after editing.
3. Clearance sanity. Session started, grips open. Torque off both arms from the cockpit and move them toward each other by hand: the clearance number must shrink as they approach and go negative just before they touch. If it does not move, the mounts are wrong.
4. Direction check, one arm at a time. Squeeze ONE grip, hold still through the countdown, nudge your hand 5 cm outward. The arm must move outward. Moving opposite means the arm-mounting parity is wrong (this rig is identical, side by side). The wrong arm moving means left/right naming is crossed.
5. Gripper and speed feel. Trigger through its range, then a slow reach. Anything that feels fast: lower motion.max_speed_deg_s — the teleop path obeys it at the handle.
6. Provoke the guard once, gently. Both hands slowly toward the centre. The arms must stop with a buzz before they meet, the HUD must read COLLISION HOLD, and pulling back out must work instantly.
7. Only then: real speed, real manipulation, recording.

Troubleshooting

Design reference

• Position mode: hmi/backend/haller_hmi/vr_pose_mode.py — the module docstring is the design rationale, including why angle-copying was abandoned.
• Collision guard: hmi/backend/haller_hmi/collision.py, verified against MuJoCo by tests/sim/test_collision_sim.py.
• Session, authority, staleness: hmi/backend/haller_hmi/human_teleop.py; the pure engage/release policy is in safety.py.
• Scene composition: hmi/backend/haller_hmi/sim/builder.py, sim/assets/scenes/workbench.xml, sim/assets/so101/so_arm100.xml.
• End-to-end smoke: scripts/vr_smoke.py — 38 checks, no headset required. Run it after any backend change that could touch teleop or recording.
• Operator quickstart: hmi/QUICKSTART-QUEST.md.
• Desktop-only rig runbook: docs/setup/desktop-real-weekend.md.