SO-101 MuJoCo simulation

The HMI can drive a simulated SO-101 instead of a physical one. The browser surface is identical (per-arm panels, leader↔follower teleop, human-pose and Quest VR teleop, dataset recorder, MJPEG camera streams) — you just point the backend at a sim config file and lerobot's SO101Follower is replaced by a SimArmHandle that writes joint goals to a MuJoCo world running in the same process.

Three preset configs ship in hmi/backend/: solo follower, bimanual, and leader+follower. Each is a single yaml.

Architecture

The substitution lives at the ArmManager layer. Anything above it (REST endpoints, telemetry WS frames, teleop sessions, recording) is identical between real and sim.

Why sim-first: measurability, not safety

The usual argument for starting in simulation is that nothing can break. That is true, and it is not the point.

The point is that only sim can tell you whether the policy works. Training loss — L1 on actions against a held-out slice of demonstrations — is a proxy. It answers "does this policy predict what the operator did", which is not the question. The question is "does this policy accomplish the task", and the only honest measurement of that is task success over closed-loop rollouts: put the policy in charge, let it run, count how often the cube ends up on the pad.

On the real rig that measurement costs a human. Someone resets the bench between rollouts, watches each one, and decides whether it counted. That is a few tens of trials a day at best — not enough resolution to tell two checkpoints apart.

In sim the same measurement is N seeded resets and a predicate, at zero human cost — and because the resets are seeded, two checkpoints are compared on the same N scenes rather than on whatever the bench happened to look like that afternoon.

That is what POST /sim/scene/reset and the task monitor exist for, and it is why they were built before any policy was trained: an objective you cannot measure cheaply is an objective you will end up optimising by vibes.

The safety is real too. It is just the second reason.

Install

The mujoco Python package is pinned in hmi/backend/pyproject.toml. Editable installs already have it; otherwise:

source ~/venvs/haller-hmi/bin/activate-haller-hmi
pip install -e hmi/backend

The HMI runs headless by default and uses EGL for offscreen rendering. If your host lacks EGL, fall back to OSMesa:

export MUJOCO_GL=osmesa   # or 'egl' (default), or 'glfw' for the interactive viewer

The three presets

Bring any one up with the --config flag (see HMI overview for the flag itself):

./scripts/run_hmi.sh --config hmi/backend/config.solo-sim.yaml

Then open the HMI at http://localhost:3000. Joint sliders, presets, and the overhead camera all work against the sim. The MJPEG stream is at http://localhost:8000/cameras/overhead_sim/stream.

Watching the physics

Two options:

Headless (default) — watch through the HMI camera

The HMI's sim cameras stream the scene to the browser exactly like a real opencv camera. The bimanual preset renders five views — the two operator cameras at 960×720, the overhead and both wrist cams at 640×480 — and every fps is set to match telemetry.hz (30) so each recorded frame carries a fresh render. Good for desktop dev and for any flow that uses the dataset recorder (which needs a camera in the config either way).

Interactive MuJoCo viewer

MUJOCO_VIEWER=1 ./scripts/run_hmi.sh --config hmi/backend/config.leader-follower-sim.yaml

Opens a desktop MuJoCo window with mouse-drag perturbation. Required for the leader+follower preset's mouse-drag mode (below) — there's no way to drag arm joints without the viewer.

Leader+follower modes

The leader+follower preset has three operating modes for the leader side. They differ only in what the new /teleop/sim/start endpoint is told to read from:

Mouse-drag (default)

MUJOCO_VIEWER=1 ./scripts/run_hmi.sh --config hmi/backend/config.leader-follower-sim.yaml

Then start a sim teleop session:

curl -X POST http://localhost:8000/teleop/sim/start \
    -H 'Content-Type: application/json' \
    -d '{"follower":"right","leader":{"source":"mouse","arm_name":"left"},"hz":60}'

Drag the LEFT arm's joints in the MuJoCo viewer — the RIGHT arm mirrors at 60 Hz.

Dataset replay

curl -X POST http://localhost:8000/teleop/sim/start \
    -H 'Content-Type: application/json' \
    -d '{"follower":"right","leader":{"source":"replay","dataset_path":"/path/to/lerobot/dataset"},"hz":30}'

Replays the action stream from a recorded LeRobotDataset on the sim follower. Useful for sanity-checking that a dataset captures what you think it does, without spinning up the real arms.

Real leader → sim follower

Edit hmi/backend/config.leader-follower-sim.yaml and switch the LEFT arm from source: sim to source: real, with the right port and calibration_id. Then use the regular /teleop/start endpoint — the HMI's existing TeleopSession does the rest:

curl -X POST http://localhost:8000/teleop/start \
    -H 'Content-Type: application/json' \
    -d '{"leader":"left","follower":"right","hz":60}'

This is one direction: real → sim. The opposite (sim leader → real follower) intentionally requires /teleop/sim/start and a MouseDragSource / DatasetReplaySource — you don't want a sim arm driving a physical one accidentally.

What's in the MJCF

Each preset's world is composed at boot from three pieces:

• sim/assets/scenes/workbench.xml — the arena: bench, floor, backdrop, place zone and lighting.
• The vendored trs_so_arm100 MJCF for each arm, instanced once per arms: entry with source: sim.
• Cubes, dealt from sim/assets/scenes/cube.xml into measured slots (sim_cubes).

hmi/backend/haller_hmi/sim/builder.py is the composer, and it also emits the two cameras. CamelCase ↔ snake_case translation between MJCF joint names and HMI joint names lives in hmi/backend/haller_hmi/sim/arm.py.

The arena

The cameras

camera_xyaxes(pos, target) derives the MJCF basis from where the camera is and what it looks at, so a viewpoint stays readable instead of being six magic numbers nobody can adjust later. overshoulder is listed first in the bimanual config, making it the cockpit BASE tile and the headset HUD default; tests pin every cube slot, the place zone and both mounts inside both operator cameras' frusta.

Five render, three are recorded — threequarter → top, plus the two wrist cams — and that split is a training decision, not plumbing. The full argument (π0.5's camera slots, armnetbench's key names, and why every recorded camera costs sample rate) is in dataset collection.

Grasp friction

The finger pads run slide friction 2.0 where the vendored MJCF ships 1.0 — upstream tuned for bare PLA, the real pads are rubber TPU, and at 1.0 a pinched 4 cm cube slips out mid-transport. MuJoCo combines a contact pair by max, so this one value governs pad-cube friction on its own.

Driving this arena from a headset is Quest VR teleop — including the collision guard's sim margins and why the place zone sits where it does.

Collecting episodes in sim

The per-episode loop

POST /sim/scene/reset  {"seed": 1000+i}    →  deal the bench, reproducibly
POST /record/start     {"repo_id", "task"} →  begin the episode
                                              (teleop the demonstration)
POST /record/stop      {"save": true}      →  save
GET  /sim/task/status                      →  did it actually succeed?

Repeat with seed incremented. Because the seed is part of the loop, a run is re-creatable from its seeds alone: the same seed and the same RandomSpec produce byte-identical cube poses, colours and lighting. (Warm-start accelerations and qacc are zeroed on every reset, which is what makes "same seed, same trajectory" true rather than approximately true.)

The recorder's own success / success_frames — in GET /record/status, and in the saved episode's next.reward column — is the same signal latched over the take; /sim/task/status is the live view of it. See dataset collection for the schema those episodes land in.

POST /sim/scene/reset

curl -X POST http://localhost:8000/sim/scene/reset \
    -H 'Content-Type: application/json' \
    -d '{"seed": 1042, "randomize": true, "home_arms": false}'

Returns the same body as GET /sim/scene: every cube's pos/quat/rgba, every light's pos/diffuse, the fixed cameras' pos, plus last_seed, randomized and reset_count — read back from the model rather than replayed from the plan, so what it reports is what the simulator actually holds.

The reset also clears the task monitor's accumulated held time, so a cube still sitting on the pad when the last episode ended cannot carry its qualifying streak into the next one.

sim_seed in the config

sim_seed: 20260809     # null (the default) means "don't reset at startup"

Seeds the first reset, applied at startup, so a run is re-creatable from its config alone. Per-episode resets pass their own seed and ignore it. null leaves the bench on the builder's home slots, exactly as it was before this existed.

What randomization varies

Camera jitter is off on purpose: those cameras are what the recorder saves, so moving them changes the observation distribution itself and throws away the framing solved for in sim/builder.py. Turn it on only if you specifically want viewpoint robustness and are willing to re-check that the bench still fills the frame.

Cube height is not randomized — it stays on the builder's vertical stagger, which is what keeps two cubes on different laps from ever sharing space.

What needs a restart instead

Cube count (sim_cubes) and cube size. Everything the reset touches — free-joint qpos/qvel, geom_rgba, light_*, cam_pos — is safe to write on a live model. Changing the number or size of cubes means rebuilding the model, and a rebuild orphans every object holding the old one: the arm handles (built from joint ids), each SimCamera's renderer (constructed from world.model on its own EGL thread), and every arm handle's world reference. So those are restart-time config, not randomizable parameters.

The success predicate, in words

A frame scores when all of these hold at once for the cube being watched:

1. the cube is in contact with the place_zone geom — from MuJoCo's contact list, which the step that just ran already populated;
2. its centre is inside the pad's half-extent shrunk by zone_inset_m (0.01 m by default; the pad's half-extent is 0.06 m, so the acceptance box is 0.05 m) — a cube balanced half off the edge doesn't count;
3. it is settled: linear speed < lin_vel_eps (0.01 m/s) and angular speed < ang_vel_eps (0.1 rad/s — looser, because a cube rocking to rest spins fast at tiny amplitude and would otherwise never qualify);
4. the robot has let go (require_release, on by default): no arm geom is touching the cube.

…and that has held continuously for settle_s (0.5) SIM seconds — data.time, not wall clock. The stepper paces itself to real time so the two normally agree, but a test driving mj_step in a tight loop advances sim time far faster than the wall, and a paused world advances it not at all. Sim time is the clock that matches what the physics actually did.

Two of those four deserve their reasons spelled out:

• Contact, not height. The obvious test — "is the cube's z above the pad?" — does not survive the numbers. A cube dropped on the pad settles at z ≈ 0.0219; the same cube on the bare bench settles at z ≈ 0.0199. That is a 2 mm discriminator against a 1 mm-thick pad, well inside the noise of a cube that landed on a corner and rocked. The contact list says exactly which geoms touch which, and costs nothing to read.
• Release, not just rest. Without it, success fires while the gripper is still pressing the cube onto the pad — the cube is on the zone and not moving precisely because the robot is holding it there. That labels the middle of a place as the end of one. The release test uses the whole arm, not just the jaws: a cube pinned under a forearm is no more placed than one still in the fingers.

GET /sim/task/status returns success, held_s, per_cube (each cube's instantaneous placed and its held_s), target, settle_s and sim_time_s. The thresholds it ran with are written into every recorded dataset's haller_scoring block, because the thresholds are the label definition.

REST surface (sim-only)

There is no global "sim mode" flag: the MuJoCo world exists iff some arm is source: sim, and that is the test every one of these routes makes.

Everything else (arm goals, mode, calibration, leader/follower, human teleop, dataset recorder, cameras) reuses the existing endpoints documented in HMI overview.

Troubleshooting

• GLFWError: X11: Failed to open display — MUJOCO_GL=glfw requires a display. Use MUJOCO_GL=egl (default) or MUJOCO_GL=osmesa for headless.
• Sim camera frame is all black — check the <light> in sim/assets/scenes/workbench.xml is in the composed MJCF (it always is — the builder includes the workbench unconditionally) and that the camera's pos and aim actually point at the scene. The camera definitions live in hmi/backend/haller_hmi/sim/builder.py.
• observation.effort is all zeros in a sim take — sim effort is only meaningful with torque on. A torque-off actuator still applies its bias term and would read as saturated, so the sim handle reports 0.0 instead of the raw number. Check the arm's torque state before blaming the recorder.
• lerobot.policies import error on the dev laptop — unrelated to the sim. Local scipy/numpy ABI quirk; VLA policy code lives on RunPod, not on the laptop.

Out of scope (for now)

• Automated closed-loop rollouts. Everything needed to score them exists (seeded reset + task monitor); what's missing is the driver that loads a policy and runs the loop N times. Belongs on RunPod alongside scripts/runpod/.
• Randomizing anything that requires a model rebuild — cube count, cube size, textures.
• Bench-material randomization. The cube geoms carry no material, which is why writing geom_rgba works on them; the bench does (bench_mat), and MuJoCo renders a material's colour for a geom that has one — writing its geom_rgba is a silent no-op. Randomizing the bench means model.mat_rgba.

Design reference

• Design spec: docs/superpowers/specs/2026-05-23-so101-mujoco-sim-trio-design.md
• Implementation plan: docs/superpowers/plans/2026-05-23-so101-mujoco-sim-trio.md
• Vendored arm MJCF: AlexanderKoch-Koch/trs_so_arm100