HMI overview

The Haller HMI is a single web application that operates both SO-101 arms and the base. It replaces the legacy web_teleop.py single-file Python HTML server with a proper full-stack app.

• Frontend: Next.js 16 + shadcn/ui + Tailwind v4 (Node 20+).
• Backend: FastAPI, wrapping lerobot (arms) and rclpy (base) in one process.
• Wire protocol: REST for commands, WebSocket for ~20 Hz telemetry.

Prerequisites

• Python venv with ROS 2 Jazzy access and lerobot installed — see LeRobot environment.
• At least one SO-101 follower configured and calibrated — see SO-101 arms. The default config expects calibration id haller_follower reachable at /dev/haller_arm_follower.
• Node 20+ and pnpm 9.x on whichever host runs the frontend.

Quick start (operator laptop)

Two terminals. First, the backend:

source ~/venvs/haller-hmi/bin/activate-haller-hmi
cd ~/haller_ws/hmi/backend
haller-hmi                       # uvicorn on http://0.0.0.0:8000

Second, the frontend:

cd ~/haller_ws/hmi/frontend
pnpm install                     # first time only
pnpm dev                         # Next dev server on http://localhost:3000

Open http://localhost:3000. You should see:

• A "live" badge top-right (green) once the WebSocket connects.
• A Base panel with a joystick on the left.
• An "Arm: right" panel on the right with six joint sliders.
• A red E-STOP pinned top-right of every page.

Pointing the frontend at a remote backend

NEXT_PUBLIC_BACKEND_URL=http://orin.local:8000 pnpm dev

Choosing a config

The backend reads hmi/backend/config.yaml by default. To run with a different config — most commonly one of the MuJoCo sim presets — pass --config to run_hmi.sh:

./scripts/run_hmi.sh --config hmi/backend/config.solo-sim.yaml

The flag is exported as HALLER_HMI_CONFIG, which haller_hmi.config.load_config honors. Useful for switching between real-hardware and sim configs without editing the systemd unit.

Operating an arm

Each arm card has a header strip (id + mode toggle), a wrist-camera placeholder, the six joint sliders, an Actions row, a Pose row, and a Saved-poses chip strip.

1. Switch the arm to manual. Top-right of the Arm card, click manual. The joint sliders, Home button, and preset replay are enabled. Manual writes are rejected (HTTP 409) in auto.
2. Drag a joint slider. Each drag debounces at 50 ms and posts POST /arm/{id}/goal — the arm tracks toward the slider value, clamped to the calibrated joint limits.
3. home — sends every joint to its calibrated 0° (range midpoint). Manual mode only. If the arm was free-drive, torque re-engages first.
4. free-drive ↔ engage — toggles torque on the entire arm. With torque off, you can hand-move the arm and the sliders just track the live position. Clicking a slider or home (or pressing engage) re-engages torque.
5. Record a pose. Type a name in the Pose input, click save. Current joint positions are written to ~/.haller/presets.json keyed by (arm id, name).
6. Replay a pose. Click any chip in the Saved strip to drive the arm to that pose, or type a name and click go to. Click × on a chip to delete the pose.
7. Hand back to autonomy. Click auto. A VLA process / external driver can now write goals; HMI manual writes return HTTP 409.
8. Emergency. Click E-STOP (top-right of every page). Torque drops on every arm, the base goes to zero, any active teleop session stops, no confirmation modal.

Operating the base

The Base panel ports the existing teleop UX onto shadcn:

• Joystick (left): drag a 2D pad — vertical = linear, horizontal = angular. 10 Hz polling to POST /base/cmd_vel.
• WASD or arrow keys (anywhere on the page): same effect as moving the joystick.
• Speed slider (right): scales the commanded velocity 0.1×–1.0× — a safety margin while learning the feel.
• STOP button: zeros the command. Independent of E-STOP (which also touches the arms).

What else is on the dashboard

• Calibration banner — shown when an arm has no calibration file. Click Calibrate <arm> to open the calibration wizard.
• Teleop card — leader↔follower mirroring between the two arms at 60 Hz. See Leader ↔ follower teleop.
• Human teleop link — opens /teleop/human for webcam-driven bimanual control. See Human-pose teleop.
• Cameras strip — every camera configured in backend/config.yaml, live MJPEG thumbnails.
• Recording panel — builds the exact lerobot-record command for a dataset collection session.

Production (Jetson Orin Nano)

The HMI runs alongside the existing haller-robot.service (which brings up the ROS hardware stack — motors, lidar, odom). They are NOT in conflict; the HMI subscribes to /odom, /scan and publishes /cmd_vel, all topics the hardware stack already exposes.

1. Build the frontend standalone bundle:

   cd hmi/frontend
   pnpm install
   pnpm build

2. Install the systemd unit:

   sudo cp scripts/haller-hmi.service /etc/systemd/system/
   sudo systemctl daemon-reload
   sudo systemctl enable --now haller-hmi.service

3. Watch the logs:

   journalctl -u haller-hmi.service -f

scripts/run_hmi.sh (the unit's ExecStart) activates the backend venv, copies the Next.js static assets into .next/standalone/ (Next 16's standalone output doesn't bundle them by default), then launches both uvicorn (:8000) and the prebuilt Next server (:3000).

REST endpoints

The backend exposes a stable REST surface plus one WebSocket. See the unified-HMI design spec for frame schemas and design rationale.

Troubleshooting

Common HMI runtime issues (live badge red, 409 on goal, calibration banner stuck, joint sliders frozen) are listed on the Troubleshooting page.