VELMA

Vision-Enabled LLM Manipulation Arm. VELMA is a tabletop SO-101 arm in MuJoCo that runs a full pick-and-place from vision. A fixed side camera locates the object to within a centimetre or so, and the wrist camera orbits that seed over five views and triangulates it to about 1 mm. The arm grasps and places from those coordinates, and can be driven by an LLM over MCP, or by hand from a terminal control panel.

What's here

so101-Models/
  main.py            entry point: loads config.yaml, starts the MCP server (or hardware backend)
  config.yaml        runtime config (backend, viewer on/off, port, LLM provider)
  requirements.txt   robot venv deps
  server/            the MCP server — 15 tools (see below)
  perception/        find_object (coarse) + wrist triangulation + the sidecar client
    requirements.txt   perception sidecar venv deps (torch CPU + ultralytics + transformers)
  percept_venv/      torch/SAM/Depth-Anything sidecar interpreter (CPU)
  wrist_refine.py    wrist-camera hover/state helpers used by perception/wrist_triangulate.py
  robot/, kinematics/, models/   backend, IK, MuJoCo scenes
  control_panel.py   terminal UI to drive the arm by hand through the MCP server
  agent.py           thin MCP client that connects any LLM (Ollama/OpenAI-compatible) to the arm
  SKILL.md           system-prompt guide for an LLM driving the arm (fed in by agent.py)
  .mcp.json          MCP server entry for CLI tools (Claude Code, Codex CLI, ...)
  venv/              robot interpreter (MuJoCo, no torch)

Install

The two virtual environments are the only heavy pieces. Recreate them from the pinned deps (run from inside so101-Models/):

# robot venv (renders MuJoCo, runs the MCP server — no torch)
python3 -m venv venv && ./venv/bin/pip install -r requirements.txt

# perception sidecar venv (torch CPU + ultralytics + transformers; needs Python 3.11/3.12)
python3.11 -m venv percept_venv && ./percept_venv/bin/pip install -r perception/requirements.txt

Then, optionally, run the setup wizard to write config.yaml (backend, LLM connection, server port) — it's a config helper, not part of the install:

./venv/bin/python setup.py

A committed config.yaml with sane defaults already ships in this repo, so setup.py is only needed to change something (e.g. switch to the hardware backend, or add an API key).

Run the server (headless — required for perception)

Perception needs offscreen rendering, so run the server without the viewer:

SO101_VIEWER=0 MUJOCO_GL=egl ./venv/bin/python main.py

SO101_SCENE=models/so101/<scene>.xml overrides the scene (default pick_and_place_scene.xml). Set SO101_VIEWER=1 to watch the arm instead — but perception is then disabled by design (the offscreen and on-screen renderers cannot coexist on this box).

Drive it by hand

In a second terminal:

./venv/bin/python control_panel.py

It connects to the running server, lists the 15 tools with their live schemas, and lets you call any of them with guided input. Any MCP client must set a long read timeout (the panel uses read_timeout_seconds=300) — grasp/place/stop_recording take several seconds, and a client that times out mid-call takes the server down with it.

Connect an LLM

With the headless server running (above), pick one of two paths:

A CLI tool that speaks MCP (Claude Code, Codex CLI, Gemini CLI, Cursor, ...): this repo's .mcp.json already points at http://localhost:3001/mcp — just run the CLI tool from inside so101-Models/ (e.g. claude). If you changed config.yaml's server.port, re-run ./venv/bin/python setup.py (Server settings) to regenerate .mcp.json to match.

Any Ollama / OpenAI-compatible model, via the bundled thin agent:

./venv/bin/python agent.py --model gemma3:4b --task "pick up the red cube and place it in the container"

agent.py feeds the model SKILL.md as a system prompt plus the live tool schemas from the server (never hard-coded, so it can't drift from what the server actually exposes), and relays tool calls both ways — it doesn't inject a procedure or second-guess the model. See agent.py --help for endpoint/model flags (LLM_BASE_URL, LLM_API_KEY, --mode auto|native|text).

The pick-and-place path (what an LLM calls)

The object's position is measured, never given:

1. find_object("the red cube") — coarse (x, y) from the fixed side camera (~cm, enough to aim).
2. triangulate("the red cube", x, y) — refine to a grasp-ready 3D point with the wrist camera (~1 mm). If confidence < 0.6 the views disagree — re-run find_object or decline.
3. grasp(x, y, z, trust_coords=true, grip_width_mm=width_mm+10) — gentle, never-bat top grasp.
4. find_object("the container") — the container's position is measured too, not assumed.
5. place(container_x, container_y, 0.05) — put it down; check cube_in_container.

The 15 tools

Perception: find_object, triangulate. State: get_robot_state, capture_cameras, get_initial_instructions. Manipulation: grasp, place. Motion: move_to_position, set_joint_angles. Gripper: set_gripper, open_gripper, close_gripper. Harness: reset_scene, start_recording, stop_recording.

Honesty note

The perception path (find_object → triangulate) is measurement-only — nothing in it reads ground truth. get_robot_state and every tool's returned state deliberately omit the cube's and container's true positions, so an LLM cannot shortcut perception by reading the answer out of state (a benchmark harness that needs ground truth for scoring can opt back in by starting the server with SO101_EXPOSE_TRUTH=1 — never set this when an LLM is connected).

Two internal reads of ground truth remain inside grasp itself (object shape for the never-bat decline, and a position drift-abort guard) and are pending removal. Full audit, exact locations, and severity: research/HONESTY_REVIEW.md.