Gorked

Computer-use agent framework for Grok Build CLI: see the screen, box every text region, target center pixels, and act — with a transparent thought process and a high-tech chat HUD.

Gorked does not call the xAI HTTP API for planning. The brain is the local grok binary (headless mode: streaming thoughts, tools, MCP). The hands and eyes are Gorked (OCR, mouse/keyboard, safety gates).

┌──────────────┐    ┌─────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Capture     │ →  │  OCR text   │ →  │  Headless Grok   │ →  │  Mouse/keyboard │
│  primary mon │    │  boxes+ctr  │    │  (CLI + MCP)     │    │  (dry-run/live) │
└──────────────┘    └─────────────┘    └──────────────────┘    └─────────────────┘
                                              │
                                              ▼
                               ┌──────────────────────────────┐
                               │  gorked-hud (C++ chat panel) │
                               │  thoughts · plans · actions  │
                               └──────────────────────────────┘

---

Features

Area	What you get
Screen vision	Fast screenshots (primary monitor only), RapidOCR boxes, center-pixel click targets
Annotation	Color boxes, labels, crosshairs + (x,y) chips on PNGs
Computer control	Move, click, double/right-click, type, click-and-type, hotkey, drag, scroll
Safety	Dry-run by default; live needs explicit flags + confirmation phrase
Brain	Headless grok CLI — streaming thoughts, tool use, JSON-schema plans
MCP tools	Grok drives Gorked via MCP (analyze_screen, click_text, hotkey, …)
HUD	Frameless C++ messenger (bottom-right): chat in, thoughts out, Esc aborts
Thought journal	OBSERVE / REASON / PLAN / ACT / RESULT traces → JSON + HUD bubbles

---

Requirements

• Linux with X11 (Wayland-only sessions may need XWayland for capture/input)
• Python 3.11+
• Grok Build CLI on PATH (grok), authenticated (grok login)
• C++17 toolchain + CMake (only if you build the HUD)
• X11 libraries: libX11, libXft, libXrender, libXrandr, fontconfig, freetype

Optional: dual-monitor setups are fine — only the primary display is captured (secondary/TV ignored to save tokens).

---

Install

git clone https://github.com/iBerry420/gorked.git
cd gorked

python3 -m venv .venv
source .venv/bin/activate
pip install -e .

# Register project MCP for Grok Build
gorked mcp --install
# Trust the folder once if needed: open with `grok --trust` in this repo

Build the HUD (recommended)

cmake -S hud -B hud/build
cmake --build hud/build -j
# binary: hud/build/gorked-hud

---

Quick start

# Synthetic UI + OCR + heuristic plan (no Grok call required)
gorked demo

# Live OCR: boxes + center coordinates on primary monitor
gorked analyze
gorked monitors          # which display is captured

# Find UI text → absolute click coords
gorked find "Save"

# High-tech chat HUD (leave open)
gorked hud
# Live mouse/keyboard when you send tasks from the HUD:
gorked hud --live

# Headless Grok + MCP tools (dry-run)
gorked drive "Click the Settings button"

# Live computer use (real input)
gorked drive "Open Chrome, search cats" --live

HUD controls

Input	Action
Enter	Newline in composer
Ctrl+Enter	Send task → spawns gorked drive
Esc	Abort running agent process tree
Drag header	Move window (frameless tool window)
−	Minimize to pill (flush bottom-right)
Restore (on pill)	Expand full panel
×	Quit

---

Architecture

Two ways to run the agent

1. Drive mode (recommended) — Grok is in charge via MCP:

User / HUD  →  gorked drive "goal"  →  grok -p (streaming-json, --yolo, --trust)
                                         ↓ MCP tool calls
                                    gorked MCP server
                                         ↓
                              capture · OCR · mouse · keyboard
                                         ↓
                              output/ + HUD bubbles

2. Agent/plan mode — Gorked OCRs, Grok returns structured PlannedAction[], Gorked executes:

gorked agent "goal"   # or gorked plan "goal"

Primary-only capture

mss index	Meaning	Default
0	Virtual all-monitors	Blocked
1…	Physical displays	Primary only (is_primary)

Env overrides: GORKED_ALLOW_ALL_MONITORS=1, GORKED_ALLOW_MONITOR_OVERRIDE=1, GORKED_MONITOR=primary.

Live vs dry-run

Layer	Dry-run	Live
CLI	default	--live
HUD	gorked hud	gorked hud --live
MCP process env	GORKED_DRY_RUN=1	GORKED_LIVE=1 + GORKED_DRY_RUN=0
Explicit toggle	—	set_live_mode(true, confirm_phrase="I_ACCEPT_LIVE_CONTROL")

Live mode is not pinned in .grok/config.toml so each run inherits the correct mode from the parent process.

---

MCP tools

Exposed to headless Grok as server gorked:

Tool	Purpose
get_status	dry_run / live, monitor, paths
set_live_mode	Gate real input (or no-op if already live)
thought	User-visible reasoning → HUD
list_monitors / capture_screen	Display info / raw shot
analyze_screen / analyze_image	OCR boxes + centers
find_text / click_text	Locate / click labels
mouse_move / mouse_click / mouse_drag / mouse_scroll	Pointer
type_text / click_and_type / hotkey	Keyboard
wait_ms / recent_actions	Timing / audit

grok mcp list
grok mcp doctor gorked

---

CLI reference

gorked capture [options]      Screenshot primary monitor
gorked analyze [image]        OCR + annotate + scene JSON
gorked find QUERY             Match text → center coords
gorked plan GOAL              Plan only (Grok CLI / heuristic)
gorked agent GOAL             Observe → plan → act loop
gorked drive GOAL             Full Grok+MCP computer-use loop
gorked demo                   Synthetic UI end-to-end
gorked monitors               List displays + capture policy
gorked mcp [--install]        Run or register MCP server
gorked hud [--live]           Launch C++ chat HUD

Useful flags: --out, --monitor, --heuristic, --no-tools, --model, --max-turns, --json.

---

Python API

from gorked.pipeline import VisionPipeline
from gorked.agent import AgentConfig, ComputerUseAgent
from gorked.drive import run_drive
from gorked import computer

# Vision
pipe = VisionPipeline("output")
scene, raw, annotated, journal = pipe.analyze_screen()
for r in scene.regions:
    print(r.text, r.click_point.as_tuple())

# Structured agent (dry-run)
agent = ComputerUseAgent(AgentConfig(goal="Click Settings", dry_run=True))
result = agent.run()

# Drive with Grok + MCP
run_drive("Open the browser settings", live=False)

# Low-level actions
ctrl = computer.ComputerController(dry_run=True)
ctrl.execute(computer.click_at(100, 200, rationale="example"))

---

Project layout

gorked/
├── src/gorked/
│   ├── capture.py      # mss screenshots (primary-only policy)
│   ├── ocr.py          # RapidOCR → TextRegion + center
│   ├── annotate.py     # boxes / crosshairs
│   ├── computer.py     # pynput mouse + keyboard
│   ├── models.py       # BBox, ScreenScene, PlannedAction, …
│   ├── thought.py      # thought journal → HUD bus
│   ├── hud_bus.py      # JSONL events for the C++ HUD
│   ├── grok_cli.py     # headless `grok` runner (streaming-json)
│   ├── planner.py      # structured plans + heuristic fallback
│   ├── agent.py        # observe → plan → act loop
│   ├── drive.py        # Grok + MCP drive mode
│   ├── mcp_server.py   # FastMCP tool surface
│   ├── pipeline.py     # capture → detect → annotate
│   └── cli.py          # typer entrypoints
├── hud/                # C++17 X11/Xft frameless messenger
│   ├── src/
│   ├── CMakeLists.txt
│   └── README.md
├── .grok/skills/       # computer-use skill for Grok Build
├── AGENTS.md           # project rules for Grok
├── pyproject.toml
└── README.md

Artifacts under output/ (gitignored): captures, annotated PNGs, scene JSON, thought traces, hud.jsonl.

---

Environment variables

Variable	Purpose
GORKED_MONITOR	primary (default) or index
GORKED_LIVE	1 → real input at MCP boot
GORKED_DRY_RUN	1 → simulate input
GORKED_OUTPUT_DIR	Artifact directory
GORKED_HUD_STREAM	Path to HUD JSONL stream
GORKED_HUD_LIVE	HUD-spawned drives use --live
GORKED_BIN	Path to gorked CLI for the HUD
GORKED_GROK_BIN	Path to grok if not on PATH
GORKED_ALLOW_ALL_MONITORS	Allow virtual dual-desktop capture
GORKED_ALLOW_MONITOR_OVERRIDE	Allow non-primary indices

Copy .env.example for a starting point (never commit real secrets).

---

Safety notes

• Dry-run is the default. Live control can move the mouse and type for real.
• Prefer keeping the Grok TUI / HUD from covering the UI the agent must click.
• Esc in the HUD aborts the active drive process group (SIGTERM → SIGKILL).
• Do not run live mode unattended on sensitive sessions without allowlists.

---

Development

source .venv/bin/activate
pip install -e ".[dev]"

# Reinstall after code changes (editable)
pip install -e .

# Rebuild HUD
cmake --build hud/build -j

---

Roadmap

• [x] Capture + OCR boxes + center click targets
• [x] Computer primitives + dry-run
• [x] Thought journal + HUD stream
• [x] Headless Grok CLI planner + drive mode
• [x] MCP tool surface for Grok
• [x] Frameless C++ chat HUD (send / abort / live)
• [ ] Multi-step re-OCR verification loops
• [ ] Action allowlists / forbidden zones
• [ ] Native Grok computer-use handoff when available

---

License

MIT — see LICENSE.

---

Acknowledgments

• Grok Build / xAI — headless agent CLI
• RapidOCR — ONNX text detection
• mss — screenshots
• pynput — input
• MCP — tool protocol