Mobile Agent Guardian

A five-level safety and verification layer for embodied mobile agents running in pi.

Mobile Agent Guardian turns raw Appium/MCP phone control into an auditable, verified protocol:

observe → classify → authorize → act → verify → record → anchor

It was designed for the unusual same-device flow where pi runs in Termux and controls that same Android phone through wireless ADB and Appium. It also works when Appium targets another Android or iOS device.

Architecture

                     ┌──────────────────────────────────────┐
                     │           Planning Agent              │
                     │  (decomposes tasks, delegates work)   │
                     └────────┬────────────────┬────────────┘
                              │ delegation     │ delegation
                              │ token          │ token
                     ┌────────▼────────┐  ┌────▼───────────┐
                     │  Mobile Worker  │  │  Browser Worker │
                     │  (Appium MCP)   │  │  (Playwright)   │
                     └────────┬────────┘  └────┬───────────┘
                              │                │
                     ┌────────▼────────────────▼───────────┐
                     │         Mobile Agent Guardian        │
                     │                                      │
                     │  L1  Vision classifier               │
                     │  L2  Verify-after-act loop            │
                     │  L3  Playwright policy bridge         │
                     │  L4  Cryptographic anchoring          │
                     │  L5  Multi-agent delegation           │
                     │                                      │
                     │  Policy · Audit chain · Redaction     │
                     └──────────┬───────────────┬──────────┘
                                │               │
                     ┌──────────▼──┐  ┌─────────▼──────────┐
                     │ Appium MCP  │  │ Remote Playwright   │
                     │ UiAutomator │  │ (Modal / headless)  │
                     └──────┬──────┘  └─────────┬──────────┘
                            │                   │
                     ┌──────▼──────┐  ┌─────────▼──────────┐
                     │   Device    │  │  Cloud Chromium     │
                     │ (ADB/WiFi)  │  │  (DOM + JS + Net)   │
                     └─────────────┘  └────────────────────┘

Five levels

Level 1: Vision classifier (src/vision.ts)

Sends screenshots to an OpenAI-compatible vision endpoint for structured risk classification. Returns labels: login, credentials, consent, payment, destructive, safe. Falls back to text heuristics from accessibility output when no vision API is configured.

GUARDIAN_VISION_API_KEY=sk-... GUARDIAN_VISION_BASE_URL=https://api.openai.com/v1 GUARDIAN_VISION_MODEL=gpt-4o-mini

Level 2: Verify-after-act loop (src/verifier.ts)

After every non-trivial mutation, captures a post-action fingerprint, diffs against the pre-action state, and classifies the outcome as expected, unexpected, unchanged, failed, or timeout. Two consecutive unexpected outcomes freeze the verifier—blocking all further mutations until the human observes the screen or runs /mobile-guard unfreeze.

Level 3: Playwright bridge (src/playwright-bridge.ts)

Exposes guardian_browse for web DOM access through a remote Playwright MCP endpoint. All calls pass through Guardian's policy engine and are audited. Configure with:

GUARDIAN_PLAYWRIGHT_URL=https://your-workspace--pi-frontend-mcp-web.modal.run/mcp

Level 4: Cryptographic anchoring (src/anchor.ts)

Periodically posts the latest audit hash + sequence number to a remote append-only endpoint, signed with HMAC-SHA256 using a device-local secret. Enables third-party verification that no events were removed from the chain.

GUARDIAN_ANCHOR_URL=https://your-anchor-endpoint/api/anchor
GUARDIAN_ANCHOR_INTERVAL=50

Level 5: Multi-agent delegation (src/delegation.ts)

Authority tokens scope what sub-agents can do:

• Allowed tool patterns (glob syntax)
• Maximum risk ceiling (critical is never delegatable)
• TTL and usage limits
• Auto-revocation on policy violation (configurable violation threshold)

Tools: guardian_delegate, guardian_revoke, guardian_check_delegation.

Install

pi install git:github.com/salus-ryan/mobile-agent-guardian

Or test a checkout:

npm install
pi -e ./src/index.ts

Quick start

# Minimal: text heuristics only, no external dependencies
pi -e ./src/index.ts

# With vision classification
GUARDIAN_VISION_API_KEY=sk-... pi -e ./src/index.ts

# With remote Playwright
GUARDIAN_PLAYWRIGHT_URL=https://... pi -e ./src/index.ts

# Monitor mode (log without enforcing)
MOBILE_GUARD_MODE=monitor pi -e ./src/index.ts

Behavior

Observation and navigation remain low-friction. When screen content contains authentication, OAuth, credential, permission, financial, or destructive indicators, Guardian changes state:

1. Classify: vision model or text heuristics label the screen.
2. Gate: sensitive mutation requires a fresh observation (< 30s) and human confirmation.
3. Act: the mutation executes.
4. Verify: post-action fingerprint is compared against pre-action state.
5. Freeze (if needed): two consecutive unexpected outcomes block further mutations.
6. Record: every step is hashed into the tamper-evident audit chain.
7. Anchor: periodically signed and posted to remote storage.

OAuth URLs trigger a human handoff boundary: the agent may open the provider, but subsequent taps or typing cross the guarded boundary.

Commands and tools

Name	Type	Purpose
/mobile-guard	Command	Show state, verifier, audit integrity
/mobile-guard unfreeze	Command	Clear verifier freeze
mobile_guard_status	Tool	Model-callable status and audit verification
guardian_browse	Tool	Policy-gated remote Playwright navigation
guardian_delegate	Tool	Create scoped delegation token
guardian_revoke	Tool	Revoke a delegation token
guardian_check_delegation	Tool	Check token authorization

Audit evidence

Logs are written with mode 0600 to:

~/.pi/agent/mobile-guardian/audit-YYYY-MM-DD.jsonl

Every record includes the previous record's SHA-256 hash. Editing, deleting, or reordering an interior event breaks verification. Raw page source and sensitive argument values are not stored; the log records hashes and byte counts.

Development

npm install
npm test        # 27 tests
npm run typecheck

Status

Reference implementation. Not a substitute for Android device policy, app sandboxing, or careful review of OAuth consent.

See THREAT_MODEL.md for security boundaries and limitations.

License

Apache-2.0