Unity Autonomous Agent MCP

🤖 A comprehensive autonomous agent framework for Unity 2022.3.22f1 that combines the power of Model Context Protocol (MCP) with advanced AI decision-making capabilities.

🌟 Key Features

🧠 Autonomous Agent Capabilities

• Intelligent Task Planning: AI-driven task decomposition and execution planning
• Decision Making: Context-aware decision trees with learning capabilities
• Self-Improvement: Continuous learning from user interactions and outcomes
• Multi-Agent Coordination: Support for multiple autonomous agents working together

🎮 Unity Integration

• Full Editor Control: Complete Unity Editor automation and manipulation
• Runtime AI: In-game autonomous NPC behavior and debugging
• Asset Management: Intelligent asset creation, optimization, and organization
• Scene Management: Automated scene setup, optimization, and testing

🔧 Advanced MCP Features

• Extensible Architecture: Plugin-based system for custom tools and handlers
• Real-time Communication: Low-latency TCP/IP communication with external AI services
• Multi-Provider Support: Compatible with Claude, GPT, Gemini, and custom LLM providers
• Resource Management: Intelligent resource allocation and optimization

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Unity Autonomous Agent MCP              │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐ │
│  │   Agent Core    │  │   Task Planner  │  │ Decision     │ │
│  │                 │  │                 │  │ Engine       │ │
│  │ - State Mgmt    │  │ - Task Decomp   │  │ - ML Models  │ │
│  │ - Learning      │  │ - Priority      │  │ - Context    │ │
│  │ - Memory        │  │ - Scheduling    │  │ - Reasoning  │ │
│  └─────────────────┘  └─────────────────┘  └──────────────┘ │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐ │
│  │  MCP Server     │  │  Unity Bridge   │  │ Plugin       │ │
│  │                 │  │                 │  │ System       │ │
│  │ - Protocol      │  │ - Editor API    │  │ - Custom     │ │
│  │ - Handlers      │  │ - Runtime API   │  │ Tools        │ │
│  │ - Transport     │  │ - Asset Mgmt    │  │ - Extensions │ │
│  └─────────────────┘  └─────────────────┘  └──────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                    Unity Engine (2022.3.22f1)              │
└─────────────────────────────────────────────────────────────┘

📋 Requirements

• Unity 2022.3.22f1 or later (tested up to Unity 6.1)
• .NET/C# 9.0
• Node.js 18.0.0+ with npm (for TypeScript server)
• Python 3.8+ (for ML/AI components)
• External LLM Provider (Claude, OpenAI, Gemini, or custom)

🚀 Quick Start

Re-analysis details and capability gaps are documented in docs/capability-matrix.md.

1. Installation

# Clone the repository
git clone https://github.com/KinofSin/UnityAutonomousMCP.git
cd UnityAutonomousMCP

# Install Unity Package
# In Unity Editor: Window > Package Manager > Add package from git URL
# Enter: file:///path/to/UnityAutonomousMCP/com.autonomous-unity.mcp

2. Setup TypeScript Server

npm install
npm run build
npm run smoke

npm run smoke validates planner + executor behavior (including run_tests -> get_test_job polling) for both successful and failed test-job terminal paths, without requiring a live Unity Editor.

3. Run modes

# Autonomous bootstrap (local dry-run using mock Unity bridge)
npm run dev -- -- "inspect scene and update scripts"

# MCP stdio mode (for Claude/Cursor/Windsurf MCP client config)
npm run dev -- --mcp

# Force mock bridge explicitly
npm run dev -- --mock -- "inspect scene and update scripts"

4. Transport bridge configuration (real Unity connection)

Unity package host (inside Unity Editor) serves both transports:

• HTTP endpoint: POST /mcp/tool on UNITY_HTTP_PORT (default 8080)
• TCP endpoint: newline-delimited JSON on UNITY_TCP_PORT (default 8081)

Node-side bridge environment variables:

UNITY_TRANSPORT=http       # http | tcp | mock
UNITY_HOST=127.0.0.1
UNITY_HTTP_PORT=8080
UNITY_TCP_PORT=8081
UNITY_TIMEOUT_MS=10000
UNITY_TEST_POLL_ATTEMPTS=30
UNITY_TEST_POLL_INTERVAL_MS=1000
UNITY_TEST_POLL_TIMEOUT_MS=120000

5. Configure Claude Desktop

{
  "mcpServers": {
    "unity-autonomous-agent": {
      "command": "node",
      "args": ["path/to/server/dist/index.js"],
      "env": {
        "UNITY_HOST": "localhost",
        "UNITY_TRANSPORT": "http",
        "UNITY_HTTP_PORT": "8080",
        "UNITY_TCP_PORT": "8081",
        "AI_PROVIDER": "anthropic",
        "AI_API_KEY": "your-api-key"
      }
    }
  }
}

6. Start Using

1. Open Unity Project
2. Navigate to Edit > Preferences > Autonomous Agent MCP
3. Configure host + HTTP/TCP ports
4. Click "Connect" to start Unity transport host
5. Start MCP server (npm run dev -- --mcp)
6. Begin interacting with AI through your preferred MCP client

✅ Current implementation status (this repo)

• Capability matrix + gap analysis: docs/capability-matrix.md
• Autonomous planner core: server/src/planner.ts
• Policy guardrails: server/src/policy.ts
• Plan executor: server/src/executor.ts
• MCP server tools (autonomous_plan, unity_tool_call, list_capabilities): server/src/mcpServer.ts
• Real Unity bridge transports (HTTP/TCP + env-driven mode): server/src/unityBridge.ts
• Concrete step contracts for autonomous_plan: server/src/contracts.ts
• Unity 2022.3.22f1 package scaffold: com.autonomous-unity.mcp/
  • Editor settings/provider: Editor/AutonomousMcpSettingsProvider.cs
  • Real transport host: Editor/AutonomousMcpTransportHost.cs
  • Tool dispatcher: Editor/AutonomousMcpToolDispatcher.cs
  • Runtime entry component: Runtime/AutonomousMcpRuntime.cs

🔁 End-to-end autonomous_plan contract mapping

autonomous_plan now emits concrete Unity tool payloads:

1. read_console → { level, limit }
2. manage_scene → { action: "inspect_active_scene" }
3. manage_script → { action: "create_or_update", scriptPath, contents } (when goal implies code changes)
4. validate_script → { strict } (after script edits)
5. run_tests → { mode } (when goal mentions tests; returns jobId)
6. get_test_job → { jobId } (executor polls until completed or failed)
7. batch_execute → { operations: [{ tool: "manage_scene", params: { action: "save_active_scene" } }] }

Test execution flow

1. run_tests starts an async Unity Test Runner job (editmode or playmode)
2. Server receives { jobId, status: "queued" }
3. Executor automatically calls get_test_job until terminal state
4. Final test summary (passed/failed/skipped + per-test details) is included in execution results

Test polling policy limits

• UNITY_TEST_POLL_ATTEMPTS: max polling iterations for get_test_job (clamped 1..300)
• UNITY_TEST_POLL_INTERVAL_MS: delay between polls in milliseconds (clamped 100..60000)
• UNITY_TEST_POLL_TIMEOUT_MS: overall timeout for polling cycle (clamped 1000..1800000)

🤖 Autonomous Agent Features

Intelligent Task Planning

• Goal Decomposition: Break complex tasks into manageable subtasks
• Priority Management: Dynamic task prioritization based on context
• Resource Allocation: Optimal distribution of system resources
• Dependency Resolution: Handle task dependencies and conflicts

Learning & Adaptation

• Pattern Recognition: Learn from user behavior and preferences
• Performance Optimization: Improve efficiency over time
• Error Recovery: Learn from mistakes and avoid repetition
• Context Awareness: Adapt to project-specific requirements

Multi-Agent Coordination

• Agent Communication: Coordinate multiple specialized agents
• Task Distribution: Distribute workload across agent instances
• Conflict Resolution: Handle competing priorities and resource conflicts
• Collaborative Problem Solving: Combine multiple agent perspectives

🔧 MCP Tools & Capabilities

🎮 Unity Editor Tools

• Scene Management: Create, modify, optimize scenes automatically
• Asset Operations: Intelligent asset creation and organization
• GameObject Control: Advanced object manipulation and optimization
• Component Management: Dynamic component addition and configuration
• Build Automation: Automated build processes and optimization

🧩 Advanced Development Tools

• Code Generation: AI-assisted script writing and optimization
• Testing Automation: Automated test creation and execution
• Performance Analysis: Real-time performance monitoring and optimization
• Debugging Assistant: Intelligent error detection and resolution
• Documentation Generation: Auto-generate technical documentation

🎯 Runtime AI Features

• NPC Behavior: Dynamic, intelligent NPC behavior systems
• Game Balance: Automated game balance testing and adjustment
• Player Analytics: Real-time player behavior analysis
• Dynamic Difficulty: Adaptive difficulty adjustment systems
• Content Generation: Procedural content creation and optimization

🔌 Plugin System

Create custom plugins to extend functionality:

[AutonomousPlugin("custom-tool")]
public class CustomToolPlugin : IAutonomousTool
{
    public async Task<ToolResult> ExecuteAsync(ToolContext context)
    {
        // Your custom tool logic
        return new ToolResult { Success = true, Data = result };
    }
}

📊 Performance & Monitoring

Real-time Metrics

• Agent Performance: Monitor agent efficiency and accuracy
• Resource Usage: Track CPU, memory, and network utilization
• Task Completion: Measure task success rates and timing
• Learning Progress: Track agent improvement over time

Optimization Features

• Caching System: Intelligent caching for frequently used data
• Batch Processing: Optimize multiple operations together
• Predictive Loading: Anticipate and preload required resources
• Adaptive Performance: Adjust performance based on system capabilities

🛡️ Security & Safety

Safety Measures

• Sandboxed Execution: Isolate agent operations from critical systems
• Permission System: Granular control over agent capabilities
• Audit Logging: Complete audit trail of all agent actions
• Rollback Capability: Undo system for agent modifications

Best Practices

• Regular Backups: Automatic backup before major operations
• Validation Checks: Verify operation safety before execution
• User Confirmation: Require confirmation for destructive operations
• Error Handling: Comprehensive error recovery mechanisms

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

1. Clone repository
2. Install dependencies (npm install in server directory)
3. Open Unity project
4. Enable Developer Mode in preferences
5. Start contributing!

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built upon the excellent foundations of Unity-MCP and UnityMCP
• Inspired by the latest advances in autonomous agents and AI decision-making
• Community feedback and contributions have been invaluable

---

🚀 Ready to transform your Unity development with AI-powered autonomy?

Start building intelligent, self-improving Unity projects today!