🐝 HiveMind - MCP Subagent Orchestrator

A high-concurrency, asynchronous MCP (Model Context Protocol) Orchestrator that coordinates multiple AI coding agents using the Copilot CLI as the underlying intelligence engine. Built on an Actor Model architecture for non-blocking operations, robust fault tolerance, and strict resource synchronization.

📋 Table of Contents

• Overview
• Key Features
• Architecture
• Components
• Installation
• Usage
• Configuration
• API Reference
• Development
• Project Structure
• Roadmap
• Contributing

🎯 Overview

HiveMind enables a Main Agent to spawn and coordinate multiple Subagents, each running isolated Copilot CLI instances. This allows for:

• Parallel task execution across multiple files/modules
• Hierarchical planning with dependency-aware scheduling
• Safe concurrent file access via lock management
• Unified observability with structured logging and tracing

Problem Solved

Traditional single-agent coding assistants struggle with:

• Large codebases requiring parallel analysis
• Multi-file refactoring with dependencies
• Long-running tasks that block the main thread
• Resource contention when multiple tools access files

HiveMind solves these by orchestrating a swarm of specialized agents that work concurrently while respecting file locks and execution order.

✨ Key Features

Feature	Description
🚀 Autonomous Execution	Submit a task and let the orchestrator plan, schedule, and execute without polling
🔒 File Locking (Warden)	Pessimistic locking with deadlock detection and wait queues
📊 Hierarchical Planning	Break complex tasks into dependency graphs with parallel execution groups
🔌 MCP Integration	Exposes orchestration capabilities as MCP tools
📈 Observability	Structured logging, OpenTelemetry tracing, and real-time metrics
💾 Persistence	SQLite-based checkpointing with crash recovery
🎛️ Dashboard	REST API + WebSocket for real-time monitoring

🏗️ Architecture

HiveMind uses a three-layer architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                              CONTROL PLANE                                  │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────┐  ┌─────────────────────┐ │
│  │ Orchestrator│  │Task Planner  │  │ Scheduler  │  │ Resource Warden     │ │
│  │    Core     │──│ (Decomposer) │──│            │──│ (Lock Manager)      │ │
│  └─────────────┘  └──────────────┘  └────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                               DATA PLANE                                    │
│  ┌──────────────────────────┐            ┌──────────────────────────────┐   │
│  │      Event Bus (IPC)     │            │      Trace Aggregator        │   │
│  │  Pub/Sub Message Passing │            │  Logs, Metrics, Spans        │   │
│  └──────────────────────────┘            └──────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             EXECUTION PLANE                                 │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐       ┌───────────────────┐   │
│  │ Agent Pod │  │ Agent Pod │  │ Agent Pod │  ...  │  Copilot CLI      │   │
│  │     1     │  │     2     │  │     N     │       │  (Child Process)  │   │
│  └───────────┘  └───────────┘  └───────────┘       └───────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘

Request Flow

1. Main Agent calls MCP tool (e.g., submit_task_auto)
2. Planner analyzes task, creates dependency graph
3. Scheduler creates execution groups respecting dependencies
4. Warden acquires file locks before each group executes
5. Agent Pods spawn Copilot CLI processes in parallel
6. Results aggregated and returned to Main Agent

🧩 Components

Core Components

Component	Location	Description
Orchestrator Core	src/core/	Central coordination, lifecycle management
Agent Manager	src/agents/	Agent pool, pod lifecycle, message routing
Event Bus	src/ipc/	Pub/sub messaging, async communication
Lock Manager	src/locks/	File locking, deadlock detection, wait queues
Planner	src/planner/	Task decomposition, dependency graphs
CLI Adapter	src/cli/	Copilot CLI spawning, I/O handling
MCP Layer	src/mcp/	Tool definitions, request handling
Tracing	src/tracing/	Structured logging, OpenTelemetry
Persistence	src/persistence/	SQLite, checkpointing, recovery
Dashboard	src/dashboard/	REST API, WebSocket server

Type System

Module	Description
src/types/agent.ts	Agent status, config, results
src/types/job.ts	Job lifecycle, subtasks, progress
src/types/lock.ts	Lock modes, handles, conflicts
src/types/events.ts	Event bus messages, channels
src/types/mcp.ts	MCP tool schemas, responses
src/types/cli.ts	CLI command construction, options

📦 Installation

Prerequisites

• Node.js 20+
• npm or yarn
• GitHub Copilot CLI installed and authenticated
• Git for version control

Install

# Clone the repository
git clone https://github.com/sureshsankaran/mcp-subagent-orchestrator.git
cd mcp-subagent-orchestrator

# Install dependencies
npm install

# Build the project
npm run build

Verify Installation

# Run tests
npm test

# Check build output
ls dist/

Quick Demo

Run the end-to-end demo to see all features in action:

# Run comprehensive demo showcasing all phases (1-7)
npm run demo:e2e

# Run basic component demo
npm run demo

The demo showcases:

• Event Bus - Pub/sub messaging with wildcard patterns
• State Machine - Agent lifecycle transitions
• Lock Manager - Resource locking with deadlock detection
• DAG - Task dependency graphs with parallel execution groups
• MCP Tools - 14+ orchestration tools exposed via MCP
• Structured Logging - Winston logger with JSON/pretty formats
• Distributed Tracing - OpenTelemetry integration
• Metrics Registry - Counters, histograms, and gauges
• Health Checks - Component health monitoring
• Full Orchestration - End-to-end task execution simulation

🚀 Usage

As an MCP Server

Register HiveMind as an MCP server in your AI tool configuration:

{
  "mcpServers": {
    "hivemind": {
      "command": "node",
      "args": ["path/to/mcp-subagent-orchestrator/dist/index.js"],
      "env": {
        "GH_TOKEN": "your-github-token"
      }
    }
  }
}

Programmatic Usage

import { OrchestratorCore, OrchestratorConfig } from 'mcp-subagent-orchestrator';

// Create orchestrator
const config: OrchestratorConfig = {
  maxConcurrentAgents: 5,
  workspacePath: '/path/to/workspace',
  persistence: {
    enabled: true,
    dbPath: '.hivemind/state.db'
  }
};

const orchestrator = new OrchestratorCore(config);
await orchestrator.initialize();

// Spawn a single agent
const agent = await orchestrator.spawnAgent('Fix the authentication bug in auth.ts');
const result = await orchestrator.waitForAgent(agent.id);

// Submit an autonomous job
const job = await orchestrator.submitJob('Refactor the entire API layer to use async/await');
const status = await orchestrator.waitForJob(job.id);

MCP Tool Examples

// Spawn a focused subagent
await mcp.callTool('spawn_agent', {
  task: 'Review auth.ts for security vulnerabilities',
  config: {
    system_instruction: 'You are a security expert. Be thorough.',
    allowed_paths: ['src/auth/'],
    read_only: true
  }
});

// Submit autonomous job (orchestrator handles everything)
await mcp.callTool('submit_task_auto', {
  instruction: 'Add comprehensive unit tests to all service files',
  strategy: 'parallel',
  max_agents: 4
});

// Monitor job progress
await mcp.callTool('get_job_status', { job_id: 'job-123' });

// Acquire file lock (for manual coordination)
await mcp.callTool('acquire_lock', {
  resource: 'src/api/users.ts',
  mode: 'exclusive',
  timeout_ms: 30000
});

⚙️ Configuration

Environment Variables

Variable	Description	Default
GH_TOKEN	GitHub token with Copilot access	Required
HIVEMIND_LOG_LEVEL	Logging level (debug/info/warn/error)	info
HIVEMIND_MAX_AGENTS	Maximum concurrent agents	10
HIVEMIND_DB_PATH	SQLite database path	.hivemind/state.db
HIVEMIND_PORT	Dashboard server port	3000
HIVEMIND_PREFERRED_CLI_TYPE	Override detected CLI type (copilot_v2, copilot_v1, codex, gh_copilot, claude)	auto-detect
HIVEMIND_PREFERRED_CLI_PATH	Absolute path to CLI binary (bypass detection)	auto-detect
OTEL_ENABLED	Enable OpenTelemetry	false

Orchestrator Config (file-first)

Configuration is loaded in this order: user overrides → env → hivemind.config.json → defaults.

Add a hivemind.config.json at the repo root to set project-wide defaults (used for both decomposition and spawned agents):

{
  "cli": {
    // Force a specific adapter type instead of auto-preference (copilot_v2 > copilot_v1 > codex > gh_copilot > claude)
    "preferred_cli_type": "codex",
    // Optional explicit binary path (skips detection)
    "preferred_cli_path": "/usr/local/bin/codex",
    // Existing options
    "cli_command": "copilot",
    "default_timeout_ms": 300000,
    "auto_approve_tools": true
  }
}

interface OrchestratorConfig {
  // Agent management
  maxConcurrentAgents: number;       // Max parallel agents
  agentTimeout: number;              // Agent execution timeout (ms)
  agentPoolSize: number;             // Pre-warmed agent pool size
  
  // Workspace
  workspacePath: string;             // Root workspace path
  trustedFolders: string[];          // Pre-trusted Copilot folders
  
  // Locking
  lockTimeout: number;               // Default lock acquisition timeout
  deadlockDetection: boolean;        // Enable deadlock detection
  
  // Persistence
  persistence: {
    enabled: boolean;
    dbPath: string;
    checkpointInterval: number;      // Checkpoint frequency (ms)
  };
  
  // Observability
  logging: {
    level: 'debug' | 'info' | 'warn' | 'error';
    format: 'json' | 'pretty';
  };
  
  // Security
  toolApproval: {
    allowedShellCommands: string[];
    deniedShellCommands: string[];
  };

  // CLI (decomposition + spawned agents)
  cli: {
    cli_command: string;
    preferred_cli_type?: 'copilot_v2' | 'copilot_v1' | 'codex' | 'gh_copilot' | 'claude' | 'unknown';
    preferred_cli_path?: string;
    default_timeout_ms: number;
    auto_approve_tools: boolean;
  };
}

📖 API Reference

MCP Tools

Tool	Description
spawn_agent	Spawn a single subagent with a task
submit_task_auto	Submit job for autonomous execution
get_job_status	Get current job status and progress
get_job_tree	Get hierarchical view of job subtasks
cancel_job	Cancel a running job
acquire_lock	Manually acquire a file lock
release_lock	Release a held lock
list_locks	List all active locks
stream_logs	Stream logs for job/agent
orchestrator_health	Get orchestrator health status

REST API Endpoints

Endpoint	Method	Description
/api/v1/agents	GET	List all agents
/api/v1/agents	POST	Spawn new agent
/api/v1/agents/:id	GET	Get agent details
/api/v1/agents/:id	DELETE	Kill agent
/api/v1/jobs	GET	List all jobs
/api/v1/jobs	POST	Submit new job
/api/v1/jobs/:id	GET	Get job details
/api/v1/jobs/:id/cancel	POST	Cancel job
/api/v1/locks	GET	List active locks
/api/v1/system/health	GET	Health check
/api/v1/system/metrics	GET	Prometheus metrics

🛠️ Development

Scripts

# Build TypeScript
npm run build

# Watch mode
npm run build:watch

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

# Lint code
npm run lint
npm run lint:fix

# Format code
npm run format

# Clean build artifacts
npm run clean

Testing

# Run all tests
npm test

# Run specific test file
npm test -- tests/unit/locks/LockManager.test.ts

# Run with coverage
npm run test:coverage

Code Quality

• ESLint for linting
• Prettier for formatting
• Jest for testing
• TypeScript strict mode enabled

📁 Project Structure

mcp-subagent-orchestrator/
├── src/
│   ├── agents/          # Agent pod management
│   ├── cli/             # Copilot CLI adapter
│   ├── core/            # Orchestrator core
│   ├── dashboard/       # REST API & WebSocket
│   ├── errors/          # Custom error classes
│   ├── ipc/             # Event bus & messaging
│   ├── locks/           # Lock manager (Warden)
│   ├── mcp/             # MCP tool definitions
│   ├── persistence/     # SQLite & checkpointing
│   ├── planner/         # Task decomposition
│   ├── tracing/         # Logging & telemetry
│   ├── types/           # TypeScript interfaces
│   ├── utils/           # Shared utilities
│   └── index.ts         # Main entry point
├── tests/
│   ├── unit/            # Unit tests
│   └── integration/     # Integration tests
├── tasks/               # Phase task breakdowns
├── dist/                # Compiled output
├── spec.md              # Full specification
├── TASKS.md             # Development roadmap
└── package.json

🗺️ Roadmap

The project is developed in 9 phases:

Phase	Name	Status	Tasks
1	Project Skeleton & Interfaces	✅ Complete	68
2	Nervous System (EventBus/IPC)	🔄 In Progress	52
3	CLI Abstraction Layer	⏳ Planned	68
4	File Warden (Lock Manager)	⏳ Planned	58
5	Hierarchical Planner	⏳ Planned	72
6	MCP Exposure	⏳ Planned	58
7	Observability	⏳ Planned	52
8	Persistence & Recovery	⏳ Planned	62
9	Dashboard & REST API	⏳ Planned	55

Total: ~542 detailed subtasks

See TASKS.md for detailed task breakdowns.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Follow TypeScript strict mode
• Write tests for new features
• Update documentation as needed
• Use conventional commit messages

📄 License

This project is licensed under the ISC License - see the LICENSE file for details.

---

<p align="center"> Built with 🐝 by the HiveMind team </p>