Prometheus MCP Server

An MCP (Model Context Protocol) server for querying AWS Managed Prometheus (AMP) with SigV4 authentication. This server enables AI assistants to execute PromQL queries and discover metrics in a secure, VPC-isolated environment.

Features

• SigV4 Authentication: Automatically signs requests using AWS credentials (supports EKS Pod Identity/IRSA)
• 5 MCP Tools:
  • query_instant - Execute instant PromQL queries
  • query_range - Execute range queries for time series data
  • list_labels - Get all label names
  • get_label_values - Get values for a specific label
  • list_metrics - Get all metric names with optional metadata
• VPC Isolation: Designed to run inside a VPC with no public exposure
• Production Ready: Includes Terraform, Kubernetes manifests, and comprehensive testing

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Customer VPC                            │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                      EKS Cluster                          │  │
│  │  ┌─────────────────┐    ┌─────────────────────────────┐   │  │
│  │  │  Prometheus MCP │◄───│  ClusterIP Service :8080    │   │  │
│  │  │  (Pod Identity) │    │  (Internal Only)            │   │  │
│  │  └────────┬────────┘    └─────────────────────────────┘   │  │
│  │           │                                                │  │
│  └───────────┼────────────────────────────────────────────────┘  │
│              │ SigV4 Signed HTTP                                │
│              ▼                                                   │
│  ┌───────────────────────┐                                      │
│  │ AWS Managed Prometheus│                                      │
│  └───────────────────────┘                                      │
└─────────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

• Python 3.11+
• AWS CLI configured with credentials
• Docker (for building container images)
• Terraform 1.5+ (for infrastructure deployment)
• kubectl (for Kubernetes deployment)
• An SSH key pair in AWS

Local Development

# Clone and install
cd prometheus-mcp
pip install -e ".[dev]"

# Set environment variables
export PROMETHEUS_WORKSPACE_ID="ws-your-workspace-id"
export AWS_REGION="us-east-1"

# Run the server
python -m prometheus_mcp.server

Run Tests

# Install dev dependencies
pip install -e ".[dev]"

# Run unit tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=prometheus_mcp --cov-report=html

Production Deployment

Step 1: Deploy Infrastructure with Terraform

cd deploy/terraform

# Initialize Terraform
terraform init

# Review the plan
terraform plan -var="ssh_key_name=your-key-name"

# Apply (creates VPC, EKS, AMP, ECR, Bastion)
terraform apply -var="ssh_key_name=your-key-name"

Resources Created:

• VPC with public/private subnets
• EKS cluster with managed node group
• AWS Managed Prometheus workspace
• ECR repository
• Bastion host for SSH access
• IAM roles with Pod Identity

Step 2: Build and Push Docker Image

# Get ECR login command from terraform output
$(terraform output -raw docker_login_command)

# Build the image
docker build -t prometheus-mcp .

# Tag and push
ECR_URL=$(terraform output -raw ecr_repository_url)
docker tag prometheus-mcp:latest $ECR_URL:latest
docker push $ECR_URL:latest

Step 3: Deploy to Kubernetes

# Update kubeconfig
$(terraform output -raw kubeconfig_command)

# Get values for K8s manifests
export ECR_URL=$(terraform output -raw ecr_repository_url)
export WORKSPACE_ID=$(terraform output -raw amp_workspace_id)
export ROLE_ARN=$(terraform output -raw prometheus_mcp_role_arn)

# Update manifests with actual values
sed -i '' "s|\${ECR_URL}|$ECR_URL|g" deploy/k8s/deployment.yaml
sed -i '' "s|\${WORKSPACE_ID}|$WORKSPACE_ID|g" deploy/k8s/deployment.yaml
sed -i '' "s|\${PROMETHEUS_MCP_ROLE_ARN}|$ROLE_ARN|g" deploy/k8s/service-account.yaml

# Apply manifests
kubectl apply -f deploy/k8s/

Step 4: Verify Deployment

# Check pods are running
kubectl get pods -n prometheus-mcp

# Check logs
kubectl logs -n prometheus-mcp -l app=prometheus-mcp

Testing via SSH Tunnel

Since the MCP server is only accessible within the VPC, use SSH tunneling to test from your laptop.

Method 1: Manual Setup (3 Terminals)

Terminal 1 - SSH to Bastion and Port-Forward:

# SSH to bastion
ssh -i ~/.ssh/your-key.pem ec2-user@<BASTION_IP>

# On the bastion, set up kubectl
~/setup-kubectl.sh

# Start port-forward
~/port-forward-mcp.sh

Terminal 2 - SSH Tunnel:

ssh -i ~/.ssh/your-key.pem -L 8080:localhost:8080 ec2-user@<BASTION_IP>

Terminal 3 - MCP Inspector:

npx @anthropic/mcp-inspector http://localhost:8080

Method 2: Using the Test Script

cd deploy/scripts
./test-via-tunnel.sh

This script will display all the commands you need to run.

Verify VPC Isolation

# This should FAIL (timeout) - proves VPC isolation
kubectl get pod -n prometheus-mcp -o jsonpath='{.items[0].status.podIP}'
curl http://<POD_IP>:8080/health --connect-timeout 5
# Expected: Connection timed out

# This should SUCCEED (via SSH tunnel)
curl http://localhost:8080/health
# Expected: {"status": "healthy"}

MCP Tools Reference

query_instant

Execute an instant PromQL query at a single point in time.

{
  "name": "query_instant",
  "arguments": {
    "query": "up",
    "time": "2024-01-15T10:00:00Z"  // optional
  }
}

query_range

Execute a range query to get time series data.

{
  "name": "query_range",
  "arguments": {
    "query": "rate(http_requests_total[5m])",
    "start": "2024-01-15T00:00:00Z",
    "end": "2024-01-15T12:00:00Z",
    "step": "1m"
  }
}

list_labels

Get all label names.

{
  "name": "list_labels",
  "arguments": {
    "match": ["up", "http_requests_total"]  // optional
  }
}

get_label_values

Get all values for a specific label.

{
  "name": "get_label_values",
  "arguments": {
    "label_name": "job",
    "match": ["{namespace='production'}"]  // optional
  }
}

list_metrics

Get all metric names.

{
  "name": "list_metrics",
  "arguments": {
    "with_metadata": true  // optional, slower but includes type/help/unit
  }
}

Configuration

Environment Variables

Variable	Description	Default
PROMETHEUS_WORKSPACE_ID	AMP workspace ID (required)	-
AWS_REGION	AWS region	us-east-1

Terraform Variables

Variable	Description	Default
aws_region	AWS region	us-east-1
ssh_key_name	SSH key pair name (required)	-
allowed_ssh_cidr	CIDR for SSH access	0.0.0.0/0
eks_cluster_version	Kubernetes version	1.29
eks_node_instance_type	EKS node instance type	t3.medium
bastion_instance_type	Bastion instance type	t3.micro

Project Structure

prometheus-mcp/
├── prometheus_mcp/
│   ├── __init__.py
│   ├── server.py              # MCP server entry point
│   ├── amp_client.py          # SigV4-authenticated AMP client
│   └── promql/
│       ├── __init__.py
│       ├── models.py          # Pydantic response models
│       └── tools.py           # MCP tool implementations
├── deploy/
│   ├── terraform/             # Infrastructure as Code
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   ├── vpc.tf
│   │   ├── eks.tf
│   │   ├── amp.tf
│   │   ├── ecr.tf
│   │   ├── iam.tf
│   │   └── bastion.tf
│   ├── k8s/                   # Kubernetes manifests
│   │   ├── namespace.yaml
│   │   ├── service-account.yaml
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── scripts/
│       ├── push-sample-metrics.sh
│       ├── test-via-tunnel.sh
│       └── cleanup.sh
├── tests/
│   ├── conftest.py
│   ├── test_amp_client.py
│   └── test_promql_tools.py
├── Dockerfile
├── pyproject.toml
└── README.md

Cleanup

To destroy all resources:

cd deploy/scripts
./cleanup.sh

Or manually:

# Delete K8s resources
kubectl delete -f deploy/k8s/

# Destroy Terraform infrastructure
cd deploy/terraform
terraform destroy

Security Considerations

1. VPC Isolation: The MCP server is only accessible via ClusterIP service within the EKS cluster
2. Pod Identity (IRSA): Uses AWS IAM roles for service accounts instead of static credentials
3. Least Privilege: IAM role only has permissions to query AMP, not write
4. No Public Endpoints: All access is via SSH tunnel through bastion
5. Container Security: Runs as non-root user with read-only filesystem

Troubleshooting

Pod not starting

kubectl describe pod -n prometheus-mcp -l app=prometheus-mcp
kubectl logs -n prometheus-mcp -l app=prometheus-mcp

IAM permissions issues

# Verify the service account has the correct annotation
kubectl get sa -n prometheus-mcp prometheus-mcp -o yaml

# Check if Pod Identity is working
kubectl exec -n prometheus-mcp -it <pod-name> -- aws sts get-caller-identity

Cannot connect via SSH tunnel

# Verify bastion is running
aws ec2 describe-instances --filters "Name=tag:Name,Values=prometheus-mcp-bastion" --query "Reservations[].Instances[].State.Name"

# Check security group allows SSH
aws ec2 describe-security-groups --group-ids <bastion-sg-id>

License

MIT