Vision MCP Server

A Model Context Protocol (MCP) server that gives vision understanding to agents connected to non-multimodal models (DeepSeek, older GPT-4, local small models, etc.): the agent hands an image to the MCP tool, the server calls a vision model, and returns text.

Supports major providers in China and the US plus any OpenAI-compatible endpoint. Official SDKs first, abstraction before implementation, zero-intrusion provider additions.

中文文档见 README.zh-CN.md

Features

• 4 tools: analyze_image / describe_image / ocr_image / list_providers, all returning plain Markdown text
• 13 built-in providers: OpenAI / Anthropic / Google Gemini / Qwen (DashScope) / Zhipu / Doubao (Volcengine) / ERNIE (Qianfan) / StepFun / Ollama / Alibaba Bailian / SiliconFlow / OpenRouter / custom OpenAI-compatible endpoint
• Three image inputs: local path / http(s) URL / base64 (data URI or raw base64), auto-sniffed
• Three-tier fallback chain: official SDK → OpenAI-compatible endpoint → native fetch (see SPEC §1)
• Stateless: every call is independent; images and results are never cached; keys are read from environment variables only

Quick start

Option A: npx (published to npm, no repo needed)

npx -y @inferai/vision-mcp

Option B: local build

git clone <repo> && cd vision-mcp
pnpm install
pnpm build
node dist/index.js

MCP configuration examples (stdio)

The server speaks the stdio transport: the MCP client spawns the process and exchanges JSON-RPC messages over stdin/stdout. Configure it wherever your client defines MCP servers:

• Claude Code: project-level .mcp.json or user-level ~/.claude.json (mcpServers key)
• Claude Desktop: claude_desktop_config.json
• Any MCP client (Cursor, self-built agents, etc.): same structure

npx version (available after the package is published):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": ["-y", "@inferai/vision-mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}

Local development (adjust the path; --env-file-if-exists=.env loads .env natively):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "node",
      "args": ["--env-file-if-exists=.env", "/absolute/path/to/vision-mcp/dist/index.js"],
      "env": {
        "OPENAI_API_KEY": "sk-..."
      }
    }
  }
}

With startup arguments (override provider defaults via argv, see below):

{
  "mcpServers": {
    "vision-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "@inferai/vision-mcp",
        "--default-provider=dashscope",
        "--siliconflow-api-key=sk-...",
        "--siliconflow-model=Qwen/Qwen2.5-VL-7B-Instruct"
      ],
      "env": {
        "DASHSCOPE_API_KEY": "sk-..."
      }
    }
  }
}

stdio notes:

• stdout carries the MCP protocol only — the server never prints logs there; diagnostics go to stderr
• the client manages the process lifecycle (spawn on start, kill on exit); no daemon needed
• first npx run downloads the package and may take a few seconds
• env variables can also come from the shell environment if the client inherits it (no env block needed)

Debug with MCP Inspector:

pnpm dlx @modelcontextprotocol/inspector node dist/index.js --xxx-api-key=xxx --xxx2-api-key=xxx

Setting variables

1. MCP config env block (recommended, most reliable across platforms) — write the variables into the env object above
2. .env file (local development) — copy .env.example to .env, fill it in, then node --env-file-if-exists=.env dist/index.js (Node 22 native, no dotenv needed)
3. Shell export — export OPENAI_API_KEY=sk-xxx then run

Providers without keys show as unavailable in list_providers and report the missing variable when called.

Publishing (before npx works)

pnpm publish          # or pnpm release (changeset flow)

Environment variables

Every provider's API_KEY, BASE_URL, and MODEL support environment overrides (convention: <PROVIDER_PREFIX>_API_KEY / <PROVIDER_PREFIX>_BASE_URL / <PROVIDER_PREFIX>_MODEL):

Provider	Environment variables	Default model
OpenAI	OPENAI_API_KEY, OPENAI_BASE_URL, OPENAI_MODEL	gpt-4o
Anthropic	ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, ANTHROPIC_MODEL	claude-sonnet-4-5
Google Gemini	GEMINI_API_KEY, GEMINI_BASE_URL, GEMINI_MODEL	gemini-2.5-flash
Alibaba DashScope	DASHSCOPE_API_KEY, DASHSCOPE_BASE_URL, DASHSCOPE_MODEL	qwen-vl-max
Zhipu	ZHIPU_API_KEY, ZHIPU_BASE_URL, ZHIPU_MODEL	glm-4v-flash (free)
Volcengine Doubao	VOLCENGINE_ARK_API_KEY, VOLCENGINE_ARK_BASE_URL, VOLCENGINE_ARK_MODEL	doubao-1.5-vision-pro
Baidu Qianfan	QIANFAN_API_KEY, QIANFAN_SECRET_KEY, QIANFAN_BASE_URL, QIANFAN_MODEL	ernie-4.5-vl-8k
StepFun	STEPFUN_API_KEY, STEPFUN_BASE_URL, STEPFUN_MODEL	step-1v
Ollama (local)	OLLAMA_BASE_URL, OLLAMA_MODEL	— (no built-in default; endpoint and model must be set)
Alibaba Bailian	BAILIAN_API_KEY, BAILIAN_BASE_URL (default DashScope compatible mode), BAILIAN_MODEL	qwen-vl-max
SiliconFlow	SILICONFLOW_API_KEY, SILICONFLOW_BASE_URL (default https://api.siliconflow.cn/v1), SILICONFLOW_MODEL	Qwen/Qwen2.5-VL-72B-Instruct
OpenRouter	OPENROUTER_API_KEY, OPENROUTER_BASE_URL (default https://openrouter.ai/api/v1), OPENROUTER_MODEL	openai/gpt-4o
Custom compatible	OPENAI_COMPAT_BASE_URL, OPENAI_COMPAT_API_KEY?, OPENAI_COMPAT_MODEL	—

? = optional (has a built-in default); * = required.

Global configuration:

Environment variable	Default	Description
VISION_MCP_DEFAULT_PROVIDER	first available	Default provider
VISION_MCP_DEFAULT_MODEL	provider default	Default model
VISION_MCP_PROVIDER_PRIORITY	table order	Provider priority (comma-separated, high first, e.g. openai,dashscope,zhipu)
VISION_MCP_MAX_RETRIES	0 (off)	Per-provider retry count before falling back
VISION_MCP_MAX_FALLBACKS	0 (off)	Max provider fallbacks before giving up
VISION_MCP_MAX_IMAGE_BYTES	20 MB	Image size limit
VISION_MCP_TIMEOUT_MS	60000	Download & request timeout (ms)

Fallback chain

When multiple providers are available, calls walk the priority chain: configured default → VISION_MCP_PROVIDER_PRIORITY list → table order (unavailable providers are skipped).

• each provider is retried up to VISION_MCP_MAX_RETRIES times on provider errors (upstream failures, timeouts)
• after a provider exhausts its retries, the next available provider in the chain is tried, up to VISION_MCP_MAX_FALLBACKS fallbacks
• only provider errors trigger retry/fallback; config or image errors fail fast
• an explicitly requested provider argument is tried alone (no fallback)
• when everything fails, the error lists every provider attempted and its last error

Also available as argv: --provider-priority=..., --max-retries=N, --max-fallbacks=N (beat env vars).

MCP startup arguments (argv)

Every provider's apiKey / baseUrl / model can be overridden via startup arguments (higher priority than environment variables), format --<provider>-<field>:

node dist/index.js \
  --openai-api-key=sk-xxx \
  --openai-base-url=https://my-gateway.example.com/v1 \
  --openai-model=gpt-4o-mini \
  --dashscope-api-key=sk-xxx \
  --default-provider=dashscope

• Global: --default-provider <name> / --default-model <name>
• Per provider: --<provider>-api-key, --<provider>-base-url, --<provider>-model (equals or space form both work)
• Any OpenAI-compatible third-party service: wire it up in one line with --openai-compat-base-url + --openai-compat-api-key + --openai-compat-model; or point any built-in provider's base-url at a mirror/proxy

Priority: tool args provider/model > startup args (per-provider > global default) > environment variables > provider built-in defaults.

Tools

Tool	Arguments	Description
analyze_image	image*, prompt?, provider?, model?	General image analysis
describe_image	image*, provider?, model?	Describe image content (default instruction)
ocr_image	image*, language? (auto/zh/en/zh-en), provider?, model?	OCR, preserving layout
list_providers	—	Provider list and configuration status

image accepts: local path / http(s):// URL / data: URI / raw base64, auto-sniffed.

Security note: URL downloads are SSRF-protected — every hop (including redirects) is validated and URLs resolving to loopback, private, or link-local addresses are blocked (hint in the error explains why).

Provider integration (three-tier fallback chain)

provider	Integration	Notes
openai / stepfun / ollama / bailian / siliconflow / openrouter / openai-compat	OpenAI-compatible adapter (openai SDK)	One adapter, configurable baseURL
anthropic	Official SDK @anthropic-ai/sdk	messages + image content block
gemini	Official SDK @google/generative-ai	generateContent + inlineData
dashscope	Native fetch	official npm package has no vision; direct multimodal-generation API
zhipu	Native fetch	official SDK accepts string content only; direct v4 API
volcengine	Native fetch	official openapi is a management plane; direct Ark API
qianfan	Native fetch	official SDK is string-only; AK/SK → token → v2 API

Adding a provider: for OpenAI-compatible endpoints, add one row to RULES in src/core/config.ts plus one mapping in the factory table in src/index.ts — zero new code. Official SDK or native fetch implementations: see SPEC §1.

Development

pnpm check        # biome checks
pnpm test         # rstest unit tests (injected mocks, no network)
pnpm build        # rslib build

Real-call smoke tests (only run against providers whose keys are configured; skipped otherwise):

OPENAI_API_KEY=sk-... pnpm exec rstest tests/e2e

Architecture

src/
├── index.ts            # Entry: composition root, stdio startup
├── core/               # Abstraction: interfaces / image loading / config / registry
├── providers/          # Adapters: official SDK or compatible endpoints, protocol conversion only
└── server/tools.ts     # MCP tool layer: zod validation + error mapping

Full spec: SPEC.md.