mcp-local-vision

Lets a text-only LLM read images and PDFs by delegating to a vision model running on your own GPU, through llama.cpp.

Built for the case where your coding model has no vision at all — GLM on the Z.AI coding plan, DeepSeek, most local models. The coding model stays where it is; this server becomes its eyes.

• No API keys, no cloud. Images never leave the machine.
• Free PDF text. Pages with a text layer are extracted by poppler — zero model calls, exact output, instant.
• Scanned pages fall back to vision. Rendered at 200 DPI and read by the VLM automatically.
• Gives the GPU back. llama-server is started on demand and stopped after an idle period, so a shared card is not held hostage between screenshots.
• Verbatim-first prompting. Tuned to transcribe exactly, not to summarize helpfully.

Why this exists

A model can only see an image if the provider serves a vision-capable model. On the Z.AI coding plan every model reports attachment: false:

zai-coding-plan:  glm-5.2, glm-5.2-highspeed, glm-4.7, glm-5-turbo   → text only

No prompt trick fixes that. Something with eyes has to do the looking, and on a 12GB card that something can be local.

Requirements

GPU	8GB VRAM or more (developed on an RTX 3060 12GB)
llama.cpp	built with your GPU backend, providing llama-server
Model	a vision GGUF plus its mmproj projector
poppler	sudo apt-get install -y poppler-utils — needed only for scanned PDFs
Node	20+

Building llama.cpp with CUDA

llama.cpp ships no Linux CUDA prebuilt (the CUDA release assets are Windows-only), so build it:

sudo apt-get install -y cmake ninja-build gcc-13 g++-13 nvidia-cuda-toolkit
git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/app/llama.cpp
cd ~/app/llama.cpp
cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86 \          # 86 = RTX 30xx; 89 = 40xx; 120 = 50xx
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-13 \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --target llama-server -j $(nproc)

gcc-13 matters: Ubuntu's CUDA 12.4 nvcc rejects GCC 15, which is the system default on recent Ubuntu. Pointing CMAKE_CUDA_HOST_COMPILER at 13 is the whole fix.

Getting the model

mkdir -p ~/models && cd ~/models
B=https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main
curl -L -O $B/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf         # 2.5GB — the language model
curl -L -o mmproj-4B-F16.gguf $B/mmproj-F16.gguf           # 0.8GB — the vision encoder

Both files are required. Without mmproj, llama-server loads fine and is simply blind — it will answer about an image it never saw. The projector stays at F16 on purpose: it is small, and quantizing the vision tower is what costs you fine print.

The 4B is the default because ~3.4GB of weights leaves the card usable for other work. Swap in the 8B (unsloth/Qwen3-VL-8B-Instruct-GGUF, 5.1GB + 1.2GB) via VISION_MODEL_PATH if the GPU is yours alone; the OCR gain is small (DocVQA 96.1 vs 95.3).

Install

git clone https://github.com/jshsakura/mcp-local-vision ~/app/mcp-local-vision
cd ~/app/mcp-local-vision && npm install
npm test        # builds fixtures, then exercises every tool over real MCP

Register with opencode

In ~/.config/opencode/opencode.json (global) or ./opencode.json (per project):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "local-vision": {
      "type": "local",
      "command": ["/home/you/app/mcp-local-vision/bin/launch.sh"],
      "enabled": true,
      "timeout": 15000
    }
  }
}

Use bin/launch.sh rather than node src/index.mjs: MCP clients spawn servers with a minimal environment, and if your node lives under nvm it will not be on PATH. The launcher finds it and survives nvm upgrades.

Register with Claude Code

claude mcp add local-vision -s user -- /home/you/app/mcp-local-vision/bin/launch.sh

Optional: a chat UI on the same model

Because llama-server speaks the OpenAI API, anything that talks to OpenAI can share it — including Open WebUI, which gives you a browser chat window with image upload against the same local model your coding agent uses.

This only works with the idle-sleep setup below, not the managed-process fallback: a chat UI needs something listening even when nothing has been asked yet.

docker run -d --network=host \
  -e PORT=8090 \
  -e OPENAI_API_BASE_URL=http://127.0.0.1:8080/v1 \
  -e OPENAI_API_KEY=sk-local \
  -e ENABLE_OLLAMA_API=False \
  -v open-webui:/app/backend/data \
  --name open-webui --restart unless-stopped \
  ghcr.io/open-webui/open-webui:main

--network=host is required so the container can reach a llama-server bound to 127.0.0.1; PORT=8090 moves Open WebUI off its default 8080, which llama-server already holds. Open http://localhost:8090 and create the admin account on first visit.

The model appears as qwen3-vl-4b-instruct with capabilities: ["completion","multimodal"], so the image attach button works. Idle polling from the UI does not wake the GPU.

Serving the tools to Open WebUI as well

Open WebUI (v0.6.31+) speaks MCP, but only over Streamable HTTP — it cannot spawn a stdio server. Run this one in HTTP mode alongside the stdio one:

VISION_MANAGED=0 mcp-local-vision --http 38765

Then register it in Admin Settings → External Tools → + Add Server:

• Connection Type: MCP (Streamable HTTP) — this part matters. The field defaults to OpenAPI, and pointing an OpenAPI connection at an MCP server gives you a crash or an infinite loading spinner rather than a useful error.
• URL: http://127.0.0.1:38765/mcp

Only administrators can add MCP servers; the permission that lets ordinary users add OpenAPI tools does not extend to these.

To check the server rather than the UI, run the handshake by hand — a healthy server answers 200 with an SSE event: message frame:

curl -i -X POST http://127.0.0.1:38765/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}'

Know what this does and does not give you. read_image takes a path on the server's filesystem. An image you drag into the chat window is an upload, not a path, so the tools cannot see it — that case is what the direct model connection above is for. The tools earn their keep on files already on disk ("read ~/docs/contract.pdf"), which is the same job they do in a coding agent. The two are complements, not alternatives.

Pass absolute paths over HTTP. In stdio mode the client spawns the server, so a relative path resolves against the client's working directory. Over HTTP the server is a long-lived process with a working directory of its own — under systemd, your home — and a relative path quietly resolves there instead. The tool descriptions say so, but it is worth knowing when a file "is not found" that plainly exists.

The HTTP endpoint has no authentication of its own and these tools read the filesystem, so it binds to 127.0.0.1 by default. If you move it off loopback with VISION_HTTP_HOST, set VISION_ALLOWED_ROOTS too.

Tools

read_image(path, question?)

Reads png, jpg, webp, gif, bmp, tiff, avif, heic. EXIF rotation is applied, transparency is flattened onto white (dark-mode screenshots with alpha otherwise read as black-on-black), and the long edge is capped at 1568px.

Omit question for a full transcription. Pass one when you only need a single detail — the model writes a short answer instead of a whole document, which keeps both latency and returned context small.

read_pdf(path, pages?, question?, mode?)

• pages — "1-5", "2,7,9", "3". Omit for the whole document.
• mode — auto (default: text layer where present, vision for scanned pages), text (never call the model), vision (render every page — use when the text layer is garbled or you need to read figures).

Text-layer pages are not capped; a 200-page report comes back in about a second. Only pages that need the vision model are capped (VISION_MAX_PAGES, default 5), and any page dropped by the cap is named in the output so the model knows to ask again with pages narrowed.

vision_doctor()

Reports the binary/model/projector paths and whether each exists, server state, and whether poppler is installed. Call it first when something fails.

GPU lifecycle

There are two ways to keep the GPU free between uses. Prefer the first.

llama.cpp's own idle sleep (recommended)

Recent llama-server takes --sleep-idle-seconds N: after N seconds with no task it unloads the model and KV cache, and the next request reloads it. The process stays up, so the endpoint answers the whole time — which is what lets a chat UI sit connected without pinning the card.

GET /health, /props and /models are explicitly exempt: they neither wake it nor reset the idle timer. A UI polling for the model list therefore costs nothing.

Measured on a 3060 12GB with the 4B model:

State	llama-server VRAM
Loaded	4676 MiB
Asleep	126 MiB (CUDA context only)

Waking costs ~5.7s on the next call. Run it under systemd and point this server at it with VISION_MANAGED=0:

# ~/.config/systemd/user/llama-vision.service
[Service]
Type=exec
ExecStart=/home/you/app/llama.cpp/build/bin/llama-server \
  --model /home/you/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj /home/you/models/mmproj-4B-F16.gguf \
  --alias qwen3-vl-4b-instruct \
  --host 127.0.0.1 --port 8080 --ctx-size 8192 \
  --sleep-idle-seconds 300 --no-warmup
Restart=on-failure

[Install]
WantedBy=default.target

systemctl --user daemon-reload && systemctl --user enable --now llama-vision
sudo loginctl enable-linger $USER    # survive logout

Process lifecycle managed here (fallback)

If your llama-server predates --sleep-idle-seconds, leave VISION_MANAGED=1 (the default). This process then starts llama-server on the first vision call and kills it after VISION_IDLE_MS. It frees VRAM just as well, but the endpoint is gone while idle — fine for MCP, useless for a chat UI that expects something listening.

Either way the idle timer resets on every call, so a burst of reads never expires mid-flow.

Two things keep the managed mode from crashing on a busy card:

• -ngl is not passed by default. Forcing -ngl 99 disables llama.cpp's automatic layer fitting — it logs n_gpu_layers already set by user to 99, abort and then dies with a CUDA OOM rather than offloading the overflow to CPU. Left unset, it sizes itself to whatever VRAM is actually free.
• Startup failures explain themselves. llama-server's stderr is captured, and an OOM is reported as an OOM with the tail of its log, not as exited with code 1.

If you would rather run llama-server yourself under systemd or in a tmux pane, set VISION_MANAGED=0 and this process will only ever health-check it.

Configuration

All optional, all via environment variables.

Variable	Default	Purpose
VISION_SERVER_URL	http://127.0.0.1:8080	llama-server base URL
VISION_MANAGED	1	Start/stop llama-server here. 0 = you run it
VISION_IDLE_MS	300000	Release the GPU after this idle time. 0 = never
VISION_SERVER_BIN	~/app/llama.cpp/build/bin/llama-server	Binary to launch
VISION_MODEL_PATH	~/models/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf	Language model GGUF
VISION_MMPROJ_PATH	~/models/mmproj-4B-F16.gguf	Vision projector GGUF
VISION_GPU_LAYERS	unset	-ngl. Leave unset — setting it disables auto-fit
VISION_CONTEXT	8192	-c context size
VISION_EXTRA_ARGS	—	Extra llama-server flags, space separated
VISION_MAX_EDGE	1568	Long-edge cap in px. Raise for dense small print
VISION_MAX_TOKENS	4096	Output cap per call
VISION_PDF_DPI	200	Rasterization DPI for scanned pages
VISION_MAX_PAGES	5	Cap on vision pages per call
VISION_TIMEOUT_MS	300000	Per-request timeout
VISION_STARTUP_TIMEOUT_MS	180000	How long to wait for a cold load
VISION_MAX_FILE_BYTES	209715200	Reject larger inputs
VISION_TEXT_PAGE_MIN_CHARS	50	Below this, a PDF page counts as scanned
VISION_ALLOWED_ROOTS	unset	Colon-separated directory allowlist. Unset = unrestricted
VISION_VERBOSE	0	Pass llama-server stderr through
VISION_API_KEY	unset	Bearer token, if llama-server runs with --api-key
VISION_HTTP_PORT	unset	Serve MCP over Streamable HTTP on this port instead of stdio
VISION_HTTP_HOST	127.0.0.1	Bind address for HTTP mode
VISION_HTTP_TOKEN	unset	Bearer token required on /mcp. Mandatory off loopback
VISION_NODE	unset	Explicit node path for bin/launch.sh

Model choice

Any vision GGUF with a matching mmproj works. Qwen3-VL is the default because its OCR is the strongest of the small models (DocVQA 96.1 at 8B) and it handles CJK well.

Model	Size	Notes
Qwen3-VL-4B-Instruct UD-Q4_K_XL	2.5GB + 0.8GB mmproj	Default. DocVQA 95.3
Qwen3-VL-8B-Instruct UD-Q4_K_XL	5.1GB + 1.2GB mmproj	DocVQA 96.1, if the card is free
MiniCPM-V 4.5	~6GB	Similar profile

Use the Instruct build, not the Thinking one. Thinking builds spend thousands of tokens reasoning before they transcribe — far slower for what is fundamentally a transcription job.

Limits

• One image per call; multi-image comparison is not exposed.
• Video is not supported even though the model accepts it.
• Handwriting is materially worse than print.
• The server reads any file the user running it can read unless VISION_ALLOWED_ROOTS is set.

License

MIT