tokenizer-mcp

tokenizer-mcp is a small MCP server that lets an LLM harness like Claude Code count the exact tokens in files (or any string) trivially easily, plus report basic file metrics — lines, characters, and size in KB. Every tool returns a single integer. For token counting the server receives the file/text and a model name and routes to the appropriate backend; the file-metric tools need no tokenization backend at all.

Installation

uv sync

Running it directly

uv run server.py

API key setup

Counting tokens for Claude requires an Anthropic API key, since the exact count comes from messages.count_tokens. Without a key, the Claude path silently falls back to tiktoken's o200k_base encoding. GPT counts work offline and need no key.

Set the key as an environment variable:

export ANTHROPIC_API_KEY=sk-ant-...

On Windows (PowerShell):

$env:ANTHROPIC_API_KEY = "sk-ant-..."

Or drop a .env file next to server.py and the server will load it on startup:

ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_TOKEN_COUNT_MODEL=claude-opus-4-8

ANTHROPIC_TOKEN_COUNT_MODEL sets the default Claude model when the caller omits one; it defaults to claude-opus-4-8. The endpoint tokenizes under the model you pass, so use a current model — the tokenizer changed at Opus 4.7 (~30% more tokens than older models for the same text).

What it exposes

Six tools, each returning an integer.

Token counting (uses the model routing / backends described below):

• count_tokens(text, model="") — count tokens in a string.
• count_tokens_file(file_path, model="") — count tokens in a UTF-8 file at an absolute path.
• count_tokens_folder(folder_path, model="") — sum of per-file token counts across every text file in a folder, recursively. Dependency/VCS/build directories (.git, node_modules, .venv, __pycache__, dist, ...) are excluded, and binary files are skipped — first by extension, then by a content sniff (NUL bytes / control-character ratio) for unrecognized extensions. Files that aren't UTF-8 are decoded as UTF-16 (when a BOM is present) or cp1252 rather than skipped.

When model is omitted, these use ANTHROPIC_TOKEN_COUNT_MODEL.

File metrics (no tokenization backend; file-only — there are deliberately no raw-string variants):

• count_lines_file(file_path) — number of lines, using Python str.splitlines() semantics (a trailing newline terminates the last line rather than adding an empty one).
• count_chars_file(file_path) — number of characters in the decoded UTF-8 text (Unicode code points).
• count_kb_file(file_path) — file size in kilobytes, ceil(bytes / 1024) with 1 KB = 1024 bytes (the same convention as the Tokenizer app). The size is read from the filesystem, so it works for any file regardless of encoding.

These are useful when a file's token count is only one of several limits — lines, characters, raw size — that decide whether it fits a given budget (e.g. a harness tool-call payload).

How model routing works

The lowercased model is matched against a short set of rules:

What you pass	Backend used
A raw tiktoken encoding (o200k_base, cl100k_base, p50k_base, p50k_edit, r50k_base, gpt2)	tiktoken.get_encoding
Anything starting with an OpenAI prefix (gpt-5, gpt-4, gpt-3.5, gpt-3, chatgpt, o1, o3, o4, text-davinci, text-embedding, code-davinci, text-curie, text-babbage, text-ada)	tiktoken.encoding_for_model
Everything else (Claude models)	Anthropic messages.count_tokens

Wiring it into an MCP client

Add an mcpServers entry pointing at this directory:

{
  "mcpServers": {
    "tokenizer": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/tokenizer-mcp", "run", "server.py"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

The env block is optional; omit it and Claude counts fall back to o200k_base, as above.