redact — PII Redaction & Summarization Service

A self-contained service for extracting text from documents, detecting and redacting PII across multiple languages, summarizing the clean output, and exposing all capabilities to AI agents via MCP. All activity is traced to Azure Log Analytics Workspace.

---

Project structure

├── src/
│   ├── api.py                    # FastAPI app initialisation + router registration
│   ├── models.py                 # All Pydantic request/response DTOs
│   ├── constants.py              # Env var keys + hardcoded constants
│   ├── errors/
│   │   ├── exceptions.py         # Domain exceptions
│   │   └── exception_handlers.py # Global HTTP error handlers
│   ├── routers/
│   │   ├── health.py             # GET /health, /entities, /languages
│   │   ├── redaction.py          # POST /redact, /process
│   │   ├── summarization.py      # POST /summarize (quota enforced)
│   │   └── patterns.py           # GET|POST /patterns, DELETE /patterns/{id}, POST /patterns/test
│   ├── redaction/
│   │   ├── extractor.py          # File → plain text
│   │   ├── redactor.py           # Presidio PII detection + redaction
│   │   └── custom_patterns.py    # Per-business-unit pattern CRUD + persistence
│   ├── summarization/
│   │   └── summarizer.py         # Claude-powered summarization
│   └── util/
│       ├── auth.py               # API key dependency
│       ├── logging_config.py     # Structured logging + Azure Log Analytics handler
│       ├── rate_limit.py         # slowapi limiter + per-key identification
│       └── quota.py              # Daily LLM quota tracker (in-memory / Redis)
│
├── mcp_server/                   # MCP server — AI agent access layer
│   ├── app.py                    # Shared FastMCP instance
│   ├── server.py                 # ASGI app factory (SSE + auth middleware + /health)
│   ├── middleware.py             # MCP_API_KEY auth middleware
│   └── tools/
│       ├── redact.py             # redact_text, list_entities, list_languages
│       ├── summarize.py          # summarize_text
│       └── patterns.py           # list/add/delete/test custom patterns
│
├── tests/
│   ├── conftest.py
│   ├── unit/                     # Mocked, no external services needed
│   ├── integration/              # Full app via TestClient
│   └── utils/                    # Shared fixtures + mock data
│
├── data/
│   └── custom_patterns.json      # Persisted user-defined patterns (auto-created)
│
├── quota_config.json             # Per-team rate limits and LLM quotas
├── index.html                    # Single-page UI (no build step)
├── main.py                       # REST entry point shim → src.api:api
├── mcp_main.py                   # MCP SSE entry point → mcp_server.server:app (port 8001)
├── mcp_stdio.py                  # MCP stdio entry point → for Claude Desktop
├── Dockerfile                    # Single image, two startup commands
├── docker-compose.yml            # REST + MCP as separate scalable services
├── requirements.txt
└── .env.example

---

Quick start

1. Install dependencies

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_lg

2. Configure environment

cp .env.example .env
# Edit .env — set ANTHROPIC_API_KEY and optionally Azure + auth credentials

3. Run the REST API

uvicorn src.api:api --host 0.0.0.0 --port 8000 --reload

4. Run the MCP server (separate process)

uvicorn mcp_main:app --host 0.0.0.0 --port 8001 --reload

5. Open the UI

open index.html
# or: python -m http.server 3000 then visit http://localhost:3000

The UI expects the backend at http://localhost:8000.

6. Run both with Docker

docker compose up                        # start REST + MCP
docker compose up --scale mcp-server=3  # scale MCP independently

---

Architecture

Browser (index.html)                   AI Agents / Claude Desktop
        │                                        │
        │  REST (port 8000)                      │  MCP stdio / SSE (port 8001)
        ▼                                        ▼
FastAPI  src/api.py                    MCP Server  mcp_server/server.py
        │                                        │
        ├── src/routers/               ├── mcp_server/tools/redact.py
        ├── src/redaction/             ├── mcp_server/tools/summarize.py
        ├── src/summarization/         └── mcp_server/tools/patterns.py
        └── src/util/                            │
            ├── rate_limit.py                    │
            ├── quota.py                         │
            └── logging_config.py               │
                │                               │
                └──────── shared ───────────────┘
                       src/redaction/
                       src/summarization/
                       src/redaction/custom_patterns.py
                               │
                       Azure Log Analytics

Two transport layers, one business logic layer

Transport	Entry point	Port	Used by
REST (HTTP)	main.py → src.api:api	8000	Browser UI, direct REST clients
MCP SSE	mcp_main.py → mcp_server.server:app	8001	Remote AI agents, Trimble-wide access
MCP stdio	mcp_stdio.py	—	Claude Desktop (local subprocess)

Both MCP transports expose the same 9 tools backed by the same src/ services.

Request flow

Router / Tool → Service → (external call if needed)
    │                  └→ logger.info(..., extra={trace fields})
    │                          └→ AzureLogAnalyticsHandler → Log Analytics
    │
    ├── Rate limit check (slowapi — burst guard per API key)
    └── Quota check (quota.py — daily LLM calls per team)

---

Environment variables

Variable	Required	Description
ANTHROPIC_API_KEY	No	Claude API key — omit to use extractive summarization fallback
API_KEY	No	REST API key — all requests must include it in X-API-Key header. Unset = auth disabled
API_KEY_HEADER	No	Header name for REST API key (default: X-API-Key)
MCP_API_KEY	No	MCP server key — agents must send Authorization: Bearer <key> or X-MCP-Key: <key>. Unset = auth disabled
REDIS_URL	No	Redis connection URL for distributed rate limiting across pods (e.g. redis://localhost:6379/0). Unset = in-memory
AZURE_LOG_WORKSPACE_ID	No	Log Analytics workspace ID — omit to log to stdout only
AZURE_LOG_WORKSPACE_KEY	No	Log Analytics primary/secondary shared key (base64)
AZURE_LOG_TYPE	No	Custom table name in Log Analytics (default: PIIRedactionTrace)

---

Rate Limiting & Quotas

The service enforces two independent limits to prevent abuse and control LLM costs.

1. Burst rate limits (slowapi)

Applied per API key (falls back to IP when auth is disabled). Each key gets its own counter bucket — teams don't share limits.

Endpoint	Default limit
POST /redact	60 requests / minute
POST /process	30 requests / hour
POST /summarize	10 requests / hour

When exceeded, the response is HTTP 429 with standard X-RateLimit-* headers.

2. Daily LLM quota (cost control)

Tracks Claude API calls per team per UTC day. Resets automatically at midnight — no cron job needed. Uses in-memory storage by default; set REDIS_URL for multi-pod deployments.

Team	Summarize calls / day
Default (unlisted key)	50
Construction / Agriculture / Geospatial	200
HR / Legal	100
Admin	1000

Quota response headers

Every /summarize response includes:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 7
X-Quota-Summarize-Limit: 50
X-Quota-Summarize-Used: 12
X-Quota-Summarize-Remaining: 38
X-Quota-Reset: 2026-04-06

Configuring per-team limits (quota_config.json)

Edit quota_config.json at the project root. Changes take effect on server restart.

{
  "default": {
    "requests_per_minute": 60,
    "process_per_hour": 30,
    "summarize_per_day": 50
  },
  "teams": {
    "their-api-key": {
      "name": "New Team",
      "requests_per_minute": 120,
      "process_per_hour": 60,
      "summarize_per_day": 200
    }
  }
}

Production — Redis backend

# .env
REDIS_URL=redis://your-redis-host:6379/0

Rate limit counters are then shared across all pods — no per-pod drift.

Testing rate limits

# Single call — inspect quota headers
curl -i -X POST http://localhost:8000/summarize \
  -H "Content-Type: application/json" \
  -d '{"redacted_text": "A person works at a company.", "length": "short"}' \
  | grep -E "X-Quota|X-RateLimit|HTTP"

# Python test — 6 calls against a quota of 3 (set summarize_per_day: 3 in quota_config.json)
python3 - << 'EOF'
import httpx, time
for i in range(6):
    r = httpx.post("http://localhost:8000/summarize",
        json={"redacted_text": "Test text.", "length": "short"}, timeout=15)
    used = r.headers.get("X-Quota-Summarize-Used", "?")
    rem  = r.headers.get("X-Quota-Summarize-Remaining", "?")
    lim  = r.headers.get("X-Quota-Summarize-Limit", "?")
    print(f"Call {i+1}: HTTP {r.status_code} | quota {used}/{lim} | remaining {rem}")
    time.sleep(0.2)
EOF

Expected output:

Call 1: HTTP 200 | quota 1/3 | remaining 2
Call 2: HTTP 200 | quota 2/3 | remaining 1
Call 3: HTTP 200 | quota 3/3 | remaining 0
Call 4: HTTP 429 | quota 3/3 | remaining 0   ← blocked
Call 5: HTTP 429 | quota 3/3 | remaining 0
Call 6: HTTP 429 | quota 3/3 | remaining 0

---

MCP Server

Overview

The MCP server runs as a separate process on port 8001 and exposes PII redaction capabilities to AI agents via the Model Context Protocol. It shares all business logic with the REST API — only the transport layer differs.

MCP tools

Tool	Description
redact_text	Detect and redact PII from plain text (all operators, all languages)
list_entities	List all supported PII entity types including active custom patterns
list_languages	List supported languages and spaCy model install status
summarize_text	Summarize PII-free text via Claude (or extractive fallback)
list_business_units	List the four supported business units
list_patterns	List patterns filtered by business unit
add_custom_pattern	Create and persist a new custom PII pattern
delete_custom_pattern	Delete a user-defined pattern by ID
test_regex_pattern	Test regex(es) against sample text without persisting

Authentication

# SSE transport (remote agents)
Authorization: Bearer <MCP_API_KEY>
# or
X-MCP-Key: <MCP_API_KEY>

The /health endpoint on port 8001 is always public (no auth required).

Claude Desktop setup

1. Open ~/Library/Application Support/Claude/claude_desktop_config.json
2. Add:

{
  "mcpServers": {
    "pii-redaction": {
      "command": "/path/to/project/.venv/bin/python3",
      "args": ["/path/to/project/mcp_stdio.py"]
    }
  }
}

3. Restart Claude Desktop — the 🔨 hammer icon will show all 9 tools under pii-redaction.

Remote agent setup (SSE)

{
  "mcpServers": {
    "pii-redaction": {
      "url": "http://mcp-server:8001/sse",
      "headers": {
        "Authorization": "Bearer your-mcp-secret-key"
      }
    }
  }
}

Example agent prompts (Claude Desktop)

Using pii-redaction, redact then summarize:
"John Smith (EMP-10029) submitted bid BID-2024-008821 from john@trimble.com.
Contract CNT-2024-REF00812, SSN 123-45-6789."

Using pii-redaction, add a custom pattern for construction:
entity CONSTR_PROJECT_CODE, regex \bPROJ-[A-Z]{2}-\d{4}-\d{4}\b,
context "project code, proj id". Then redact: "Project PROJ-TX-2024-0091 assigned."

---

Custom PII Patterns

Per-business-unit regex patterns extend the built-in Presidio recognizers. Patterns persist to data/custom_patterns.json and activate on the next redact call.

Business units

Unit	ID	Built-in entities
Construction	construction	CONSTR_BID_NUMBER, CONSTR_PERMIT_NUMBER, CONSTR_SUBCONTRACTOR_ID, CONSTR_SITE_CODE
Agriculture	agriculture	AGRI_FARM_ID, AGRI_PARCEL_ID, AGRI_YIELD_RECORD
Geospatial	geospatial	GEO_SURVEY_LICENSE, GEO_CONTROL_POINT_ID
HR / Legal	hr_legal	HR_EMPLOYEE_ID, HR_COST_CODE, HR_CONTRACT_REF

Admin UI

Click Custom Patterns (gear icon) in the top-right header of the UI:

• Browse built-in patterns per business unit
• Add custom patterns with regex, context keywords, and confidence score
• Test regex against sample text before saving
• Delete user-defined patterns

API

# List all patterns for a unit
curl http://localhost:8000/patterns?unit=construction

# Add a custom pattern
curl -X POST http://localhost:8000/patterns \
  -H "Content-Type: application/json" \
  -d '{
    "entity_type": "CONSTR_PROJECT_CODE",
    "label": "Project Code",
    "unit": "construction",
    "patterns": ["\\bPROJ-[A-Z]{2}-\\d{4}-\\d{4}\\b"],
    "context": ["project code", "proj id"],
    "score": 0.85
  }'

# Test a regex
curl -X POST http://localhost:8000/patterns/test \
  -H "Content-Type: application/json" \
  -d '{"patterns": ["\\bPROJ-[A-Z]{2}-\\d{4}\\b"], "sample_text": "Project PROJ-TX-2024 approved"}'

# Delete a pattern
curl -X DELETE http://localhost:8000/patterns/{id}

---

Tracing & Logging

Every API action emits a structured trace record via Python's standard logging module.

Destinations

• Stdout — always active, human-readable. Captured by any container log driver.
• Azure Log Analytics Workspace — active when AZURE_LOG_WORKSPACE_ID and AZURE_LOG_WORKSPACE_KEY are set. Records land in PIIRedactionTrace_CL.

Trace fields

Field	Type	Description
timestamp	ISO 8601	UTC time of the event
level	string	Log level (INFO, WARNING, ERROR)
event_type	string	process | redact | summarize
client_ip	string	Caller IP (respects X-Forwarded-For)
session_id	string	UUID grouping related events for one document
filename	string	Uploaded filename (process events only)
file_type	string	File extension (pdf, txt, etc.)
detected_language	string	ISO 639-1 code auto-detected by langdetect
entity_counts	object	{"PERSON": 2, "EMAIL_ADDRESS": 1, ...}
pii_count	int	Total PII spans found
operator	string	Redaction style used (replace, mask, etc.)
text_length	int	Character count of input text
summarization_run	bool	Whether Claude was called
llm_model	string	Model ID used for summarization

KQL queries

// All PII processing events in the last 24 hours
PIIRedactionTrace_CL
| where TimeGenerated > ago(24h)
| where event_type_s == "process"
| project TimeGenerated, client_ip_s, filename_s, detected_language_s, pii_count_d

// Top PII types detected this week
PIIRedactionTrace_CL
| where TimeGenerated > ago(7d)
| mv-expand entity_counts_s
| summarize total = sum(todouble(entity_counts_s)) by tostring(entity_counts_s)
| order by total desc

// Summarize quota usage per team key
PIIRedactionTrace_CL
| where TimeGenerated > ago(1d)
| where event_type_s == "summarize"
| summarize calls = count() by client_ip_s
| order by calls desc

---

API Reference

GET /health

{ "status": "ok" }

GET /entities

Returns all supported PII entity types including active custom patterns.

GET /languages

Returns supported languages and whether their spaCy NLP model is installed.

POST /process

Upload a file, extract text, auto-detect language, and redact PII.

Request — multipart/form-data

Field	Type	Description
file	file	.txt, .pdf, .docx, .doc, .csv, .md
language	string	ISO 639-1 code or "auto" (default)
session_id	string	Optional — groups trace events

curl -X POST http://localhost:8000/process \
  -F "file=@document.pdf" -F "language=auto"

Response

{
  "original_text": "My name is John Smith...",
  "redacted_text": "My name is <PERSON>...",
  "entities_found": [{"type": "PERSON", "start": 11, "end": 21, "score": 0.85, "original": "John Smith"}],
  "entity_counts": {"PERSON": 1},
  "detected_language": "en",
  "filename": "document.pdf",
  "file_type": "pdf",
  "session_id": "a1b2c3d4-..."
}

POST /redact

Redact PII from raw text.

Field	Type	Default	Description
text	string	required	Plain text to redact
language	string	"auto"	ISO 639-1 code or "auto"
entities	list	all	Entity IDs to detect
operator	string	"replace"	replace | redact | mask | hash
session_id	string	—	Groups trace events

curl -X POST http://localhost:8000/redact \
  -H "Content-Type: application/json" \
  -d '{"text": "Contact Sarah at sarah@example.com", "operator": "mask"}'

POST /summarize

Summarize PII-free text via Claude. Subject to hourly burst limit and daily LLM quota.

Field	Type	Default	Description
redacted_text	string	required	PII-free text
length	string	"short"	short | medium | detailed
session_id	string	—	Groups trace events

Returns HTTP 429 when daily quota is exceeded. Quota headers are included on every response.

GET /patterns

List custom PII patterns. Filter with ?unit=construction.

GET /patterns/units

List the four business units.

POST /patterns

Create a custom pattern.

Field	Type	Description
entity_type	string	Uppercase ID e.g. CONSTR_PROJECT_CODE
label	string	Human-readable label
unit	string	construction | agriculture | geospatial | hr_legal
patterns	list	Regex pattern strings
context	list	Context keywords (optional)
score	float	Confidence 0.0–1.0 (default 0.80)

DELETE /patterns/{id}

Delete a user-defined pattern. Returns 403 for built-in patterns.

POST /patterns/test

Test regex patterns against sample text without persisting.

---

Multi-language PII detection

Language is auto-detected via langdetect. The detected language drives spaCy NLP (names, locations, orgs). Country-specific IDs use regex and work regardless of which NLP model is installed.

Region	Languages	NLP model
Default	English	en_core_web_lg ✓
Europe	German, French, Spanish, Dutch, Italian, Swedish	*_core_news_lg
Americas	Portuguese (Brazil), Spanish (LATAM)	pt/es_core_news_lg
APAC	Japanese, Chinese, Korean	ja/zh/xx models

python -m spacy download de_core_news_lg   # German
python -m spacy download fr_core_news_lg   # French
python -m spacy download pt_core_news_lg   # Portuguese

---

Scaling

Both services share the same Docker image. Scale them independently:

# Scale MCP server for more concurrent agent connections
docker compose up --scale mcp-server=3

# Scale REST API for higher HTTP throughput
docker compose up --scale rest-api=2

For Azure Container Apps:

az containerapp update --name mcp-server \
  --min-replicas 2 --max-replicas 20

Add REDIS_URL when running multiple REST API pods so rate limit counters are shared across replicas.

---

Development

# Run unit tests
pytest tests/unit/ -v

# Run integration tests
pytest tests/integration/ -v

# Run with coverage
pytest tests/unit/ --cov=src --cov-report=html

# Lint and format
ruff check --fix src tests mcp_server
black src tests mcp_server
isort src tests mcp_server

---

Interactive API docs

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc
• MCP health: http://localhost:8001/health