<!-- Replace YOUR_USERNAME/YOUR_REPO once pushed, or the badge will be broken. -->

WhatsApp MCP Assistant: RAG search + ML analysis + message sending

An MCP server over WhatsApp chat history combining all three things discussed: hybrid-retrieval RAG search over the conversation, ML analysis (sentiment, topic clustering, real stats), and WhatsApp Cloud API message sending -- exposed as five real MCP tools.

Builds directly on the design proven in the docs-RAG project (hybrid BM25 + TF-IDF/LSA retrieval, real MCP protocol testing), retargeted at conversational data, which has its own genuinely different challenges from documentation (short messages, informal language, multi-speaker attribution, timestamps).

A real bug found live: iPhone export format wasn't supported

While actually testing this against a real, live-exported chat (not the synthetic sample data), the pipeline crashed with a cryptic ZeroDivisionError three layers deep inside the BM25 library. Root cause, confirmed directly: iPhone WhatsApp exports use a completely different format than Android:

Android:  12/01/24, 09:15 - Priya: message text
iPhone:   [12/01/24, 9:15:32 AM] Priya: message text

Bracketed instead of dash-separated, includes seconds, uses 12-hour AM/PM instead of 24-hour time -- the parser only handled the Android format, so every line in a real iPhone export silently failed to match, producing zero parsed messages, which then crashed deep inside a third-party library instead of failing with a clear message at the actual source of the problem.

Fixed properly, not patched around: the parser now tries both formats, correctly converts 12-hour AM/PM to 24-hour time, and handles a real, easy-to-miss quirk -- iPhone exports commonly use U+202F (a narrow no-break space, invisible when eyeballing the file) instead of a plain space before "AM"/"PM", which a plain-ASCII regex silently fails to match. Also added: ChatPipeline now raises a clear, specific error ("Parsed zero messages... check that this is really a WhatsApp export file") if a chat file produces zero messages, instead of letting that surface as an unrelated crash inside a dependency three call-stack frames away from the actual cause.

Attempted improvement that honestly didn't help: a trained reranker

The docs-RAG project's biggest quality lever was a trained reranker on top of hybrid retrieval. It was missing here, so it was built (wa_mcp/reranker.py, same design: logistic regression trained from scratch on BM25/dense scores, term overlap, chunk length, plus a chat-specific "distinct senders in this chunk" feature) and rigorously measured, not just added and assumed to help.

Real result, on the held-out test queries:

Hybrid alone:       precision@5=0.200  recall@5=1.000  mrr=0.815
Hybrid + reranker:  precision@5=0.200  recall@5=1.000  mrr=0.726

The reranker measurably does NOT help here, and slightly regresses MRR. Most likely cause: only 8 training queries (120 examples, 8 positive) is too small a training set for the reranker to learn reliable weights -- smaller than the docs project's training set, on a task where hybrid retrieval alone is already close to its ceiling at this corpus size.

Decision: not wired in as the default, specifically because shipping something that measures worse while calling it an "improvement" would be dishonest. The code is real, tested for correctness (tests/test_reranker.py -- training produces valid, imbalanced labels; predictions are valid probabilities), and available (wa_mcp/reranker.py) for a much larger real chat where a bigger, real-usage-derived training set might behave differently -- but it's not claimed as a proven improvement, because it isn't one, measured honestly.

The improvement that DID measurably help was the filler-message filtering described above -- that one is wired in as the default, because it was actually measured to help.

Attempted improvement #2: a trained filler classifier -- helped on its own test, then failed at real scale

The fixed FILLER_PHRASES list has an honest, stated limitation: it can only catch filler wording someone thought to write down. So a trained alternative was built (wa_mcp/filler_classifier.py, logistic regression on TF-IDF, same pattern as the sentiment classifier) and compared directly against the fixed list -- including on filler phrasings deliberately absent from the fixed list ("hbu", "no cap", "word", "true story", "for real").

On its own held-out test set, the trained classifier looked like a clear win:

accuracy: 0.9   precision: 1.0   recall: 0.8

It correctly caught 4 of 5 novel slang variants the fixed list would have completely missed, while never wrongly flagging a real, substantive message.

Then it was tested against the actual 60,000-message stress-test corpus used earlier -- and it did measurably WORSE than the fixed list:

Fixed list:          2 chunks remaining after filtering
Trained classifier:  1,728 chunks remaining after filtering

The trained classifier let obvious filler through repeatedly ("call me later", "wait what" appearing in its own top search results, unfiltered).

Root cause, confirmed directly, not guessed: checked exactly which filler phrases the 60k-message test corpus actually used against what the classifier's 28-example training set had ever seen -- 14 of the 22 real filler phrases in that corpus were never in the classifier's training data at all ("yeah", "wait what", "sounds good", "call me later", "checking now", and 9 others). The classifier isn't learning a general concept of "this message carries no retrieval-worthy content" -- with only 28 examples, it's memorizing its own narrow training vocabulary, the same fundamental limitation as the fixed list, just less predictable and harder to inspect/edit than a plain list you can read and extend directly.

This is a genuinely important, humbling lesson, not a wasted attempt: a small held-out test set drawn from a similar, narrow vocabulary can make a tiny trained model look like it generalizes well, right up until it meets real data with a genuinely different vocabulary distribution -- exactly the gap between "measures well on its own test set" and "actually works in the real world" that's worth knowing about as a general ML lesson, not just for this one feature.

Decision: the fixed list remains the default (use_trained_classifier=False in chunk_messages()), specifically because it measurably performed better on the real, at-scale test that actually matters. The trained classifier is kept available (use_trained_classifier=True) and tested for correctness (tests/test_filler_classifier.py) -- but not claimed as an improvement, because real testing found the opposite. A genuinely larger training set (hundreds of real labeled examples, not 28) would likely change this finding; not built here, since that's real, substantial labeling effort, not a quick fix.

A real scaling investigation: tested at 60,000 messages, not assumed

This project was originally built and tested against a 66-message synthetic sample. A real question came up during use: does this actually work at real chat scale (tens of thousands of messages)? Rather than guess, a realistic 60,000-message synthetic chat was generated (mostly short filler messages -- "lol", "ok", "one sec" -- with one real, findable conversation planted in the middle) and tested directly.

First result: it does NOT crash and is NOT slow -- built in 5.9s, searched in 0.035s even at 60k messages. But retrieval QUALITY genuinely degraded: the planted conversation about booking "Cotton Villa" got diluted across noisy chunks full of filler messages, and none of the top-5 results was a clean match.

Root cause: with ~20,000 sliding-window chunks at this scale, a 5-message chunk that's 80% filler ("yep", "one sec", "wait what") has far less distinguishing content for BM25/TF-IDF to match against than one where every message carries real content -- filler doesn't just fail to help, it actively dilutes genuine signal sitting in the same window.

Fix: wa_mcp/chunking.py now filters out common filler messages before chunking (they're still counted in message-count stats, just excluded from retrieval chunks specifically). Re-tested against the exact same 60k-message file: chunk count dropped from 19,999 to 2, and both top results were now clean, correct matches for the planted conversation.

Honest limitation, not glossed over: the filler list is a fixed set of phrases, not a trained classifier -- it will always miss some real-world phrasings. One was caught and fixed during this very investigation ("haha true" wasn't in the initial list, only found by testing against the large synthetic file and inspecting actual results). A more robust version would train a small classifier the same way the sentiment model was trained, rather than hardcode a list -- not implemented here, stated as a real next step rather than presented as solved.

Also worth noting honestly: the synthetic test chat was closer to a worst-case (nearly all filler except one planted signal) than a typical real chat, which usually has a higher proportion of substantive messages. The mechanism (filtering filler improves signal-to-noise at scale) is validated; the exact magnitude of improvement on any specific real chat will vary.

Data: synthetic, not anyone's real private chat

data/sample_chat.txt is a hand-written, fictional conversation (three people planning and taking a trip), written in the REAL WhatsApp export text format (DD/MM/YY, HH:MM - Name: message). The parser handles this real format, so a genuine exported chat (Settings > Export Chat in WhatsApp) would work too -- deliberately not used here, same reasoning as writing a fictional docs corpus instead of scraping real documents.

Part 1: RAG search over chat history

Individual WhatsApp messages are often 1-5 words ("yes", "lol", "on my way") -- too short to retrieve meaningfully alone. wa_mcp/chunking.py groups messages into overlapping sliding-window chunks (window=5, stride=3) so retrieval has something substantive to match against, with the overlap ensuring a conversational moment near a window boundary is still captured whole in at least one chunk.

Retrieval reuses the proven hybrid design (BM25 + TF-IDF/LSA + Reciprocal Rank Fusion, with Porter stemming) from the docs-RAG project.

Honest note on retrieval quality on chat data specifically: casual chat text is measurably harder to retrieve well than clean documentation prose. A real example found during testing: the query "who was stressed about work" found the correct chunk at rank 4 out of 22, not rank 1 -- reasonable (top 20%), but not perfect, and reported as such rather than only testing easy queries that happen to rank first.

Part 2: ML analysis

• Sentiment classifier (wa_mcp/sentiment.py): logistic regression TRAINED FROM SCRATCH on hand-labeled, informal chat-style examples (not clean prose) -- same "no pretrained model" reasoning as the docs project's reranker (this sandbox has no access to huggingface.co). Real, honestly reported result on a held-out test set: 70% accuracy, 62.5% precision, 100% recall. Clearly better than random guessing on a small, hand-labeled dataset -- and clearly limited by that small dataset size, not presented as more than it is.
• Topic clustering (wa_mcp/topics.py): TF-IDF + KMeans, classical, no download needed. Tested against a real, checkable claim: every chunk mentioning "Cotton Villa" (the accommodation) lands in the same cluster, distinct from the beach-day and stress-venting conversations.
• Chat statistics (wa_mcp/stats.py): message counts per sender, average response time between different senders, busiest day -- all directly computed, including honestly handling a real tie (two participants are tied at 23 messages each in the sample data; tested as "one of the tied leaders," not asserting a single winner as if the tie didn't exist).

Part 3: WhatsApp message sending -- the honest limitation

wa_mcp/whatsapp_client.py is written to Meta's actual, currently- documented Cloud API request format (POST /{phone_number_id}/messages, Bearer token auth, messaging_product/to/type/text JSON body -- verified via Meta's developer docs, not assumed from memory).

This has never been verified against the live API. This sandbox has no network access to graph.facebook.com (confirmed directly: a plain curl request to it returns 403, same category of block as huggingface.co earlier), and there's no real WhatsApp Business Account or access token available regardless. Tests (tests/test_whatsapp_client.py) mock the HTTP layer to verify the client builds requests correctly against Meta's documented contract -- that is NOT the same claim as "this works against the real API," and this README says so explicitly rather than blurring the two.

One genuinely interesting thing found while building this: calling send_whatsapp_message with a placeholder token, without any mocking, actually attempted a REAL network request and came back with HTTP 403 -- confirming this sandbox's network block directly, and confirming the client fails cleanly (returns a proper error result) rather than crashing, even against a real, if blocked, network attempt -- not just against mocked failures.

Build & run

python3 -m venv venv
./venv/bin/pip install -r requirements.txt

Run tests:

./venv/bin/python -m pytest tests/ -v

Try it live (uses the fictional sample chat by default):

./venv/bin/python -m wa_mcp.interactive_demo

Or point it at a real chat you've exported yourself (Settings > Export Chat in WhatsApp, no API needed -- both Android and iPhone export formats are supported):

./venv/bin/python -m wa_mcp.interactive_demo ~/Downloads/_chat.txt

Run the MCP server:

./venv/bin/python -m wa_mcp.server

To actually send real WhatsApp messages (untested against the live API, see above), set real credentials first:

export WHATSAPP_PHONE_NUMBER_ID="your-real-id"
export WHATSAPP_ACCESS_TOKEN="your-real-token"
./venv/bin/python -m wa_mcp.server

Test suite

tests/test_parser.py            10/10 -- both Android + iPhone export formats,
                                          incl. multi-line messages and the
                                          narrow-no-break-space quirk
tests/test_chunking.py           6/6  -- sliding window + overlap + filler filtering
tests/test_pipeline.py           2/2  -- clear error on unrecognized format
tests/test_retrieval.py          3/3  -- hybrid search on real chat queries
tests/test_sentiment.py          5/5  -- trained-from-scratch classifier, real train/test split
tests/test_stats_and_topics.py   5/5  -- real computed stats + topic separation
tests/test_whatsapp_client.py    3/3  -- mocked HTTP layer, documented contract only
tests/test_reranker.py           3/3  -- reranker mechanics (see honest finding above:
                                          not wired in as default, doesn't measurably help)
tests/test_filler_classifier.py  4/4  -- trained classifier mechanics (see honest
                                          finding above: fixed list wins at real scale)
tests/test_mcp_server.py         6/6  -- real MCP protocol, all 5 tools

47/47 passing.

Known limitations (stated plainly)

• WhatsApp sending is untested against the live API -- the single biggest, most important honest caveat in this project. See Part 3.
• Small, synthetic chat data (66 messages). Real stats/retrieval numbers, but from a small sample -- a genuine multi-thousand-message export would behave somewhat differently at scale.
• Small hand-labeled sentiment dataset (28 train / 10 test examples) -- 70% accuracy reflects that data size honestly, not a ceiling on the approach itself.
• No template message support -- Meta requires pre-approved templates for messages sent outside a 24-hour customer service window; only free-form messages are implemented here.

Layout

data/sample_chat.txt      hand-written, real-format synthetic chat export
eval/sentiment_data.py    hand-labeled TRAIN/TEST sentiment examples
wa_mcp/
  parser.py               real WhatsApp export format parser
  chunking.py             sliding-window conversational chunking
  retrieval.py            hybrid BM25 + TF-IDF/LSA + RRF
  sentiment.py            logistic regression, trained from scratch
  topics.py               TF-IDF + KMeans clustering
  stats.py                message counts, response times, busiest day
  whatsapp_client.py      Cloud API client (mocked-tested, see Part 3)
  pipeline.py             ties it all together
  server.py               the actual MCP server (FastMCP)
  interactive_demo.py     "watch it work" CLI
tests/                    one file per layer, incl. real MCP protocol tests