The chat on this site, explained

There’s a chat in the corner of every page on this site. Ask it about my work and it answers from what’s published here, with numbered citations, and tells you to email me when the site doesn’t cover something. It took about three days to build. Each question costs me less than a cent.

The actual chatbot — build a prompt, call a model, stream the answer back — is roughly 120 lines. I want to be careful with the word “simple” though, because the full system is 1,062 lines plus 375 lines of tests, and the breakdown is the interesting part:

Purpose	Lines	Share
The chatbot: prompt, model call, streaming	~120	11%
Trust: citation validation, grounding rules, eval suite, corpus builder	~250	24%
Abuse and cost: human check, rate limiting	~106	10%
UX: the panel — streaming render, queue states, links, accessibility	~480	45%
Observability: anonymous question logging	~100	10%

What happens when you ask a question

At build time, a script extracts 52 sections from the rendered site into a corpus of about 5,600 tokens. When a question comes in, a serverless function checks you’re human, checks you haven’t hit the hourly limit, and sends the corpus plus your question to Claude Haiku. The instructions pin three behaviors: answer only from the corpus, cite every claim by section id, refuse anything the site doesn’t cover. The model writes citation markers inline, and the server checks each one against the real section list before it reaches your browser, so a citation can’t point at nothing. Questions are logged anonymously — question, outcome, latency, tokens — so I can see what visitors actually ask.

There’s also an eval suite: ten questions that run against the real API whenever the prompt changes. Seven check that citations land on the right pages, two check that off-topic questions get refused, one checks that “ignore your instructions” gets deflected. It was 10/10 before launch.

Then launch day happened

Real use found five problems in the first day. A verification race stacked twelve copies of the same notice in the message list. The human-check box came back after every single answer. Answers ran three times longer than instructed. The “book a call” link had never rendered once — the model says “book a 15-minute call” and my regex matched “book a call”, so the feature shipped dead and nobody knew. And long URLs overflowed the chat bubble.

None of these were hard fixes. Hours each, because the primitives are off the shelf: Turnstile, KV and analytics from Cloudflare, caching and streaming from Anthropic. But all five lived in the envelope, not the core, and none of them showed up until a real person used the thing. The eval suite caught zero of them, because the eval suite tests answers, and these were everything-but-the-answer bugs.

So if you’re scoping a project like this, the ratio to carry in your head is one part chatbot, nine parts trust, abuse handling, UX and logging. The nine parts are mostly ordinary engineering. That’s the good news, depending on how you feel about ordinary engineering.