01Three real levers, not one
Instruction modules for behaviour, golden examples bound to a task's schema, and retrieval tuning — each one measured, not guessed at.
instruction modules: minutesgolden examples: biggest jumpretrieval tuning
AI quality & regression
Your agent is 80% right and nobody knows how to move it
The first two weeks felt like magic. Then it plateaued, and the only lever anyone has left is rewriting the prompt until this week's failing case passes — with no way to check what else just broke.
A reviewer's correction on finding F-03 becomes a bound golden example and an eval case in the same click. The eval suite score across five releases — 74% to 96% — is the same 36 cases run every time, not an impression.
Sound familiar?
It got good fast, then it stopped
The first two weeks felt like magic. Then accuracy plateaued around 80% and every attempt to push past it either does nothing or breaks something that used to work.
Fixing one case breaks another
Someone rewrites the prompt to fix a case that came up in a demo. Three other cases that used to pass now fail, and nobody notices until a customer does.
Reviewers catch mistakes that vanish into nothing
A human corrects a wrong output every day. The correction fixes that one instance and teaches the system nothing — the same mistake shows up again next week.
Why the usual fixes don't move the number
Prompt-tweak roulette
Changing wording until this week's failing case passes, with no way to check what else moved. Improvement becomes indistinguishable from luck, because nothing is measured before or after.
A bigger, more expensive model
A stronger model narrows a capability gap. It does nothing for a retrieval gap or a missing example — and design-notes.md's own finding is that most real failures are exactly that, not model capability.
Fine-tuning on your traffic
Needs labelled volume most deployments don't have yet, adds serving cost and latency, and bakes behaviour into weights nobody can inspect — which directly contradicts the auditability the rest of this platform sells.
A feedback form nobody reads
Corrections get logged somewhere, reviewed by nobody, and never make it back into the system. The knowledge that a human caught a mistake evaporates the moment they close the tab.
How we solve it
02A correction becomes a golden example, in one click
A reviewer's fix doesn't just patch one output — it's promoted into a bound example the next release trains against and an eval case that gates it.
correction → golden example→ eval case, same click
03Every release is scored against the same suite
Not a vibe check — a fixed set of acceptance cases, run every release, so "did this actually get better" has a number instead of an impression.
36/36 cases runsame suite, every releaseregression, not folklore
What you get on every document
A score that climbs, release over release
74% to 96% isn't a claim — it's the same 36 acceptance cases run against every version, so the trend line is measured, not asserted.
A growing example library that's actually yours
Golden examples come from your real corrected cases, tenant-scoped — not a generic library that teaches the model about someone else's documents.
Regressions get caught before release, not after
A change that fixes today's complaint but breaks yesterday's fix fails the eval suite before it ships, instead of surfacing in next week's support ticket.
Nothing a reviewer catches gets thrown away
Every correction has somewhere to go — a golden example, a regression test — so the work of the person who caught the mistake compounds instead of evaporating.
