A/04Accelerators
LLM Evaluation Harness
“Is it better?” — with an answer that is a number.
The problem it solves
Without evaluation, every prompt tweak and model upgrade is judged by whoever tried it last. Results change, and nobody can say whether they improved. Teams then learn about a regression from a customer instead of from their pipeline.
Regression testing for prompts, models and retrieval: a versioned set of cases drawn from your own history, scored on every change.
What’s inside
- ✦A versioned case set built from your real, anonymised history
- ✦Scoring matched to the task: exact checks where facts must be exact, rubric review where judgement is needed
- ✦A run on every change in CI, compared against the last release
- ✦Side-by-side comparison of models and prompt versions
- ✦Reports a non-engineer can read
How a project uses it
- 01Pick the decisions the system makes that matter most
- 02Collect and label cases with the people who own those decisions
- 03Wire the harness into your delivery pipeline
- 04Agree the thresholds that block a release
What it is not
- A public benchmark or leaderboard
- A one-off accuracy report
- A substitute for people reviewing the cases that matter
Works well with
- A/01
Governed RAG Foundation
Answers from your documents — only from the ones the person asking may read.
Retrieval-augmented generation over company data, with access control carried from the source system and a citation on every answer.
- RAG
- Vector search
- Access control
- Citations
- A/02
MCP Server Kit
Give agents narrow, typed tools — never a shell.
A starting point for Model Context Protocol servers with a small, typed, read-first tool surface, a log of every call, and an approval step for anything that changes state.
- MCP
- Agents
- Audit trail
- Least privilege
Start a project
Want this under your project?
Tell us what you are building and what you already have. We will tell you whether this accelerator fits, what it would need adapting, and where it would not help.
No form and no automated follow-up sequence. An email reaches an engineer, and the first reply is either a real question or an honest “not us.” We work in English, remotely, across European time zones.
We reply to every message that isn’t automated.