-
Eval sets built from real work
Test cases come from the customer's own documents and the questions their experts actually ask, not from synthetic prompts. Across a seven-part proof-of-concept portfolio, every workflow was validated by subject-matter experts on real documents against one evaluation template: time saved, rework reduced, result quality, practical applicability, improvement potential.
-
Deterministic checks before model-graded ones
Testbed runs deterministic cases over AI processes and retrieval pipelines first — structure, required fields, output language, refusal behaviour — because those failures are cheap to catch and unambiguous. Model-graded scoring comes after, for the judgements a rule cannot express.
-
Retrieval measured separately from generation
Whether the right passage was retrieved is a different question from whether the answer reads well. Over technical archives of up to 300 pages per document, retrieval used top-k vector search, a reranking step with a stated relevance justification, and citations down to document, file and page — so a wrong answer can be traced to retrieval or to generation.
-
An error taxonomy, not an error count
Failures get classified: missing extraction, wrong attribute, fabricated value, wrong output language, unsupported claim. On contract deliverables, that taxonomy showed a single wide table schema hallucinating, so extraction moved to an n-step design — identify first, then extract attributes.
-
Human-review thresholds written down
Every workflow states what a person still checks and when. Partner-report pre-checks carry a three-star consistency rating per indicator with comments explaining each deviation; contract extraction keeps human-in-the-loop validation at the accuracy the customer accepted.
-
Launch criteria agreed before the build
A workflow ships against numbers set in the specification, not against a demo reaction: on contract deliverables the targets were at least 70% recall, over 75% precision and more than 50% time reduction, with the projected review time falling from five hours to one at the agreed accuracy.
-
Quality, latency and cost as one trade-off
Model choice follows quality, latency, cost and data sovereignty per client, on a provider-independent layer. For a multilingual self-help assistant, the per-interaction cost against a static decision tree was part of the evaluation rather than a surprise after rollout.
