A repeatable way to find, build and evaluate AI workflows with domain experts

Context

The development-finance teams of a German development bank, and a humanitarian NGO it funds, run on documents: annual ministry reports, partner reports, contracts, evaluation reports, technical design archives and self-help guides for refugees.

Business problem

Each workflow consumed expert hours in summarising, cross-checking and drafting. The question was not whether a language model could write a paragraph, but where generative AI removes manual effort reliably enough that experts would adopt it.

My mandate

I designed the solution architecture and the multi-step AI processes for every proof of concept, wrote the prompts for extraction, reconciliation, generation, retrieval and the chatbot, and was accountable for each result meeting the acceptance criteria the experts set.

Decisions

• Every proof of concept used one specification template (background, problem, objective, scope, methodology, success metrics, dependencies) and one evaluation template (time savings, rework reduction, result quality, applicability, improvement potential).
• Processes were designed as fan-out / fan-in: parallel extraction nodes producing JSON, a reconciliation step, then generation with source references preserved in every output. The ministry-report PESTLE draft alone was an eight-node process with six parallel research analyses.
• Knowledge retrieval over 300-page technical documents used vector retrieval, a reranking step with relevance justification, and citations by document, file and page, exposed through a chat interface with follow-up questions.
• Contract deliverables extraction moved from a single wide table schema (which hallucinated) to an n-step design: identify first, then extract attributes, with human-in-the-loop validation.
• The refugee self-help guide became a stateful, multilingual assistant (English, Arabic, Ukrainian, Bengali, Russian) by injecting the exported decision tree into the prompt and tracking the step pointer in the conversation state.

Delivery

Rapid prototyping on the platform, agile iterations, weekly reviews with the domain experts, every proof of concept validated on real documents.

Outcome

Experts estimated about a day saved per ministry report, a fifth to a third less drafting effort on an evaluation chapter, and a quarter of an hour per partner-report pre-check; contract review was projected to drop from five hours to one at the targeted accuracy. Retrieval over the technical archive enabled research that had been impractical, and the experts asked for a production rollout to a wider group.

Retrieval over the technical archive enabled research that had been impractical, and the experts asked for a production rollout to a wider group.

Reuse

The specification and evaluation templates, the process patterns, the cited-retrieval design and the decision-tree assistant pattern were reused across the portfolio and later deployments.

Evidence

Time savings are the experts’ own estimates from the evaluation template; the contract-review figure is a projection at the targeted recall and precision. Outputs carried source references so every claim could be traced.