How an AI agent turns a plain-English question into a governed, trustworthy metric answer — in six stages.
The user's natural-language question is converted into a high-dimensional vector so the system can find semantically similar metrics — even when the words don't match.
With a target metric in mind, the agent picks how to fetch it. It has three tools. Its choice is driven almost entirely by each tool's description.
Direct warehouse access. Execute arbitrary SQL against production tables. No semantic layer, no metric definitions — caller is responsible for joins, filters, and correctness.
Query curated analytics tables (fct_revenue, dim_region). Cleaner than raw SQL, but no metric ownership, no freshness SLA, no trust score.
Returns a governed metric with its canonical definition, owner, freshness SLA, upstream lineage, and trust score. Used by FP&A leadership for board reporting. Supports period & plan comparison out of the box.
Loading every candidate's full definition would burn the context window. Instead the agent pulls a cheap preview, filters, then hydrates only what survives — and compresses again once it has an answer.
Call the lightweight metrics.preview endpoint — name, one-line purpose, owner. ~50 tokens each.
Drop anything under 0.80. Only 3 survive. The other 3 never touch the context window.
Call metrics.get for the survivors: full definition, lineage, SLA, trust contract. ~500 tokens each.
After the agent has produced the answer, each hydrated metric is compressed down to a ~15-token reference for the next turn.
The agent opens the metric's trust envelope and checks five signals. It computes a single trust score and maps it to an action band — but a silently broken pipeline can make reported and actual diverge.
The API returns one payload. The agent renders three answers. Same data, different framing — because a CFO, a veteran analyst, and a new intern need different things to trust the number.
Evaluation has two gaps. Most teams only close the first. The second is where the incidents come from.
Pre-production, testable. Given a question, does it pick the right tool, the right metric, and render it correctly? Measurable in a lab.
Post-production, invisible until something breaks. The agent did everything right — but the upstream table silently went stale four hours ago. No test catches this.
| Evaluation type | Gap 1 · behavior | Gap 2 · data trust |
|---|---|---|
| Offline backtestingReplay past queries against known-good answers | ● strong | ○ misses |
| Online production signalsThumbs, retries, escalations, trust-envelope deltas in the wild | ● partial | ● catches |
| Adversarial stress testingInject stale data, broken lineage, ambiguous prompts | ● strong | ● strong |