How an agent turns a question into a trustworthy answer

In the last post an agent built a semantic model and showed its work — it investigated the data rather than trusting it, gathered the definitions the business had already settled elsewhere, and flagged every decision that wasn't its to make. This post is about what happens when someone asks that model a question.

Here is the failure mode to worry about. Ask an agent "how is revenue trending?" against the ecommerce model we'll use below, and a careless one comes back with: revenue fell 56% in November. The number is real — it comes straight out of a query that ran without error. It is formatted like an answer, delivered with confidence, and wrong in a way that could start a very bad meeting. The data simply ends on November 14th.

Agents on data rarely fail loudly. The query compiles, the result looks like money, and nothing anywhere signals that the answer is broken. You cannot tell a right answer from a wrong one by looking at the answer. You can only tell by looking at the process that produced it — which is why the open-source analysis skills in Malloy Publisher are not about making an agent faster at querying. They are about making it behave like an analyst.

Where the discipline comes from

I spent time as a data analyst before joining Credible, and nothing in these skills will surprise anyone who has done that job. Restate the question before touching data. Never trust a column name. Check the denominator. Make the parts sum to the whole. Recompute the number that matters a second way before you put it in front of anyone. It is the checklist a senior analyst runs a junior's work through — the difference between someone who queries data and someone whose answers you act on.

The problem was never that this discipline is secret. It's that it lived in analysts' heads, and was applied only when an analyst was in the room — and for most questions, no analyst ever was. Written down as skills, it runs on every question, asked by anyone, including all the ones nobody would have staffed an analyst on. That is not a replacement for the expertise. It is a translation of it, and the translation is what opens data analysis to people who were never going to write a query.

So here is what the discipline looks like running. One question, against a demo ecommerce model — a different dataset than the last post, deliberately, because the discipline is the point and it shouldn't matter what it runs on. Every number below is from a query the agent actually ran.

The question

"How is revenue trending? Which channels are driving the growth?"

The analysis skill makes the agent restate this before anything runs: the metric is revenue, the breakdown is acquisition channel, the time grain is monthly, and no time range was given — which is itself a decision the agent will have to make and disclose. It also asks what a correct answer would look like: a monthly series, a per-channel comparison against a prior period, magnitudes in the hundreds of thousands. Knowing the shape of a right answer is how you notice a wrong one.

Resolve the words, never guess the names

"Revenue" is not a field name. It's a concept to resolve. The agent's first call is getContext — the retrieval tool from earlier in this series — with a description of what it needs, and the phrase-detection skill governs how that description gets written: translate the question into concepts, don't echo its words.

The model comes back with two defensible revenues, each carrying the documentation a modeler attached to it:

• gross_revenue — "Total revenue from all line items in USD, including returned and cancelled items"
• net_revenue — "Revenue from line items that are not returned or cancelled, in USD"

For January through October 2022 those are $4,510,963 and $4,305,542 — 4.8% apart. Which one is "revenue"? The last post made the case that this is a business decision, not a data fact, and here is the payoff: someone already made the decision and wrote it down. The agent takes net_revenue, says so, and reports the returns and cancellations ($44,143 and $161,279 — which account for the gap, to within a dollar of rounding) rather than silently absorbing them.

"Which channels" resolves the same way: to a traffic_source dimension whose docs enumerate its five values — Organic, Search, Display, Email, Facebook. And docs aren't the only place values live: mark a dimension #(index) and getContext searches its actual values too, so a question about "Facebook" lands on traffic_source by what the field contains rather than what it's called. That matters because the words in a question are as often data values as field names, and an agent that only matches names is blind to half of them. The skill is blunt about why having the real values matters: a filter on a value that doesn't exist in the data doesn't error, it silently returns nothing. In this dataset, filtering order status on 'complete' matches zero of the 255,883 rows that say 'Complete'. Both queries run. Only one of them is about anything.

Ground the scope

Before interpreting any result, the skill has the agent establish what the dataset even is: the time range and the row count. Every number is meaningless without them.

run: order_items -> {
  aggregate:
    first_order is min(created_at)
    last_order is max(created_at)
    line_items is count()
    orders is count(order_id)
}

271,019 line items across 264,071 orders, running from January 1, 2019 to November 14, 2022. That last timestamp is the whole ballgame. November is a half-month, and any trend that includes it will show a cliff that isn't there.

This is where the confident guesser dies. October's net revenue is $505,857; November's is $224,791. Down 55.6% — if you don't know the month is 14 days long. Normalize per day and October ran at $16,318 a day, November at $16,057: down 1.6%, which is to say flat. The agent reports the trend through October, the last complete month, and says why.

And that trend is real: net revenue grew from $384,124 in January 2022 to $505,857 in October, up 32% across the year with a clear ramp from August on.

Verify before trusting

The skill's phrasing is the one I'd put on the wall: your first result is a draft, not an answer. Ask what would make it wrong, then run the query that would expose exactly that.

The channel breakdown, January–October, against the same window in 2021:

Three checks before this table is allowed to mean anything, all from one checklist in the skill:

Do the parts sum to the whole? The five channels total $4,305,543, against the $4,305,542 the trend query produced from a different source down a different join path. The dollar between them is rounding — every figure here is rounded from cents on its own, so a column of them need not add to the rounded total. That is what a clean reconciliation looks like. A grain or filter problem doesn't cost you a dollar, it costs you a percentage, and it's a problem to fix rather than a caveat to footnote.

Does the key number survive recomputation? The agent recomputed Search's $3,186,546 a second way — from the raw line-item price and status flag instead of the model's named measure. Same figure to the cent. In SQL, this cross-source join is exactly where fan-out silently inflates a total; the last post showed that failure giving answers up to 27.9% high on data that looked fine. Malloy's aggregates are grain-aware, and the reconciliation proves it held.

Does the conclusion survive a different metric? "Which channel is growing fastest?" has two true answers. By percentage it's Email, +49.7% — and if that's the headline, someone doubles the Email budget. But Email is $122,572 on a $4.3M base: 2.8% of revenue. By dollars the answer is Search, which added $1.0M year over year — 74% of all new revenue — while every channel grew between 43% and 50%. The honest reading is that growth is broad-based and Search pays the bills. When a ranking flips under a different reasonable metric, the skill's instruction is to surface that, not pick the exciting one.

Present with the assumptions attached

The answer that finally comes back is short: revenue is up 32% through October and growing across every channel; Search drives three-quarters of the growth in dollars. But it arrives wearing its assumptions — net revenue, not gross; January through October, because the data ends November 14th; channel values confirmed against the data. Which means the person reading it can disagree with it. Maybe finance counts revenue gross. An assumption you can see is one you can argue with; the dangerous ones are the ones you inherit without knowing they were made.

The pitfalls are a catalog, not folklore

Half of what I just described isn't cleverness — it's remembering the ways answers go wrong. That memory is now a file: fan-out that multiplies totals through a join, filters that match nothing and return zero rows without erroring, count() versus distinct count, percentages with unexamined denominators, the partial period at the edge of every live dataset. Alongside it sit the query patterns and the SQL habits Malloy makes unnecessary.

Every analyst carries a version of this list, assembled from their own scar tissue and shared over shoulders, one review at a time. Written down, it's better than what any of us carried: it's the union of everyone's scars, it applies on every question rather than when someone remembers, and when it's wrong you can open a pull request against it.

Meaning plus method

This post and the last one are two halves of one claim. The semantic model supplies meaning: what revenue is, which joins are safe, what the statuses actually say. The analysis skills supply method: how to find the right definition, ground the scope, and verify before presenting. Either alone fails — a well-built model queried carelessly still produces the November cliff, and flawless discipline against undocumented columns just verifies its way to the wrong field.

And a verified answer shouldn't evaporate when the chat ends. Capture it as a Malloy notebook — the narrative and the live queries together — so the next person's follow-up starts from verified work instead of from scratch. There's a skill for chatting with a notebook too. But a notebook is an analysis at rest. Some answers get asked once; the ones a team returns to every Monday want to become something people open, not something they re-ask — and that is a post of its own.

Try it

Start where the last post left you: a model over your own data, built by an agent that documented the fields and flagged its assumptions. That is exactly what this post's discipline resolves against, so pick it back up:

cd my-model && npm start

Start your agent there and lead with the question this time, instead of "model my data": ask it how revenue is trending. Then watch what it does before it answers. Which definition it resolves "revenue" to — the one that got written down last time. Whether it establishes the date range before reading a trend. What it recomputes before showing you a number.

Starting from scratch instead? The tutorial has both ways in: the bundled storefront package, a documented ecommerce model that needs nothing of your own, and the one-command path that wraps any CSV, Parquet, JSON or Excel file you have.

The skills are just files, and the ones linked throughout this post are the versions your agent runs. Read them, tighten the checks where your business needs more, and when your own scar tissue finds a pitfall the catalog is missing — that's a pull request.

• Browse the skills: the skills/ directory in malloydata/publisher
• Install without cloning: @malloy-publisher/skills on npm
• Share what you built, and join the community: Malloy community Slack

Down 55.6% and down 1.6% came out of the same model, in the same format, with the same confidence — and as arithmetic, both are correct. The model computed exactly what it was asked, both times. They are answers to two readings of the same question, and only one reading survives contact with the calendar. The skills exist so the judgment that tells them apart runs on every question — not just the ones an analyst was in the room for.