Your Data Model Is Part of Your AI System. Evaluate and Optimize It Like One.

Malloy Publisher is the open-source server for Malloy data models. It serves a model to people, data apps, and AI agents through one REST API and one MCP endpoint, so every question runs through the same definitions. Alongside the server, we've open-sourced the skills that teach an agent to work with it. A skill is a markdown file of instructions an agent follows for one kind of task. Earlier posts covered setting Publisher up against your own data, how an agent builds a data model and shows its work, and how an agent turns a question into an answer you can trust.

When an agent answers a question about your data, the answer depends on more than its large language model. It depends on the data model, which supplies the meaning: what revenue means, which customers count, and how orders join to products. And it depends on the skills, which supply the method. The Malloy analysis skills spell out how the agent should work from question to answer. For example, they tell it to look up what a word like "revenue" means in the model before it writes a query, check which dates the data covers before it reads a trend, and confirm the parts add up to the total before it reports a number. The data model and the skills are part of your AI system, and like every part of an AI system, they need evals.

Evaluation is the first lesson of machine learning engineering: before you optimize anything, decide what good looks like and measure against it. Without a measure, you can't tell whether a change helped, hurt, or broke something you weren't looking at. With one, you can optimize: make a small change, measure again, keep it if it helped and undo it if it didn't. Data models usually skip both steps. A model gets built, reviewed, and shipped, and the first sign of a problem is a number you don't trust. An agent can run both steps, and run them every time the data, the definitions, or the skills change.

Even a good model has gaps:

• A concept your team cares about may never have been modeled.
• A question may match two modeled concepts, with nothing to say which one it means.
• A definition may simply be wrong.

And a model that was right when it shipped doesn't stay right. The data changes: a new order status, a renamed product line, a region that didn't exist last year. The definitions change: finance moves the fiscal year, sales redraws the season. The system around the model changes too: a skill gets edited, the LLM gets upgraded. Each of these can move an answer without anyone touching the model, and nobody notices until a number looks wrong.

When a question hits one of those gaps, the agent's skills decide what happens next. Good skills have the agent ask what you mean, or say the model can't answer. Weaker skills let it pick a meaning and answer anyway. If that meaning isn't what your team means, the number won't match the one your team would report.

This post shows how to evaluate and optimize a data model and the skills an agent uses with it: build a test set from the questions your team asks, score the model and the skills against it, trace each miss to the model, the agent's skills, or the eval itself, and let an agent fix what it finds, one small edit at a time, keeping only the edits that hold.

What an Eval Is

An eval is a list of questions, each paired with an answer you've verified, called the golden. An agent answers every question without seeing the goldens. A second agent, the judge, reads each answer and decides whether it matches. One run tests the whole system at once: whether the model can express the answer, whether concept retrieval found the right concepts in the model, and whether the skills lead the agent to build the right queries from them.

The idea will be familiar to anyone who has onboarded an analyst. A new analyst gets everything the team knows, and for the first few weeks someone checks their work. Sometimes the right move is to ask instead of answer: does "summer" mean the sales season, or June to August? A good analyst asks. An eval is that same check, written down so it runs again every time the model or the skills change.

Establishing a golden is the hard part, and an agent can't do it alone:

• A certified figure. If finance publishes the number, that's the golden.
• A person's decision. "How many customers do we have?" has three real counts in Publisher's storefront demo: 1,000 in the customer file, 974 with an order, and 943 with a delivered order. The data can't pick one. Someone on your team has to decide, for example that a customer is someone the company has delivered to. With that decision made, the data gives one answer: 943.
• Two independent derivations. Once the definition is decided, an agent derives the number two ways from the raw tables, with no model in between. For example, summer sales is the season summed directly, and the year's total less everything outside the season. Agreement makes the number a candidate, not a golden. Two queries can share the same wrong assumption. A person still confirms the number before it becomes a golden.

A golden answer can also be a refusal. Sometimes the data model can't answer the question. An agent that says so, instead of guessing, scores as correct. A golden answer can be a checklist, too. Take "Who are our best customers?" when the model doesn't define "best". One customer might lead on spend but return half of it. Another might buy only on clearance and earn no margin. A third might buy less often, at full price, and keep everything. The checklist passes an answer that asks which kind of customer your team means, or shows who leads under each measure. Picking one measure and answering confidently doesn't pass.

How It Works

eval-loop runs the steps below in order, and most of them are skills of their own. The steps stay separate, because an agent that scores an answer while explaining it tends to write the explanation into the score.

1. Collect the questions. The best come from your logs: what people actually asked, in their own words. If the model isn't in use yet, start with the questions your team already asks every week. eval-import turns them into a set and locks the wording of each one, because rewording a vague question until the agent passes deletes the test.

2. Split the set. Each question is marked dev or holdout. Diagnosis and fixes read only the dev questions. The holdout questions are kept back to check a fix against, so it can't be tuned to the questions it was judged on.

3. Answer. A fresh agent answers each question with exactly what a real user's agent has: the Malloy analysis skills and the MCP connection to the model. It never sees the goldens, the eval files, or the model's source. If it could reach any of those, it could get the right number without going through the model, and the score would mean nothing.

4. Score. eval-answer re-runs the agent's final query, and a judge following eval-judge compares the answer with the golden. The judge also says whether it believes the golden.

5. Find where it broke. A score says something is wrong, not what to change. So each miss is checked at the same three points, in order: could the model express the answer at all; did concept retrieval hand the agent the right concepts from the model; and did the skills lead the agent to build the right query from them. The first point that fails is the one to look at, because everything after it was working with what it was given.

6. Decide where the fix belongs. eval-diagnose reads the whole transcript and asks one question: if the model's documentation were perfect, would the agent have done the right thing? The answer names one of three owners.

   • The model: a definition is missing, ambiguous, or wrong, or it's described in words nobody would use to ask for it, so concept retrieval never finds it.
   • The agent's skills: the agent had what it needed and still did the wrong thing. The fix is in the package's guide or the shared analysis skill, and no model edit will reach it.
   • The eval itself: the agent did the right thing and the answer key called it wrong.

   Don't assume the model is the default. On one 35-question run, nine of the first ten fixes were in the skills and one was in the model, and more answer keys were wrong than the model had defects.

7. Fix and check. eval-improve makes the smallest edit that would close a diagnosed gap, and backs every claim in the edit with a query it ran. Then an acceptance check decides whether the edit stays: the questions it targeted now pass, no question that passed before has broken, the holdout questions it never saw still hold, and all of that holds on a second run. Accept means a git commit. Reject means revert. Then the loop goes again. A skill edit gets no shortcut, because a guide written next to the question it just failed is the easiest thing in the loop to overfit. The question stays in the set either way, so the gap can't quietly reopen.

Every run ends with eval-report, which turns the run's logs into something a person can read: a row per question with its verdict, and behind each row the golden, the agent's answer, the judge's reasoning, and every query the agent ran. The summary reports four numbers, in order: how many questions the model could express an answer to, how often concept retrieval delivered what the answer needed, how many answers were right, and what the run cost. It also keeps the model's failures separate from the eval's own, such as an agent that ran out of turns or a golden nobody confirmed. The two look identical in a pass rate and have different owners.

An Example Run

Publisher ships a demo you can run yourself, called storefront: a small online store's data, a model over it, and an example eval set of twelve questions. Most of the questions are ordinary, like which category sells the most.

Three of the twelve questions were designed to find gaps. Each of the three asks about a business rule we deliberately left out of the model:

• Summer is the sales season, 25 May to 15 September.
• Net sales leave out order lines that were cancelled or returned.
• A customer is someone who has had an order delivered.

The three rules came with the questions, in a note from the business team, the way they would in real life. The model doesn't encode any of the three. One exception: the package's short guide for agents explains the net-sales filter.

We ran the example set against Publisher. 8 of the 12 questions passed, 3 failed, and 1 came close. Two of the three failures were questions we designed to find gaps: summer sales and customer count. The net-sales question passed, because the guide covers it. The third failure, top customer by spend, was not planned, and it taught us the most. A second run failed the same three questions for the same reasons, which is what makes them worth fixing. A question that fails once and passes once is a different finding: usually the model leaves two readings open, and the agent picks one at random. Answering and judging cost $3.34, and diagnosing cost $1.49.

Summer sales: a concept the model never defines. The agent read summer as June to August and answered $213,939. The golden is $267,423, so it was 20% low. The agent did say it assumed June to August, but the headline number was wrong. The judge's test is whether a reader who acted on the figure would be wrong. Stating the assumption makes a wrong answer easier to diagnose; it doesn't make it right. The diagnosis found that the whole error is the difference between the two date windows, and that neither the model nor the guide mentions a season. The fix belongs in the model.

Customer count: two counts, nothing to choose between them. The agent led with 1,000 and offered 974. Neither matches the golden of 943, and the agent never asked which customers your team counts. The diagnosis found a second problem: the model describes the customer file as "people who have placed orders", which is false for the 26 customers who never ordered. Both problems, the missing definition and the false description, need fixes in the model.

Top customer by spend: a fix that broke a neighbor. The package's guide warns that total_sales includes cancelled and returned lines. That warning is why net sales passed, at $834,216. But asked for the biggest customer by total spend, the agent applied the same filter, which nobody asked for. It named customer 395 at $7,736.59 instead of customer 884 at $8,817.56. The diagnosis put the fix in the model: say that "spend" means total_sales unless the question asks for net.

Best customers: a fix in the skills, not the model. The agent said the model has no definition of "best". Then it picked a measure anyway: spend without cancelled and returned lines, which puts customer 395 first. Customer 395 is correct on that measure, so the answer scored a near match rather than a pass. Total spend, order count, margin, and delivered spend all put customer 884 first instead. The answer that passes stops at "the model doesn't define best" and asks. The agent knew that and answered anyway, because the analysis skill tells it to state an assumption rather than stop. No model edit fixes that; the skill has to say when to ask. A definition would settle it too. If your team says "we score customers on margin over the trailing five years, less returns and restocking costs," that becomes a measure in the model, and the question has one right answer.

The best-customers answer key needed a fix of its own. Written before the package's guide existed, the key said every measure puts customer 884 first. That was true of the four measures the key listed. The guide led agents to a fifth measure, spend without cancelled and returned lines. The judge flagged the key's claim, we checked it against the raw tables, and we added the fifth measure to the key. The fix belonged in the eval, not the model.

Five fixes from one run: three in the model, one in the skills, one in the eval.

Closing the Gap, and Keeping It Closed

The summer fix is a few lines, in the one place every agent reads:

dimension:
  #(doc) The company's summer sales season: 25 May to 15 September
  #(doc) inclusive, set by the buying calendar. Use for any question
  #(doc) about "summer" sales.
  in_summer_season is
    (month(created_at) = 5 and day(created_at) >= 25)
    or (month(created_at) ? 6 | 7 | 8)
    or (month(created_at) = 9 and day(created_at) <= 15)

In the loop, that edit is a proposal. The agent makes it, reloads the model, and runs the set again. If summer sales now passes, the other eleven questions still pass, and the holdout questions hold on two runs, the edit is committed. If not, it's reverted and the finding stays open. It's hill climbing: one small edit at a time, kept only when the score goes up and nothing that passed goes down, with git as the record of every step.

The top-customer failure shows why the set keeps running after the fixes land. The guide was added to fix questions like net sales, and it did. It also broke the top-customer question, which nobody was looking at. That's how models go stale: not one big change, but a series of small, reasonable ones, each moving an answer somewhere else. A set that runs on every change catches that break before your users do, and the loop fixes it the same way it fixed summer.

The same loop works beyond answer quality. The loop can show which #@ persist annotations would speed up the slowest questions, give evidence for tuning concept retrieval, and tune a model's descriptions for the LLM your team uses. Today you start the loop, say how far to run it, and an agent runs it. In a later post we'll look at the loop running on its own, on a schedule, and proposing fixes as pull requests for a person to approve.

Try It

You need Node 20 or newer, Bun, a Java runtime, and Python 3.11 or newer. Open Claude Code, or another coding agent, in an empty directory and say:

Clone malloydata/publisher and run the eval loop over the storefront example.

Expect summer sales and customer count to fail, top customer to fail or come close, and best customers to come close unless the agent asks what "best" means. Answers can vary from run to run, so run it twice before reading the score. Concept retrieval also works better with an embedding API key. Without one, retrieval falls back to keyword matching and finds fewer of the concepts each question needs.

Then point it at your own model. Bring the questions your team already asks, in any form, and say "set up an eval set for my model from these questions." The agent turns them into a set, derives the answers it can, and asks you to confirm every one.

• The eval skills: the skills folder in Malloy Publisher, starting with eval-loop
• The example set: the storefront eval set, with the questions, the answer key, and both derivations of every golden
• Join the community: Malloy community Slack

Your data model and your skills are part of your AI system. An eval tells you whether they work today. The loop keeps them working as the data, the definitions, and the skills change, one real question at a time.