Guide

7 MIN READ

Why don't AI answers match our dashboards?

An AI agent's total disagrees with the dashboard when it assumes a different definition, double-counts across a join, counts rows instead of orders, reads a partial month as a drop, or rebuilds a filter differently. How each happens, and how to check an answer.

James Swirhun

James Swirhun

Published Nov 5, 2026

James Swirhun is Head of Product at Credible, an AI analytics engine that gives AI agents, data apps, dashboards and reports one data model to answer from, so everyone gets the same numbers.

An AI agent's number can disagree with your dashboard because the two made different assumptions, because the agent made a mistake, or because a calculation was rebuilt differently. The dashboard can also be the one that's wrong. Both numbers look plausible, and neither query raises an error.

The figures below come from real queries on real datasets. Unless another post is named beside it, a figure is from our post How an agent turns a question into a trustworthy answer.

Why does the AI's number differ from the dashboard?

The question may have more than one valid answer, and the agent picked a different one than whoever built the dashboard. The two may have read different data. The agent may have made a mistake, such as a join that double-counts or a partial month read as a drop. Or a calculation written fresh for the question came out differently this time.

It assumed a different definition

On one ecommerce dataset, gross revenue for January to October 2022 is $4,510,963 and net revenue is $4,305,542. Net leaves out returned items ($44,143) and cancelled items ($161,279). A dashboard built on net and an agent that sums every sale will disagree by $205,421, and neither number is wrong. They keep disagreeing until someone writes down which one the business means. The same happens with "customers" (every account, or only paying ones?) and "summer" (June to August, or the company's sales season?).

A join counted things twice

Six orders total $955 across twelve line items. Join orders to line items in SQL and sum the order amount, and the result is $1,445, 51% too much, because each order's amount repeats for every one of its items. From our post The Semantic Layer Is an Interface, Not a Compiler.

It counted rows instead of orders

The same ecommerce dataset has 271,019 line items across 264,071 orders. Asked how many orders there were, an agent that counts rows would answer 271,019, which is 2.6% high. A dashboard that counts distinct order IDs says 264,071.

It read different data than the dashboard

A dashboard often reads a copy of the data refreshed overnight, while an agent queries live tables that include today's orders. One may read a cleaned table and the other the raw one, or a table that a migration replaced. The dashboard and the agent may date an order by when it was placed or when it shipped, or end the day in UTC instead of local time.

It compared a partial month with a full one

Asked how revenue was trending, an agent that doesn't check the date range would report a 56% drop in November: $224,791 against October's $505,857. But the data ends on November 14th. Per day, the two months are within 2% of each other. The agent in our run reported the trend through October, the last complete month, and said why.

It guessed a filter value

A filter on a value the data doesn't contain returns no rows and raises no error. In the ecommerce data, 255,883 rows have the status 'Complete'. A filter for 'complete', lowercase, matches none of those rows. The open-source analysis skill gives another case: a user types "ray ban", and the data stores Ray-Ban and Ray-Ban Junior.

It rebuilt a calculation, or picked a different reading

Steps written fresh for each question, such as a text filter or the denominator of a percentage, can come out differently each time. A saved query can also carry a filter, such as recent orders only, that a fresh query leaves out. When the model leaves two readings open, the agent can pick either one. In our eval runs, that was the usual reason a question passed on one run and failed on the next. From our post Your Data Model Is Part of Your AI System.

How do I check an AI answer in five minutes?

Ask the agent to show the query and the definitions it used, then run six checks from the open-source analysis skill, which tells an agent to run the same checks before it answers. All but the fourth need only the answer and its query.

  1. Check the time period. Is it the period you meant, and is the last period in it finished?
  2. Ask which definition was used. Gross or net? Which customers count? A stated assumption can still be the wrong one.
  3. Add up the parts. A breakdown of a sum should match the total, and a gap of a percent points to a join or a filter. Distinct counts and averages don't add up across groups.
  4. Recompute the headline number another way, such as from the raw rows instead of the named metric.
  5. Question an empty result. Check the filter value's spelling and case, the date range and the table before believing zero.
  6. Ask "percent of what?" A small channel can grow fastest in percent while adding few dollars.

How do you stop AI answers from disagreeing with the dashboard?

Write the definitions, joins and cleanup steps into one data model, and have every dashboard and agent query through it. Give the agent a method for checking its own work. Then test the whole system against questions with verified answers, and re-run the test on every change.

Write the definitions down where the agent reads them. A data model holds what revenue means, the company's calendar, how tables join, which values a field holds, and cleanup steps such as which orders count as complete. Every agent that queries through the model uses the same definitions. You decide what agents can reach, and connecting them only to the model keeps every answer on those definitions. In Malloy, each total knows which table its values come from, so a join can't count a total twice (why Malloy). A model can't decide what a question needs, such as whether a partial month matters, and an agent can still choose its own measure for a word the model doesn't define, such as "best".

Give the agent a method. The open-source analysis skills tell an agent to restate the question, look up definitions instead of guessing field names, and check its answer before it presents anything.

Test the whole system on your own questions. An eval is a list of real questions, each with an answer someone has verified. In one run, an agent read "summer sales" as June to August and was 20% low. The company's season runs from May 25 to September 15, and a few lines in the model that state the season fixed the answer. In the same run, a note added to fix one question made another go wrong, which is why the test re-runs on every change. From our post Your Data Model Is Part of Your AI System.

What do checked AI answers look like in production?

Teams that run agents in production test them against answers they have already verified, and re-run the tests on every change. bars.com scores its agent on twenty questions from real analyst work. Each answer is checked against a figure derived separately in SQL. The suite has run three times, with 20 of 20 correct each time.

VideoAmp puts AI answers in front of clients who buy and sell advertising on its numbers. Its teams wrote more than 2,000 definitions, business rules and warnings into the model. Before launch, they tested the AI against real analyst questions whose answers were certified in existing reporting, and metrics matched to the decimal. Queries that would double-count a metric return an error. When the AI mishandled a question in production, the fix went into the model, and every later answer used it.

How should a team start?

Everything this guide describes is open source. Malloy is MIT-licensed, and Malloy Publisher serves a model to agents over MCP, with the analysis and eval skills in the same repository. The tutorial gets an agent answering questions over your own files in a few minutes. You don't need to buy anything to use it.

The skills are plain files. If you've seen a mistake the skills don't catch, open a pull request against the skills in Malloy Publisher, and every agent that uses the skills will check for that mistake.

Credible runs the same models and skills as a managed service. Dashboards, data apps, chat and agents all query one published model, so they use the same definition of revenue and the same joins. A change to a definition is reviewed like code, then published as a new version that every dashboard and agent switches to at the same time. See how Credible governs data, pricing, or the docs.

Frequently asked questions

Why do AI revenue totals disagree with my dashboard?

The agent may have assumed a different definition than the dashboard, such as gross revenue instead of net. Or it made a mistake: a join that repeats rows, a count of rows instead of distinct orders, or an unfinished month read as a drop. The dashboard can also be the one that's wrong.

Why can text-to-SQL give wrong answers?

Text-to-SQL works from table and column names, which don't say what the business means. A column can look right but have been replaced in a schema migration. A table can be meant only for certain questions. A join can need two keys, not one, to match rows correctly. The query still runs and returns a number.

How do I stop an AI from making up metrics?

Give it definitions to read instead of columns to guess from. A data model states what each metric means, which joins are safe, and what values each field holds, so the agent has less to guess. An agent can still improvise, so test the agent with questions that have verified answers, and re-run those questions on every change.

How do I check whether an AI's answer to a data question is correct?

Ask it to show the query and the definitions it used. Check that the last period is complete, so a half month isn't read as a drop. Check that a breakdown of a sum adds up to the total. Recompute the key number another way. If a filter returned nothing, check the value's spelling and case.

Can I trust AI for financial analysis?

Only when the agent's numbers are tied to the same definitions finance uses and the answers are tested against figures finance has certified. VideoAmp wrote more than 2,000 definitions and business rules into its model, then tested its customer-facing AI against certified reporting before launch. Metrics matched to the decimal. Without that kind of test, treat an AI figure as a draft. How VideoAmp tested its AI →

The AI Analytics Engine

Make your data Credible

Model your data's meaning, and the AI Analytics Engine delivers answers you can trust to every agent, data app, and dashboard that asks.

Get started