Materialization Is an Annotation, Not a Pipeline

Agents run far more queries than people do. An analyst might run a dozen queries in an afternoon. An agent runs a dozen sequential queries to answer a single user question. Each one is time the user spends waiting, and each one lands on your warehouse bill. So queries have to be fast and efficient.

The usual fix is to build summary tables ahead of time, with a job to fill them and a schedule to run the job. It works until someone changes a definition. Say revenue should now leave out cancelled orders. The summary table was built from the old definition. The data team will rebuild it after the ticket works its way through their queue. Until then, every query that reads the table still counts cancelled orders.

In Credible, the data model is the only thing you build. Add one line next to what you want to be fast, and the engine builds the tables, picks the right one for each query, keeps them fresh, and drops them when nothing needs them. Change the definition of revenue, and the engine knows the old table no longer matches. Every query runs live against the new definition until the engine rebuilds the table. Slower for a while, but never wrong. Nobody files a ticket.

Inside the AI Analytics Engine made that claim in a few sentences. This post shows it working on Malloy Publisher's storefront example: what you write, which table each query reads, what happens when a definition changes, and what Credible adds on top. At the end there's a prompt you can hand to a coding agent to run the whole thing on your laptop in about three minutes.

The short version:

Materialization is an annotation, not a pipeline, and the package is what ships.

Two Annotations, No Pipeline

Two things in a data model are worth making fast. A source that joins several tables, because every query against it pays for those joins. And a measure that queries keep rolling up (aggregating) the same way, like revenue by day, because each one re-reads the whole fact table to add it up. Malloy has an annotation for each.

• #@ persist goes on a source. The engine computes the source ahead of time, joins resolved, and stores the result as a table. Queries against the source read that table.
• #@ preaggregate goes on a measure, with a grain: the dimensions to roll it up by. The engine stores the measure rolled up at that grain. A query that asks for the measure at that grain, or a coarser one, reads the small rollup instead of the fact table.

That is all materializing means: computing part of the data model ahead of time and storing the result as a table. A stored table covers a query when it holds every field the query groups and filters by, and was built inside the freshness window you set, if you set one.

Build the same tables yourself and you need a job to fill each one, a schedule for the job, a place in the DAG so it runs after the tables it reads, and a name everyone downstream has to learn. Each piece is reasonable. But the table now holds opinions about your data (which grain matters, which filters to assume, what "active" means) in a second place, on a second release cycle. Nothing has to break for the two to drift apart. An agent that runs hundreds of queries a day sends most of them to that table, so whatever it assumes becomes the answer.

An annotation can't drift, because it doesn't define anything. Delete every #@ persist and #@ preaggregate in a model and every query returns the same answer, only more slowly.

How It Works

1. Annotate. Add #@ persist to a source or #@ preaggregate to a measure. The engine checks each annotation when the model loads. A grain has to name a dimension the source declares, and only sum, count, min and max can be pre-aggregated, because those are the aggregates whose per-group results add up to a correct total. An annotation that breaks a rule is refused, with a message that says why.
2. Build. Publishing builds every table the model declares, in an order worked out from the model's references, so a rollup is built after the table it rolls up. The build order (DAG) still exists. Nobody had to write it. Each table is identified by a fingerprint of the definition that produces it, not by its name. On the next build, an unchanged definition matches the table already there, so the engine keeps it. A changed definition gets a new fingerprint and a fresh build.
3. Serve. A query reads the cheapest table that covers it. The engine skips a table for three reasons: it isn't built yet, it's older than its freshness window, or it lacks a field the query needs. Then it tries the next cheapest: the rollup first, then the persisted table, then the raw source.
4. Keep fresh. On Credible, a freshness window is a goal, not a schedule. Say "never more than a day old," and the engine schedules the rebuilds to meet it.
5. Ship and clean up. On Credible, the tables publish with the model as one versioned package. A table that no published version needs anymore is dropped.

That serving rule is what makes it safe to hand all of this to the engine. Materialization is an optimization, never a correctness dependency, and that's why it can be an annotation.

The one thing the engine won't decide for you is how old is too old. A table with no freshness window counts as current until its next build. Set freshness.fallback="stale_ok" and the engine serves an out-of-window table on purpose, the right call when recomputing live is the expensive part. Either way, the tradeoff between speed and freshness is written in the model, in a diff someone reviewed, not discovered from a stale dashboard.

An Example Run

Publisher ships a demo package called storefront: a small online store's data, with a model over it. Its fact table, order_items, holds 25,356 order lines over three years, joined to products, customers and regions. We added one source:

#(doc) Order lines with their category, brand and region resolved, stored as a table
#@ persist name="order_lines"
source: order_lines is order_items -> {
  select: id, order_id, customer_id, created_at, status, sale_price, category, brand, region, gross_margin
} extend {
  dimension: order_day is created_at.day

  measure:
    #@ preaggregate grain="order_day"
    revenue is sale_price.sum()
    order_count is count(order_id)
}

order_lines keeps every order line with its joins resolved. revenue is rolled up by day. order_count counts distinct orders, so it has no #@ preaggregate. Publisher returns the SQL it ran with each result, so the FROM clause shows which table served each query.

Before the build, everything runs live. Revenue by month read the raw parquet file, joined the three dimension tables, and returned 36 months, from $29,013.04 in January 2023 to $125,923.84 in December 2025.

The build. One request built both tables in about a fifth of a second: order_lines, one row per order line, then the daily rollup, one row for each of the 1,096 days.

Revenue by month reads the rollup. The same query now read 1,096 rows instead of 25,356 and returned the same 36 values to the cent. So did revenue by year ($503,428.14, $681,892.66 and $912,857.17) and total revenue, $2,098,177.97.

Revenue by category reads the table. The rollup has no category, so it doesn't cover the query. The query read order_lines instead, with no joins left to compute, and put Outerwear first at $464,861.08.

Distinct orders read the table too. Daily distinct counts can't be added up, because an order with lines on two different days would be counted twice. So order_count has no rollup, and orders by month read order_lines. Annotate it anyway and Publisher refuses to load the model:

A documentation change rebuilds nothing. We reworded the source's #(doc) line and built again. Both tables were reused, because neither definition had changed.

A definition change can't read the old table. We added where: status != 'Cancelled' to the source. Before the next build, total revenue ran live against the new definition and returned $2,010,604.49. It couldn't read the old table, because that table carries the old definition's fingerprint. The next build rebuilt both tables, and the same query read the new rollup and returned the same $2,010,604.49.

That is the cancelled-orders problem from the intro, closed. A table is identified by the definition it was built from, so a changed definition is never served from a table built for the old one.

The run doesn't show speed. The storefront is small enough that every query returns in a few milliseconds either way. What changes is how much each query reads: 1,096 rows instead of 25,356 for revenue by month, and no joins for revenue by category. On a fact table with a few billion rows, that difference is a big bill and a long wait.

What Credible Adds

The same annotations work in open-source Publisher, which builds and serves the tables. Credible runs them as a managed service and adds three things.

Freshness. Credible schedules each rebuild to keep a table inside its freshness window. A window can go on the annotation (freshness.window="24h") or apply to a whole package in publisher.json.

Releases. The model, its data apps, the docs, the access rules and the stored tables publish together as one versioned package. The definition change above would be one diff and one review, shipped as a new version that builds its own two tables. While those build, "latest" keeps serving the previous version, so nobody sees a half-built answer. If the new numbers look wrong, you point "latest" back at the previous version, whose tables are still there. There's nothing to keep in sync, because there's only one thing.

Cleanup. Every stored table carries a reference count of the published versions that can still reach it, and when none can, Credible drops it (storage reclamation). Auto-archive is on by default and retires a version a set time after it stops being latest. That window is yours to set, in hours, days or weeks. Set it to 7 days, and a week after the definition change ships, the old version is archived and its two tables are dropped. The 7 days you could roll back are the 7 days the old tables are kept.

Compare that with a pipeline nobody needs anymore. The job keeps running, the table keeps filling and the warehouse keeps billing, because nobody is sure what reads it. Working that out across a whole stack is most of a migration project. In the engine, a data stack migration becomes a garbage collection problem, and the engine runs the collector.

What It Costs

A source that re-scans a fact table on every query pays for that scan every time. Persist it, and queries read a smaller pre-built table instead. Where that table lives is your choice. Built in your own warehouse, it's your warehouse's storage and your warehouse's bill, and Credible charges nothing for the queries that read it. Built in the engine's storage, Credible bills the table by the GB-month and the queries it answers from it by compute time, per second. The pricing page has the rates.

Try It

You need Node 20 or newer, git and curl. Open Claude Code, or another coding agent, in an empty directory and say:

It takes about three minutes. Expect the build to make two tables in well under a second. Revenue by month should read the daily rollup and start at $29,013.04 for January 2023. Revenue by category and order_count by month should read order_lines, with Outerwear first at $464,861.08 and 158 orders in January 2023. Those are the same numbers the queries return live.

Then point it at your own model. The malloy-materialization skill covers the annotations, the build, and how to confirm a query read the table. Its tuning guide reads a package's build history and recommends what to persist and what to stop persisting. The persistence docs have the full syntax, and Rethinking Data Transformation with Malloy lays the language out against dbt.

Making a query fast used to mean building something underneath the model: a table, a job, a schedule. Now it's a line in the model, checked when the model loads, built in the right order, and served only when it covers the query. You own the what. The engine generates the how. And making it fast is part of the how, not a separate project.