The 4 Questions to Ask Before Deploying Semantic Layers in Production

Editor's note: This is a detailed, hands-on comparison of the Credible and dbt semantic layers — query vocabulary, business context, agent skills, deterministic control, information hiding, correctness guardrails, auth, and pricing — by Adam Ribaudo, founder of Noise to Signal, drawn from two production deployments. It first appeared on noisetosignal.co and is reproduced here with his permission.

Recently, semantic layers have come roaring back onto the scene. Previously characterized as nice-to-have infrastructure that only enterprise data teams could afford to maintain, they've since proven themselves as a critical ingredient for agentic analytics.

Despite this, it's relatively uncommon for data practitioners to find mature, production deployments of semantic layers in the wild. Over the last year, I've rolled out two different agentic analytics systems backed by semantic layers from two different vendors and thought I could share my experience. These are lessons learned from real deployments and all of the messy and surprising ways that business users interact with them.

I've organized the comparison across four questions you should ask of your overall solution design:

• What can an agent ask?
• What can an agent know?
• What can't an agent do?
• How does the platform run?

The semantic layer you choose has a material impact on the answers to these questions. While the comparisons below are applied to two specific vendors, you can bring these questions to any solution you may be developing.

First, let me introduce the vendors.

dbt

dbt hardly needs any introduction as it has dominated the data tooling conversation for half a decade. However, its capabilities as a semantic layer are less well known. This feature arrived through the 2023 acquisition of Transform, whose MetricFlow engine powers it today. MetricFlow itself went Apache 2.0 in late 2025, though the serving APIs remain a paid dbt Cloud feature. The 2026 Fivetran merger repositions the combined company around trusted data for AI agents.

Malloy and Credible Data

Malloy is a semantic modeling and query language created by Lloyd Tabb, Looker's founder and the creator of LookML. It was open-sourced under the MIT license in 2021. Credible Data, founded in 2025 by Kyle Nesbit (who led BI and data analytics for Google Cloud), is the commercial company behind it. Credible maintains the open-source Malloy Publisher server, raised a $10M seed in July 2026, and open-sourced its entire agent layer the same month.

The Four Questions

1. What can an agent ask?

Query language vs. metric catalog

Different semantic layers expose different vocabularies for how an agent might request data. As a comparison point, raw SQL has been proven to be such an unbounded vocabulary as to introduce errors in agentic systems. A semantic layer intentionally narrows the vocabulary to reduce error. In order of vocabulary size from smallest to largest we have:

• dbt's metric catalog
• Malloy
• SQL (no semantic layer)

dbt's semantic layer accepts a parameterized request. These are pre-defined metrics, group-by dimensions, and filters that are compiled to SQL. There are no custom aggregations, no nesting, and no query chaining. Contrast that with Malloy which is a query language. It provides the grammar necessary to express grouping, aggregation, nested breakdowns, window functions, and multi-stage pipelines. For an agent translating a messy business question, the test is two-fold:

• Can the vocabulary express the analysis required?
• Is the vocabulary so large as to introduce errors?

Too small a vocabulary and you risk not meeting the needs of your users. Too large of a vocabulary and your agent may introduce errors made possible by the language complexity. The examples below are what your agent will need to produce to successfully request data from each tool.

dbt request shape:

{
  "tool": "query_metrics",
  "arguments": {
    "metrics": ["revenue", "order_count"],
    "group_by": [
      { "name": "metric_time", "grain": "MONTH" },
      { "name": "product__category" }
    ],
    "where": "{{ Dimension('order__status') }} = 'completed'",
    "order_by": ["metric_time"]
  }
}

Malloy query shape:

run: orders -> {
  where: status = 'completed'
  group_by:
    order_month is order_date.month
    product.category
  aggregate: revenue, order_count
  order_by: order_month
}

In my experience, Malloy provides a goldilocks effect whereby you have the expressibility of SQL with the guardrails in place to prevent the errors introduced by text-to-SQL approaches. The choice will depend on the needs of the business, however.

Metrics vs. sources as the atomic unit of reuse

Semantic layers are built up from atomic, reusable units of work. The shape and mechanics of these units define what you can express and what you're left governing once your semantic layer is "done".

dbt's atom is the metric. Define "revenue" once and dbt can calculate it correctly across any combination of pre-defined dimensions and filters at the specified time grain. For a catalog of straightforward aggregations serving static assets (ie. a dashboard) this is exactly the right shape. This shows strain, however, when the framing of a question changes the metric definition. Supplying "% of revenue from new customers" cannot be derived from your "revenue" metric. Instead, we need two new metrics: "pct_revenue_new_customers" and "revenue_new_customers". As the domain of questions grows, so does the metric catalog. The resulting variations of "revenue" then undercut the promise of a single, clean, easy-to-govern metric catalog.

Malloy's atom is the source. A source wraps an entity like orders, customers, or products with its joins, dimensions, and measures. In this case, "revenue" is defined once as a measure associated with "orders" rather than published as a standalone metric. "% of revenue from new customers" therefore requires no new modeling even if the question was not anticipated. It's the same governed measure with filtered ratio components composed at query time. The net effect is less churn within the semantic model itself. The model changes when the business changes, not when the questions change.

run: orders -> {
   aggregate: pct_new is revenue { where: customer.is_new } / all(revenue)
 }

2. What can an agent know?

Business context: where it lives and how it reaches the agent

Accurate answers to messy questions depend on the agent understanding the business context behind the data. These are the caveats, assumptions, and traps that a veteran analyst holds in their head. Getting that knowledge to the agent at query time, inside a finite context window, becomes the next challenge after establishing the agent's query vocabulary.

dbt gives this knowledge exactly one place to live: description strings associated with metrics. These are delivered alongside the full list of metrics returned from the "list_metrics" MCP tool. With this information loaded into context, the agent judges the relevance of each metric based on the description. For small catalogs, this is easy to reason over. At scale, descriptions swell into multi-paragraph operating manuals which can impact LLM attention and token costs.

Malloy attaches text annotations to any primitive: sources, dimensions, measures, pre-defined queries, or views. Its serving layer, Malloy Publisher, then provides a getContext function that accepts a natural language question and returns only the relevant sources, views, dimensions, and measures using text embeddings. This becomes a major advantage in scaled environments where large lists of metrics and business entities compete for agent attention. However, for smaller data models, the question-embeddings-results loop adds unnecessary weight and latency. Fortunately, operators can always fall back to parsing the Malloy annotations directly which produces similar full-text results as dbt.

The skill layer

The semantic layer alone is never enough for an agent to meaningfully engage with a business user. The consuming agent needs a second layer: vocabulary, disambiguation rules, query strategy, verification habits, tone. Both dbt and Credible now publish agent skills, but they ship different things. dbt's dbt-agent-skills is a collection primarily focused on operating dbt itself. This includes skills for building models, migrating engines, and running commands. The craft an agent requires remains the data team's to write and hand-install. Credible ships with that layer: thirty skills co-designed with five MCP tools ("agents reason, skills guide, tools retrieve"), decomposing general analytical craft into governed, portable pieces, while business meaning stays in the model's annotations. That said, neither vendor understands your business and there is always work remaining to be done in bringing that domain knowledge to bear.

3. What can't an agent do?

Deterministic control

Some controls over agent behavior are too important to entrust to a prompt. For example, masking columns, or limiting queries to specific data snapshots. Malloy's model parameters ("givens") let the host application inject those values alongside the query. These parameters are outside the LLM's reach and applied no matter what query the agent writes. dbt has no request-time equivalent, so the modeler's choices are fixed at build time.

While not a requirement for every project, deterministic control wrapped around a probabilistic behavior can level up the overall reliability of the system.

Information hiding

Context engineering is a game of hide and seek. Your goal as the engineer is to hide unnecessary information from the agent until it's necessary. dbt and Credible approach this problem differently.

dbt hides information "for free" in that nothing is visible to the agent until components of your data model are promoted to the semantic model. This closed-by-default world ensures that SL information is exposed to an agent as an intentional act from an engineer. This avoids the possibility of accidentally exposing internal mechanics/transformations that would waste agent context and attention.

Malloy serves as both a data model AND a semantic layer. When both are exposed to an agent, you risk sharing details that are unnecessary at best or misleading at worst. There are two mechanisms that help prevent this:

• Access Modifiers - Field-level controls that change the visibility of fields. Set to: public, internal, or private.
• Access Control - Source-level controls that change the visibility of sources, rows, or columns. Affords role-based row-level access or column masking.

In this case, dbt's rigidity pays a dividend in that visibility controls are structural from the start. But Malloy's expressiveness affords more variety in that query inputs (such as the user executing the query) can dynamically change model visibility.

Correctness guardrails

Confidently wrong answers are the death knell of any agentic analytics pilot. The first time the CEO uses your chat agent and spots an obvious flaw is also the last time the CEO will use your chat agent. For this reason, you need to understand what guardrails are in place to avoid common query pitfalls. The most common trap being incorrect aggregations due to query fan-out.

Both dbt and Malloy target this failure but with opposite philosophies. dbt prevents the query by building the join graph itself and avoiding fan-outs entirely. With this, the agent can't write the dangerous join because it can't write joins at all.

Malloy makes the query safe: aggregates across join_many compile to symmetric aggregates, so totals stay correct during fan-out. The risk ends up not being correctness, but cost. The warehouse still materializes the joined product, and a multi-way fan-out can balloon to tens of millions of intermediate rows before aggregation.

4. How does the platform run?

API & MCP access and auth

How your agents and users connect to the semantic layer can have a major impact on your security and traceability posture. Both dbt and Malloy/Credible offer direct API access as well as MCP endpoints. However, the identity mechanisms are entirely different. dbt's hosted server is provisioned with a single static service token which means that every connection shares an identity. This also results in a plaintext token written into every MCP client's configuration. In contrast, Credible's hosted MCP service provides per-user OAuth which affords auditability, individual revocation, and least privilege.

dbt's service token is dead simple and makes it easy to get up and running quickly. For a pilot, the service token is fine. For an org-wide rollout, you'll likely want to wrap the API in your own per-user authenticated flow or use a service that allows OAuth connections.

Cloud pricing

Costs shape your incentives. Are you incentivized to tightly control access to your conversational agent or share it with the whole organization? Are you incentivized to keep your semantic layer covering one problem domain or many? In this category, the comparison between dbt and Credible is asymmetrical for an obvious reason: dbt (now Fivetran) is the incumbent and Credible is the venture-funded challenger. This puts each vendor in different corners of the ring: dbt maintains per-seat + queried metric pricing while Credible offers unlimited seats, a generous free tier and usage-based pricing.

Do I expect these positions to hold? Not really. dbt has famously tinkered with its pricing model several times over the last few years. Credible hasn't changed its pricing yet, but it's much younger which makes stability harder to assess. The reality of building agentic systems in 2026 is that you're building on sand. What's true today, whether it's token costs or platform fees, may not be true tomorrow.

Closing Thoughts

This article focuses on two vendors, but these comparison categories are universal. When embarking on a semantic layer and agentic analytics solution, you'll need to consider how to balance context, correctness, and access along with many other attributes.

A theme in many of the vendor comparisons above is that dbt offers a near-turn-key offering compelling for smaller semantic models when you're already running dbt Cloud. Malloy / Credible on the other hand is available as a pure OSS offering with a higher ceiling in terms of the capabilities it affords. I tend to reach for those more advanced capabilities and have a preference for deploying Malloy, but the irony is that I still use dbt daily for the nuts & bolts of data modeling and transformation. What's right for you will depend on your project's requirements.

Originally published at noisetosignal.co. Adam Ribaudo is the founder of Noise to Signal, a data and AI consultancy.