Performance & Cost

Credible keeps managed, derived copies of your data to make queries faster and cheaper. You opt parts of your model in with an annotation, and the engine builds each copy, keeps it fresh, reuses it across versions, and serves from it.

A derived copy changes how fast a query returns, not what the query means. Between refreshes it can lag the source, by no more than the freshness window you set. For what to index for AI retrieval, see Discovery; this page covers how indexes are built and served.

Deciding What to Persist

For an expensive, frequently queried source, persist the source, index its dimensions, and pre-aggregate its hot measures. The bare annotation is usually enough: the defaults let the engine deduplicate copies across versions and schedule refreshes. Add a freshness window only for data with a real staleness requirement.

Access-controlled sources have limits on materialization; see Value Search and Materialization.

Serving Behavior

A query serves from a derived copy when one covers it, and otherwise runs live against the source database. A copy that isn't built, is outside its freshness window, or is at a grain the query can't use is skipped, so the query is slower, not wrong.

• A materialized table stores a source's data as a physical table and routes queries to it.
• A search index makes a dimension's values findable by value search, filter suggestions, and agents.
• A pre-aggregation stores a measure rolled up to a declared grain, and answers queries at that grain or any coarser one it can correctly re-aggregate to.

The copies build on each other automatically. On a materialized source, indexes and pre-aggregations are built from the materialized table rather than the warehouse, so:

• Values are consistent — the index matches what queries return.
• Refreshes are aligned — a table-backed index refreshes when its table does. An index on an unmaterialized source refreshes on publish, on demand, and on its freshness window if you set one.
• Builds are cheaper — no second warehouse scan.

Credible sequences the builds, so an index or rollup is always built after the table it derives from.

Configuration

Annotations

Options are optional key="value" pairs on the annotation; omit them to accept the defaults.

##! experimental.persistence

#@ persist name="orders_daily" refresh="incremental" watermark="order_date" freshness.window="24h"
source: orders_daily is orders -> {
  group_by: order_date is order_time.day, status
  aggregate: revenue is amount.sum()
}

• Persist a query, not a table. #@ persist goes on a source defined with ->. A plain table source (conn.table(...) extend { }) has nothing to materialize: the annotation is refused with a warning at load, and a build that covers it fails. To keep a filtered copy of a table, persist raw -> { select: * } and filter that.
• Turn on the experiment. Persistence is still experimental in Malloy, so a file with a #@ persist starts with ##! experimental.persistence. Without it, the file's persist annotations are skipped. #@ preaggregate needs no flag.

Package Manifest

Reuse scope and refresh cadence are set once per package in publisher.json (see Publishing):

{
  "materialization": {
    "scope": "package",
    "freshness": { "window": "24h", "fallback": "live" }
  }
}

scope belongs inside materialization. It still loads from the root of publisher.json with a deprecation warning, but never set both: two different values fail the package load.

• scope: "package" (default) — a derived copy is reused across versions that define the same thing. Lowest cost.
• scope: "version" — each version keeps its own copies. Use it to own an exact rebuild schedule for a version.

Set either a freshness objective or a materialization.schedule, not both. A fixed schedule requires scope: "version".

Incremental Refresh

A full refresh recomputes the whole table. For a large, append-mostly source, refresh="incremental" reads only rows newer than the last build:

#@ persist refresh="incremental" watermark="order_date"
source: daily_revenue is orders -> {
  group_by: order_date
  aggregate: revenue is amount.sum()
}

The source body and queries against it are unchanged. Search indexes are always incremental.

Publishing fails with a targeted error if a required key is missing, a named dimension doesn't resolve, or a key names a measure.

Choosing a Shape

Limits

Two gaps are reported at publish rather than handled automatically:

• Late data — a row arriving with a watermark below the range already covered is never picked up.
• Hard deletes — a deleted row never appears in a delta, so a merge_key source keeps it. Prefer soft deletes: keep the tombstone flag in the persisted output and filter on it downstream.

To repair either, correct the row upstream and advance its watermark, or force a full rebuild with Rerun on the package page (or forceFullRebuild on the runs API). Any model change triggers a full rebuild automatically.

Pre-Aggregations

Mark a hot measure with its rollup grain, and Credible maintains a rolled-up copy and answers coarse queries from it. No query names the rollup; the engine routes automatically.

dimension: order_day is order_time.day

#@ preaggregate grain="order_day, category"
measure: total_revenue is amount.sum()

• grain (required) — the dimensions the rollup stores, named as dimensions the source declares. An inline expression such as order_time.day is refused at publish, so declare the truncation as a dimension and name it. A query uses the rollup when everything it groups and filters by is covered, including coarser truncations of a stored time dimension (order_day.month over an order_day grain). Otherwise it runs live.
• #@ -preaggregate — clears every grain declared above it. It's how a source that extends another opts out of the base's rollups.

Routing refuses rather than guesses. Only measures whose partial results merge can be pre-aggregated:

A source with a join_many or join_cross can't be rolled up at all, because stored partials would double-count. Some queries are always answered live even when a rollup covers them: a query that runs a named view:, a query that passes a given, and a query that groups by an expression instead of a declared dimension.

Measures with the same grain share one rollup table. A rollup is built from the materialized table when the base is persisted, and from the warehouse when it isn't — often the right choice when a full copy of the base isn't worth keeping. A rollup is never fresher than its base; a stale rollup is skipped, not served.

Freshness

freshness.window is an objective, not a fixed refresh time: it sets how stale a derived copy may get, and the engine schedules refreshes to meet it — batching work, running off-peak, and skipping refreshes a recent publish or run already covered.

fallback sets what happens when a materialized table is older than its window. live, the default, runs the query against the warehouse instead of serving stale data. stale_ok keeps serving the stale table, for sources where recomputing live is the expensive part. Set either on the persist annotation (freshness.fallback="stale_ok") or for the whole package in publisher.json. A table with no window is never considered stale; it serves until its next build.

Search indexes report their staleness on the version page and in search and retrieval responses, so agents can tell when suggestions come from an older snapshot.

Builds and Refreshes

The version page lists Materialized sources and Indexed dimensions with their status and build history.

What It Costs

Derived copies meter as described on the pricing page:

• Storage — tables, rollups, and indexes, billed per GB-month
• Compute time — queries the engine answers from them, billed per second
• Queries run directly on your warehouse — no compute charge from Credible

Storage Reclamation

Credible garbage-collects unused derived copies: a copy is kept only while an unarchived version references it. Archive versions you no longer use to reclaim their storage — auto-archive, on by default, does this on a retention window you control.

Next Steps