Performance & Cost

Materialize sources, index dimensions, and pre-aggregate measures for fast, cheap, searchable models

Your model is documented and secured — the last stage before publishing is optimizing how your modeled data is stored and served, for performance and cost. Credible maintains managed, derived copies of your data, and you opt parts of your model into them with a single annotation. The engine builds the copy, keeps it fresh, reuses it where it can, and wires it into serving — so your model gets faster, cheaper, and more searchable without you managing any of the machinery.

There are three kinds of derived copy, each opted into at its natural grain:

Derived copy	Goal	What you opt in	Annotation
Materialized table	Make a source fast and cheap to query	A source	`#@ persist` on the source
Search index	Make a dimension's values searchable	A dimension	`#(index)` on the dimension
Pre-aggregation	Make a measure fast and cheap at coarse grains	A measure	`#@ preaggregate` on the measure

Each is derived from your published model, kept fresh by Credible, and reused automatically.

#(index) appears in two stories. For what to index and how it improves AI retrieval, see Discovery Metadata. This page covers the other side: how the index is built, kept fresh, and served.

Serving Behavior

A query serves from a derived copy when one covers it, and otherwise runs live against the source database. The result is always correct — if a derived copy isn't available yet, or doesn't cover the query, the query is simply slower, not wrong.

A materialized table persists a source's data as a physical table and routes queries to it.
A search index embeds a dimension's values so they are findable by value search, filter suggestions, and the AI agent.
A pre-aggregation stores a measure rolled up to a declared grain and answers queries at that grain, or any coarser one it can correctly re-aggregate to, from the rollup instead of the base source.

How the Derived Copies Compose

The three features work together automatically, with no extra configuration. If you index a dimension on a source you have also materialized, Credible builds the search index from the materialized table instead of re-scanning the warehouse — and reverts to the warehouse if you un-materialize the source. A pre-aggregation on that source is built the same way: from the materialized table when there is one, straight from the warehouse when there isn't.

You can rely on three properties:

Consistent — the index reflects the materialized snapshot, so searchable values match what queries return.
Stable — a table-backed index refreshes exactly when its source table refreshes. An index on a source you have not materialized refreshes on publish and on demand, and — if you declare a freshness window — on that window.
Cheap — indexing reuses the table you already built instead of paying to re-scan the warehouse.

The recommended pattern for an expensive, frequently-queried source is "persist the source, index its dimensions, and pre-aggregate its hot measures." Credible sequences the work for you so an index or rollup is always built after the table it derives from.

Deciding What to Persist

Materialize a source

Queried often or expensive to compute? Add #@ persist.

Index a dimension

Values that users or the agent search or filter by? Add #(index).

Pre-aggregate a measure

A hot measure queried at coarse grains? Add #@ preaggregate with its grain.

Leave it live

No annotation: queried live from the source database, not searchable.

The annotation alone is the intended usage. Defaults are chosen so the engine can optimize on your behalf — deduplicating copies across model versions and scheduling refreshes to meet a freshness objective. At most, add a freshness window for data with a real staleness requirement.

Two guardrails are enforced when you publish an indexed dimension: it may be partitioned by at most one required filter, and it may not sit on a source that requires parameters. Both surface as publish-time errors rather than silent wrong answers.

A gated source can use #@ persist, but not the other tiers. A colocated #@ persist on a source carrying an #(authorize) gate builds and serves normally: the build freezes the source's own relation and never evaluates the gate, and the gate is applied per request as a filter over the frozen copy. The caller's identity and the gate expression both stay live.

Still refused, because in each case no gate would be left to evaluate:

storage= — the built table is served to every caller as-is, carrying no gate.
#@ preaggregate — a rollup groups away the column the gate reads, so the grain can't express it.
A gate reached only through a join — the source must carry the gate at the entry point callers use. Persist the ungated base instead and let the gated source read through it live.

What a persisted gated source costs you is freshness, not access. The gate is re-evaluated on every request, but the column values it reads are frozen at build time. So a row that changes hands — a cost_center reassigned, an owner changed — keeps serving to its former owner until the source rebuilds.

Bound that with a materialization.freshness window plus fallback: live: an artifact that ages past the window drops out of serving and the query runs live instead. A gated source with neither a freshness window nor a rebuild cadence is one whose access decisions are as old as its last build.

Three things decide whether that window actually binds:

Declare it on the source's own tag — #@ persist name="..." freshness.window="24h" freshness.fallback="live". Window and fallback resolve independently, per property, per layer, so a package-level fallback: stale_ok silently defeats a window set on the source; and a package-wide window forces every other persisted source to recompute once stale.
refresh="incremental" does not bound revocation. A delta only re-reads rows past the watermark, so a row that changes owner without its watermark advancing is never re-read again — while the source keeps reporting an advancing boundary and reads as healthy. Only a full rebuild recomputes the gating column.
A content-identical sibling shares the artifact, and the window. Reuse is keyed on a content address that folds the connection and the SQL but not the source name, so two persist sources whose bodies compute the same SQL resolve to one table carrying one freshness policy. The tightest window any of them declares governs all of them. If two sources need genuinely different windows, give them genuinely different SQL.

Configuration

Annotations

Add the annotation to the source, dimension, or measure you want to persist. Options are optional key="value" pairs; omit them to accept the default.

#@ persist name="orders_fast" refresh="incremental" watermark="order_date" freshness.window="24h"
source: orders is conn.table('sales.orders') extend {
  #(index)
  dimension: status is order_status
}

name — choose where the materialized table lands (optional; container-qualifiable).
refresh — "full" (the default) rebuilds the whole copy; "incremental" applies only what changed and requires a watermark (see Incremental Refresh).
watermark / merge_key — how an incremental table finds and applies new rows (see Incremental Refresh).
freshness.window — the staleness objective the engine schedules against (see Freshness).

Package Manifest

Reuse scope and the refresh cadence are declared once for the whole package in publisher.json (the same manifest described in Publishing):

{
  "scope": "package",
  "materialization": { "freshness": { "window": "24h", "fallback": "live" } }
}

scope: package (the default) — a derived copy is reused across the package's versions whenever they define the same thing. Maximal reuse, lowest cost.
scope: version — each version keeps its own copies, with no cross-version reuse. Choose this when you want to own an exact rebuild schedule for a version.

Declare either a freshness objective or an explicit materialization.schedule, never both. A fixed schedule is the power-tier option and is only valid under scope: version.

Incremental Refresh

By default (refresh="full") every refresh recomputes the whole table. For a large, append-mostly source — a fact table that grows daily — that means re-reading years of data to add one day. Declare refresh="incremental" instead, and each refresh reads only the rows that are new since the last build and applies them to the existing table:

#@ persist refresh="incremental" watermark="order_date"
source: daily_revenue is orders -> {
  group_by: order_date
  aggregate: revenue is amount.sum()
}

The whole declaration lives on the #@ persist tag — the source body is exactly what you would write with no persistence at all, queries against the source are unchanged, and search indexes are already incremental with no declaration needed.

Key	Means
`refresh`	Set to `"incremental"` to advance the table with a bounded delta instead of a full recompute. Requires `watermark`.
`watermark`	Names the one output dimension a refresh derives its range from — an event timestamp, an order date, an ingestion time. Its values must be monotone: a given row's watermark value never decreases.
`merge_key`	Declare this only when a row's watermark value moves (e.g., `watermark="updated_at"` on a mutable table). Names the row's stable identity — one or more output dimensions, comma-separated — so a changed row is merged in place of its stale copy instead of appended beside it.

The three keys form a chain — merge_key requires watermark, and watermark requires refresh="incremental" — and publishing fails with a targeted error if any link is missing, if a named dimension doesn't resolve to an output column, or if it names a measure. You find out where you declared it, not by watching a table that never advances.

Which shape is yours

A rollup or an append-only fact — no merge_key. A row's order_date or ingested_at never changes, so each refresh replaces its date range outright. This also picks up rows deleted upstream within the refreshed range:

#@ persist refresh="incremental" watermark="ingested_at"
source: events is conn.table('raw.events') -> {
  select: ingested_at, event_id, user_id, payload
}

A mutable table — watermark plus merge_key. updated_at moves when a row changes, so the engine needs id to find and replace the stale copy:

#@ persist refresh="incremental" watermark="updated_at" merge_key="id"
source: accounts is conn.table('raw.accounts')

No monotone dimension — leave refresh unset. A small lookup table overwritten wholesale upstream has nothing to order rows by; full-copy is a supported, cheap answer.

Limits and repairs

Incremental trades completeness for cost, and two gaps are disclosed at publish rather than solved:

Late data. A row that arrives with a watermark value below the range already covered is never picked up automatically.
Hard deletes. A row deleted upstream can't appear in any delta, so a merge_key source retains it. Prefer soft deletes — a tombstone flag arrives as an ordinary update — and keep the flag in the persisted source's output, filtering it in consuming views instead.

Both are repaired the same two ways: correct the row upstream and advance its watermark so the next refresh applies it, or force a full rebuild with a Rerun from the package page (or forceFullRebuild on the runs API). Changing the model always triggers a full rebuild automatically — a delta is never applied across a logic change.

Non-additive measures — an exact count_distinct, a median — are safe in incremental sources: each refresh recomputes affected output rows from the full input rather than merging stored partial aggregates. Window calculations that look forward along the watermark (lead(), whole-partition percentages) are rejected at publish, because rows already materialized would go silently stale; trailing windows are fine.

Pre-Aggregations

A pre-aggregation makes a measure fast and cheap at coarse grains. You mark a hot measure and its rollup grain; Credible maintains a rolled-up copy and silently answers coarse queries from it. You never hand-write a rollup source, and no query ever names one — routing is a property of the engine, not a judgment the caller (or the AI agent) makes per query:

#@ preaggregate grain="order_time.day, category"
measure: total_revenue is amount.sum()

grain — required: the dimensions the rollup stores. A query is served from the rollup when everything it groups by and filters on is covered by the grain — a coarser truncation of a stored time dimension (order_time.month over a day grain) counts. Anything else falls back to the base source and runs live: "unsupported" and "unaccelerated" are the same, correct outcome.
#@ -preaggregate pins a measure to the base even when a covering rollup exists — the escape hatch for a consumer that can't tolerate the rollup's freshness.

Routing is correctness-aware — the engine never serves a silently wrong number from a rollup:

Additive measures (sum, count, min, max) and avg are re-aggregated from the rollup at any covered grain.
Non-additive measures (count(distinct), median, percentiles) can't be correctly re-aggregated to a coarser grain, so a rollup answers them only at exactly its declared grain — other grains run live. You're told this once, as a publish-time warning on the measure.

Everything else on this page applies unchanged. Measures declared at the same grain pack into a single rollup table, built by the same runs — from the materialized table when the base source is also #@ persist-ed, straight from the warehouse when it isn't (often the right choice: a rollup is frequently worth maintaining when a full copy of the base is not). Rollups share their base's freshness window — a rollup is never fresher than the table it was built from, and a stale rollup is skipped in favor of the base, never served — and they're reused and garbage-collected like any other derived copy.

For an expensive source with hot measures, this is the same three-part pattern described above — persist, index, and pre-aggregate — with Credible sequencing all three annotations for you.

Freshness

freshness.window is an objective, not a fixed refresh time: it tells Credible how stale the derived copy is allowed to get, and the engine schedules refreshes to meet it. This lets Credible batch work, run off-peak, and skip a refresh any recent publish or on-demand run already covered.

The fallback setting controls what a query does when a materialized table is older than its window — live runs the query against the warehouse instead of serving stale data.

Search indexes surface their staleness on the version page and in search and retrieval responses, so the agent can tell when suggestions come from an older snapshot.

Builds and Refreshes

On publish — Credible builds every persisted source, search index, and pre-aggregation for the new version automatically.
On demand — trigger a Rerun from the package page (or the runs API) to force a rebuild. You can rerun a whole version, or a single source or dimension — optionally including its upstream persisted sources.
On a schedule — the engine refreshes derived copies to meet their freshness objectives.

The version page shows Materialized sources and Indexed dimensions side by side, each with a simple status and its build and refresh history, so you can see at a glance whether a version is fully built.

What It Costs

Derived copies meter the way the pricing page describes: the tables, rollups, and indexes the engine keeps are storage, billed per GB-month; a query the engine answers from them meters compute time, billed per second; a query that runs directly on your warehouse incurs no compute charge from Credible. Storage is the optimization that makes the other two meters small.

Storage Reclamation

Credible garbage-collects every unused derived copy — a materialized table, index, or rollup is kept only while an unarchived package version references it. Archiving a version releases its references, and any copies no longer referenced by another version are reclaimed automatically. So the way to keep storage costs down is to archive package versions you no longer use — auto-archive (on by default) does this for you on a retention window you control.