Abstract

Manta’s agents reason over a knowledge graph — an in-memory graph that runs inference to plan and take actions on a business’s behalf. Reasoning engines like this are fast precisely because they hold their working set in RAM, which means they cannot hold everything: the raw, row-level operational data streaming in from external platforms is orders of magnitude too large to fit, and would be pointless to load fact-by-fact anyway.

What the graph actually needs is statistics — aggregates, rollups, rates, moving windows — kept fresh. So we compute those statistics where the data already lives (a columnar analytics warehouse) and feed only the summaries into the graph. The hard part is doing this continuously and cheaply when data arrives from many customers, across many platform connections, at unpredictable times.

Our answer is StarRocks asynchronous materialized views (AMVs for short): partitioned by integration type and integration key together — a multi-column partition key aligned with the raw tables, where the key is the thing data actually arrives as: a single connected account. Because that connection lives in the partition key, a load for one account refreshes only that account’s partition and leaves every other connection — the same customer’s other accounts, and every other customer — untouched. Refresh cost tracks the size of the change, per connection, all day long.

1. Why a knowledge graph needs statistics

Manta doesn’t just store a business’s data — it reasons over it. Each customer’s world is modeled as a knowledge graph — a live context graph for the agent: an in-memory RDF store running a rule-based inference program that derives new facts, validates them against a schema, and drives an agent loop that proposes and executes changes.

For that reasoning to be useful, the graph has to know things like:

• “This campaign has already spent its entire daily budget in the first hour of the day.”

• “This campaign’s click-through rate over the last 7 days is 40% below the account’s trailing 30-day median.”

• “This budget line has spent 80% of its allocation with 3 days left in the flight.”

• “This creative’s cost-per-conversion is trending up week over week.”

None of those are raw facts. They’re aggregates over time windows and entity keys — and the first one is the reason we care about grain: a runaway campaign that burns a full day’s budget in an hour is only visible if the statistics both resolve to the hour and reach the agent while that hour is still young — fine grain and near-real-time freshness, together. Daily rollups, or hourly ones refreshed only overnight, would surface it the next morning: too late to matter. So we want statistics that are hourly-grained and refreshed in near-real-time. They’re also exactly the kind of thing a rule engine should consume, not compute: a rule that has to sum millions of raw metric rows before it can fire is a rule that will never fire fast enough to matter.

The design philosophy is that the graph should be a lean, live model of the world — a small, continuously refreshed context an agent can read on every turn — not a data lake it has to scan.

2. The larger-than-RAM problem

The tension is a size mismatch that holds even for a single tenant, before multiplying across all of them.

An in-memory reasoner holds its graph — and every fact it infers on top — in RAM, with a fixed capacity sized up front per tenant. That is the source of its speed, and also its ceiling: overrun the capacity and the load fails.

Meanwhile the raw side is genuinely large. A single performance-metrics feed produces one row per entity × hour, per connected account — and each of those rows already carries a couple of dozen metric columns (impressions, clicks, spend, conversions, revenue, and derived rates). Across every entity, every hour, every account, every tenant, and all of history, that is billions of rows. Materializing all of that into graph facts, row by row, would blow past any reasonable in-memory budget while burying the reasoner in data it never queries directly.

So this isn’t a wall — never aggregate in the graph — it’s a placement decision on a gradient:

1. Data the reasoner touches directly belongs in RAM. Facts an agent reads on nearly every turn, entities it reasons over one at a time, anything that changes often and is small enough to hold: load it straight into the graph. In-memory inference over it is the fast path — that’s the whole point of the engine, and we lean on it.

2. Aggregation that would blow the RAM budget belongs where the data already sits. When an answer needs a GROUP BY over millions of rows the reasoner never reads individually, holding those rows in RAM and grinding them down on the hot path is the wrong end of the trade — it works, it’s just costly. A columnar MPP warehouse is built for exactly this — SUM/AVG/window functions over columnar storage is its home turf — so push the aggregation down and load only the result.

So the design principle is a dial, not a rule: keep in the brain what the brain reasons over directly; compute the heavy aggregates next to the data and load only the summaries. This piece is about the second setting of that dial — the case where deferring to a materialized view clearly wins.

3. Why materialized views — and why asynchronous ones

A statistic that has to be fresh is, definitionally, a view that must be maintained. That’s the textbook case for a materialized view: define the aggregate once as SQL, let the engine keep the physical result up to date.

But a naive materialized view has a fatal cost profile for us. Consider the shape of our ingestion:

• Data arrives one load at a time, per platform connection — a single connected ad account.

• A “load” might be one account’s hourly metrics refresh — a few thousand rows restating the last few days (platforms keep revising recent numbers as conversions settle).

• These arrive all day long — roughly hourly per connection, but staggered and unpredictable across many accounts and many tenants.

If every one of those small loads triggered a full recomputation of a view spanning all accounts and all history, we’d be doing terabyte-scale scans to absorb a kilobyte-scale change. The refresh cost would be decoupled from the change size — the classic way to make “materialized views” a synonym for “expensive.”

This is where StarRocks Asynchronous Materialized Views fit this problem closely. Three properties matter — and the first is the one everything else hangs on.

3.1 Refresh happens at partition granularity

This is the fact the whole cost model rests on: StarRocks refreshes a materialized view one partition at a time. When data in a base-table partition changes, StarRocks refreshes only the corresponding view partition(s); the rest is untouched. The partition_refresh_number property caps how many partitions a single refresh task will touch, so large refreshes split into batches. (Create a partitioned materialized view)

The consequence is a design rule we lean on hard: whatever you want to refresh independently must live in the partition key. Refresh isolation is a property of the partition key, so to give a single account’s load its own refresh boundary we make that account part of the partition key.

3.2 Partitioned, multi-column, incremental refresh

A StarRocks AMV can be partitioned, with its partition columns aligned to the base table’s — that alignment is what makes per-partition, incremental refresh possible. The partition key can span multiple columns, so we key it on the two that identify a connection: the integration type (meta, google, …) and the integration key (the specific connected account, acct_1234). Both the raw table and the AMV are partitioned by exactly this pair: (StarRocks docs: Asynchronous materialized views)

That is what lets us refresh one connection at a time — one account’s partition. Time lives as a plain GROUP BY column inside the view (hourly-grain rows), and keying on the connection keeps the partition count bounded.

3.3 Automatic refresh on data change

An AMV declared REFRESH ASYNC is refreshed automatically when its base tables change — “each time the base table data changes, the materialized view is automatically refreshed” — with no external scheduler polling “is there new data yet?”, and refresh tasks target only the affected partitions. (REFRESH MATERIALIZED VIEW)

This is why we land the raw feeds in native StarRocks tables: native OLAP tables version-track every mutation — insert, upsert, or delete — so every change bumps the affected partition’s version and triggers a refresh of exactly that partition.

The refresh mechanics are worth understanding for capacity planning: the frontend (FE) detects changed partition versions and enqueues refresh tasks, which are then scheduled against finite cluster resources (partition_refresh_number batches the partitions within a single task, and an MV can be pinned to a dedicated resource_group that caps its CPU/memory/concurrency). Each task is cheap and with many connections loading small batches all day, the dimension to size is aggregate refresh-task throughput. Combined with a partition key that carries the connection, the outcome is exactly what we want.

A small load touches only one connection’s data → that single (integration_type, integration_key) partition is the only one StarRocks refreshes → the cost of the refresh is proportional to the size of that connection’s partition, not the size of the whole view.

4. The architecture, end to end

Here’s how a byte of raw data becomes a fresh statistic inside a tenant’s reasoning graph.

The AMV layer sits at the top of a materialization pipeline that already knows how to react to new data and stream it into the graph. Instead of scanning raw tables, the pipeline scans materialized-view output — and gets pre-aggregated, always-fresh statistics for free.

Let’s walk the stages.

4.1 The materialized view

We define the statistics we want as AMVs over the raw tables. A representative one — hourly performance, partitioned by (integration_type, integration_key) and clustered by campaign, refreshed automatically and incrementally.

A few deliberate choices:

• The connection is in the partition key — that is what buys refresh isolation. StarRocks refreshes at partition granularity, so making the connection the partition key gives each account its own refresh boundary. A load for (meta, acct_A) lands in that one partition; StarRocks refreshes exactly it and leaves every other connection — including the same customer’s other accounts and every other customer — untouched.

• Campaign is the distribution key. We hash-distribute by campaign_id within each connection’s partition, which clusters one campaign’s rows into few tablets for read locality. The one cost: an account’s load re-aggregates that account’s other campaigns and its other retained hours too — a small over-refresh, confined to a single connection.

• Additive summaries + hourly rates, never raw rows. The SUM columns compose: a trailing-7-day total is just the sum of that window’s hourly rows. The hourly ctr / cost_per_conversion are conveniences for the “how did this hour do” question — but a windowed rate like a 7-day CTR is recomposed as a ratio of the summed components (Σclicks / Σimpressions over the window), because an average of hourly ratios is a different, wrong number. So the graph receives additive ingredients plus the hourly answer; any window is one cheap recomposition away, either in a trailing-window AMV or in the reasoner. What never crosses the boundary is a raw row.

• Partition count is bounded by connections. The number of partitions tracks how many accounts customers have connected, so it stays bounded and predictable. Per-partition refresh cost is bounded in turn by the raw feed’s rolling retention window.

Because query rewrite is transparent, the same AMV can also accelerate ad-hoc analytics that hit the raw table — a genuine bonus, though subject to StarRocks’ automatic-rewrite eligibility rules (the query’s aggregation has to be derivable from the view, and freshness settings have to permit it). Useful when it applies, not a free lunch, and not why we built it.

4.2 Reacting to a change

The change announces itself, and it does so in the view’s own terms. When a load moves account A’s last two days, the refresh touches exactly one partition — (meta, acct_A) — and it becomes a discrete unit of work. StarRocks records which partitions each incremental refresh touched (visible through the refresh metadata / task-run history), and the materialization pipeline keys off exactly that set: a changed partition → a slice to process → materialization fires. Because the partition already encodes the integration type and key, the unit of work is self-describing — the changed connection’s partition, keyed and ready to process, with no polling and no synthetic batch tag.

4.3 Read → map → compact triples

The pipeline reads exactly one changed slice, over a columnar transport (StarRocks exposes an Arrow Flight SQL endpoint, so the read and transfer stay columnar right up to the point where we explode each row into triples — no row-by-row roundtrip).

Because the source is an AMV, the result is small — one row per campaign-hour, not per raw event. And because both key predicates hit partition columns, StarRocks prunes straight to the single (integration_type, integration_key) partition; the event_hour filter then trims within it. Each slice is split into modest chunks and mapped, declaratively, into RDF triples. The mapping is pure specification: a table of columns to graph predicates, with keys forming a stable identity per grain. One hourly-stats row becomes triples the reasoner can query directly.

The chunks are serialized to a compact, streamable RDF encoding and staged in object storage, so the load step is cheap, replayable, and restartable after a failure.

4.4 Load into the graph

Finally the triple chunks are loaded into the tenant’s graph through a write path that serializes writes per tenant (the in-memory store is a single writer) and handles capacity growth and idempotency. Each connected account maps to exactly one tenant, so a changed partition routes unambiguously to one graph; and because each tenant maps to its own isolated graph, one customer’s statistics load never blocks another’s — the per-connection separation that starts as a partition key in the AMV is preserved all the way into the brain.

5. What this buys us

• Cost proportional to the change — per connection. Because the connection is in the partition key, a small recent load for one account refreshes only that account’s partition. Every other connection — the same customer’s other accounts, and every other customer — is never re-read. Across many accounts loading independently all day, total refresh cost is the sum of many small per-connection refreshes, not one fleet-wide recompute every few minutes. This is the difference between “continuously affordable” and “recompute the world on every load.”

• Near-real-time freshness. Auto-refresh-on-change means the statistic is ready shortly after the data lands, and the pipeline pulls it into the graph on its next tick.

• The graph stays lean. Only summaries cross the boundary into RAM. The reasoner reads pre-computed hourly rates and composes any window it needs from additive counts — never a raw-row GROUP BY on the hot path.

• Isolation, end to end. The connection is a partition key in the warehouse, so refreshes are physically independent per account; statistics flow to a per-tenant prefix in object storage, and into a separate graph in memory. And when a single account’s volume grows enough to warrant its own partition budget, it peels off into its own AMV without touching the pipeline.

• Declarative all the way down. The statistic is one SQL view; the graph shape is one mapping specification; the orchestration is the pipeline that already exists. Adding a new statistic is adding a view and a mapping, not a new service.

6. Trade-offs and honest limitations

No architecture is free. The ones we watch:

• Eventual, not transactional, freshness. There are two asynchronous hops — the warehouse refresh, then the materialization pipeline. The graph is near-real-time (seconds to minutes), not point-in-time consistent with the warehouse. For steering decisions, that’s the right trade; anything needing an exact-now number should query the warehouse directly.

• The partition key is a cardinality budget — and we spend it on the connection. One partition per connected account, and StarRocks has a practical ceiling (~100k partitions) on how many a cluster wants to track. (Partitioning best practices) Connection count is naturally bounded — it tracks how many accounts customers have connected and plateaus — so the budget stays comfortable. Two escape hatches when a single customer connects an enormous number of accounts: bucket integration keys into a fixed number of shards — trading exact per-account isolation for a hard cap on partitions — or peel a genuinely huge account into its own dedicated AMV.

• Refresh is per partition. A load can never be cheaper than one partition’s refresh: it re-aggregates that connection’s whole retained history — all its campaigns and every hour still in the raw retention window — even if it moved a single campaign-hour. That’s a bounded, per-connection cost, kept small by the raw feed’s rolling retention, and confined to one account.

• Refresh-task throughput is a cluster dimension, not just per-partition cost. Async refresh enqueues a task per changed partition; per task is cheap, but many connections loading all day produce many tasks, and the FE scheduler runs them against finite cluster resources (bounded, e.g., by the MV’s resource_group). Sizing the cluster means sizing aggregate refresh throughput and queue depth, not only the cost of a single partition’s recompute.

• Aggregation grain is fixed at view-definition time. A view grouped by (integration_type, integration_key, campaign, hour) can’t answer a sub-hourly (minute-level) question, and — as above — its hourly rates aren’t windowed rates. We expect a small family of AMVs at different grains — hourly rollups, coarser daily and trailing-window stats (which recompose windowed rates from the additive hourly counts), account-level benchmarks — rather than one universal view. Nested materialized views can compose these, though nested incremental refresh has its own partition-alignment caveats, so we lean on it deliberately rather than by default.

• Incremental refresh has rules. Partitioned incremental refresh works cleanly only when the view’s partition column(s) are derived from the base table’s — so the base table must be partitioned by the same (integration_type, integration_key) multi-column key, which is a real constraint on the ingestion layout and the price of the isolation. Some SQL shapes (certain joins, non-partition-aligned expressions) force broader refreshes. Views have to be designed for incrementality, not just correctness.

7. Where this goes

The pattern generalizes well past performance metrics. Any time the graph needs a derived, windowed, or cross-entity number over data too large to hold, the same shape applies: define it as a partitioned AMV whose key carries the dimension you want to refresh independently, let the warehouse maintain it incrementally, let the materialization pipeline stream summaries into each tenant’s reasoner. Trailing-window statistics, account-level benchmarks, pacing and anomaly signals — all are views waiting to be written.

The deeper point is architectural: a reasoning engine should reason, not aggregate. By putting the aggregation where the data lives, choosing a partition key that matches how the data actually arrives — one connection at a time — and making the maintenance incremental, we keep the brain small, fast, and — most importantly — current, without pretending RAM is infinite.