@oxy-hq/cli 0.1.0

package/skills/oxy-custom-apps/SKILL.md

@@ -0,0 +1,330 @@
+---
+name: oxy-custom-apps
+description: Use when building, debugging, reviewing, or shipping an Oxy custom-app bundle (a code-first React/Vite app served by oxy), including writing Oxy Functions and driving them from the UI. Encodes the contract rules, pitfalls, performance guardrails, and the self-serve `oxy publish` ship flow.
+---
+
+# Customer-app development
+
+Customer-app bundles are arbitrary JS apps served by oxy at
+`https://<env>/customer-apps/<org>/<slug>/`. Oxy is a data gateway:
+serves static assets, gates cookie auth, and proxies queries. The
+bundle is opaque above the data plane — no per-app server config.
+
+## The Oxy App Kit (use it)
+
+Two small packages remove the footguns — default to them rather than wiring a
+bundle by hand:
+
+- **`@oxy-hq/sdk`** — the data layer (`useQuery`, `useSemanticQuery`,
+  `useAgentRun`, `<OxyChat>`); see rule 2 and `oxy-hq/customer-apps:
+  docs/sdk-flow.md`.
+- **`@oxy-hq/vite-plugin`** (`oxyApp()` in `vite.config.ts`) — bakes the asset
+  **base path** from the manifest, copies `oxy-app.json` into `out/`, proxies
+  `/api` in dev, and injects the `window.__OXY_APP__` shim. Dropping it
+  re-introduces the #1 bug — asset 404s → blank dashboard (rule 3).
+
+Scaffold both with `@oxy-hq/create-oxy-app` or by copying `oxy-hq/customer-apps:
+examples/hello-oxy/`. The
+platform is kit-agnostic (any bundle with a correct base path works), but
+hand-rolling means owning the base-path + dev-proxy wiring yourself.
+
+(**`oxy-hq/customer-apps:` prefixes a path in that repo**, not in whatever repo
+you are sitting in. This skill loads globally, so every such pointer is
+qualified.)
+
+## Three rules
+
+1. **`oxy-app.json` (project root, not `public/`) is identity only.**
+   Schema:
+
+   ```json
+   {
+     "schemaVersion": 2,
+     "slug": "url-slug",
+     "orgSlug": "your-org",
+     "name": "Display Name"
+   }
+   ```
+
+   Required: `schemaVersion` (must be `2`), `slug` (non-empty).
+   Strongly recommended: `orgSlug` (lets `@oxy-hq/vite-plugin` derive
+   the asset base path automatically, so plain `pnpm build` produces
+   a bundle ready to drop into the linked org with no env var).
+   Optional: `name` (display), `projectId` (dev-time hint).
+
+   At serve time, oxy injects the authoritative org + project from
+   the admin row via `window.__OXY_APP__` — the manifest's `orgSlug`
+   and `projectId` are advisory hints only, never enforcement.
+
+   Do not invent `products`, `writers`, or `from:`-blocks. Those
+   don't exist in v2 and the probe rejects them.
+
+   File location: **`<root>/oxy-app.json`**, sibling to
+   `vite.config.ts`. The vite-plugin reads it from there; legacy
+   bundles that kept it under `public/` won't see the plugin's
+   base-path derivation kick in.
+
+2. **All data flows through SDK hooks → oxy HTTP endpoints.** The
+   surface is intentionally small:
+
+   | Hook / component | Endpoint | When |
+   | ---- | -------- | ---- |
+   | `useQuery({sql})` | `POST /api/projects/:id/query` | Raw SQL, 10k row cap |
+   | `useSemanticQuery({topic, …})` | `POST /api/projects/:id/semantic-query` | Reference topics/measures by name |
+   | `useAgentRun({agentId})` | `POST /api/projects/:id/agents/:id/asks` + SSE | Streaming chat / Q&A |
+   | `useProcedureRun({procedureId})` | `POST /api/projects/:id/procedures/:id/runs` ⚠️ beta | Long-running batch |
+   | `useFunction(name)` | `POST /customer-apps/<org>/<slug>/fn/<name>` + SSE | Invoke a server-side **Oxy Function** (see below) |
+   | `<OxyChat agentId="…"/>` | wraps `useAgentRun` | Drop-in chat UI |
+   | `<OxyAnswer {...run}/>` | renders any run | Drop-in answer renderer |
+
+   ```tsx
+   import { OxyChat, useQuery } from "@oxy-hq/sdk";
+   const { rows } = useQuery({
+     sql: "SELECT col FROM t WHERE x = {{ params.x | sqlquote }}",
+   }, { params: { x: someState } });
+   // ...
+   <OxyChat agentId="analytics" />
+   ```
+
+   See `oxy-hq/customer-apps: docs/sdk-flow.md` for the full reference,
+   including `OxyApiError` parsing and beta-surface caveats.
+
+   **Server-side work → Oxy Functions, not the browser.** `useQuery` &
+   friends run in the client; for work the browser shouldn't do
+   (warehouse writes, ELT kick-offs, external API calls, secret-bearing
+   logic) ship a **TypeScript handler under `functions/`** and invoke it
+   with `useFunction(name).invoke(body?)`. Declare it in `oxy-app.json`:
+
+   ```json
+   { "schemaVersion": 2, "slug": "my-app",
+     "functions": { "daily-rollup": { "route": true, "timeoutSeconds": 30 } } }
+   ```
+
+   ```ts
+   // functions/daily-rollup.ts — runs on Oxy's isolate runtime
+   export default async (req, ctx) => {
+     const { rows } = await ctx.query("SELECT … FROM …");   // data-plane ctx
+     return Response.json({ rows });
+   };
+   ```
+
+   `ctx` bridges to the data plane: `ctx.query` / `ctx.queryStream`,
+   `ctx.semantic.query`, `ctx.warehouse.{insert,exec,upsert}`,
+   `ctx.airway.run`, `ctx.fetch` (SSRF-allowlisted), `ctx.env`,
+   `ctx.user`, `ctx.log`. `oxy publish` bundles `functions/*.ts` with
+   esbuild — no separate backend to stand up. Worked example +
+   author-facing `ctx` types: `oxy-hq/customer-apps:
+   examples/hello-oxy/functions/`.
+
+3. **`OXY_APP_BASE_PATH` must match the path the bundle is linked
+   under.** `@oxy-hq/vite-plugin` resolves it from (in order):
+   `OXY_APP_BASE_PATH` env → `orgSlug + slug` in `oxy-app.json` →
+   `/`. `oxy publish` sets that env to `/customer-apps/<org>/<slug>/`
+   for the build automatically, and the vite-plugin derives the same
+   from the manifest for `pnpm dev` / `pnpm build`. Mismatch → asset
+   404s → blank dashboard.
+
+## Four pitfalls
+
+- **CORS errors are usually wrong-URL errors in disguise.** The SDK's
+  base URL must be `${OXY_URL}/api`, not `OXY_URL` bare. The API
+  router (which carries CORS headers) lives under `/api/*`. Hitting
+  `/<projectId>/...` lands in the static fallback and the missing
+  headers look like a CORS failure.
+
+- **Cookie auth requires `credentials: "include"`.** The SDK's
+  `useQuery` sets this. If you fetch directly, set it yourself or the
+  `oxy_session` cookie won't be forwarded.
+
+- **`projectId` in the manifest is advisory, not enforcement.** The
+  SDK uses it to construct the query URL. Authorization is the user's
+  cookie → user → org_member of the project's org, decided
+  server-side. A bundle that ships a stolen `projectId` can't read
+  data the user can't already read.
+
+- **Cloud mode is the target.** Don't branch on local vs. cloud; don't
+  rely on the nil UUID `00000000-...` as a default project; don't
+  assume `OXY_STATE_DIR` is writable.
+
+## Performance guardrails
+
+Any app that ships an Oxy Function or drives a batch of them should keep these —
+cheap up front, hard to retrofit. (Established by the pokehouse `post-je` function.)
+
+**In an Oxy Function — count the round-trips.** Each isolate run is stateless and
+pays a fresh network hop per `ctx.query` / `ctx.warehouse.*`, so serial *dependent*
+calls dominate latency.
+- Run independent data-plane calls with `Promise.all`, not sequential `await`
+  (e.g. resolve reference data **and** provision tables concurrently — only the
+  final step needs both).
+- Fold a check into a query that already runs: a duplicate/existence guard is a
+  scalar subquery on the main `SELECT`, not a second round-trip.
+- `Promise.all` idempotent DDL (`CREATE TABLE IF NOT EXISTS`); better, keep
+  provisioning off the hot path if a migration can own it.
+- A "cheap" extra `ctx.query`/`ctx.log` is N× the cost across N invocations.
+
+**Driving many function calls from the UI (bulk actions) — don't go serial.**
+- Independent calls run with **bounded concurrency** (~3–4) via a worker-pool over a
+  queue, not one-at-a-time — turns an O(N) wall-clock batch into a few waves.
+- If the calls share mutable server state that races under concurrency (a rotating
+  token, a lazily-created table), the best fix is to **remove the shared mutation**
+  (e.g. a single-writer scheduled refresher owns the token; requests only read).
+  When you can't, warm the state with one call first, then fan out.
+- Don't refetch derived state (a status ledger) after *every* item — the progress UI
+  is the live feedback; **throttle** the refetch (~1/s) and flush once at the end.
+- Guard `setState` after the component may have unmounted mid-batch (`mounted` ref).
+
+**Trust the server, not the client body.** Environment / tenant / which-secret
+selection is derived server-side (a function secret, a server-side mapping table,
+the authed identity), never from the request body — validate or override anything
+the client sends that picks a target.
+
+**Reuse the SDK's data-plane caching.** `useQuery` already dedups concurrent identical
+queries and short-caches results — don't hand-roll a fetch that bypasses it. Mark
+read-only functions `cache: { ttlSeconds }` in the manifest; **never** cache a
+side-effecting one (it silently drops writes). `idempotencyKey` = a fresh UUID
+per ATTEMPT (click), never a stable per-subject key: the server stores each
+key's response indefinitely and replays it WITHOUT running the function, so a
+`store:period` key silently breaks any post → compensate (delete) → post-again
+cycle — the second post "succeeds" with the stale response and writes nothing.
+The key shields one request from transport duplication; cross-attempt dedup
+belongs in the function itself (a server-side guard over real state, e.g. a
+posted-ledger row the compensating action clears).
+
+**Watch-outs when optimizing (each surfaced in review):**
+- **Clear UI running-state in `finally`.** An async batch driver must reset its
+  `running`/loading flag (and do its final refetch) in a `finally`, not after the
+  `await` — a stray throw otherwise leaves the bar stuck at "…" with the button
+  disabled.
+- **Parallelizing can move a side effect ahead of a gate.** Running an independent
+  side-effecting call concurrently with a validation query means it can now fire
+  even when validation would reject. Fine if the side effect is benign — just know
+  you traded the ordering for the latency.
+- **Additive migration for a scoped guard → backfill legacy rows to their TRUE value,
+  not a constant.** When a dedup/uniqueness guard starts keying on a new column,
+  `ADD COLUMN … DEFAULT <constant>` silently defeats the guard for pre-existing rows.
+  Derive the real value from an existing column (e.g. a DocNumber prefix) so old rows
+  stay correctly scoped.
+
+## Minimal starter
+
+Easiest path: `pnpm dlx @oxy-hq/create-oxy-app@latest my-app --template vite`.
+The scaffolded layout:
+
+```
+my-app/
+├── oxy-app.json            ← project root, not public/
+├── src/
+│   ├── App.tsx
+│   └── main.tsx
+├── package.json
+├── vite.config.ts
+└── tsconfig.json
+```
+
+`oxy-app.json`:
+```json
+{ "schemaVersion": 2, "slug": "my-app", "orgSlug": "test", "name": "My App" }
+```
+
+`src/main.tsx`:
+```tsx
+import { OxyAppProvider } from "@oxy-hq/sdk";
+import { createRoot } from "react-dom/client";
+import { App } from "./App";
+
+createRoot(document.getElementById("root")!).render(
+  <OxyAppProvider><App /></OxyAppProvider>
+);
+```
+
+`src/App.tsx`:
+```tsx
+import { useQuery } from "@oxy-hq/sdk";
+
+export function App() {
+  const { rows, loading, error } = useQuery({
+    sql: "SELECT 1 AS hello",
+  });
+  if (loading) return <p>loading…</p>;
+  if (error) return <p style={{ color: "red" }}>{error.message}</p>;
+  return <pre>{JSON.stringify(rows, null, 2)}</pre>;
+}
+```
+
+`vite.config.ts`:
+```ts
+import oxyApp from "@oxy-hq/vite-plugin";
+import react from "@vitejs/plugin-react";
+import { defineConfig } from "vite";
+
+export default defineConfig({
+  plugins: [react(), oxyApp()],
+});
+```
+
+The plugin handles `outDir = "out"`, base-path resolution from the
+manifest, dev-server proxy to `/api`, and the `window.__OXY_APP__`
+dev shim. Plain `pnpm build` produces a bundle with the correct
+asset base baked in.
+
+## Publishing
+
+Ship with `oxy publish` — **no CI, no S3 sync, no folder picker**. From
+the app directory (`oxy-hq/customer-apps: examples/hello-oxy/` is the worked
+shell; note its manifest deliberately carries no `orgSlug`, leaving the org to
+be resolved at publish time):
+
+```bash
+oxy login --env production    # once; browser flow, caches a token,
+                              # and prints whether you're an app-admin
+oxy publish --env production           # build → tar → upload (draft channel)
+oxy publish --env production --promote # …straight to live
+```
+
+- **Identity** is the manifest's `slug` + `orgSlug` (override with
+  `--app` / `--org`). The server upserts the app row — there is no
+  separate `oxy apps create` step.
+- **First publish of a new app** needs `--project <workspace-uuid>`: the
+  build-config lookup can't resolve a project until the app exists. Drop
+  the flag after the first publish — it's auto-resolved from then on.
+- **`--env` targets:** `local` → the oxy web-app Vite dev server `:5173`
+  (run `pnpm dev` there; it serves `/cli-auth` and proxies `/api` to
+  oxy on `:3000`), `dev`/`staging`/`production` →
+  `app{-dev,-staging,}.oxygen-hq.com`.
+- `--dir out` skips the build and uploads a prebuilt directory.
+- Promote / roll back / unpublish from the oxy admin app-detail console;
+  every publish is audited (who shipped each build, when).
+
+Full hands-on guide: `oxy-hq/customer-apps: docs/local-development.md`.
+
+## Reading data while vibe-coding
+
+You don't need an `X-API-Key` to poke at the data API by hand — your
+`oxy login` token authenticates as a bearer. From an app directory:
+
+```bash
+oxyc login --env local       # once; caches a token per host
+
+# gh-api-style client (resolves the target + bearer for you, no headers):
+oxyc api user --env local                                   # who am I?
+oxyc api projects/<projectId>/query --env local -f sql='select 1'
+
+# raw curl with your token (e.g. to script or debug):
+curl -H "Authorization: Bearer $(oxyc token --env local)" \
+     -H 'content-type: application/json' \
+     localhost:5173/api/projects/<projectId>/query -d '{"sql":"select 1"}'
+```
+
+`oxyc api <path>` takes the path relative to `/api/`; `-X` sets the method
+(POST when a body is given), `-f`/`-F` set the body (string / JSON-typed),
+`--input` sends a raw one, `-i` includes response headers. `--env` resolves the
+same targets as `oxy publish` (`local` → the Vite dev server `:5173`, which
+proxies `/api` → oxy `:3000`).
+
+Do not guess a path: `oxyc routes <filter>` lists what exists and
+`oxyc schema <path>` gives the body it expects.
+
+Access is your **normal org membership** — fine for your own test
+workspace, not a cross-customer backdoor. The `projectId` is the one the
+vite-plugin already resolved for the app (it's in `window.__OXY_APP__`).

package/skills/oxy-data-architecture/SKILL.md

@@ -0,0 +1,225 @@
+---
+name: oxy-data-architecture
+description: Use when deciding where an aggregate gets materialised — writing an `execute_sql` rollup, chaining one after an `airway` ingest step, choosing rollup vs semantic `pre_aggregations`, or whether raw rows must be retained — and on the symptoms: a dashboard that aggregates at query time, a load that wrote every row yet failed, preagg that cannot bucket to 15 minutes or join two views, or no "bronze"/"silver"/"gold" in the schema. Pipeline internals: `oxy-airway-pipelines`; view syntax: `oxy-semantic-modeling`.
+---
+
+# Oxy data architecture: where results get materialised
+
+An OLAP store wants **materialised results, not aggregation at query time.** A
+dashboard that groups four million order rows on every page load is not a slow
+query to tune — it is a rollup that was never built. So the work that produces
+those tables has to be systematic: one place it runs, one guarantee about when it
+may read, one rule for what gets kept.
+
+That spans Airway ingest, the semantic layer, and scheduled automations, which is
+why it is its own skill rather than a section in one of them — and it does not
+restate them. Fold mechanics, duplicate rows and the pipeline lease are
+**`oxy-airway-pipelines`**; views and measures **`oxy-semantic-modeling`**; task
+syntax and schedules **`oxy-automations`**.
+
+## The source doc, and why half of it is not teachable
+
+The frame comes from **`internal-docs/medallion-architecture.md`** in
+**`oxy-hq/oxygen-internal`** — a **private** repo (often checked out under a
+directory named `oxy-internal`). You cannot fetch it like a URL and it may not be
+checked out at all where you are reading this, which is why every stable fact
+below is carried here rather than pointed at. Do not copy more of it in: it is
+actively maintained there, and a duplicate rots and then argues with the original.
+
+**Its status is `design`, with an open "Decisions needed" section.** So this
+skill splits it: **current state** is taught below as fact, re-verified against
+source and cited; the **target tiering, gap list, and open decisions** are named
+at the end and deliberately *not* taught. Building against a convention that does
+not exist yet is worse than building against no convention at all.
+
+## The vocabulary bridge
+
+**Nothing in the code says bronze, silver, gold, or medallion.** The doc
+introduces that vocabulary as a *lens*, not as names you will find. Grep
+`oxy-internal` for "bronze" or "silver" and you get Tailwind classes for
+leaderboard medal ranks in the observability UI (`bg-rank-bronze`); grep "gold"
+and you get golden-snapshot test names (`crates/oxy-compile/src/compile.rs:1042`)
+and a restaurant in seed data. **Not one hit is a data tier.** An agent that
+greps, finds nothing, and invents a tier structure has invented it.
+
+| medallion word | what is actually there |
+|---|---|
+| (transient) bronze buffer | `<schema>_raw.<table>` append buffer — **`replacing` disposition only**, not a tier |
+| **silver** | `<schema>.<table>`, the normalized read table — every disposition lands here |
+| **gold** | `execute_sql` rollup tables (`*_daily_metrics`) **in the same schema** |
+| the promotion | the **fold** (watermark-bounded, latest-wins, merge-on-read) — inline at end of load **and again** on the scheduled vacuum, identical SQL |
+| storage | Airhouse = DuckLake on DuckDB (S3 Parquet + catalog) — **for pipelines that land in Airhouse** |
+
+That last row is a default, not a law. A rollup lands wherever its `execute_sql`
+step's `database:` points, so gold is not necessarily DuckLake: the canonical
+exemplar view in `oxy-semantic-modeling` sits on a gold table
+(`…___sales_daily_metrics`) whose `datasource:` is **`clickhouse`**. That view is
+where the two skills meet — a view models a rollup an automation built — and it
+is also the reminder to read the `database:`/`datasource:` rather than assume the
+warehouse.
+
+## Current state: silver on ingest, gold in place
+
+**Normalization happens at ingest, so the landed table is already silver.** It is
+not a later stage. Airway flattens nested JSON into parent/child relational
+tables, snake_cases every name (`normalizer/mod.rs:23`), coerces types, and
+injects dlt-style lineage columns — `_aw_id`, `_aw_load_id`, `_aw_parent_id`,
+`_aw_root_id`, `_aw_list_idx` (`normalizer/relational.rs:153`). There is no
+separate "raw → typed" step to insert your own logic into.
+
+**The `_raw` buffer is a write-throughput optimization, not a tier.** It exists
+only for `write_disposition: replacing`, to keep the write path O(batch) instead
+of scanning the target (airway `types.rs:117`). It is drained on fold, so steady
+state is roughly empty — and when a fold fails the rows stay durable there,
+which is what makes the re-run below safe rather than what makes the failure
+harmless. **You cannot rebuild silver from it.**
+
+> **Location trap.** The buffer lives at **`<schema>_raw.<table>`** — a sibling
+> *schema*, where `<schema>` is the pipeline's `dataset_name`, so one landing in
+> `toast_pos` buffers to `toast_pos_raw.orders` (airway
+> `connector/destinations/airhouse.rs:1003`). Several in-tree comments still say
+> `<table>_raw`, oxy's own `crates/agentic/airway/src/source_factory.rs:475`
+> among them. That is the pre-`raw_schema_name` location, healed in place by
+> `migrate_legacy_raw`; the comments never caught up. **This is not version
+> drift** — the migration predates every airway rev oxy has pinned. Follow the
+> comment and you look in the wrong schema and conclude the buffer is missing.
+
+**Gold is a `CREATE OR REPLACE TABLE` in an automation**, reading the
+Airway-landed tables and writing aggregates into the *same* dataset schema —
+`toast_pos.orders_daily_metrics` beside `toast_pos.orders`. No marts schema
+exists today. The `execute_sql` task:
+<https://www.oxygen-hq.com/docs/guide/build/automations/task-types>
+
+## The sequencing guarantee — the mechanism that makes this systematic
+
+This is the whole answer to "how do we do the background work systematically",
+and it is one sentence in the source:
+
+> A step completes only once the pipeline's end-of-load fold has committed, so a
+> following `execute_sql` step reads a queryable table rather than a half-folded
+> one.
+> — `crates/core/src/config/model.rs:2435`
+
+So ingest → rollup is a plain two-step automation, and the ordering is real
+rather than hoped-for:
+
+```yaml
+tasks:
+  - name: load
+    type: airway
+    pipeline: pipelines/restaurant_analytics.airway.yml
+  - name: rollup
+    type: execute_sql
+    database: pokehouse
+    sql_query: |
+      CREATE OR REPLACE TABLE toast_pos.orders_daily_metrics AS
+      SELECT business_date, restaurant_id, sum(net_amount) AS net_sales
+      FROM toast_pos.orders GROUP BY 1, 2
+```
+
+Two things to know about that step:
+
+- **`type: airway` is undocumented.** The task-types page above lists
+  `execute_sql` thoroughly and mentions airway **zero** times (checked
+  2026-08-19). It is also not run by `step_executor` like the other I/O tasks —
+  it is delegated as a `TaskSpec::Airway` (`model.rs:2428`). Reading the docs
+  and concluding you must trigger ingest some other way is the expected mistake.
+- **A failed fold fails the run — and the fix is to re-run, never to
+  re-extract.** Every row can be written and the load still fail, because the
+  fold that promotes them into the public table is a separate step: *"what
+  distinguishes 'written' from 'queryable'"*
+  (`crates/agentic/airway/src/events.rs:163`). Airway emits `LoadCompleted`
+  anyway, with correct counts and the per-table outcome in its `folds` field —
+  the counts are real — *"and the run reports an error"* (`events.rs:153`). It
+  then returns `Err`, which is *"what makes the RUN terminal state failed"*
+  (airway `pipeline/mod.rs:1378`; the returns at `:1376` and `:1881`,
+  `TaskOutcome::Failed` at `crates/agentic/airway/src/worker.rs:311`). So the
+  run goes red and the message names the cause outright: *"load wrote every row
+  but N table(s) failed the end-of-load fold … rows are durable in the staging
+  buffer but not yet visible in the public schema; re-run to fold them"*
+  (`pipeline/mod.rs:1291`). Take that literally. Cursors already advanced, so
+  *"the re-run re-extracts nothing and the fold sweeps the pending buffer"*
+  (`pipeline/mod.rs:1372`) — a full re-extract, or a Reset schema, is the
+  expensive wrong move on a run that already landed every row.
+
+## Rollup is not preagg
+
+Asked to make a slow dashboard fast, an agent working in the semantic layer
+reaches for `pre_aggregations`. Nothing warns it off: `pre_aggregations` appears
+in no docs page, and `oxy-semantic-modeling` does not mention it either — so the
+first thing that tells you it is the wrong tool is the SQL failing. It is a
+**different mechanism** and it cannot do what a rollup does:
+
+| | `execute_sql` rollup | semantic `pre_aggregations` |
+|---|---|---|
+| lives in | an automation | a `.view.yml` block |
+| reads | any SQL — joins, CTEs, window functions | **one view**: `generate_build_sql` takes a single `view.source_sql()` and emits no JOIN (airlayer `engine/preagg.rs:266`) |
+| time buckets | anything you can write | a calendar part only — `granularity` is interpolated straight into `date_trunc('{g}', …)` (airlayer `dialect/mod.rs:75`), so `year`/`quarter`/`month`/`week`/`day`/`hour`/`minute`/`second` and nothing between |
+| refresh | your automation's schedule | `refresh_key`, by a background worker |
+
+So **15-minute buckets and any two-table join must be a rollup.** `RollupSpec`
+carries `dimensions`, `measures`, `time_dimension`, `granularity` — there is
+nowhere to name a second view. Use preagg for the narrow case it fits: one view,
+calendar grain, speeding up a measure that already resolves. Use a rollup for
+everything else, which in practice is most gold.
+
+## Bronze policy is a decision rule, not a preference
+
+The right policy is a function of **re-extractability**, and it is chosen per
+domain rather than applied uniformly:
+
+| source | policy | why |
+|---|---|---|
+| **re-extractable** — Toast, QuickBooks, most APIs | transient `_raw` buffer is fine; **the source system is the bronze** | you can always re-pull to rebuild silver; a durable warehouse copy is redundant cost |
+| **non-re-extractable** — video events, sensor streams | the raw event table **is durable bronze and must be retained** | you cannot re-pull yesterday's frames; drop the events and the history is gone |
+
+The video domain is the worked example, and it diverges visibly: `oxy_cam_events`
+is written by a direct append (`crates/cameras/src/service/ingest.rs:106`) into a
+plain table with no `_raw` buffer and no fold
+(`crates/cameras/src/airhouse/schema.rs:16`) — it never goes through Airway at
+all. That is the shape durable bronze takes here. Before proposing a retention or
+cleanup job on any raw table, run the test above; the answer for cameras is the
+opposite of the answer for Toast.
+
+## Why the shapes are these shapes: DuckLake
+
+`PRIMARY KEY`, `UNIQUE`, and `CREATE INDEX` are all rejected
+(`crates/cameras/src/airhouse/schema.rs:6`). That single constraint is why
+`merge` stages then `MERGE INTO` and why `replacing` dedups on fold — no
+`ON CONFLICT` to lean on, and scan pruning comes from partition/sort, not an
+index. **`oxy-airway-pipelines` owns the consequences** (duplicate rows, the
+single-flight lease, reset); do not re-derive them here. The one that reaches
+gold: DDL in an `execute_sql` rollup must carry no constraints either.
+
+**A corollary for rollup SQL:** join parent to child on the **propagated
+business key** (`invoice_id` on `invoices__line`), never on `_aw_parent_id` — it
+is regenerated every load (airway `types.rs:114`), so a join on it matches within
+one load and silently stops matching across loads.
+
+## Proposed, not built — cite, do not build against
+
+The rest of that doc is a roadmap. Say so out loud if someone asks you to follow
+it; none of the following exists:
+
+- **A conformance layer** — shared `dim_store` / `dim_date` / `dim_camera` and
+  cross-source id maps, materialised once. Cross-source keys are wired ad hoc
+  per domain today, mostly hand-maintained `VALUES` tables. One camera dimension
+  has since shipped in the video domain: proof the gap is real, **not** a
+  convention to copy.
+- **Separating gold from silver** by schema or a `*_metrics` suffix tooling
+  treats as derived. They co-mingle today, and whether to adopt a convention now
+  or wait for a second consumer is an **open decision** in the doc.
+- **Codifying the re-extractability test as policy** — the rule above is
+  reasoning that has held, not something enforced anywhere.
+
+If a task needs one of these, that is a design conversation with the doc's
+owners, not a pattern to invent in a customer workspace.
+
+## Version anchor
+
+Checked 2026-08-19 against oxy-internal **0.5.126**, airway **0.1.32**
+(`55e2794`), airlayer **0.3.6** (`7e884e77`) — both pinned in `Cargo.lock`.
+Unprefixed `src/…` paths above are inside those external crates, not
+oxy-internal; `crates/…` paths are oxy-internal. Both engines are fetched by git
+rev, so a bump can move this ground with no oxy-internal commit — re-check
+against the pin in `Cargo.lock`, not against memory.