dataspring-cli 0.4.2__py3-none-any.whl → 0.4.4__py3-none-any.whl

cli/_skills/dataspring-author/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: dataspring-author
-description: Conventions for writing dbt YAML descriptions and the org's business_context doc. Use when editing dbt/models/**.yml, dbt/context/business_context.md, or any file that feeds DataSpring's semantic layer or org context.
+description: Building and documenting an org's data in DataSpring - the datacore loop (pipelines, connections, dbt models, metrics; pull, push, check, deploy, build, run, backfills), dbt YAML descriptions and the business_context doc. Use when editing a datacore workspace or dbt/models/**.yml, writing metrics, loading or backfilling a pipeline, or any file that feeds DataSpring's semantic layer or org context.
 ---
 
 # Authoring DataSpring's data-layer context
@@ -9,6 +9,28 @@ You are editing files that an analytical agent will read at query time.
 Lazy descriptions silently degrade every query that touches the
 affected entity. This skill exists so the conventions stay tight.
 
+## Before you start: sign in and name the org
+
+Every command and connection acts in ONE org, named each time; nothing
+switches it for later. CLI: `dataspring auth login` prints a URL and a code
+at once (run it in the background and hand both to the person), then
+`--org <org>`, `DATASPRING_ORG`, or `{"org": "<org>"}` in a
+`.forspring.json` at the workspace root, which is the natural place:
+
+```bash
+echo '{"org": "acme"}' > .forspring.json && dataspring whoami --json
+```
+
+MCP: one connection per org, `https://dataspring.app/api/mcp/?org=<org>`, or
+`Authorization: Bearer ds_...` for a headless agent (the key's own org).
+
+A refusal leads with a reason and ends with its fix; match the reason. The
+ones you meet while building: `role_too_low` (connections, pipelines,
+deploy, build, run, cancel and reset need admin; a member can edit models
+and check), `org_required` (ask which org, never guess), `not_signed_in`
+(log in again). The full table, with the CLI exit codes, is in
+`dataspring-consume`.
+
 ## 1. The contract
 
 The descriptions on metrics, dimensions, and semantic models are
@@ -206,7 +228,51 @@ version produced each issue. Edit the board and the next issue follows.
 and is what stops later history columns moving. Rendering and checking
 prose is the consumer's job and is covered in `dataspring-consume`.
 
-## 8.2 Iterating on a dbt model (managed tenants)
+## 8.2 The datacore loop (managed tenants)
+
+A managed org's pipelines, connections and dbt project are one workspace of
+files. Edit, `check`, `deploy`; nothing an edit or a check does reaches
+production.
+
+```bash
+dataspring datacore pull ./ws            # deployed files, each exactly as pushed
+dataspring datacore push ./ws            # changed files only, into a new draft
+dataspring datacore check --validate-only  # seconds: parse + semantic validation, no build
+dataspring datacore check                # waits; exits 1 unless it passes
+dataspring datacore deploy --then-build  # admin; rebuilds changed models too
+```
+
+- **A new org** pulls its starting workspace (`datacore.yaml`, empty
+  `pipelines/` and `connections/`, and in `models/` a starting dbt project,
+  see 8.7).
+- **A check** compiles every pipeline, builds the changed models and their
+  tests into `<org>_dev`, and dry-runs their SQL. It runs as an identity
+  that can write `<org>_dev` only: a project whose `generate_schema_name`
+  puts a model in another dataset is refused, naming the models. A
+  workspace with no `models/dbt_project.yml` yet passes with dbt `skipped`,
+  so land raw data first and add models after.
+- **The semantic manifest is checked per item**, each finding as `FAIL metric
+  <name>: <message>` (`check.dbt.failures` on the run). It is also
+  dry-imported as a load would import it, so what passes the check imports.
+  A ratio metric's `numerator` and `denominator` are metrics, not measures
+  (MetricFlow's definition), as are a derived metric's inputs:
+
+```yaml
+- name: gross_margin_pct
+  type: ratio
+  type_params: {numerator: gross_margin, denominator: revenue}  # two metrics
+```
+
+- **Validate-only while you edit** (`datacore_run` `check` with `build:
+  false`; CLI `datacore check --validate-only`): pipelines compiled,
+  connections checked, `dbt parse`, the per-metric semantic validation and
+  the import's dry run, and no `dbt build`, so a broken metric or model is
+  reported in well under a minute and it never waits behind a check or a
+  load. It is never a green check: run a full `check` before `deploy`.
+- **Deploy** needs a green check of that exact draft (any edit makes a new
+  draft) and starts a run of each windowed pipeline it changed.
+
+## 8.3 Iterating on a dbt model (managed tenants)
 
 A model change does not need a load. The loop is `datacore_edit` (kind
 `model`), `datacore_run` `check`, `deploy`, then `build`: a run that loads
@@ -218,6 +284,156 @@ under `models/`. Do not use `run` to see a model change: it reloads every
 resource of the pipeline first. CLI: `dataspring datacore build --select
 "marts.finance+" --wait`.
 
+**Look before you write, and verify before you deploy.** `run_sql`
+(`dataspring sql`) runs as the org's read identity, which reads the org's
+own `<org>_raw` (what the loads wrote), `<org>_staging` (the staging views
+over raw), `<org>_core`, `<org>_marts` and `<org>_dev` (what the last
+`check` built), and nothing else; `<org>` is the org id with `-` as `_`.
+
+- Before writing a staging model, list the raw columns that actually
+  exist: `dataspring datacore schema --table <t>` (or
+  `dataspring://datacore/schema`). A field that was null on every row
+  loaded has no column at all, so a model written against the source API's
+  documentation can fail with `Unrecognized name`; select a typed null for
+  it instead. `dataspring datacore freshness` gives row counts and the run
+  that wrote each table; `dataspring datacore sample <t>` some rows.
+- After a `check`, the models it built (the changed ones and everything
+  downstream) are tables and views in `<org>_dev` for 7 days: compare their
+  numbers with the source before `deploy` puts them where dashboards read,
+  e.g. `dataspring sql "SELECT * FROM <org>_dev.fct_project LIMIT 20"` or
+  `dataspring datacore schema --dataset dev`.
+- A script of several statements runs as one job and answers the last
+  one's rows; `CREATE TEMP TABLE` in it is scratch space for an analysis
+  that ends with the script.
+
+## 8.4 Workspace files keep the customer's text
+
+The customer's comments in a pipeline, connection or dbt YAML file are
+its documentation. To replace a connection or pipeline whole, send
+`datacore_edit` `apply` the YAML text as `content`: it is validated like a
+`document` and stored byte for byte, so `dataspring datacore pull` returns
+it as written. A `document` (an object) is stored as a fresh dump with no
+comments; use it only for a file you are creating. `patch`, and a
+`metric_edit` / `semantic_model_edit` update or delete in a shared dbt
+file, replace only the lines of the keys or list entries that changed;
+the rest of the file, mixed indentation included, stays byte for byte.
+When that cannot be done (a flow-style document, say) the answer carries
+`reformatted`: the whole file was rewritten in one indentation style.
+Tell the user the diff is large for that reason. `dataspring datacore
+push` sends each changed file as its text the same way, and skips a file
+whose bytes equal the stored one's.
+
+## 8.5 Loads, windows and backfills
+
+- **Windows are whole inclusive days**, the run's and every chunk's. When an
+  API reads its end date as exclusive (`to=2025-03-31` misses 31 March), the
+  resource says so and is sent the day after each window's last day:
+
+  ```yaml
+  window: { lookback: 7d, param_start: createdFrom, param_end: createdTo, end: exclusive }
+  ```
+
+  Test an API's end bound with a one-day window before a backfill: a wrong
+  guess silently drops the last day of every chunk.
+- **A window is one request** unless it has a `step`. For a source that caps
+  rows per request and does not page (e-regnskab: 20,000 item lines), a long
+  re-pull is cut into steps, one request each (`7d`, `2w`, `1M`; calendar
+  months, as backfill chunks are cut). `end: exclusive` holds per step, and
+  the scheduled window, a manual run and every backfill chunk all step:
+
+  ```yaml
+  window: { lookback: 365d, step: 1M, param_start: createdFrom, param_end: createdTo, end: exclusive }
+  ```
+
+  A rest resource only, and not a `resolve` child: step its parent.
+- **`resolve` fans out over the parent's own window** (its lookback, or the
+  run's window), not the child's. When the child's rows lack the parent's
+  key, carry it: `include_from_parent: [hnInvoiceSalesID]` adds the column
+  `_<parent resource>_<field>` (`_invoices_hnInvoiceSalesID`) to every child
+  row; use it in the child's `primary_key` or rename it in `shape`.
+  `param` fills a `{param}` placeholder in the endpoint, or else is sent as
+  a query parameter.
+- **An endpoint that answers a list of scalars** (`["budget", "forecast"]`)
+  loads one row per element in a column `value`; fan out over it with
+  `resolve: { field: value, ... }`.
+- **A backfill** is `dataspring datacore run <pipeline> --start ... --end ...
+  --chunk 1M`: one child run per chunk, each started as the previous one
+  ends. The chunk is `<n>d`, `<n>w` or `<n>M` (`7d`, `2w`, `1M`, `3M` a
+  quarter, `12M` a year; no `Y`). A chunk before the last loads only the
+  windowed resources (and those they read rows from) and skips dbt; the last
+  chunk loads everything and builds the models once. A failed child fails
+  the backfill. One backfill per pipeline at a time.
+- **One request per window.** A rest resource without a `paginator` sends
+  one request per window and nothing splits it, so a source that caps an
+  answer at N rows needs a `lookback`, and a `--chunk`, under N rows.
+  --chunk 1M` (or `3M`, `7d`, `2w`): one child run per chunk, each started as
+  the previous one ends. A chunk before the last loads only the windowed
+  resources, their `resolve` children (fetched for that chunk's parent
+  rows) and those they read rows from, and skips dbt; the last chunk
+  loads everything and builds the models once. A failed child fails the
+  backfill. One backfill per pipeline at a time.
+- **Stop** a queued or running run, or a whole backfill, with `dataspring
+  datacore cancel <run_id>` (admin); the schedule fires again at its next slot.
+- **`reset`** drops a pipeline's raw tables and load state. It is
+  irreversible: ask first.
+
+## 8.6 Writing a pipeline file
+
+Do not read DataSpring's source for the grammar: it is served, cut from the
+files the runner validates against.
+
+```bash
+dataspring datacore reference                   # the schemas, every connector: secrets, tables
+dataspring datacore reference catalog kanpla    # one connector, whole, with a file that uses it
+dataspring datacore reference pipeline          # the pipeline JSON schema (also: connection)
+```
+
+MCP: `dataspring://datacore/reference[/<part>]` or `read(kind=
+"datacore_reference", id=<part>)`; REST `GET /api/datacore/reference`.
+Check the catalog first: a source with a connector is `connector: <name>`
+plus `config` and `secrets`. Otherwise write an inline `source` (or
+`sources`) and `resources`. The rules that bite:
+
+- **Secrets are names**, `t-<org>-<connection>-<field>`, lowercase letters,
+  digits and hyphens only (no underscore). Set each with `dataspring secret
+  set <name> --stdin` before pushing the file that names it.
+- **Windows** are whole inclusive days, `lookback` before today through
+  `lookahead` after, in the pipeline's timezone; declare `end: exclusive`
+  for an API whose end date is exclusive.
+- **Loads**: `replace` is a full refresh; `merge` needs `primary_key` (or
+  `merge_key`); `window_scope: replace` deletes by the `partition` date, so
+  use it only when that is the date the API filters on.
+- **Checks**: `min_rows` is refused on a `cursor` resource; use `freshness`.
+- **A `resolve` child of a windowed parent** needs its own `window`, and the
+  parent its window in `params` (`{{ window.start }}`), or backfill chunks
+  skip the child.
+
+The full guide with complete examples is "Writing a pipeline" in
+`dataspring docs --full`.
+
+## 8.7 The managed dbt project
+
+The workspace's `models/` is a dbt project (dbt-core 1.12, BigQuery). A new
+org starts with one that already keeps the contract; `dataspring datacore
+pull` shows it. The contract:
+
+- `profile: 'dataspring'` (the runner writes the profile; dataset
+  `<org>_marts`).
+- Sources at `schema: "{{ env_var('DBT_RAW_SCHEMA') }}"` and `database:
+  "{{ env_var('DBT_GCP_PROJECT') }}"`, never a dataset name.
+- `generate_schema_name` returns `DBT_SCHEMA_OVERRIDE` when set (a check:
+  `<org>_dev`), else `DBT_SCHEMA_PREFIX` + the layer (`+schema: staging` is
+  `<org>_staging`).
+- A `metricflow_time_spine` model with `time_spine:` configured; include it
+  in `then.dbt_select`.
+- Semantic models and metrics in YAML anywhere under `models/models/`;
+  `metric_edit` creates them in `models/models/semantic/metrics/`.
+- `packages.yml` is fetched by `dbt deps` on every run (`dbt_utils`).
+
+A deploy builds nothing: `deploy --then-build`, `datacore build`, or a
+pipeline's `then.dbt_select` does. The full contract is "The managed dbt
+project" in `dataspring docs --full`.
+
 ## 9. Parameter reference
 
 <!-- dispatch-params: semantic_model_edit metric_edit board_edit briefing_edit import_manifest business_context_edit warehouse_edit datacore_edit datacore_run -->
@@ -227,14 +443,14 @@ _Generated from the dispatch registry (`GET /api/dispatch`); do not edit by hand
 
 MCP tool `semantic_model_edit`; `POST /api/dispatch/semantic_model_edit`. Edit semantic models in the org manifest. Admin/owner only for writes. The body is `{"action": {"action": "<verb>", ...}}`, one verb per table below.
 
-**`create`** — 
+**`create`** — Create a semantic model (admin/owner).
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `model_data` | object | yes | Full semantic-model definition (name, measures, dimensions, entities) |
 | `expect_version` | string | no | Managed tenants only: the datacore version (draft snapshot id) this edit was written against, from the previous edit's `version` or dataspring://datacore/files; refused with the current one when stale |
 
-**`update`** — 
+**`update`** — Change a semantic model (admin/owner). On an imported one a change to how it computes answers a dbt YAML proposal and writes nothing; descriptions write at once.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -243,7 +459,7 @@ MCP tool `semantic_model_edit`; `POST /api/dispatch/semantic_model_edit`. Edit s
 | `confirmed` | boolean | no, default `False` | set true only after the user confirmed a redefinition of a NATIVE entity whose impact you showed them; has no effect on imported entities |
 | `expect_version` | string | no | Managed tenants only: the datacore version (draft snapshot id) this edit was written against, from the previous edit's `version` or dataspring://datacore/files; refused with the current one when stale |
 
-**`delete`** — 
+**`delete`** — Delete a semantic model (admin/owner).
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -260,14 +476,14 @@ MCP tool `semantic_model_edit`; `POST /api/dispatch/semantic_model_edit`. Edit s
 
 MCP tool `metric_edit`; `POST /api/dispatch/metric_edit`. Edit metrics in the org manifest. Admin/owner only for writes. The body is `{"action": {"action": "<verb>", ...}}`, one verb per table below.
 
-**`create`** — 
+**`create`** — Create a metric, native to DataSpring (admin/owner). A ratio's numerator and denominator, a derived metric's inputs and a cumulative or conversion metric's base are metrics, not measures.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `metric_data` | object | yes | Metric definition (name, type, type_params) |
 | `expect_version` | string | no | Managed tenants only: the datacore version (draft snapshot id) this edit was written against, from the previous edit's `version` or dataspring://datacore/files; refused with the current one when stale |
 
-**`update`** — 
+**`update`** — Change a metric (admin/owner). On an imported one a change to how it computes answers a dbt YAML proposal and writes nothing; descriptions write at once.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -276,14 +492,14 @@ MCP tool `metric_edit`; `POST /api/dispatch/metric_edit`. Edit metrics in the or
 | `confirmed` | boolean | no, default `False` | set true only after the user confirmed a redefinition of a NATIVE entity whose impact you showed them; has no effect on imported entities |
 | `expect_version` | string | no | Managed tenants only: the datacore version (draft snapshot id) this edit was written against, from the previous edit's `version` or dataspring://datacore/files; refused with the current one when stale |
 
-**`delete`** — 
+**`delete`** — Delete a metric (admin/owner; irreversible).
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `name` | string | yes |  |
 | `expect_version` | string | no | Managed tenants only: the datacore version (draft snapshot id) this edit was written against, from the previous edit's `version` or dataspring://datacore/files; refused with the current one when stale |
 
-**`preview`** — 
+**`preview`** — Test a metric before saving (any role; writes nothing): sample query results or validation errors, plus the `dbt_yaml` that would be committed for it.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -430,17 +646,17 @@ MCP tool `warehouse_edit`; `POST /api/dispatch/warehouse_edit`. Manage the org's
 
 MCP tool `datacore_edit`; `POST /api/dispatch/datacore_edit`. Edit the datacore workspace: connections, pipelines and the dbt project of a managed tenant, as files. ``kind`` picks the family: The body is `{"action": {"action": "<verb>", ...}}`, one verb per table below.
 
-**`apply`** — Write one workspace file whole (create or replace): a connection or pipeline as a ``document`` (validated against its schema and the catalog), a dbt project file as ``content``.
+**`apply`** — Write one workspace file whole (create or replace): a dbt project file as ``content``; a connection or pipeline as ``content`` (its YAML text, stored exactly as sent, comments and layout kept) or as a ``document`` (an object, stored as a fresh dump), validated against its schema and the catalog either way.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `kind` | one of `connection`, `pipeline`, `model` | yes | connection (admin), pipeline (admin) or model (member) |
 | `name` | string | yes | For kind connection or pipeline: the name, i.e. the file name without .yaml (connections/<name>.yaml, pipelines/<name>.yaml). For kind model: the path under models/, the dbt project (models/models/marts/orders.sql, models/models/staging/schema.yml, models/seeds/x.csv, models/dbt_project.yml); no traversal, .sql .yml .yaml .csv .md .txt .json, never target/ or dbt_packages/ |
 | `document` | object | no | kind connection: the whole document (connection.schema.json: kind, base_url, auth with secret NAMES). kind pipeline: the whole document (pipeline.schema.json: a catalog `connector` with config, secrets, overrides; or a `connection` reference, an inline `source` or named `sources`, with `resources`; schedule, checks, then). Secrets are named, never valued; every name must exist |
-| `content` | string | no | kind model: the whole file content |
+| `content` | string | no | The whole file as text. kind model: required. kind connection or pipeline: the YAML text, validated like a document and stored byte for byte (comments and layout kept); give content or document, not both |
 | `expect_version` | string | no | The datacore version this edit was written against: the `version` the previous edit returned, or dataspring://datacore/files. An edit against a stale version is refused with the current one. Omit to skip the check. |
 
-**`patch`** — Apply a JSON merge patch (RFC 7386: a key set to null is removed) to an existing YAML file: a connection, a pipeline (a schedule, a window, a check), or a YAML file under models/ (not SQL).
+**`patch`** — Apply a JSON merge patch (RFC 7386: a key set to null is removed) to an existing YAML file: a connection, a pipeline (a schedule, a window, a check), or a YAML file under models/ (not SQL). Only the lines of the keys the patch changes are rewritten; every other line, comments and indentation included, stays as written (an answer with ``reformatted`` says the file had to be rewritten whole).
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -461,13 +677,14 @@ MCP tool `datacore_edit`; `POST /api/dispatch/datacore_edit`. Edit the datacore
 
 MCP tool `datacore_run`; `POST /api/dispatch/datacore_run`. The datacore's check-then-deploy loop and its operations. The body is `{"action": {"action": "<verb>", ...}}`, one verb per table below.
 
-**`check`** — Check the current draft; no arguments beyond an optional note.
+**`check`** — Check the current draft (member): compile every pipeline, build the changed dbt models into <org>_dev with their tests, and validate the semantic manifest per metric. Nothing reaches production.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `note` | string | no | Optional note, recorded on the check run |
+| `build` | boolean | no, default `True` | false = validate only: compile the pipelines, check the connections, dbt parse, the semantic validation per metric and the manifest import's dry run, with no dbt build (an answer in well under a minute, beside a running check or load). It is NEVER the green check deploy or auto_deploy needs |
 
-**`deploy`** — 
+**`deploy`** — Promote the checked draft to deployed (admin), or roll back to an earlier deployed snapshot; then_build also rebuilds changed models.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -476,14 +693,14 @@ MCP tool `datacore_run`; `POST /api/dispatch/datacore_run`. The datacore's check
 | `then_build` | boolean | no, default `False` | When the deploy changed any file under models/, also queue a models-only build (the `build` action) of the newly deployed snapshot on the raw tables already loaded; build_select narrows it. Without it, a deploy that changed models answers with the command that does it |
 | `build_select` | string | no | With then_build: the dbt selector to rebuild (default: every model, `fqn:*`) |
 
-**`run`** — 
+**`run`** — Load a pipeline now (admin): its own window, a given window, or a chunked backfill whose chunks run back to back.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `pipeline` | string | yes | Pipeline name (a deployed pipelines/<name>.yaml) |
-| `window` | RunWindowParams | no | A date window to load instead of the pipeline's own; wider than backfill_chunk becomes a chunked backfill. Both ends are inclusive days, and so is every chunk; a resource whose API reads its end bound as exclusive declares `window.end: exclusive` in the pipeline and is sent the day after each chunk's last day |
+| `window` | RunWindowParams | no | A date window to load instead of the pipeline's own; wider than backfill_chunk becomes a chunked backfill. Both ends are inclusive days, and so is every chunk; a resource whose API reads its end bound as exclusive declares `window.end: exclusive` in the pipeline and is sent the day after each chunk's last day. A rest resource whose window has a `step` sends one request per step of the window or chunk, not one for the whole window |
 | `cursor` | RunCursorParams | no | {from: <value>}: a cursor to re-pull from |
-| `backfill_chunk` | string | no | Chunk size for a windowed run: 7d, 2w or 1M (calendar months, the default) |
+| `backfill_chunk` | string | no | Chunk size for a windowed run: <n>d, <n>w or <n>M (days, weeks or calendar months; 1M is the default, 3M a quarter, 12M a year; there is no Y). Every chunk before the last loads only the windowed resources (with their resolve children and the ones they read rows from) and skips its dbt step, each skip recorded with its reason on the child run; the last chunk loads every resource and builds the models once |
 
 **`build`** — Rebuild the deployed dbt models on the raw tables already loaded: no resource is loaded.
 
@@ -492,14 +709,14 @@ MCP tool `datacore_run`; `POST /api/dispatch/datacore_run`. The datacore's check
 | `pipeline` | string | no | A deployed pipeline whose `then` block names what to build (its dbt_select, dbt_exclude and tests). Omit to build every model of the deployed project |
 | `select` | string | no | A dbt selector (`dbt build --select`), e.g. `marts.finance+` or `+fct_postings`; replaces the pipeline's dbt_select and dbt_exclude. Default: the pipeline's then.dbt_select, or every model (`fqn:*`) when no pipeline is named |
 
-**`reset`** — 
+**`reset`** — Delete a pipeline's dlt state and drop its raw tables (admin; irreversible).
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
 | `pipeline` | string | yes | The pipeline whose dlt state and raw tables to remove |
 | `approve` | boolean | no, default `False` | Must be true: the reset is irreversible. Ask the user first |
 
-**`cancel`** — 
+**`cancel`** — Stop a queued or running run, or a whole backfill (admin); the schedule fires again at its next slot.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|

cli/_skills/dataspring-consume/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: dataspring-consume
-description: How to query DataSpring correctly to answer analytical questions. Use when the user is asking analytical/business questions, when an agent is connected to a DataSpring MCP server, when running `dataspring query ...` from the terminal, or when scripting against DataSpring's REST API. Covers the three surfaces (MCP, CLI, REST) — same principles, different invocations.
+description: How to query DataSpring correctly to answer analytical questions. Use when the user is asking analytical/business questions, when an agent is connected to a DataSpring MCP server, when running `dataspring query ...` from the terminal, or when scripting against DataSpring's REST API. Covers the three surfaces (MCP, CLI, REST) — same principles, different invocations — and signing in, naming the org, and what to do with a refusal.
 ---
 
 # Querying DataSpring to answer analytical questions
@@ -24,10 +24,10 @@ traces — departing from them produces misleading numbers.
 | List dimensions | `list_dimensions()` | `dataspring dimensions list` | `GET /api/semantic-layer/dimensions` |
 | Explain one metric | `explain_metric(metric_name)` | `dataspring metrics show <name>` | `GET /api/semantic-layer/metrics/<name>` |
 | Query | `query_metrics(metrics, dimensions?, grain?, start_date?, end_date?, where?, ...)` | `dataspring query -m <name> [-d ...] [-g month] ...` | `POST /api/dispatch/query_metrics` with the same fields |
-| Explain what was computed | `explain_query(metrics, ...)` (same fields as `query_metrics`) | (no CLI command yet) | `POST /api/dispatch/explain_query` |
+| Explain what was computed | `explain_query(metrics, ...)` (same fields as `query_metrics`) | `dataspring explain_query --metrics <name> ...` | `POST /api/dispatch/explain_query` |
 | What DataSpring learned / undo | `learned_edit list` / `undo` | `dataspring learned list` | `GET /api/learned` |
-| Verified queries | `verified_query_edit list` / `record` | (no CLI command yet) | (no dedicated route) |
-| Read business_context | `dataspring://context` resource | `dataspring business-context get` | `GET /api/context` |
+| Verified queries | `verified_query_edit list` / `record` | `dataspring verified_query_edit list` | `POST /api/dispatch/verified_query_edit` |
+| Read business_context | `business_context_edit get` (also in the connection's instructions) | `dataspring business-context get` | `GET /api/context` |
 | Check business_context size | `business_context_edit show_size` | `dataspring business-context size` | (no dedicated route — use `GET /api/context` and measure) |
 | Org's published skill | (this skill, served via SKILL.md) | `dataspring skill instructions consume` | `GET /api/v1/skill/consume/instructions` |
 | Be woken when a number moves | `watch_edit apply` | `dataspring watch_edit apply --document-json @watch.yaml` | `POST /api/dispatch/watch_edit` |
@@ -51,15 +51,44 @@ reads is the one this deployment was built with:
 
 ```bash
 curl -fsSL https://dataspring.app/install.sh | sh    # Windows: irm https://dataspring.app/install.ps1 | iex
-dataspring login
+dataspring auth login
 dataspring skills install --all                      # into ~/.claude/skills/<name>/, stamped
 ```
 
 Run `dataspring skills install` again after `dataspring upgrade` (the CLI
-upgrades itself in place; `dataspring --version` says when the server has
-operations this CLI does not know). `dataspring skills diff` shows what
-drifted between the installed copy and the served one; `--offline` on any
-`skills` command uses the copy bundled in the CLI instead of the server.
+upgrades itself in place through uv tool, pipx or pip, and prints the
+commands to run when it cannot tell which installed it; `dataspring
+--version` says when the server has operations this CLI does not know).
+`dataspring skills diff` shows what drifted between the installed copy and
+the served one; `--offline` on any `skills` command uses the copy bundled in
+the CLI instead of the server.
+
+## Signing in and naming the org
+
+People sign in with their Forspring account; who is in which org, with
+which role, is the account's answer on every request. DataSpring has no
+"current org": every command and every connection names one.
+
+- **CLI.** `dataspring auth login` prints a URL and a code at once and waits
+  until a person approves it in a browser (the machine running the CLI needs
+  none). Run it in the background, hand the person the URL and the code, and
+  wait for it to exit. It stores a `ds_` key for the org they approve
+  (`--all-orgs`: one per org). With one key stored, commands act in that org.
+  With several, name the org on every command, in this order of precedence:
+  `--org <org>`, `DATASPRING_ORG` (then `FORSPRING_ORG`), or a
+  `.forspring.json` holding `{"org": "<org>"}` in the working directory or a
+  parent. `DATASPRING_API_KEY=ds_...` (CI) acts in the key's own org.
+  `dataspring whoami --json` says which org a command here acts in and why
+  (`org_from`). There is no command that switches org for later commands.
+- **MCP.** One connection per org: `https://dataspring.app/api/mcp/?org=<org>`
+  (the slash before `?` is required), signed in with the Forspring account.
+  To work in two orgs, add two connections; no tool moves a connection. A
+  headless agent connects to the same URL with `Authorization: Bearer ds_...`;
+  the key acts in its own org, and `?org=` may only repeat it.
+- **REST.** `Authorization: Bearer ds_...`; the key is the org.
+
+Never pick an org for the person. When the org is ambiguous you are refused
+with `org_required` and the list of their orgs (§11): ask which one.
 
 ## 1. Knowledge layers — consult before querying
 
@@ -271,6 +300,31 @@ two queries (before/after) and calling out the methodology change rather
 than silently quoting one misleading trend. Same rule regardless of
 surface — same warehouse, same boundaries.
 
+## 6.1 Say when a period is partial
+
+A grouped query is clamped to the window you asked for, so its first and
+last buckets can hold only part of their period. The answer then carries
+`effective_window`: `covered` (the days the numbers cover, always the window
+asked for), `partial_periods` (each cut-short bucket with the days it holds)
+and a `note`. It is absent when every bucket is whole. Quarter to date at
+quarter grain, or January to September at year grain, comes back as ONE
+row labelled with the whole period:
+
+```json
+"effective_window": {"covered": {"start": "2025-01-01", "end": "2025-09-30"},
+  "partial_periods": [{"bucket": "2025-01-01", "start": "2025-01-01", "end": "2025-09-30"}]}
+```
+
+Label every partial bucket with the days it holds, beside its number, and
+never compare it with a whole period as if it were one:
+
+> 2025 to date (1 January to 30 September): 4.2M DKK, against 3.9M DKK for
+> the same nine months of 2024.
+
+For a like-for-like comparison send `compare_to: same_period_last_year` (or
+`previous_period`), which compares the same days. `explain_query` reports the
+same `effective_window` before you run anything.
+
 ## 7. Insights, not just data
 
 Don't just return raw query results. Interpret what the number
@@ -488,8 +542,8 @@ Examples of honest refusals:
 
 ## 11. Tool / response error recovery
 
-Errors come back with a `code` and a `suggestion` (canonical name; older
-paths still expose `hint` as an alias). Follow the suggestion.
+A failed query comes back with an `error`, a `code` and a `suggestion`
+(`hint` is an alias older clients read). Follow the suggestion.
 
 Codes you'll see:
 
@@ -523,10 +577,34 @@ replaced the sentence the reader actually needed. If you could not
 recover, say what you could not find out and what you would need — that
 is about THEM. If you did recover, just answer.
 
-Surface shape: **MCP** returns `{error, code, suggestion}` in the tool
-response (`hint` mirrors `suggestion` for back-compat); **CLI** prints
-"✗ Error: …" with a hint line, exit code 1; **REST** is HTTP 4xx/5xx with
-`{"detail": {"message", "code", "suggestion", ...}}`.
+Surface shape: **MCP** returns `{error, code, suggestion}` as the tool
+result; **REST** (`POST /api/dispatch/<tool>`) answers the same body with
+HTTP 200, and a body the tool's schema refuses (or an operation that fails
+outright) with 400 `{"detail": ...}`; **CLI** prints `✗ Error: …` with a `Hint:` line and exits 1.
+
+### Refusals: who you are, which org, what your role allows
+
+A refusal is not a query error. It has a stable `reason`, a `note` saying
+what happened and a `fix` saying what fixes it. **REST** answers it with the
+reason's HTTP status and `{"ok": false, "reason", "note", "fix"}` (plus
+`org`, and `contacts` and `fix_url` when the account named who can fix it);
+**MCP** raises a tool error whose text is `<reason>: <note> Fix: <fix>`;
+the **CLI** prints the same line and exits with the reason's code. Match on
+the reason, never on the prose, and pass the fix on verbatim.
+
+| Reason | CLI exit | What you do |
+|---|---|---|
+| `not_signed_in` | 4 | CLI: run `dataspring auth login` and hand the person its URL and code. MCP: reconnect and sign in. Never ask for a password or a key in chat. |
+| `org_required` | 2 | Several orgs, none named. Ask the person which one (the refusal lists them); then `--org`, `DATASPRING_ORG` or `.forspring.json`. Never guess. |
+| `not_a_member` | 3 | Check the org id's spelling against the list. Otherwise only an owner or admin of that org can add the person, at the Forspring account; say so and stop. |
+| `product_off` | 4 | DataSpring is not switched on for this person in that org. Tell them who can switch it on (`contacts`, `fix_url`); do not retry. |
+| `org_suspended` | 4 | Billing stopped DataSpring for the whole org. Tell them an owner can restart it on the billing page; do not retry. |
+| `role_too_low` | 4 | Their role cannot do this. Offer what it can do (a personal preference or quick metric instead of an org-wide change), or name the role needed. |
+| `org_mismatch` | 4 | The key or connection is for another org than the one named. Use that org's key or URL, or drop the conflicting `--org` / `?org=`. |
+| `account_unavailable` | 1 | Forspring's account did not answer, so access could not be checked. Retry once after a few seconds; nothing changed. |
+
+Other CLI exits: 0 success, 1 any other failure, 2 a usage error, 64 a
+removed command (it prints what replaced it).
 
 ## 12. Multi-turn / multi-call efficiency
 
@@ -606,9 +684,13 @@ How it reaches you:
   `dataspring business-context size` shows the byte budget.
 - **REST**: `GET /api/context` returns `{"content": "..."}`.
 
+- **MCP**: the connection's instructions carry it from the first turn;
+  `business_context_edit` `get` reads it again. (`dataspring://context` is
+  the user's own query preferences, not this document.)
+
 Treat `business_context` as part of the same authoritative layer
-as metric descriptions. Same provenance (dbt repo via the
-manifest workflow), same trust level.
+as metric descriptions: the same trust level. It is edited in DataSpring
+(`business_context_edit`), and an edit is live at once.
 
 That's the working agreement. Same principles, three surfaces —
 do this and the answers will be right.
@@ -699,11 +781,11 @@ MCP tool `explain_query`; `POST /api/dispatch/explain_query`. Show what a query
 
 #### `run_sql`
 
-MCP tool `run_sql`; `POST /api/dispatch/run_sql`. Run ONE read-only SQL statement against the org's warehouse and get the rows back, ungoverned.
+MCP tool `run_sql`; `POST /api/dispatch/run_sql`. Run read-only SQL (one statement, or a script whose last statement answers) against the org's warehouse and get the rows back, ungoverned.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
-| `sql` | string | yes | One SQL statement (BigQuery Standard SQL on a managed warehouse). Name tables as <org>_marts.<table>, <org>_core.<table> or <org>_staging.<table>, <org> being the organization id with every hyphen as an underscore (acme-dk is acme_dk_marts); the identity it runs as can read those and nothing else, so a write or a reach outside them is refused by the warehouse. |
+| `sql` | string | yes | One SQL statement (BigQuery Standard SQL on a managed warehouse), or a script of several separated by ';' whose last statement's rows are returned (CREATE TEMP TABLE ... ; SELECT ... works: temp tables are the scratch space, gone when the script ends). Name tables as <org>_marts.<table>, <org>_core.<table>, <org>_staging.<table>, <org>_raw.<table> (what the loads wrote) or <org>_dev.<table> (what the last datacore check built; expire after 7 days), <org> being the organization id with every hyphen as an underscore (acme-dk is acme_dk_marts); the identity it runs as can read those five datasets of its own org and nothing else, so a write to a dataset or a reach outside them is refused by the warehouse. |
 | `warehouse` | string | no | One of the org's warehouses (see dataspring://warehouses); omit for the active one. |
 | `max_rows` | integer | no, default `1000` | Rows to return at most (default 1000, cap 10000). Aggregate in SQL rather than paging. |
 | `format` | one of `json`, `csv`, `markdown` | no, default `json` | 'json' answers rows in `data`; 'csv' or 'markdown' answer one document in `content`. |
@@ -766,7 +848,7 @@ MCP tool `update_context`; `POST /api/dispatch/update_context`. Update user pref
 
 MCP tool `quick_metric_edit`; `POST /api/dispatch/quick_metric_edit`. Edit user-defined quick metrics (arithmetic expressions over existing metrics). The body is `{"action": {"action": "<verb>", ...}}`, one verb per table below.
 
-**`create`** — 
+**`create`** — Save a personal quick metric: arithmetic over existing metrics, queried as `qm:<name>`.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -774,7 +856,7 @@ MCP tool `quick_metric_edit`; `POST /api/dispatch/quick_metric_edit`. Edit user-
 | `expression` | string | yes | Arithmetic expression over existing metrics (e.g., "total_revenue / order_count") |
 | `description` | string | no |  |
 
-**`update`** — 
+**`update`** — Change one of your quick metrics.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|
@@ -783,7 +865,7 @@ MCP tool `quick_metric_edit`; `POST /api/dispatch/quick_metric_edit`. Edit user-
 | `expression` | string | no |  |
 | `description` | string | no |  |
 
-**`delete`** — 
+**`delete`** — Delete one of your quick metrics.
 
 | Parameter | Type | Required | Description |
 |---|---|---|---|

cli/_skills/dataspring-correct/SKILL.md

@@ -12,8 +12,9 @@ readers see it, and be exact about what changed.
 
 Before agreeing anything is wrong, call `explain_query`; with no
 `params` it explains this session's last query. Quote back the metric it
-resolved, the filters applied (standing preferences included), the effective
-window, and briefly the SQL. Often the number is right and the disagreement
+resolved, the filters (standing preferences included), the window, and
+briefly the SQL. A bucket in `effective_window.partial_periods` (a year to
+date labelled as the year) is often the whole dispute: name its days. Often the number is right and the disagreement
 is about which metric was meant — ambiguity, not a correction. Never
 apologise for a number you have not re-read.
 
@@ -48,7 +49,7 @@ Read the WHAT YOU MAY CHANGE block — this user's targets, nothing else. An
 admin gets the systemic fix; a member or viewer a personal preference,
 verified query or quick metric, which fixes their experience immediately.
 Both are complete answers: never route a user elsewhere, never offer what
-ends in a permission error. §7 is the same split for a coverage gap.
+ends in a `role_too_low` refusal. §7 is the same split for a coverage gap.
 
 ## 4. Ownership and ceremony
 

cli/bundled_manifest.py

@@ -6,7 +6,7 @@ the server's to say whether the server has operations this CLI does not
 know. tests/test_skills_surface.py asserts it is current.
 """
 
-MANIFEST_HASH = "b9fdc845fd9d3a22cea19e234b330e7e41d17f5d5ba6b5e1c713123d38109a6d"
+MANIFEST_HASH = "54af1f0a9d07ef692b63e0648d835ce5efc020bd9588db6d75897b36af4e0232"
 TOOL_KEYS = (
     "business_context_edit",
     "semantic_model_edit",