mostlyright-data 0.30.6__tar.gz → 0.31.1__tar.gz

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.30.6
+Version: 0.31.1
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/SKILL.md

@@ -9,6 +9,11 @@ Build the dataset the user requested, verify its contents, and report its actual
 Use hosted commands with `--json` and returned identifiers. This core workflow governs the
 references. Load the reference for the current stage only, not the entire library before starting.
 
+Every build also improves the table beyond what was asked: derive what a rule can settle, and
+offer the small models (classify, extract, match) for the judgement no rule can make. Do not
+wait for the person to name a model. Steps 3, 4, 5, 8 and 10 say where, and
+[enriching with small models](references/enriching-with-small-models.md) says how.
+
 ## Four gates
 
 Every build passes these four gates in order. Do not skip one. The numbered steps below carry
@@ -102,6 +107,10 @@ step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
    [brief reference](references/2-brief-two-to-four-questions-each-with-a-recommended-answer.md).
    In an unattended session use the established brief, record assumptions within its scope,
    and do not wait for an absent person.
+   Run `mr-data models --json` once and add one line of enhancements to the brief message: at
+   most three, each saying what it would add, with your recommendation, as
+   [enriching](references/enriching-with-small-models.md#where-the-loop-runs) describes. A
+   judgement the request already names, such as a label per row, is part of the build.
 
 ## Research, register, build
 
@@ -109,6 +118,11 @@ step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
    from publisher claims. Read [source inspection](references/3-probe-read-a-source-before-committing-to-it.md)
    for supported bounded paths and limitations. A missing capability is not evidence that the
    source is invalid. Never claim inspected rows when only documentation was read.
+   Read the sampled rows column by column for what would improve them: text stored as numbers or
+   dates, coded text with a published format, code lists, names to join on, free text a label
+   would summarise, pages or documents holding facts no column holds. Sort each by
+   [the ladder](references/enriching-with-small-models.md#the-ladder-a-rule-first-a-model-for-the-rest):
+   a rule first, a model for the rest.
 5. State the selected sources, joins, coverage and material limitations. Ask for a decision only
    if existing authorization does not cover it. Read the
    [recipe reference](references/5-draft-one-recipe-document-one-call.md) before authoring fields.
@@ -120,6 +134,10 @@ step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
    rules. If semantic search or JEV interpretation into filters/sorts is explicitly required,
    do not register or queue a recipe that omits it; report the capability limit and wait.
    Do not add search to existing recipes or rebuild existing datasets without an explicit request.
+   Put the deterministic derivations found in step 4 into the first revision. Add a classify
+   or extract source when the brief asked for that judgement or the person accepted the offer,
+   from the binding `mr-data models` lists, and let a rule settle every row it can before the
+   model reads the rest ([writing a classify source](references/enriching-with-small-models.md#writing-a-classify-source)).
    Bind `dataset.id`, describe columns, declare checks and units, and register with
    `mr-data recipe RECIPE.json --json`. Repair all lint findings before queuing work.
    The server returns the recipe digest; do not compute or invent it.
@@ -210,6 +228,9 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
    coverage instead, as [receipts](references/receipts.md) describes. Carry the
    correction into step 10. Read
    [verification](references/7-interrogate-ask-the-run-what-it-actually-delivered.md).
+   Then run `mr-data models suggest RUN_ID --json`. It reads the run's preview and names where
+   the table could use a deterministic derivation or a model call, and whether this deployment
+   serves that model. Confirm each lead against the rows before you offer it.
 9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
    even when the run succeeded. Untruncated acquisition does not prove publisher pagination, nested
    records or population coverage is complete. A null coverage window does not mean all history was acquired.
@@ -230,6 +251,8 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
     Search preparation runs independently: pending or failed indexing never means the data build
     failed, and declared metadata alone never proves observed search quality.
     Report rows and coverage actually verified, checks, limitations and the stable dataset link.
+    End with what would make the table better: the remaining enhancements, deterministic ones
+    first, each naming the column it reads and what it would add, and the model that would do it.
     Read [delivery](references/9-present-only-what-survived-inspection-with-caveats.md) for details.
 
 ## Source-specific references
@@ -240,6 +263,7 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
 | Reader coordinate and options | [Readers](references/readers.md) |
 | Connector or collection | [Sources](references/sources.md) |
 | One request per row of another source, such as a decision model scoring each row | [Per-row requests](references/sources.md#authoring-a-per-row-request) |
+| Which small models this deployment serves, the rule-first ladder, and classify and extract sources | [Enriching with small models](references/enriching-with-small-models.md) |
 | SQL and joins | [Transforms](references/transforms.md) |
 | Source credentials | [Credentials](references/source-credentials.md) |
 | Live stream recording | [Streams](references/recording-a-stream-venue.md) |

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/references/5-draft-one-recipe-document-one-call.md

@@ -141,6 +141,7 @@ The allowed IDs and labels are:
 | `people-society` | People & Society |
 | `health-medicine` | Health & Medicine |
 | `science-research` | Science & Research |
+| `ai-models` | AI & Models |
 | `technology` | Technology |
 | `energy` | Energy |
 | `agriculture-food` | Agriculture & Food |

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/references/commands.md

@@ -1,6 +1,6 @@
 ## Commands
 
-These thirty-eight are the whole surface, alongside `acquire-slices`, which names a capability
+These thirty-nine are the whole surface, alongside `acquire-slices`, which names a capability
 gap rather than doing anything, and `export-hosted-candidate`, which is a backend operator step.
 
 Every command that takes a run identifier also takes its first eight or more characters,
@@ -41,6 +41,7 @@ run in full.
 | `mr-data parts` | List one table version's parts — the identifier, the row count, the byte size and the recorded bounds of each — so a training loader can shard them across workers and read a table no single file should hold. `--version` names a version rather than the live one, and `--from`/`--to` keep only the parts a value range cannot rule out. Nothing is downloaded by listing them; the pattern is under [Reading a table that is made of parts](7-interrogate-ask-the-run-what-it-actually-delivered.md#reading-a-table-that-is-made-of-parts). |
 | `mr-data diff` | Compare two runs and say what changed. |
 | `mr-data connections` | List saved workspace connections and their current recipe coordinates; `--dataset ID` filters to connections granted to that dataset. No credential values are returned. |
+| `mr-data models` | List the small models a recipe can call on this deployment: each task (classify, extract, match), whether it is served, and each binding's route, pinned model, rate, today's usage and a request body to start from. `models suggest RUN_ID` reads the preview one run sealed and names, column by column, a deterministic derivation or a model call that would improve the table, with whether its model is served. Both read only and spend no model call. [Enriching with small models](enriching-with-small-models.md) says what to do with the answer. |
 | `mr-data keys` | Enrol a source credential by name, list the names with their created and rotated times, and delete one. `keys set` reads the value from a file or from standard input and never from the command line; no command prints a value back. |
 | `mr-data stream` | Record a public `wss://` venue so a build can read it. `stream document register --file F` admits the connector document, `stream registry create --digest D` admits a set of them, `stream probe` listens briefly and seals nothing, `stream capture --seconds N` records a bounded window, and `stream subscribe` keeps recording. `stream subscription {show|pause|resume|stop} ID` retains the legacy full-record interface. `stream subscription quiescence ID` reads the bounded server-computed proof; `pause-cas` and `resume-cas` read its ETag and make one conditional bounded write. An ambiguous CAS failure is not retried until a fresh invocation observes the resource again. `stream status ID` reads a capture or probe, and `stream recordings --document-digest D` names what a build could read and until when. `--seconds` is at most 600. `mr-data run` takes no duration: a build reads sealed batches and is bounded by them. |
 | `mr-data open` | Mint a single-use, short-lived address that continues this session in a browser at one dashboard path, and print it. `--open` hands it to the platform opener. The credential decides who mints: a device credential — what `mr-data login` stores — mints at Cloud, and a personal access token mints at Studio. |

mostlyright_data-0.31.1/skills/mr-data-build/references/enriching-with-small-models.md

@@ -0,0 +1,158 @@
+## Enriching with small models
+
+Mostly Right builds the pipeline with an agent and then improves the table with small models
+where code is not enough. Offering those improvements is part of every build. Do not wait for
+the person to name a model: most people do not know which ones exist. Look for where the table
+could be better while you probe, say what you found, and recommend what to add.
+
+### What this deployment serves
+
+Run `mr-data models --json` once per session, before the brief message. It lists three tasks and
+whether each is served here, read from Studio's platform credential listing.
+
+| Task | What it does | Binding |
+| --- | --- | --- |
+| classify | Labels each row from a text column with yes/no or choice questions: a topic, a category, a sentiment, a flag no keyword rule decides. Each answer carries a probability per label. | `classify` (Clef, self-hosted) first; `openrouter-decisions` (TypeSafe's Jev) answers the same questions |
+| extract | Pulls named fields (names, amounts, dates, places) out of page or document text into columns. Every value appears verbatim in the input or is dropped. | `extract` (the extraction model, NuExtract-based) |
+| match | Links the same entity across tables or within one. | none yet: `planned` |
+
+Each served binding comes with its route, its pinned `model`, its rate, today's usage and quota,
+and a `request_template`. Copy `origin`, `method`, `path_template` and `model` from the listing
+exactly, and write the same origin as the source's `connector.origin` and as the only entry of
+`pages.allowed_origins`. Clef's and the extraction model's origins are this deployment's own
+service addresses, which no one can guess: the platform credential is chosen by that route and pin, so a guessed origin or a model
+alias is refused at registration with `PLATFORM_CREDENTIAL_UNBOUND`. The person enrols no key.
+The person pays nothing for these calls. They count against the workspace's daily quota, which
+`mr-data recipe forecast` reports. Prefer `classify` over `openrouter-decisions` for new work:
+Mostly Right pays OpenRouter per Jev call, and Clef answers the same request body.
+
+Each binding also comes with a `pages_template`: `request_timeout_seconds` 120 and `retries` 3.
+Write both into the collection's `pages`. Clef and the extraction model scale to zero when idle,
+and their first instance takes three to five minutes to load. Studio starts loading it when the
+run is admitted, and until the service has answered once it spreads each member's attempts over
+six minutes rather than seconds. The longer timeout and the fourth attempt let the last one land
+after a slow start. Size `limits.max_requests` at `pages.max_fetches_per_run` times four. A run
+that still fails says `it never answered, likely a 3 to 5 minute cold start` in its
+`ROW_REQUEST_UNAVAILABLE` detail: tell the person the model was starting up and run it again, which keeps every saved
+answer.
+
+A task listed `not_served` or `planned` is offered as coming, never written into a recipe. For
+`match`, join on a normalized deterministic key now and say that model matching for the rows the
+key cannot settle is coming.
+
+### The ladder: a rule first, a model for the rest
+
+Improve the table with code wherever a rule can decide, and spend a model call only on the
+judgement no rule can make. Both are improvements to offer.
+
+**Deterministic, in the transform, no model:**
+
+- Cast text to its real type: numbers, booleans, dates and timestamps in UTC.
+- Decode a published format into columns with `regexp_extract`: a METAR weather report, a
+  vehicle identification number, an ICD or NAICS code, a postcode.
+- Map a documented code list to readable labels with a `CASE`, from the publisher's own list.
+- Derive categories a published standard defines from columns the table already has. The FAA
+  flight category (VFR, MVFR, IFR, LIFR) follows from ceiling and visibility; fog or thunderstorm
+  flags follow from the present-weather codes.
+- Build a normalized key (lowercase, trimmed, punctuation and legal suffixes such as Inc, LLC and
+  Ltd removed) for an exact join or a dedupe.
+
+**A model, through a per-row source:**
+
+- classify: what a row's text is about, whether it states something, its sentiment or intent,
+  which of the person's categories it belongs to.
+- extract: facts stated in prose, a page or a document that no column holds.
+- match, once served: two records that are the same entity under different spellings.
+
+**Both together.** When a rule settles most rows, settle those in the transform and let the
+model label the rest, then `coalesce` the rule's label with the model's in the transform. Say
+which rows each label came from in the column description. Where Studio serves it, the rule can
+also keep the rows it settled from being sent at all: see [Needs Studio 4.14](#needs-studio-414).
+
+### Where the loop runs
+
+1. **Brief (step 3).** Read what the table is for and which judgements the person named. Run
+   `mr-data models --json`. In the brief message add one line of enhancements: at most three,
+   each saying what it adds in the person's words, with your recommendation. A judgement the
+   brief already asks for ("which reports mention fog", "tag each filing by sector") is part of
+   the build, not an offer.
+2. **Probe (step 4).** Read the sampled rows column by column with the ladder: each free-text,
+   coded, name and document column. Note which rule or model would improve it.
+3. **Draft (step 5).** Write the deterministic derivations into the first revision: they cost
+   nothing and are checked like any other column. Write a model source when the brief asked for
+   that judgement or the person accepted the offer. In an unattended build, add a model source
+   only for a judgement the brief names, and record the others as offers.
+4. **Test (gate 3).** A model source is tested on the scratch table like any per-row source, with
+   `pages.stop_at_budget: true` and a small `pages.max_fetches_per_run`. Read a few answers
+   against the rows they label before the full.
+5. **After the run (step 8).** Run `mr-data models suggest RUN_ID --json`. It reads the run's
+   preview and names, column by column, a deterministic derivation or a model call that would
+   improve the table, each with `model_status` for the model it needs. It spends no call. Read
+   it as leads: confirm each against the rows before offering it.
+6. **Deliver (step 10).** End the report with what would make the table better: the remaining
+   offers, deterministic ones first, each with the column it reads and what it would add. One
+   concrete sentence each, not a menu of everything possible.
+
+### Writing a classify source
+
+A classify source is a [per-row request](sources.md#authoring-a-per-row-request) over the source
+that holds the text. Start from the listing's `request_template` and fill in:
+
+- `parameters` and the `{text_column}` placeholder in `state`: rename both to the upstream
+  source's column that holds the text. The template's `text_column` and `id` are placeholders,
+  and registration refuses a parameter or carried column the upstream does not have.
+- `questions`: one entry per column you want. A `choice` question names its labels as keys of
+  `criteria` (1 to 64); include a "none fits" label so a row is never forced into a wrong one. A
+  `noul` question answers one statement true or false, and its `criteria` may describe only
+  `true` and `false`. A `score` question rates the text on a list of 1 to 64 levels. Keep the
+  instructions to one sentence about the text alone. `state` takes at most 65,536 characters.
+- `carry_columns`: the upstream key and the columns the transform needs, so the transform reads
+  this source alone.
+- Answers are reused: a request already answered under the same pin and question is not sent
+  again, so a refresh pays only for new rows.
+
+The Reader reads each label's probability by JSON pointer, as the Jev example in
+[per-row requests](sources.md#authoring-a-per-row-request) shows. Declare each probability column
+`required: false`, rank them in the transform, and keep the top label and its probability. A
+table that publishes a model's label names the model pin in the column description and keeps the
+probability beside the label.
+
+### Writing an extract source
+
+An extract source has the same per-row shape with the extraction body
+([the extract binding](https://mostlyright.md/docs/reference/sources/)):
+
+- `input` is `{"view": "input", "text": "{text_column}"}`. `text` is the only placeholder, and it
+  names a column of the upstream source, never of the transform. Give it visible text, not raw
+  HTML: read the page in the upstream source with a Reader such as `html.web_extract`
+  ([Readers](readers.md)).
+- `fields` is a literal list of 1 to 64 `{name, description}` entries. Changing it changes the
+  source's capture identity, so settle the list in the brief.
+- `one: true` asks for a single object per row, for a detail page; leave it out for a list of
+  records per row.
+
+Each answer is grounded: every value appears verbatim in the input, and `spans` say where. A row
+whose text states none of the fields returns no record, which is an answer, not a failure.
+
+### Saying it
+
+Name the model by what it does, not by its binding: "a small classification model reads each
+report and labels the weather regime", "a small extraction model pulls the issuer and the amount
+out of each filing". Say how many rows it would read, that it costs the person nothing, and what
+the new columns would be called. Never say a model was used when the table has no source that
+called it, and never present a model's label as fact: it is a label with a probability.
+
+### Needs Studio 4.14
+
+Check first: run `mr-data recipe explain RECIPE.json --json` and read `studio_capabilities`. Use
+this only when `row_filter` (and, for the second item, `per_row_revisit_interval`) is true there.
+When it is false, the answer is "not available on this Studio yet": every upstream row is sent,
+and the transform settles which label wins.
+
+- State the deterministic rule as the per-row source's `row_filter`, the negation of the rows the
+  rule settles, so those rows cost no call
+  ([filter before paying](making-a-table-incremental.md#filter-before-paying-row_filter)). It
+  must never drop a row the transform keeps.
+- Where the per-row interval is served too, state `pages.revisit` as a long interval, up to a
+  year: the model is pinned and the question fixed, so an answer does not change and should be
+  carried forward ([a freshness bound](making-a-table-incremental.md#a-freshness-bound-on-per-row-answers)).

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/references/preparing-table-search.md

@@ -16,10 +16,16 @@ meaning-based retrieval over descriptions, and a planner that turns a sentence i
 | `search propose --recipe FILE` | Returns a starter spec. It never edits the file. |
 | `search validate` | Checks a spec and its evaluation cases against the pinned Studio contract, then asks Studio. |
 | `search locate` | Turns a title, a year or an id into the `[part, row]` locator an evaluation case needs. |
+| `search sample` | Reads up to 100 rows of a sealed version, with `--columns` to project. |
+| `search values` | Lists the most frequent values of one `--column`, with `--prefix` to narrow them. |
+| `search preview` | Runs `--query` or a `--plan` file as an unranked scan under a revision's spec, activated or not. |
 | `search configure` | Stores a draft revision of the spec, with its evaluation cases. |
+| `search add-cases` | Declares a new revision that adds `--cases` to the draft or to `--revision`. |
+| `search evaluate` | Reports where a revision's evaluation measurement stands. It starts nothing. |
 | `search build` | Queues a search build for a sealed version, or for one revision with `--revision`. |
 | `search status` | Reports package readiness, job progress and the failure reason. |
 | `search try` | Runs one query against the table's live search and prints rows, plan and timings. |
+| `search query` | Searches a public table through Cloud's search route, as its page and MCP do. |
 | `search activate` | Switches the version's live search to a built generation. |
 | `search rollback` | Switches back to an earlier generation. |
 | `search explain` | Shows how Studio parses one query into filters and sorts. |
@@ -28,6 +34,15 @@ meaning-based retrieval over descriptions, and a planner that turns a sentence i
 Exit codes are stable. 0 means complete. 1 means failed or invalid. 3 means still running or
 served by the fallback scan. 4 means ambiguous.
 
+`sample`, `values` and `preview` need Studio 4.41 and a warm worker. They queue one bounded query
+and wait for its rows the way `mr-data query` does. A read still running exits 3; run the same
+command again to read the same query back. `QUERY_NO_WARM_EXECUTOR` means no worker holds the
+workspace: open one with `mr-data probe`, wait the `retry_after_seconds` it names, and retry.
+`evaluate` exits 0 on `passed` and `not_required`, 1 on `failed`, `build_failed` and
+`measurement_failed`, and 3 on `no_cases` and while the revision is unbuilt, building or
+unmeasured. An older
+Studio answers these verbs with `THIN_SEARCH_CAPABILITY_UNAVAILABLE`.
+
 ## 1. Check the capability
 
 Probe the deployment before you add `table.search` to a recipe:
@@ -129,8 +144,8 @@ An alias maps a phrase to an exact filter. Use one for each category value a use
 {"phrase": "tv show", "column": "media_type", "operator": "eq", "value": "tv"}
 ```
 
-Phrases are unique and casefolded. Take the values from the built table with `mr-data query`, not
-from documentation. Ordered comparisons need a numeric or date column. `contains` needs a text
+Phrases are unique and casefolded. Take the values from the built table with
+`mr-data search values TABLE_ID --column genres` or `mr-data query`, not from documentation. Ordered comparisons need a numeric or date column. `contains` needs a text
 column. Decimal values travel as strings. Never invent a meaning such as "best" when the brief
 does not name a metric. Leave that word unresolved, or add an explicit alias such as
 "highest rated".
@@ -311,6 +326,8 @@ mr-data search build TABLE_ID --version VERSION_ID --revision REVISION_ID --json
 mr-data search status --job JOB_ID --json
 ```
 
+To add cases to the draft without restating its spec, run `mr-data search add-cases TABLE_ID
+--version VERSION_ID --cases more-cases.json --json`. It answers the new revision; build that one.
 Leave out `--spec` to declare the `table.search` spec of the recipe that sealed the version.
 Use that to add evaluation cases without retyping the spec. Without `--spec` the client cannot
 check locally that semantic cases include a paraphrase and a hard negative. Studio checks them
@@ -354,6 +371,32 @@ Add `"engine": "resident"` and `"settled": true` to `search` to use the live pac
 does. Public query prints Cloud stage durations under `result.timings`, the HTTP status and the
 request ID. Keep a refusal's `detail` with its request ID.
 
+## 15. Search a table the way its page does
+
+A public table with a ready search package answers through Cloud's search route. The dataset
+page, the Public API's `POST /api/v2/public/tables/{id}/search`, MCP `search_table` and this verb
+all reach the same search service with the same plans. There is no scan fallback.
+
+```sh
+mr-data search query PUBLIC_TABLE_ID --query "incepton" --json
+mr-data search query PUBLIC_TABLE_ID --query "dun" --unsettled --page-size 5 --json
+mr-data search query PUBLIC_TABLE_ID --query "dune" --count --json
+mr-data search query PUBLIC_TABLE_ID --query "dune" --plan plan.json --cursor NEXT_CURSOR --json
+```
+
+A query is settled by default, so the planner runs. `--unsettled` answers as the search box does
+while someone types, without the planner. `--page-size` is 1 to 100 (default 20). `--count` asks
+for the number of matching rows. To read the next page, save the answer's `plan` to a file and
+pass it with `--plan` beside `--cursor NEXT_CURSOR`; a cursor continues one plan.
+
+The answer carries the rows as `{row_id, fields}`, the applied `plan`, the `planner` report, the
+`retrieval` mode, `next_cursor`, the `package` it read, which `service` answered, the stage
+timings under `service_timings_ms` and Cloud's `Server-Timing` under `cloud_timings_ms`.
+
+A table without a ready package is refused with Cloud's `search_unavailable` and a reason:
+`preparing` (retry after the time it gives), `not_public`, `no_package`, or Studio's own reason.
+A plan or cursor the service refuses answers `invalid_plan` with diagnostics.
+
 ## Rules
 
 - Never emulate search locally, and never claim readiness from a declaration.

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/references/source-credentials.md

@@ -38,8 +38,9 @@ A [per-row request](sources.md#authoring-a-per-row-request) presents a key by it
   `request.auth` says. A saved connection cannot stand in for it.
 - `platform` presents a key Mostly Right holds, so the user enrols nothing.
 
-The platform key serves OpenRouter's decisions endpoint for TypeSafe's Jev decision model today,
-pinned to `typesafe/jev-1.13`. Do not ask the user for an OpenRouter key to use it. A user who
+The platform key serves every binding `mr-data models --json` lists: Clef for classification,
+the extraction model where it is served, and OpenRouter's decisions endpoint for TypeSafe's Jev
+decision model, pinned to `typesafe/jev-1.13`. Do not ask the user for a key to use any of them. A user who
 prefers their own key enrols it with `mr-data keys set`. The source then states
 `opaque_reference`, `credential.secret_name` and `"auth": {"kind": "bearer"}`.
 

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/skills/mr-data-build/references/sources.md

@@ -251,11 +251,15 @@ breaks a rule above.
 | `opaque_reference` | `credential.secret_name` and `request.auth` | a key the user enrolled with `mr-data keys set` |
 | `platform` | no `credential` and no `auth` | an endpoint Mostly Right holds a key for |
 
-The platform credential needs no key from anyone, and any subscribed workspace may use it. It serves
-one binding today: `POST https://openrouter.ai/api/alpha/decisions`, with `/model` set to
-`typesafe/jev-1.13`. Pin that exact model. A floating alias such as `~typesafe/jev-latest` is
-refused at registration with `PLATFORM_CREDENTIAL_UNBOUND`. Keep the origin, method and path
-exact, state no query and no literal header, and write `/model` as a literal. Write no other body
+The platform credential needs no key from anyone, and any subscribed workspace may use it. Each
+binding is one route and one pinned model, and `mr-data models --json` lists the ones this
+deployment serves: Clef for classification (`classify`), TypeSafe's Jev through OpenRouter
+(`openrouter-decisions`, `POST https://openrouter.ai/api/alpha/decisions` with `/model` set to
+`typesafe/jev-1.13`) and, where it is served, the extraction model (`extract`). Pin the exact
+model the listing names. A floating alias such as `~typesafe/jev-latest` is refused at
+registration with `PLATFORM_CREDENTIAL_UNBOUND`. Keep the origin, method and path exactly as
+listed, state no query and no literal header, and write `/model` as a literal.
+[Enriching with small models](enriching-with-small-models.md) says when to offer each. Write no other body
 member whose name differs from `model` only in case. A user who holds their own OpenRouter key can
 present it with `opaque_reference` instead.
 
@@ -287,7 +291,11 @@ The worker retries 408, 429 and every 5xx but 501 and 505. It also retries when
 arrived: a transport, resolver or TLS failure, or a response refused before its status was
 checked, such as a compressed body or a malformed header, whatever its status. It sends a request
 at most `pages.retries` more times, then fails the run with `ROW_REQUEST_UNAVAILABLE`. A
-`Retry-After` longer than 60 seconds fails the run at once, and more retries cannot help it. When
+`Retry-After` longer than 60 seconds fails the run at once, and more retries cannot help it. A
+platform credential's source that calls this deployment's own model service (Clef, the
+extraction model) waits longer before that service has answered once: a failure
+with no answer, or a 502, 503 or 504 without `Retry-After`, is that service's cold start, and the
+attempts left are spread so the last goes out six minutes after the first. When
 the request presents a key, a checked 401 or 403 stops the source with
 `ROW_REQUEST_CREDENTIAL_REFUSED`, and a checked 402 with `ROW_REQUEST_PAYMENT_REQUIRED`. Any
 other answer the upstream gave fails that one member, and the run goes on. That covers another

{mostlyright_data-0.30.6 → mostlyright_data-0.31.1}/src/mostlyright/data_harness/thin/dataset-categories-v1.json

@@ -13,7 +13,7 @@
       "id": "crypto",
       "label": "Crypto",
       "definition": "Cryptocurrencies, blockchain networks, DeFi and digital-asset markets.",
-      "exclusions": "Use Finance for conventional securities, banking and non-crypto prediction markets.",
+      "exclusions": "Use Finance for conventional securities, banking and other non-crypto financial markets.",
       "examples": "Bitcoin prices; Ethereum transactions; DeFi lending"
     },
     {
@@ -21,7 +21,7 @@
       "label": "Finance",
       "definition": "Banking, securities, investment, insurance and financial-market activity.",
       "exclusions": "Crypto assets belong to Crypto; economy-wide indicators belong to Economics & Business.",
-      "examples": "Stock prices; interest-rate products; prediction-market order books"
+      "examples": "Stock prices; interest-rate products; exchange trade records"
     },
     {
       "id": "economics-business",
@@ -58,12 +58,19 @@
       "exclusions": "Use the subject-specific category for weather, health, technology or other covered research.",
       "examples": "Astronomical observations; particle experiments; research publications"
     },
+    {
+      "id": "ai-models",
+      "label": "AI & Models",
+      "definition": "AI systems, specialized and frontier models, evaluations, training-run metadata, agent traces and model-related corpora.",
+      "exclusions": "General software, internet and telecommunications measurements belong to Technology. A dataset about a non-AI subject stays in that subject even when a model was used to build it.",
+      "examples": "Frontier-model benchmark scores; model cards; agent-workflow traces"
+    },
     {
       "id": "technology",
       "label": "Technology",
-      "definition": "Software, computing, AI, internet and telecommunications.",
-      "exclusions": "Blockchain belongs to Crypto; media consumption and gaming belong to Arts, Media & Entertainment.",
-      "examples": "Software packages; internet performance; AI benchmarks"
+      "definition": "Software, computing, internet and telecommunications.",
+      "exclusions": "AI systems, models, evaluations and agent traces belong to AI & Models. Blockchain belongs to Crypto; media consumption and gaming belong to Arts, Media & Entertainment.",
+      "examples": "Software packages; internet performance; package registries"
     },
     {
       "id": "energy",