mostlyright-data 0.31.6__tar.gz → 0.31.7__tar.gz

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.31.6
+Version: 0.31.7
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/skills/mr-data-build/SKILL.md

@@ -270,6 +270,14 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
    Then run `mr-data models suggest RUN_ID --json`. It reads the run's preview and names where
    the table could use a deterministic derivation or a model call, and whether this deployment
    serves that model. Confirm each lead against the rows before you offer it.
+   A succeeded full run's `mr-data status` and `mr-data watch` output carries a `delivery` block.
+   It names a missing cover, a description that is missing or out of the skill's shape,
+   natural-language search when the run's table has none, and always the enrichment offer.
+   Complete every step before you report the build as done. Studio marks no main table, so you
+   judge it: skip the search step for a supporting table and say so in one line. A scratch table
+   you test and demote owes none of these steps.
+   Offering enrichment is mandatory. Accepting it is the person's choice, and a no is a complete
+   answer.
 9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
    even when the run succeeded. Untruncated acquisition does not prove publisher pagination, nested
    records or population coverage is complete. A null coverage window does not mean all history was acquired.
@@ -285,7 +293,8 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
    repair, which is why the decision belongs in step 5. A table left not refresh-ready never
    refreshes unless it declares `rebuild`; it does not become ready later.
    Download and verify artifacts when the user requested bytes.
-10. For new tables with search intent, inspect search readiness and test meaningful queries against
+10. Search is on by default for the dataset's main table. Supporting tables such as lookups and
+    code lists do not need it. Enable it, inspect search readiness and test meaningful queries against
     the observed version as [search preparation](references/preparing-table-search.md) describes.
     Search preparation runs independently: pending or failed indexing never means the data build
     failed, and declared metadata alone never proves observed search quality. Before delivery,

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/skills/mr-data-build/references/5-draft-one-recipe-document-one-call.md

@@ -300,6 +300,8 @@ uses must follow from the actual fields and coverage, without promising an outco
   no bold field labels, em dashes,
   semicolons, fake quotations or unsupported claims
 - every `##` heading in DESCRIPTION.md is one of the five, in that relative order, none repeated
+- Coverage and Columns are present. A succeeded full run's `delivery` block checks the opening,
+  the headings and the 4000-character bound, and names the description until it passes
 
 **Every dataset gets a cover before its first run is presented.** Once the description is written,
 generate one with `mr-data cover DATASET_ID "what the image should depict" --json`, describing the

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/skills/mr-data-build/references/commands.md

@@ -26,9 +26,9 @@ run in full.
 | `mr-data recipe` | Register one recipe document (dataset, question, table plan, sources, transform, checks and units) in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it. `--no-lint` sends it as written instead. A recipe that misses a feature that sets how fast a run fetches (a stated `pages.concurrency` under 64, a per-row request open to `batch`, a page walk open to `pagination.parallel`, or a Parquet file open to a ranged read) is refused `THIN_RECIPE_SLOW`, with each feature's pointer and fix, before anything is sent. Apply the fixes, or pass `--accept-slow "REASON"` with one sentence on why the slow path is right for this source. The receipt prints the reason and the features it covered. Registration refuses `THIN_BRIEF_MISSING` when the dataset's record carries no answered question and no delegation, and `THIN_RECIPE_DATASET_UNBOUND` when the document names no `dataset.id`. `--delegated "their words"` writes the delegation onto the dataset and registers in the same command, and clears the first of those two and never the second. `mr-data recipe show ID` reads one back. `mr-data recipe explain FILE` (or a recipe identifier) lists every reason its table can or cannot refresh, each with a rule code and Studio's suggested repair, and exits 2 when one blocks it. `mr-data recipe forecast RECIPE_ID` says what one run would ask of each source before it runs, and exits 3 when the run would overrun today's platform quota or one attempt. `--digest` is optional there, and without it the digest the recipe was registered with is forecast. `mr-data recipe readiness --table TABLE_ID --current-only` reads the revision one table runs now. The explanation's `fast_features` names each fast feature the recipe uses or misses, with a one-line fix. [Making a table incremental](making-a-table-incremental.md#the-loop-explain-forecast-run-diagnose) says when to use each. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
 | `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID`, with `--digest RECIPE_DIGEST` when you want the run to assert which bytes it means. Neither is positional. The digest is the bare hex the registration receipt printed. Left out, the run reads it back from the registered recipe, which is the same value, because a recipe identifier names one revision. A full of a recipe that misses a fetch-speed feature is refused `THIN_RECIPE_SLOW`, as registration is, unless `--accept-slow "REASON"` is passed. Other modes print the missed features and are not refused. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted source-action plan, and a plan shape it does not refresh returns `RESYNC_REQUIRED` before acquisition. [Which plans refresh](making-a-table-incremental.md#which-plans-refresh) lists them, and `mr-data recipe explain` says which one a table is. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal. Report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. `--max-rows` cuts rows after a source is fetched and decoded, so it does not shorten a collection walk. A sample with no clamp is refused: `CLAMPS_REQUIRED` from Studio, and `THIN_REQUEST_INVALID` from this client first. A table's first succeeded run goes live, truncated or not, so test on a scratch table as [the probe rule](3-probe-read-a-source-before-committing-to-it.md#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table) says. A large run is held for a size confirmation (not a charge), which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full an existing legacy sample-first pair is holding once its preview has succeeded, naming either half of the pair. Where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` resumes a failed or cancelled progressive full from the pages it saved (`resume_capture_run_id` and the ceilings it carried; `retry_of_run_id` as well only when Studio admits a named retry). A failure that says the recipe must change is refused before anything is posted. Any other failed run is tried again when its triple carries `room_fault: true`, re-stating that run's own coordinate and ceilings and naming the failed run so Studio starts a new attempt, and is refused with the triple and the next step Studio's recovery category names when it is not. `room_fault` is Studio's `recovery.room_fault` where Studio states one. A source that kept refusing (`SOURCE_RATE_LIMITED`, `SOURCE_HTTP_5XX`, `SOURCE_PROXY_BUSY`) is sent anyway: Studio admits it 120 seconds after the failure and refuses `RUN_RETRY_TOO_SOON` with a Retry-After before then, so wait that long and run `--retry` again. `--cancel RUN` stops one. |
-| `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. `--diagnose` adds everything else Studio can say about the run: its progress, a per-row source's plan counts, its checkpoints, and how each attempt ended, with a lost attempt's cause marked as an inference. |
+| `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. `--diagnose` adds everything else Studio can say about the run: its progress, a per-row source's plan counts, its checkpoints, and how each attempt ended, with a lost attempt's cause marked as an inference. A succeeded full run carries `delivery`: the steps still owed before the build is done (a description missing or out of the skill's shape, a missing cover, natural-language search when the dataset's main table has none, and always the enrichment offer), each with its command. It never changes the exit code. |
 | `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed and whether that failure was the execution room's. `--dataset DATASET_ID` narrows it to one dataset's runs, so "what have I run for this dataset?" is one scoped read, and a value that is not an identifier is refused `THIN_ARGUMENT_INVALID`. `--status` and `--mode` narrow it too and are refused `THIN_ARGUMENT_INVALID` for a word outside the seven states or the six modes; the three combine. Studio narrows before it cuts a page, so a filtered page holds the newest runs that match however many runs the workspace has. `--limit` says how many, and pages are followed to reach it; a listing that does not end says how many were read and suggests a smaller `--limit` or a narrower filter. The answer prints `next_cursor`, and `--cursor` continues from it. |
-| `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. `mr-data watch RUN_ID RUN_ID ...` follows up to 16 runs at once, prints one line as each ends, and returns once all have finished with one receipt per run (`failed_run_ids`, `refused_run_ids`); it exits 2 when any run failed and 1 when a run could not be watched. `--from-seq` and `--last-event-id` resume one run only. |
+| `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. `mr-data watch RUN_ID RUN_ID ...` follows up to 16 runs at once, prints one line as each ends, and returns once all have finished with one receipt per run (`failed_run_ids`, `refused_run_ids`); it exits 2 when any run failed and 1 when a run could not be watched. `--from-seq` and `--last-event-id` resume one run only. A full run that succeeds carries the same `delivery` block as `mr-data status`, and the multi-run receipt merges every run's steps. |
 | `mr-data query` | Run one bounded read-only SQL statement over a finished run's own table and print the rows. Name the table `run_table` in the SQL, whatever the recipe called it. Bounded on rows, on answer bytes and on time, and refused by name when a bound is crossed. When no worker can answer, the refusal names `mr-data download RUN_ID` as `fallback_command`, which brings the table here to read locally. The question is asked through Mostly Right Cloud; while its usage paywall is on and the workspace is unsubscribed, each answered query counts toward the person's monthly allowance, and a spent allowance refuses `THIN_USAGE_LIMIT`. A refusal after the question was queued names its `query_id`; running the same command again reads that query back. |
 | `mr-data search` | `search VERB` prepares and tests versioned search for a new table through Studio; no model or index runs on this computer. The verbs are `inspect`, `sample`, `values`, `propose`, `configure`, `validate`, `add-cases`, `explain`, `preview`, `evaluate`, `estimate`, `build`, `status`, `activate`, `cancel`, `rollback`, `locate` and `try`. `inspect`, `propose` and `validate` also take `--recipe FILE` before registration. `validate --spec` checks a spec and its evaluation cases against the pinned contract without sending anything. `locate` resolves evaluation-case rows to locators, and `try` runs one query against the live search. A build returns promptly with job coordinates, and an operation Studio does not serve answers a typed capability error. [Prepare search](preparing-table-search.md) says how it fits a build. |
 | `mr-data peek` | Print the columns, their logical types and the first rows of one run's table. The type comes off the run's own column profile where the preview carries none, so `type` is null only for a run that sealed no profile. |

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/skills/mr-data-build/references/enriching-with-small-models.md

@@ -88,7 +88,11 @@ also keep the rows it settled from being sent at all: see [Needs Studio 4.14](#n
 5. **After the run (step 8).** Run `mr-data models suggest RUN_ID --json`. It reads the run's
    preview and names, column by column, a deterministic derivation or a model call that would
    improve the table, each with `model_status` for the model it needs. It spends no call. Read
-   it as leads: confirm each against the rows before offering it.
+   it as leads: confirm each against the rows before offering it. The run's `delivery` block on
+   `mr-data status` and `mr-data watch` lists this offer as required, beside a missing
+   description, cover or main-table search. Search is its own required step, not part of this
+   offer. Make the offer before you report the build as done. Offering is
+   mandatory. Accepting is the person's choice. If nothing is worth adding, say so in one line.
 6. **Deliver (step 10).** End the report with what would make the table better: the remaining
    offers, deterministic ones first, each with the column it reads and what it would add. One
    concrete sentence each, not a menu of everything possible.

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/skills/mr-data-build/references/preparing-table-search.md

@@ -57,6 +57,13 @@ missing search support only when `error` begins with
 `THIN_SEARCH_CAPABILITY_UNAVAILABLE` as a capability limit too. Any other refusal is not
 capability evidence. That includes 401, 403, 429, timeouts and invalid responses. Resolve it.
 
+Search is on by default for the dataset's main table, whether or not the brief mentions it.
+Supporting tables such as lookups, code lists and dimension tables do not need it. A succeeded
+full run's `delivery` block names the search step whenever the run's table has no search, a
+search whose build failed included. Studio marks no main table, so you judge which one it is.
+Skip the step for a supporting table, while the capability is missing, or when the table has
+nothing searchable, and say why in one line.
+
 When the brief does not mention search, search is optional only while the capability is missing.
 Then omit `table.search`, keep the drafted intent beside the recipe, and record the limit in the
 dataset's Sources and decisions note. Never put that limit in the description, title or topics.

mostlyright_data-0.31.7/src/mostlyright/data_harness/thin/delivery.py

@@ -0,0 +1,261 @@
+"""The steps a succeeded full run still owes before the build is reported as done.
+
+A full run that succeeded has data on the page. The page is not finished until it has a real
+description and a cover, and the person has been offered enrichment. This module names what is
+missing, reading the dataset once, and always names the enrichment offer. It never raises: a
+dataset read that fails degrades to the enrichment step and a note.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable, Mapping
+from typing import Any
+
+from mostlyright.data_harness.thin import description_shape
+from mostlyright.data_harness.thin.session import StudioSession
+from mostlyright.data_harness.thin.transport import ThinLaneError, Transport
+from mostlyright.data_harness.thin.v4 import identifier
+from mostlyright.data_harness.thin.v4_datasets import StudioV4DatasetClient
+
+DELIVERY_VERSION = 1
+
+#: Where the description's shape is written down.
+SKILL_REFERENCE = "the build skill's references/5-draft-one-recipe-document-one-call.md"
+
+#: What each conformance failure tells the person to fix.
+_FAILURE_SAYS: dict[str, str] = {
+    "too_long": "it is over 4000 characters",
+    "missing_bold_opening": "it does not open with a bold first sentence",
+    "unknown_heading": "a ## heading is not one of the five",
+    "repeated_heading": "a ## heading is repeated",
+    "out_of_order_heading": "the ## headings are out of order",
+    "missing_coverage": "it has no ## Coverage section",
+    "missing_columns": "it has no ## Columns section",
+}
+
+#: The step names, in the order a person does them.
+STEPS: tuple[str, ...] = ("description", "cover", "search", "enrichment_offer")
+
+_ENRICHMENT_WHY = (
+    "Offering enrichment is required on every full build. Confirm each lead against the rows, "
+    "then offer the worthwhile classification or extraction in one question. If nothing is worth "
+    "adding, say so in one line. A no is a complete answer."
+)
+
+_SEARCH_WHY = (
+    "Enable natural-language search on the dataset's main table: the one people query "
+    "directly. It is on by default, not an offer. Follow the skill's search flow in "
+    "references/preparing-table-search.md: search inspect, propose, validate, configure, build, "
+    "activate. Skip it, saying why in one line, when this is a supporting table (a lookup, a "
+    "code list, a dimension), when search inspect shows this deployment does not serve search "
+    "authoring, or when the table has nothing searchable."
+)
+
+#: Every block says this: a scratch table is a test, not the delivered dataset.
+SCOPE = "A scratch table you test and demote owes none of these steps."
+
+
+def applies(run: Mapping[str, Any], *, succeeded: bool) -> bool:
+    """Only a full run that succeeded owes delivery.
+
+    A sample is a test, and a refresh or a backfill adds rows to a dataset delivered before. The
+    caller folds a recipe test into ``succeeded``.
+    """
+
+    return succeeded and run.get("mode") == "full"
+
+
+def _step(step: str, command: str, why: str) -> dict[str, str]:
+    return {"step": step, "command": command, "why": why}
+
+
+def _enrichment_step(run_id: str) -> dict[str, str]:
+    return _step("enrichment_offer", f"mr-data models suggest {run_id} --json", _ENRICHMENT_WHY)
+
+
+def _description_why(description: Any) -> str | None:
+    """Why the description step is owed, or ``None`` when the description conforms."""
+
+    verdict = description_shape.check(description)
+    if verdict["conforms"]:
+        return None
+    sections = ", ".join(description_shape.HEADINGS)
+    if verdict["failures"] == ["empty"]:
+        return (
+            "The dataset has no description. Write one in the skill's shape: a bold first "
+            f"sentence, then ## sections from {sections}, in that order. See {SKILL_REFERENCE}."
+        )
+    said = ", ".join(_FAILURE_SAYS[code] for code in verdict["failures"])
+    return (
+        f"The description does not follow the skill's shape: {said}. Rewrite it with a bold first "
+        f"sentence, then ## sections from {sections}, in that order, Coverage and Columns always "
+        f"present. See {SKILL_REFERENCE}."
+    )
+
+
+def required_steps(
+    record: Mapping[str, Any], *, dataset_id: str, run_id: str
+) -> list[dict[str, str]]:
+    """The steps one dataset record still needs, the enrichment offer always last."""
+
+    steps: list[dict[str, str]] = []
+    why = _description_why(record.get("description"))
+    if why is not None:
+        steps.append(
+            _step(
+                "description",
+                f'mr-data dataset set {dataset_id} --name "TITLE" --topics "TOPIC,TOPIC" '
+                "--description-file DESCRIPTION.md",
+                why,
+            )
+        )
+    if not record.get("cover"):
+        steps.append(
+            _step(
+                "cover",
+                f'mr-data cover {dataset_id} "WHAT THE IMAGE SHOWS"',
+                "The dataset has no cover. Generate one.",
+            )
+        )
+    steps.append(_enrichment_step(run_id))
+    return steps
+
+
+def _search_state(answer: Mapping[str, Any], table_id: str) -> str:
+    """``active``, ``building`` (a search job is running for this version) or ``none``."""
+
+    if answer.get("schema_version") != "table-search-metadata.v1":
+        raise ValueError("not a table-search metadata answer")
+    if answer.get("table_id") != table_id:
+        raise ValueError("table-search metadata for another table")
+    if isinstance(answer.get("generation_id"), str) and answer["generation_id"]:
+        return "active"
+    # Only a running job counts: a revision whose build failed, or a spec never built, is owed.
+    if answer.get("preparation_state") == "pending":
+        return "building"
+    return "none"
+
+
+def _search_step(
+    client: StudioV4DatasetClient, record: Mapping[str, Any], run: Mapping[str, Any]
+) -> tuple[dict[str, str] | None, str | None]:
+    """The search step when the run's table has no search, and any note. Never raises.
+
+    Studio marks no main table, so the step names the rule and the agent skips it, saying why,
+    for a supporting table.
+    """
+
+    try:
+        table_id = identifier(run.get("table_id"), "table identifier")
+    except Exception:  # a record without a usable table identifier is still a delivered run
+        return None, "Search could not be checked: the run names no table."
+    try:
+        # Imported here: the search module's imports reach the run module, which imports this one.
+        from mostlyright.data_harness.thin.v4_search import TABLE_SEARCH_PATH
+
+        answer = client._call(
+            "GET",
+            TABLE_SEARCH_PATH.format(table_id=table_id),
+            query={"links": "false"},
+            expected=(200,),
+        )
+        state = _search_state(answer, table_id)
+    except ThinLaneError as refusal:
+        return None, _search_unchecked(table_id, f" ({refusal.code})")
+    except Exception:  # a failed read must never fail the command
+        return None, _search_unchecked(table_id, "")
+    if state == "active":
+        return None, None
+    if state == "building":
+        return None, (
+            f"Search is building for this version. Read mr-data search status {table_id} "
+            "until it is active."
+        )
+    why = f"This run's table, {table_id}, has no search. {_SEARCH_WHY}"
+    return _step("search", "mr-data search inspect --recipe RECIPE.json --json", why), None
+
+
+def _search_unchecked(table_id: str, why: str) -> str:
+    return f"Search could not be checked{why}. Read mr-data search status {table_id}."
+
+
+def _unchecked(run_id: str, note: str) -> dict[str, Any]:
+    return {
+        "version": DELIVERY_VERSION,
+        "scope": SCOPE,
+        "required": [_enrichment_step(run_id)],
+        "note": note,
+    }
+
+
+def block(client: StudioV4DatasetClient, run: Mapping[str, Any], run_id: str) -> dict[str, Any]:
+    """The ``delivery`` member for one succeeded full run. Never raises."""
+
+    try:
+        dataset_id = identifier(run.get("dataset_id"), "dataset identifier")
+    except Exception:  # a record without a usable dataset identifier is still a delivered run
+        return _unchecked(run_id, "Presentation could not be checked: the run names no dataset.")
+    try:
+        record = client.dataset(dataset_id)
+    except ThinLaneError as refusal:
+        record = f" ({refusal.code})"
+    except Exception:  # a failed read must never fail the command
+        record = ""
+    if not isinstance(record, Mapping):
+        why = record if isinstance(record, str) else ""
+        return _unchecked(
+            run_id,
+            f"Presentation could not be checked{why}. Read mr-data dataset show {dataset_id}.",
+        )
+    steps = required_steps(record, dataset_id=dataset_id, run_id=run_id)
+    search, note = _search_step(client, record, run)
+    if search is not None:
+        steps.insert(len(steps) - 1, search)
+    delivered: dict[str, Any] = {"version": DELIVERY_VERSION, "scope": SCOPE, "required": steps}
+    if note is not None:
+        delivered["note"] = note
+    return delivered
+
+
+def for_run(
+    session: StudioSession,
+    transport: Transport,
+    run: Mapping[str, Any],
+    run_id: str,
+    *,
+    succeeded: bool,
+) -> dict[str, Any] | None:
+    """The ``delivery`` member, or ``None`` when the run owes none."""
+
+    if not applies(run, succeeded=succeeded):
+        return None
+    try:
+        return block(StudioV4DatasetClient(session, transport=transport), run, run_id)
+    except Exception:  # the block must never fail the command
+        return _unchecked(run_id, "Presentation could not be checked.")
+
+
+def merged(blocks: Iterable[Mapping[str, Any] | None]) -> dict[str, Any] | None:
+    """One block for several runs: every step once, in run order."""
+
+    steps: list[dict[str, str]] = []
+    notes: list[str] = []
+    seen: set[tuple[Any, Any]] = set()
+    for one in blocks:
+        if not isinstance(one, Mapping):
+            continue
+        for step in one.get("required") or ():
+            key = (step.get("command"), step.get("why"))
+            if key in seen:
+                continue
+            seen.add(key)
+            steps.append(dict(step))
+        note = one.get("note")
+        if isinstance(note, str) and note not in notes:
+            notes.append(note)
+    if not steps:
+        return None
+    delivered: dict[str, Any] = {"version": DELIVERY_VERSION, "scope": SCOPE, "required": steps}
+    if notes:
+        delivered["note"] = " ".join(notes)
+    return delivered

mostlyright_data-0.31.7/src/mostlyright/data_harness/thin/description_shape.py

@@ -0,0 +1,120 @@
+"""Whether a dataset description follows the build skill's shape.
+
+One pure function over the text, held to ``tests/fixtures/description/conformance.json`` so Cloud
+can mirror it case for case. The rules a mirror must copy exactly:
+
+* One leading U+FEFF is dropped. Whitespace is the ASCII set space, tab, LF, CR, FF and VT only.
+* Lengths count Unicode code points, not UTF-16 units.
+* Lines are split on LF after CRLF becomes LF.
+* A heading is a line that starts at column one with ``## ``. Its text is the rest of the line
+  with ASCII whitespace trimmed. Lines inside a fence (a line starting with ``` or ~~~ after up
+  to three spaces opens or closes one) are body text.
+* The opening is the text before the first heading. Its first paragraph runs to the first blank
+  line, and it must start with ``**`` and close a non-blank bold span inside that paragraph.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+CHECK = "description-conformance.v1"
+
+MAX_DESCRIPTION_CHARS = 4000
+
+#: The five headings the skill allows, in the order they must appear.
+HEADINGS: tuple[str, ...] = (
+    "Coverage",
+    "Columns",
+    "Missing values",
+    "Suitable for",
+    "Source and rights",
+)
+
+#: The headings every built table has something to say under.
+REQUIRED_HEADINGS: tuple[str, ...] = ("Coverage", "Columns")
+
+#: Every failure code, in the order they are reported.
+FAILURES: tuple[str, ...] = (
+    "empty",
+    "too_long",
+    "missing_bold_opening",
+    "unknown_heading",
+    "repeated_heading",
+    "out_of_order_heading",
+    "missing_coverage",
+    "missing_columns",
+)
+
+_WHITESPACE = " \t\n\r\f\v"
+
+
+def _fence(line: str) -> bool:
+    stripped = line.lstrip(" ")
+    return len(line) - len(stripped) <= 3 and stripped.startswith(("```", "~~~"))
+
+
+def _bold_opening(opening: str) -> bool:
+    paragraph = opening.split("\n\n", 1)[0]
+    if not paragraph.startswith("**"):
+        return False
+    close = paragraph.find("**", 2)
+    return close > 2 and bool(paragraph[2:close].strip(_WHITESPACE))
+
+
+def check(description: Any) -> dict[str, Any]:
+    """``{"conforms": bool, "failures": [code, ...]}`` for one description."""
+
+    text = description if isinstance(description, str) else ""
+    if text.startswith("\ufeff"):
+        text = text[1:]
+    if not text.strip(_WHITESPACE):
+        return {"conforms": False, "failures": ["empty"]}
+    found: set[str] = set()
+    if len(text) > MAX_DESCRIPTION_CHARS:
+        found.add("too_long")
+    opening: list[str] = []
+    headings: list[str] = []
+    fenced = False
+    for line in text.replace("\r\n", "\n").split("\n"):
+        if _fence(line):
+            fenced = not fenced
+        elif not fenced and line.startswith("## "):
+            headings.append(line[3:].strip(_WHITESPACE))
+            continue
+        if not headings:
+            opening.append(line)
+    blank_normalised = "\n".join(
+        "" if not line.strip(_WHITESPACE) else line for line in opening
+    ).strip(_WHITESPACE)
+    if not _bold_opening(blank_normalised):
+        found.add("missing_bold_opening")
+    seen: set[str] = set()
+    last = -1
+    for heading in headings:
+        if heading not in HEADINGS:
+            found.add("unknown_heading")
+            continue
+        if heading in seen:
+            found.add("repeated_heading")
+            continue
+        seen.add(heading)
+        position = HEADINGS.index(heading)
+        if position < last:
+            found.add("out_of_order_heading")
+        last = max(last, position)
+    if "Coverage" not in seen:
+        found.add("missing_coverage")
+    if "Columns" not in seen:
+        found.add("missing_columns")
+    failures = [code for code in FAILURES if code in found]
+    return {"conforms": not failures, "failures": failures}
+
+
+__all__ = [
+    "CHECK",
+    "FAILURES",
+    "HEADINGS",
+    "MAX_DESCRIPTION_CHARS",
+    "REQUIRED_HEADINGS",
+    "check",
+]

{mostlyright_data-0.31.6 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_datasets.py

@@ -771,8 +771,10 @@ def declare_arguments(parser: argparse.ArgumentParser) -> None:
     )
     parser.add_argument(
         "--category",
-        choices=CATEGORY_IDS,
-        help="with set, the primary category; inspect definitions with dataset categories",
+        help=(
+            "with set, the primary category; inspect definitions with dataset categories. "
+            "Offline reference: " + ", ".join(CATEGORY_IDS)
+        ),
     )
     parser.add_argument(
         "--topics",
@@ -1760,10 +1762,8 @@ def write(
     name = checked_name(named, needed_by="set") if named is not None else None
     description = read_description(args, stdin=stdin)
     category = getattr(args, "category", None)
-    if category is not None and category not in CATEGORY_IDS:
-        raise ThinLaneError(
-            "THIN_REQUEST_INVALID", "--category must be one of: " + ", ".join(CATEGORY_IDS)
-        )
+    if category is not None and category.strip() == "":
+        raise ThinLaneError("THIN_REQUEST_INVALID", "--category needs a value")
     topics = read_topics(getattr(args, "topics", None))
     named_license = getattr(args, "license", None)
     if (