mostlyright-data 0.21.2__tar.gz → 0.22.0__tar.gz

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.21.2
+Version: 0.22.0
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -82,6 +82,11 @@ mr-data checks RUN_ID
 mr-data download RUN_ID --output ./out
 ```
 
+Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
+including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
+with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
+can reissue supported platform execution failures without changing the recipe.
+
 Use the identifiers returned by registration and run submission. `mr-data dataset create`
 creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
 

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/README.md

@@ -70,6 +70,11 @@ mr-data checks RUN_ID
 mr-data download RUN_ID --output ./out
 ```
 
+Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
+including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
+with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
+can reissue supported platform execution failures without changing the recipe.
+
 Use the identifiers returned by registration and run submission. `mr-data dataset create`
 creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
 

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.21.2"
+version = "0.22.0"
 description = "Mostly Right hosted CLI for reviewed datasets"
 readme = "README.md"
 requires-python = ">=3.11"

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/skills/mr-data-build/SKILL.md

@@ -1867,8 +1867,10 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
 
 - `collection.discovery` says how the corpus is LISTED — `json_api` with RFC 6901 pointers,
   `html_index` with the reader's closed selectors, `sitemap`, or an `explicit` member list — and
-  which pagination grammar the listing uses. The run follows only that grammar and never a link
-  found inside a detail page.
+  which pagination grammar the listing uses. `source_rows` instead names an upstream source and
+  its `url_column`, optionally `base_url_column`: acquisition runs upstream first, resolves and
+  deduplicates links, and preserves incomplete upstream coverage. Use only declared dependencies;
+  this is not arbitrary recursive crawling. It requires the matching Studio API/worker release.
 - `collection.pages` says what may be fetched and how much: `allowed_origins` (which must include
   the discovery address's own origin), `max_pages`, `max_fetches_per_run`, `concurrency` (1 or 2),
   `min_interval_seconds` as a whole number of seconds (one or more), `request_timeout_seconds`,
@@ -1877,7 +1879,8 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
 - `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
   pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
   merged relation.
-- The Reader pin is required, must be `html.web_extract`, and is the SAME settings for every page.
+- The Reader pin is required, must be `html.web_extract`, `html.tabular`, `json.tabular`, or
+  `xml.tabular`, and uses the same settings for every page.
 
 **`limits.max_requests` is charged per HTTP request, redirect hops and retries included** — a page
 that redirects once costs two. Registration only checks that
@@ -1888,6 +1891,10 @@ redirects are likely — trailing slashes, `http`→`https`, `www`, which is mos
 `limits.max_requests >= discovery.max_requests + 2 * max_fetches_per_run`. A budget larger than the
 corpus can absorb costs nothing.
 
+For a dependent graph, declare recipe-level `acquisition_limits` with `max_requests` and
+`max_source_bytes`. Every source must explicitly state both ceilings; their sums must fit the
+totals. Unused capacity does not transfer between sources.
+
 The relation the transform reads is the reader's columns plus `page_id`, `page_url`,
 `page_fetched_at`, `page_content_sha256`, `page_revision`, `page_discovered_at` and `page_ordinal`.
 Those names are reserved: a reader that declares one is a registration refusal. Select them in the
@@ -2011,6 +2018,8 @@ decoder. Zero matches and multiple matches both refuse.
 | `archive.gzip@1.1.0` | Same; also accepts a response served as `application/octet-stream`. |
 | `archive.tar@1.0.0` | Same as `archive.zip@1.0.0`. |
 | `archive.tar@1.1.0` | Same; also accepts a response served as `application/octet-stream`. |
+| `archive.tar@2.0.0` | Explicit `compression: none or gzip`, `encoding: utf-8 or cp1252`, `member` and `member_format: csv, tsv, semicolon or lines`. Lines also accepts exact `members` list, emitting member_name, line_number, raw_line. |
+| `text.lines@1.0.0` | Explicit `encoding: utf-8 or cp1252`; preserves physical lines as line_number, raw_line without delimiter or quote inference. |
 | `spreadsheet.xlsx@1.0.0` | `sheet`, `header_row`, and `range`. The heading row must be the first row of the range. |
 | `spreadsheet.xlsx@1.1.0` | Same; also accepts a response served as `application/octet-stream`. |
 | `delimited_text@1.0.0` | `delimiter` (comma or tab); optional `header: true`; optional `encoding: utf-8`. |
@@ -2022,6 +2031,7 @@ decoder. Zero matches and multiple matches both refuse.
 | `html.tabular@1.0.0` | Absolute record element path and relative text/attribute field selectors; fields are required unless `required: false`. |
 | `html.web_extract@1.0.0` | `mode: dom` with closed CSS-like record/field selectors, or `mode: json` with an inert JSON script selector and RFC 6901 field pointers. |
 | `html.web_extract@1.1.0` | Same settings as 1.0.0; extracted text follows DOM order, with one space at block-element boundaries and `<br>` and none between inline runs. |
+| `html.web_extract@1.2.0` | Same text as 1.1.0; bounded integer `:nth-child(N)` / `:nth-of-type(N)` and field `scope: ancestor` with `ancestor_selector`. Requires the corresponding deployed certification. |
 | `protobuf.tabular@1.0.0` | Digest-pinned base64 FileDescriptorSet, fully-qualified message type, `single` or `length_delimited` framing, and fields with an explicit repeated `explode` or `refuse` choice. |
 | `parquet.tabular@1.0.0` | Optional `columns` selects a bounded ordered Parquet projection. |
 | `xbrl.facts@1.0.0` | `document_kind` (`xbrl` or constrained XML `ixbrl`) and exact namespace/name fact projection; contexts, units, and metadata remain evidence fields, not joins. |
@@ -2241,3 +2251,11 @@ The floating pill represents only your active work. Start `dataset activity` imm
 When recovering, report what you are trying now; an earlier failed attempt belongs in the record. Before every final handoff, cancellation, or exhausted stop, report `--phase done` with a truthful final sentence (for example, “Dataset ready to explore” or “Stopped before the build completed”). Do this even when tables are not enabled. Only use `waiting_on_you` when a question is actually open — the brief at stage 2, the plan at stage 4, or the full build after a preview — and never under a delegation, which leaves nothing to wait on. A table going live does not finish your agent session.
 
 Stage 1 opened the dataset before research began; keep that same tab. Enable its optional Follow agent control for a watch-along experience; manual scrolling pauses it and must never be overridden.
+
+## Recovering an import failure
+
+For a deterministic reader or transform failure, use the reported source, line and reason before changing settings. Do not repeatedly submit the same failed recipe or guess an encoding. Repeated run submission may return the existing run with `status: run_returned`; inspect `run.status` and its failure rather than treating it as new work. Use `--retry` only when the CLI identifies the failure as retryable.
+
+For malformed delimited text, use the hosted `text.lines@1.0.0` reader with explicit encoding to retain physical lines, then inspect field counts and metadata before authoring the final parse. For compressed TAR files, use `archive.tar@2.0.0` with an explicit wrapper and exact member names. These readers preserve evidence; they do not repair values or prove a guessed encoding. Reader implementations are backend modules and are not importable from the thin CLI installation.
+
+Estimate the final output separately from each source. Ordinary builds retain a 10-million-row output ceiling; increasing per-source limits does not increase it. A manifest over parts does not by itself make arbitrary SQL builds incremental. Use a supported partitioned execution capability only when the deployed worker admits it.

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/src/mostlyright/data_harness/thin/v4.py

@@ -105,7 +105,7 @@ from mostlyright.data_harness.thin.transport import ThinLaneError
 #: ``JOB_INVALID`` outright, which in production was every refresh a collection epoch was offered
 #: for rather than only the collection runs. This package must not reach a Studio older than the
 #: commit it pins; ``docs/V4-WORKER-PROTOCOL.md`` states it beside the layout's own ordering rule.
-PINNED_V4_OPENAPI_SOURCE_SHA256 = "2a83dda60e84c166f95ea18af747985bfeeb5887cb48d2c72d4335e83720dc37"
+PINNED_V4_OPENAPI_SOURCE_SHA256 = "2b854c3c4fae7c30a6219ee99ea814f80c57fbf0251ed22a0b66b002b598e6f7"
 PINNED_V4_CONTRACT_VERSION = "4.7.0"
 
 # --------------------------------------------------------------------------------------------

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/src/mostlyright/data_harness/thin/v4_runs.py

@@ -1242,7 +1242,27 @@ def start(args: argparse.Namespace, *, client: StudioV4RunClient | None = None)
     body = create_run_body(args)
     selected = client or _client(args)
     try:
-        return _started(selected.create_run(body), selected.session, status="run_queued")
+        run = selected.create_run(body)
+        # The create route is idempotent and can return a previous terminal result.
+        # Report its actual state rather than promising that another worker was queued.
+        state = run.get("status")
+        payload = _started(
+            run, selected.session, status="run_queued" if state == "queued" else "run_returned"
+        )
+        if state != "queued":
+            payload["message"] = (
+                f"Studio returned a run in state {state}. Identical run coordinates reuse "
+                "the same run; this response does not establish that a new attempt was queued."
+            )
+        triple = failure_triple(run)
+        if triple:
+            payload["failure"] = triple
+            payload["next_command"] = (
+                f"mr-data run --retry {run['run_id']}"
+                if triple["room_fault"]
+                else f"mr-data status {run['run_id']}"
+            )
+        return payload
     except ThinLaneError as refusal:
         held_payload = _held_for_confirmation(refusal, body)
         if held_payload is None:
@@ -1271,7 +1291,12 @@ def start(args: argparse.Namespace, *, client: StudioV4RunClient | None = None)
 def run_exit_code(payload: Mapping[str, Any]) -> int:
     """A run held at the spend gate is a gate a script acts on, not a report it reads."""
 
-    return 2 if payload.get("status") == "run_awaiting_confirmation" else 0
+    return (
+        2
+        if payload.get("status") == "run_awaiting_confirmation"
+        or payload.get("run", {}).get("status") in {"failed", "awaiting_confirmation"}
+        else 0
+    )
 
 
 # --------------------------------------------------------------------------------------------

{mostlyright_data-0.21.2 → mostlyright_data-0.22.0}/src/mostlyright/data_harness/ux/render.py

@@ -487,6 +487,7 @@ STATUS_HEADLINES: dict[str, str] = {
     # what Studio's own state machine reached, and "the watch ended" is honest about the case
     # where the connection stopped before the run did.
     "run_queued": "The run is queued. Watch it at the address below.",
+    "run_returned": "Studio returned the run below in its current state.",
     # A run over the spend threshold is HELD, not refused and not started. Three words rather than
     # one, because "it is waiting for you", "you said yes and it is queued" and "it is stopped" are
     # three different things to have happened, and one word covering them is how a held run gets