mostlyright-data 0.15.2__tar.gz → 0.16.0__tar.gz

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.15.2
+Version: 0.16.0
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
@@ -208,21 +208,33 @@ mr-data --help
 
 ## Build a dataset
 
+Register a recipe document with the sources, table schema, transformations and checks:
+
 ```bash
-mr-data probe SOURCE_ID "does this feed carry city grain before 2021"
-mr-data recipe RECIPE.json
-mr-data run RECIPE_ID --sample --max-rows 50000
-mr-data checks RUN
-mr-data download RUN --output ./out
+mr-data recipe recipe.json --json
+mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
+mr-data status RUN_ID
+mr-data checks RUN_ID
+mr-data download RUN_ID --output ./out
 ```
 
-`probe` asks one registered source one question; the session behind it is opened, reused and closed
-for you. `recipe` registers one document and prints the identifiers the server derived from it.
-`run` starts a run and `mr-data status RUN`, `mr-data runs`, and `mr-data watch RUN` report on it.
-The current Harness worker executes unwindowed `sample` and `full` runs. It refuses a stated run
-window, and it does not execute `refresh`, `backfill`, promotion catch-up, or scheduled warehouse
-continuation. `mr-data promote TABLE_ID` is still the human-gated publication request, but claim a
-table is live and current only when the deployed Studio and warehouse services return that evidence.
+Use the identifiers returned by registration and run submission. `mr-data dataset create`
+creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
+
+The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
+update behavior: bounded source windows merge with retained history; snapshot sources are
+revalidated or acquired again. The table's transformations and checks run over the resulting
+source relations. URL windows may use declared query parameters or path placeholders; a fixed
+date URL does not advance automatically. Configure a correction lookback where the publisher
+can revise earlier observations.
+
+A source exposing only its current snapshot cannot supply a historical delta. It remains a
+supported source, with snapshot comparison and recomputation rather than a claim that only new
+rows were fetched. Recorded streams use their registered capture and continuation semantics.
+
+See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
+`mr-data promote TABLE_ID` requests publication; claim a table is live and current only when
+Studio returns that evidence.
 
 ## There is no local execution
 
@@ -252,7 +264,9 @@ Use `mr-data --help` for the full command list and options. Commands that suppor
 
 - Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
   or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
-- Hosted credential-free acquisition supports one single-address public HTTPS object. Bounded collections have no hosted route, and the route they used to have was the deleted local one.
+- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
+  collections have no route at all; the retired local lane was their only route.
+- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
 - A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.
 
 ## Verification
@@ -267,7 +281,7 @@ before review or merge.
 ## Development
 
 ```bash
-uv sync --locked --extra dev
+uv sync --locked --extra dev --extra local-search-neural
 ```
 
 The test suite drives the execution engine in-process, and requires Linux for the worker-boundary

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/README.md

@@ -45,21 +45,33 @@ mr-data --help
 
 ## Build a dataset
 
+Register a recipe document with the sources, table schema, transformations and checks:
+
 ```bash
-mr-data probe SOURCE_ID "does this feed carry city grain before 2021"
-mr-data recipe RECIPE.json
-mr-data run RECIPE_ID --sample --max-rows 50000
-mr-data checks RUN
-mr-data download RUN --output ./out
+mr-data recipe recipe.json --json
+mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
+mr-data status RUN_ID
+mr-data checks RUN_ID
+mr-data download RUN_ID --output ./out
 ```
 
-`probe` asks one registered source one question; the session behind it is opened, reused and closed
-for you. `recipe` registers one document and prints the identifiers the server derived from it.
-`run` starts a run and `mr-data status RUN`, `mr-data runs`, and `mr-data watch RUN` report on it.
-The current Harness worker executes unwindowed `sample` and `full` runs. It refuses a stated run
-window, and it does not execute `refresh`, `backfill`, promotion catch-up, or scheduled warehouse
-continuation. `mr-data promote TABLE_ID` is still the human-gated publication request, but claim a
-table is live and current only when the deployed Studio and warehouse services return that evidence.
+Use the identifiers returned by registration and run submission. `mr-data dataset create`
+creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
+
+The hosted engine executes sample, full and refresh runs. A refresh uses each source's declared
+update behavior: bounded source windows merge with retained history; snapshot sources are
+revalidated or acquired again. The table's transformations and checks run over the resulting
+source relations. URL windows may use declared query parameters or path placeholders; a fixed
+date URL does not advance automatically. Configure a correction lookback where the publisher
+can revise earlier observations.
+
+A source exposing only its current snapshot cannot supply a historical delta. It remains a
+supported source, with snapshot comparison and recomputation rather than a claim that only new
+rows were fetched. Recorded streams use their registered capture and continuation semantics.
+
+See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
+`mr-data promote TABLE_ID` requests publication; claim a table is live and current only when
+Studio returns that evidence.
 
 ## There is no local execution
 
@@ -89,7 +101,9 @@ Use `mr-data --help` for the full command list and options. Commands that suppor
 
 - Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
   or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
-- Hosted credential-free acquisition supports one single-address public HTTPS object. Bounded collections have no hosted route, and the route they used to have was the deleted local one.
+- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
+  collections have no route at all; the retired local lane was their only route.
+- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
 - A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.
 
 ## Verification
@@ -104,7 +118,7 @@ before review or merge.
 ## Development
 
 ```bash
-uv sync --locked --extra dev
+uv sync --locked --extra dev --extra local-search-neural
 ```
 
 The test suite drives the execution engine in-process, and requires Linux for the worker-boundary

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.15.2"
+version = "0.16.0"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"
@@ -84,6 +84,9 @@ mr-data-generic-api = "mostlyright.data_harness.thin.workers:generic_api_probe_m
 mr-data-session-worker = "mostlyright.data_harness.thin.workers:session_worker_main"
 mr-data-session-worker-job = "mostlyright.data_harness.thin.workers:session_worker_job_main"
 mr-data-ingestion-job = "mostlyright.data_harness.thin.workers:ingestion_job_main"
+# The shared, persistent executor for bounded incremental refreshes.  Unlike the
+# segment-delivery Job above, it claims a run directly and has no task-specific argv.
+mr-data-ingestion-worker = "mostlyright.data_harness.thin.workers:ingestion_worker_main"
 # The hosted stream venue's recorder. It runs from the same worker runtime the producer and
 # the ingestion Job run from -- the install stage puts every console script in the image and
 # that stage deletes none of them -- so this entry is the whole of what a new job shape needs.

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/skills/mr-data-build/SKILL.md

@@ -115,7 +115,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --description-file F` writes the title and the description under the version it was read at, retrying once if somebody else wrote first; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
-| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh` (a refresh re-acquires the declared addresses over the previous version and claims no window); `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported, and each bounds ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
+| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh`. A refresh narrows each source that declares a supported address window and merges its returned relation into retained source state; unwindowed sources are conditionally revalidated or re-acquired whole, then the full declared transform runs. `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported, and each bounds ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
 | `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed and where. Exits non-zero on a failed run. |
 | `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed. `--status` and `--mode` narrow it and are refused for a word outside the seven or the four; `--limit` says how many, and pages are followed to reach it. |
 | `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. |
@@ -745,10 +745,10 @@ which is the other reason the search belongs there.
 protocol, on `--sample`, `max_rows` and `window` truncate: the run succeeds at the clamp boundary
 and reports the coverage it actually delivered. `max_source_bytes` refuses in every mode, sample
 included, and on `full`, `refresh` and `backfill`, hitting any clamp is a refusal with
-`CLAMP_EXCEEDED`. The current worker admits `sample`, `full` and `refresh` (unwindowed in effect) and refuses `backfill` before applying
-those dispositions. Therefore an executable sample may truncate on `max_rows`; an executable full
-refuses a crossed row or byte ceiling; every stated window, refresh, and backfill is refused before
-a byte is fetched.
+`CLAMP_EXCEEDED`. The current worker admits `sample`, `full` and `refresh` and refuses `backfill` before applying
+those dispositions. A declared supported source window narrows that source's refresh request; it
+does not make every source or connector incremental. Therefore an executable sample may truncate
+on `max_rows`, and an executable full or refresh refuses a crossed row or byte ceiling.
 
 **A succeeded run carries `coverage`, and `truncated` is required on every entry.** That is how a
 complete acquired response is told from one cut at fifty thousand rows without opening the
@@ -760,11 +760,14 @@ the record — `mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full --c
 held answer prints back as `confirm_command`. Any authenticated surface of the paying workspace may
 settle it. `mr-data run --cancel RUN_ID` stops a run that is queued or running.
 
-Do not treat the accepted command vocabulary as execution evidence. A `refresh` restores the
-previous version, re-acquires the declared addresses in full and seals the next version, claiming
-no coverage window -- it is how an enabled table's catch-up and its scheduled refresh run. A
-`backfill` is defined by the exact window it covers, which this worker cannot narrow a source
-request to, and is refused before acquisition.
+Do not treat the accepted command vocabulary as execution evidence. Author a source window only
+when the publisher has a truthful bounded date endpoint: use query names by default or exact
+`{parameter}` markers in its path, retain a correction lookback, and keep historical sources
+revalidatable unless an explicit closure contract says they cannot change. A refresh merges each
+windowed source slice into retained source state and recomputes the complete transform; unrelated
+snapshot sources are conditionally revalidated or re-acquired as needed. Cursor, pagination and
+new-file discovery need their own supported source contract and must not be represented as a date
+window. A `backfill` remains refused before acquisition.
 
 ### 6. Interrogate — ask the run what it actually delivered
 
@@ -1276,12 +1279,12 @@ the source again. Do not lock a schedule on your own judgement — it is the sam
 promotion, and it belongs to the user.
 
 This call is a publication request, not proof of data continuation. The current Harness worker
-cannot execute the catch-up backfill, refresh mode, or windowed acquisition needed to close the
-idle interval after the last successful run. It also does not attach or execute a warehouse
-schedule. Another deployed Studio or warehouse component may do that work, but only its returned
-state and durable run evidence support a claim that the table is live, caught up, or refreshing.
-If that evidence is absent, report that publication was requested and that ongoing freshness is
-unavailable; do not imply that promotion filled the gap.
+can execute a supported refresh, including declared source windows, but it does not make a frozen
+date, a generic cursor or a file listing advance by inference. It also does not attach or execute a
+warehouse schedule. Another deployed Studio or warehouse component may do that work, but only its
+returned state and durable run evidence support a claim that the table is live, caught up, or
+refreshing. If that evidence is absent, report that publication was requested and that ongoing
+freshness is unavailable; do not imply that promotion filled the gap.
 
 `mr-data pin TABLE_ID --version VERSION_ID` freezes the pointer on an exact version for a rollback
 or a hold, `mr-data unpin TABLE_ID` resumes tracking, and `mr-data demote TABLE_ID` detaches the

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/src/mostlyright/data_harness/acquisition/http.py

@@ -3834,3 +3834,22 @@ def _require_snapshot_fd_bytes(
             "SNAPSHOT_COLLISION",
             "existing content-addressed snapshot bytes differ",
         )
+    # A write can land in an earlier chunk after that chunk was hashed.  Metadata alone cannot
+    # close that interval: some filesystems expose timestamps too coarsely for a same-size write.
+    # Re-read positionally and compare against the caller-owned expected bytes, preserving the
+    # first pass's bounded memory use and never advancing this descriptor's stream position.
+    offset = 0
+    while offset < expected_length:
+        try:
+            repeated = os.pread(file_descriptor, min(65_536, expected_length - offset), offset)
+        except OSError as error:
+            raise AcquisitionSecurityError(
+                "SNAPSHOT_COLLISION",
+                "existing content-addressed snapshot could not be re-read",
+            ) from error
+        if not repeated or repeated != expected[offset : offset + len(repeated)]:
+            raise AcquisitionSecurityError(
+                "SNAPSHOT_COLLISION",
+                "existing content-addressed snapshot bytes changed during validation",
+            )
+        offset += len(repeated)

{mostlyright_data-0.15.2 → mostlyright_data-0.16.0}/src/mostlyright/data_harness/acquisition/parsing.py

@@ -28,6 +28,7 @@ from mostlyright.data_harness.formats import (
     FORMAT_MEDIA_TYPES,
     FORMAT_SUFFIXES,
 )
+from mostlyright.data_harness.source_partition import PARTITION_KEYS, source_partition_day
 
 # Read from the single closed table in ``mostlyright.data_harness.formats``; this module
 # keeps the private names it has always used so its call sites are unchanged.
@@ -594,7 +595,7 @@ def parse_streaming_csv_window(
         or not 1 <= max_output_bytes <= MAX_FILE_BACKED_WINDOW_BYTES
     ):
         raise AcquisitionSecurityError("WINDOW_STREAM_LIMIT", "window byte ceiling is invalid")
-    if partition_key not in {"iso_date_prefix", "iso_date_value"}:
+    if partition_key not in PARTITION_KEYS:
         raise AcquisitionSecurityError(
             "WINDOW_STREAM_PARTITION", "partition key encoding is invalid"
         )
@@ -858,9 +859,12 @@ def _window_instant(value: object) -> datetime.datetime:
 
 
 def _partition_key_and_date(value: str, *, encoding: str) -> tuple[str, datetime.date]:
-    key = value[:10] if encoding == "iso_date_prefix" else value
-    if _ISO_DATE.fullmatch(key) is None:
-        raise AcquisitionSecurityError("WINDOW_STREAM_PARTITION", "partition key is not canonical")
+    try:
+        key = source_partition_day(value, encoding)
+    except ValueError:
+        raise AcquisitionSecurityError(
+            "WINDOW_STREAM_PARTITION", "partition key is not canonical"
+        ) from None
     try:
         parsed_date = datetime.date.fromisoformat(key)
     except ValueError: