mostlyright-data 0.16.2__tar.gz → 0.17.0__tar.gz

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.16.2
+Version: 0.17.0
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.16.2"
+version = "0.17.0"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/skills/mr-data-build/SKILL.md

@@ -112,7 +112,7 @@ gap rather than doing anything, and `export-hosted-candidate`, which is a backen
 | `mr-data whoami` | Use the compatibility alias for the remotely validated `auth status` answer. |
 | `mr-data clarify` | Say whether one clarifying question may be put to the user right now, and compose the record of their answer. It reaches nothing, asks nobody and waits for nothing. Exits non-zero when no person can answer or when a recipe is already registered. |
 | `mr-data probe` | Ask one registered source one question and print the answer. **Currently unusable:** it reaches `/v3/sessions`, which Studio refuses, and it takes a question identifier that no v4 receipt returns. Do not plan a source inspection around it. |
-| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --description-file F` writes the title and the description under the version it was read at, retrying once if somebody else wrote first; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
+| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`; `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the categories, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
 | `mr-data recipe` | Register one recipe document — dataset, question, table plan, sources, transform, checks and units — in one call, and print the identifiers the server derived. `mr-data recipe show ID` reads one back. |
 | `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
 | `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID --digest RECIPE_DIGEST` — both required, neither positional, and the digest is the bare hex the registration receipt printed. The command accepts `--sample`, `--full`, `--refresh`, `--backfill` and `--window`, and the worker executes `--sample`, `--full` and `--refresh`. A refresh narrows each source that declares a supported address window and merges its returned relation into retained source state; unwindowed sources are conditionally revalidated or re-acquired whole, then the full declared transform runs. `--backfill` and a `--window` on a sample or full are refused before acquisition. `--max-rows` and `--max-source-bytes` remain supported, and each bounds ONE SOURCE rather than the finished table. A large run is held for a spend confirmation, which `--confirm` settles; a sample-first held full requires Studio's interactive human approval instead. `--cancel RUN` stops one. |
@@ -307,7 +307,8 @@ and a title that never changes is fine too. Where the address cannot be minted,
 plain dashboard address rather than blocking.
 
 Hold the `dataset_id` for the rest of the session: stage 3 writes the decision record to it, stage
-4 writes its title, description and cover, and the recipe binds to it by naming it as `dataset.id`.
+4 writes its title, topics, licence, description and cover, and the recipe binds to it by naming
+it as `dataset.id`.
 
 ### 2. Probe — read a source before committing to it
 
@@ -470,18 +471,63 @@ and is settled once, at stage 4.
 Every grain, key, join and cutoff settled after stage 3 is a further `decision` block on the
 DATASET's record — `mr-data dataset note DATASET_ID` — because the fact outlives every run.
 
-**Write the title and the description before the first run, one call each.** The page has been
-open since stage 1 under a working title; this is where it settles:
+**Write the title, the topics, the licence and the description before the first run.** The page has
+been open since stage 1 under a working title. This is where it settles. One call carries the title,
+the topics and the licence, so the title and the chips land together; the description arrives after
+it, through the draft snapshots and the final commit. (The worked example below is a different
+dataset from the Kathmandu one the earlier stages follow, chosen because its numbers are easy to
+count.)
 
 ```sh
-mr-data dataset set DATASET_ID --name "Kathmandu monsoon flooding, 2015-2026" --json
-mr-data dataset set DATASET_ID --description-file DESCRIPTION.md --json
+mr-data dataset set DATASET_ID --name "Hourly weather at Prague Airport (LKPR), 2000-2026" \
+  --topics "weather,prague,czech republic,metar,iowa environmental mesonet" \
+  --license "CC-BY-4.0" --json
 ```
 
-Two calls rather than one, so the person watching sees the title land and then the text arrive.
-Each call reads the version first and writes under it, so a description you wrote against a page
-somebody else has since edited is retried once and then reported rather than silently overwriting
-them. While composing, send bounded snapshots with `mr-data dataset draft DATASET_ID --description-file DRAFT.md --expected-version VERSION --json`. Chain the returned version into the next snapshot, writing a meaningful paragraph at a time. These snapshots are temporary and do not replace the saved description. Commit the complete documentation with `mr-data dataset set DATASET_ID --description-file DESCRIPTION.md --expected-version VERSION --json`. If another edit moved that version, stop the draft and reconcile it with the user’s text; never retry by blindly advancing the version. Do not send fabricated partial text or delay completed text to simulate writing.
+The call reads the version first and writes under it, so a page somebody else has since edited is
+retried once and then reported rather than silently overwritten. The licence is the SPDX identifier
+of the rights basis the sources actually grant -- for example `CC-BY-4.0`, `ODbL-1.0`, `CC0-1.0`;
+where the sources' terms are not an open licence, leave `--license` off and say so under "Source and
+rights" instead. Never invent a licence.
+
+While composing, send bounded snapshots with `mr-data dataset draft DATASET_ID --description-file DRAFT.md --expected-version VERSION --json`. Chain the returned version into the next snapshot, writing a meaningful paragraph at a time. These snapshots are temporary and do not replace the saved description. Commit the complete documentation with `mr-data dataset set DATASET_ID --description-file DESCRIPTION.md --expected-version VERSION --json`. If another edit moved that version, stop the draft and reconcile it with the user’s text; never retry by blindly advancing the version. Do not send fabricated partial text or delay completed text to simulate writing.
+
+**What search engines and answer engines read.** These are the three fields search engines and
+answer engines index. A search engine (Google, Bing) ranks the page by the words a person types.
+An answer engine (ChatGPT, Perplexity, Claude, Google AI Overviews) quotes the page as a source
+when a person asks a question. Making the page rank is called SEO. Making the page quotable is
+called GEO. Both read the same three fields, and each field lands in fixed places with fixed
+limits. The rules below are mechanical: apply them, then run the check at the end. Do not
+interpret them.
+
+| You write | Where it appears | Limit |
+| --- | --- | --- |
+| `--name` | browser tab title (`{name} \| Mostly Right`), the page heading, the JSON-LD `name` | at most 60 characters, or search results cut it |
+| the opening paragraph of DESCRIPTION.md | the meta description, the social card, the JSON-LD `description`, the `Summary:` line answer engines fetch | search engines cut at 300 characters; answer engines cut at 240 |
+| `--topics` | category chips and the category filter on the /datasets directory, JSON-LD `keywords`, the `Topics:` line answer engines fetch, the `topic` filter in the public API and MCP | 3 to 8 topics, each at most 40 characters |
+
+**Title rules**
+1. Shape: subject, place, period. `Hourly weather at Prague Airport (LKPR), 2000-2026`.
+2. At most 60 characters. Count them.
+3. Use the words a person would type: the common place name, and the official identifier in
+   parentheses once if one exists (ICAO code, station id, ticker, constituency name).
+4. Never: a brand, the word "dataset" or "data", a colon, a pipe, an exclamation mark, a version
+   number, a run date.
+5. Sentence case. No trailing full stop.
+
+**Topic rules**
+1. 3 to 8 topics, comma-separated, no duplicates.
+2. Topic one is the category, chosen from this list and no other:
+   `weather, climate, air quality, hydrology, energy, transport, aviation, public health,
+   economics, finance, elections, sports, prediction markets, demographics, geospatial,
+   agriculture`.
+3. Then, in this order, one each where it applies: the city or region, the country, the
+   phenomenon or instrument, the publisher or programme. After those, up to three more, each a
+   distinct thing a person would type that none of the earlier topics already names.
+4. Form: lowercase; ASCII letters, digits, spaces and hyphens only; one to three words; at most
+   40 characters.
+5. Never: a column name, a unit, a bare year, a second word from the category list, a market or
+   venue name unless the dataset is about that market.
 
 **The dataset description is the documentation, and it is read by people and by search engines.**
 `dataset.description` is Markdown, rendered on the dataset page with headings and lists, and it
@@ -490,7 +536,7 @@ seconds -- not a paragraph, and not a report. Use exactly this shape, in this or
 section out only when there is genuinely nothing to say (shown indented here; the document
 itself carries the headings at column one):
 
-    One or two plain sentences: what each row is, where it comes from, and the period it covers.
+    Two plain sentences: what each row is, where it comes from, and the period it covers.
 
     ## Coverage
     - **Window:** the exact dates and the time zone.
@@ -510,13 +556,47 @@ itself carries the headings at column one):
     ## Source and rights
     The publisher, the programme, and the rights basis, in one or two sentences.
 
+That is six sections: the opening paragraph, and the five headings under it.
+
+**Opening paragraph rules** (everything before the first `##` heading)
+1. Two sentences, 150 to 240 characters together. The first sentence ends before character 200
+   and is complete on its own.
+2. Sentence one says what each row is, for where, over what window, from which publisher:
+   `Each row is one hourly METAR observation at Prague Václav Havel Airport (LKPR), January 2000
+   to August 2026, from the Iowa Environmental Mesonet ASOS archive.`
+3. Sentence two gives the size and the measured quantities, as numbers and nouns:
+   `43,812 rows of temperature, dew point, wind, pressure and visibility.`
+4. Full names on first use with the abbreviation in parentheses once. Never "this dataset",
+   "we", "it", "here".
+5. No Markdown in the opening paragraph: no bold, no links, no bullets, no code.
+
+**Body rules** (the five sections under the headings)
+1. Every sentence must stay true and complete when quoted alone, away from the page. Never
+   "as above", "see below", "it", "they", "the same".
+2. Numbers, not adjectives: `one row per hour, about 720 rows a month`, not `high-frequency`.
+3. Keep the five headings exactly as given, in that relative order, omitting only a section with
+   genuinely nothing to say. Answer engines match headings to questions: "what does it cover",
+   "what is it suitable for".
+4. Say what is absent as plainly as what is present: gaps, sentinels, the years the source did
+   not record.
+5. Everything in the paragraph below still applies.
+
 Rules that keep it readable: short sentences; no sentence over about twenty-five words; bullets
 rather than comma lists; no tables, no code blocks, no links to internal tools, no headings beyond
 the ones above. Spell out the names a reader would search for -- the place, the identifiers in
 every common form, the programme and the publisher -- once each, in the opening sentences and
-never as a keyword list. The opening sentences double as the summary cards and search engines
-show, so they must stand alone. A one-line description is a defect to fix in the revision, not a
-style choice. Give `table.description` the grain and the window.
+never as a keyword list. A one-line description is a defect to fix in the revision, not a style
+choice. Give `table.description` the grain and the window.
+
+**Check before writing.** Fix and re-check until every line passes:
+- title: at most 60 characters, contains the place, contains no colon, pipe or exclamation mark,
+  no brand, no "dataset" or "data", no version number, no run date
+- opening paragraph: 150 to 240 characters, first full stop before character 200, no Markdown
+- topics: 3 to 8, no duplicates, first one is on the category list, all lowercase, each at most
+  40 characters, none is a column name, a unit, a year, a second word from the category list, or
+  a market or venue name unless the dataset is about that market
+- licence: an SPDX identifier the sources actually grant, or `--license` left off
+- every `##` heading in DESCRIPTION.md is one of the five, in that relative order, none repeated
 
 **Every dataset gets a cover before its first run is presented.** Once the description is
 written, generate one with

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/src/mostlyright/data_harness/acquisition/http.py

@@ -17,6 +17,7 @@ import threading
 import time
 from collections.abc import Callable, Iterator, Mapping
 from dataclasses import dataclass, replace
+from datetime import UTC, datetime
 from pathlib import Path, PurePosixPath
 from typing import TYPE_CHECKING, Protocol
 from urllib.parse import unquote_to_bytes, urljoin, urlsplit
@@ -184,6 +185,71 @@ _CONTENT_RANGE = re.compile(rf"bytes ({_DECIMAL})-({_DECIMAL})/({_DECIMAL}|\*)")
 # written out again, so one edit moves both and the two headers cannot drift apart.
 _DECIMAL_ONLY = re.compile(_DECIMAL)
 
+# The statuses on which ``Retry-After`` says anything a fetch may act on. RFC 9110 defines the
+# field for 503 and for a 3xx; RFC 6585 gives 429 the same reading. A redirect is followed here
+# rather than waited out, so the two that remain are these. Read on any other status the field
+# would be either absent or about something this path does not do, and a 500 that carries one
+# must not be able to talk a caller into a wait it never reviewed.
+RETRY_AFTER_STATUSES: frozenset[int] = frozenset({429, 503})
+# Long enough for either spelling a compliant origin may send -- an IMF-fixdate is 29 characters
+# and a bounded delta-seconds is at most 16 -- and short enough that a value past it is refused
+# as a value rather than parsed. A longer field is not truncated and re-read: it is dropped, and
+# the caller falls back to the wait it reviewed for itself.
+MAX_RETRY_AFTER_CHARS = 64
+# ``delta-seconds`` is ``1*DIGIT`` (RFC 9110 5.6.7), which is NOT ``_DECIMAL`` above: that one
+# refuses a leading zero, because the value it reads is sealed or compared and one span spelled
+# two ways would be two spellings in the evidence. A wait is neither, so ``Retry-After: 07`` is a
+# perfectly ordinary seven seconds and refusing it would only lose the origin's answer.
+#
+# The digit bound is defence in depth rather than the thing that keeps this bounded:
+# ``MAX_RETRY_AFTER_CHARS`` already caps the string long before the grammar sees it, and a
+# 17-digit wait would be refused by the caller's ceiling anyway. It is here so the bound does not
+# depend on either of those staying true.
+_RETRY_AFTER_DELTA = re.compile(r"[0-9]{1,16}")
+# The one date spelling this parser admits: the same RFC 9110 IMF-fixdate FIELD that
+# ``_IMF_FIXDATE`` below holds ``Last-Modified`` to, written separately because the two do
+# different work with it. That one is a shape check on a value echoed back to the publisher
+# unchanged, so it is stricter per character and deliberately admits a 31st of February; this
+# one is turned into an instant a run will wait for, so it captures its fields and hands the
+# calendar the job of refusing a day that does not exist.
+#
+# RFC 9110 5.6.7 requires a sender to use IMF-fixdate and permits a recipient to see two
+# obsolete forms; those two are deliberately NOT parsed. The 850-style form carries a two-digit
+# year, and a windowing rule for it is a guess about which century an origin meant -- a guess
+# this module would then spend a run's seconds on. An unreadable value costs nothing, because
+# every caller of ``parse_retry_after_seconds`` treats ``None`` as "the origin said nothing" and
+# waits its own reviewed backoff instead.
+#
+# ``[0-9]`` rather than ``\d`` throughout, because ``\d`` also matches the digits of every other
+# script Unicode has, and ``float`` reads those too: an origin could otherwise name its wait in
+# Arabic-Indic digits and have it honoured. The parser table in ``tests/h3/test_http.py`` sends
+# three alphabets' worth, which is what makes this a property rather than a preference.
+#
+# The month names are written once, in ``_RETRY_AFTER_MONTHS``, and the alternation is built from
+# it. The tuple is also what turns a captured name into a month number, and the two are read one
+# after the other on the same value: a name the pattern admitted but the tuple did not hold would
+# raise inside the ``try`` below and read as "the origin said nothing", which is a silent way to
+# lose a spelling. One source, so they cannot disagree.
+_RETRY_AFTER_MONTHS = (
+    "Jan",
+    "Feb",
+    "Mar",
+    "Apr",
+    "May",
+    "Jun",
+    "Jul",
+    "Aug",
+    "Sep",
+    "Oct",
+    "Nov",
+    "Dec",
+)
+_RETRY_AFTER_DATE = re.compile(
+    r"(?:Mon|Tue|Wed|Thu|Fri|Sat|Sun), "
+    rf"([0-9]{{2}}) ({'|'.join(_RETRY_AFTER_MONTHS)}) ([0-9]{{4}}) "
+    r"([0-9]{2}):([0-9]{2}):([0-9]{2}) GMT"
+)
+
 
 @dataclass(frozen=True)
 class DatagovRateObservation:
@@ -1219,7 +1285,21 @@ class PeerAttemptError(AcquisitionSecurityError):
         # everywhere else, so a caller that reads it fails closed rather than retrying a
         # refusal whose origin never answered.
         http_status: int | None = None,
+        # The origin's own ``Retry-After``, exactly as it wrote it, and only from a status that
+        # gives the field meaning (``RETRY_AFTER_STATUSES``). It rides here for the reason
+        # ``http_status`` does: a caller deciding whether and when to ask again is answering a
+        # question about this response, and the typed code cannot carry the answer. Absent
+        # everywhere else, and never interpreted here -- ``parse_retry_after_seconds`` reads it,
+        # and the caller that will spend the wait bounds it.
+        retry_after: str | None = None,
     ) -> None:
+        if retry_after is not None and (
+            type(retry_after) is not str or len(retry_after) > MAX_RETRY_AFTER_CHARS
+        ):
+            raise AcquisitionSecurityError(
+                "TRANSPORT_EVIDENCE",
+                "peer-attempt Retry-After evidence is invalid",
+            )
         if any(not isinstance(attempt, PeerAttempt) for attempt in peer_attempts):
             raise AcquisitionSecurityError(
                 "TRANSPORT_EVIDENCE",
@@ -1233,6 +1313,7 @@ class PeerAttemptError(AcquisitionSecurityError):
         self.peer_attempts = peer_attempts
         self.hops = hops
         self.http_status = http_status
+        self.retry_after = retry_after
         self.total_response_body_size_bytes = sum(
             attempt.response_body_size_bytes for attempt in peer_attempts
         )
@@ -1772,7 +1853,8 @@ class PinnedHttpsRetriever:
 
         Redirect bodies are discarded after their digest is measured.  A peer retry starts a
         fresh request only after the failed attempt records its consumed bytes; the caller must
-        therefore use a fresh staging file per acquisition and discard it on any exception.
+        therefore use a fresh staging file per acquisition, or truncate it through ``reset_sink``
+        before re-attempting, and discard it on any exception it does not re-attempt.
         """
         if not callable(sink):
             raise AcquisitionSecurityError("STREAM_SINK", "stream sink must be callable")
@@ -1944,6 +2026,7 @@ class PinnedHttpsRetriever:
                     tuple(attempts),
                     hops=tuple(hops),
                     http_status=response.status,
+                    retry_after=_retry_after_header(response),
                 )
             media_type = _media_type(response.header("content-type"))
             if media_type not in self._limits.allowed_media_types:
@@ -2652,6 +2735,7 @@ class PinnedHttpsRetriever:
                     peer_attempts,
                     hops,
                     http_status=response.status,
+                    retry_after=_retry_after_header(response),
                 )
             content_type = self._response_header(response, "content-type", peer_attempts, hops)
             try:
@@ -2724,6 +2808,7 @@ class PinnedHttpsRetriever:
         hops: list[RetrievalHop],
         *,
         http_status: int | None = None,
+        retry_after: str | None = None,
     ) -> None:
         latest = peer_attempts[-1]
         peer_attempts[-1] = PeerAttempt(
@@ -2733,7 +2818,12 @@ class PinnedHttpsRetriever:
             response_body_size_bytes=latest.response_body_size_bytes,
         )
         raise PeerAttemptError(
-            code, detail, tuple(peer_attempts), hops=tuple(hops), http_status=http_status
+            code,
+            detail,
+            tuple(peer_attempts),
+            hops=tuple(hops),
+            http_status=http_status,
+            retry_after=retry_after,
         )
 
     def _response_header(
@@ -3026,6 +3116,76 @@ def _remaining_socket_timeout(deadline: float, *, configured: float) -> float:
     return min(configured, remaining)
 
 
+def parse_retry_after_seconds(value: str | None, *, now: float) -> float | None:
+    """Read one ``Retry-After`` field as a wait in seconds from ``now``, or decline to read it.
+
+    Both spellings RFC 9110 defines are admitted: delta-seconds, and an absolute IMF-fixdate,
+    which is subtracted from ``now`` (a POSIX timestamp) to give the same kind of answer. A date
+    already in the past is not a negative wait -- it is the origin saying "now", and it returns
+    ``0.0``, which a caller floors at its own backoff.
+
+    ``None`` means only that this field was not readable as a wait: absent, empty, one of the two
+    obsolete date spellings RFC 9110 lets a recipient see, a calendar date that does not exist, or
+    anything else. It is never a verdict about the origin, and never a longer or shorter wait than
+    the caller would have taken on its own.
+
+    The ``MAX_RETRY_AFTER_CHARS`` test below can never be the reason: the longest string either
+    grammar admits is an IMF-fixdate's 29 characters, so a value past the cap is already refused
+    as a value. It is kept because this function is public and its bound should not depend on a
+    grammar someone widens later.
+
+    NOTHING HERE BOUNDS THE ANSWER: a readable field is reported as the origin wrote it, and how
+    long a run may actually wait is the caller's ceiling to apply, because this function has no
+    idea what budget it is spending.
+    """
+
+    if value is None:
+        return None
+    candidate = value.strip()
+    if not candidate or len(candidate) > MAX_RETRY_AFTER_CHARS:
+        return None
+    if _RETRY_AFTER_DELTA.fullmatch(candidate):
+        return float(candidate)
+    match = _RETRY_AFTER_DATE.fullmatch(candidate)
+    if match is None:
+        return None
+    try:
+        moment = datetime(
+            int(match.group(3)),
+            _RETRY_AFTER_MONTHS.index(match.group(2)) + 1,
+            int(match.group(1)),
+            int(match.group(4)),
+            int(match.group(5)),
+            int(match.group(6)),
+            tzinfo=UTC,
+        )
+    except ValueError:
+        # A day the month does not have, or a leap second: a spelling this parser cannot turn
+        # into an instant is one it declines, exactly as an unparseable one.
+        return None
+    return max(0.0, moment.timestamp() - now)
+
+
+def _retry_after_header(response: TransportResponse | StreamedTransportResponse) -> str | None:
+    """The response's ``Retry-After`` field when the status is one that gives it meaning.
+
+    Reading it must never change what a refusal IS. A hostile origin that repeats the field
+    would otherwise turn a plain 429 into a ``RESPONSE_HEADER_DUPLICATE``, so a refusal to read
+    the field is answered here with ``None`` -- the caller then waits its own reviewed backoff,
+    which is exactly what it does for an origin that sent no field at all.
+    """
+
+    if response.status not in RETRY_AFTER_STATUSES:
+        return None
+    try:
+        value = response.header("retry-after")
+    except AcquisitionSecurityError:
+        return None
+    if value is None or len(value) > MAX_RETRY_AFTER_CHARS:
+        return None
+    return value
+
+
 def _datagov_rate_observation(response: TransportResponse) -> DatagovRateObservation | None:
     """Parse only the bounded numeric Data.gov rate headers; retain no arbitrary headers."""
 

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/src/mostlyright/data_harness/acquisition/parsing.py

@@ -47,6 +47,7 @@ _FORBIDDEN_REFERENCE_KEYS = frozenset(
 _SAFE_COLUMN = re.compile(r"^[^\x00-\x1f\x7f]{1,256}$")
 _ISO_DATE = re.compile(r"\d{4}-\d{2}-\d{2}\Z")
 _ISO_TIMESTAMP = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?Z\Z")
+_ISO_SPACE_MINUTE_TIMESTAMP = re.compile(r"\d{4}-\d{2}-\d{2} \d{2}:\d{2}\Z")
 _MAX_REUSABLE_CSV_ROW_BYTES = 256 * 1024
 _STREAMING_ROW_HASH_BATCH_ROWS = 1_024
 # One input character can become a six-byte JSON escape; the per-row estimate also includes the
@@ -1084,17 +1085,25 @@ def _partition_key_and_date(value: str, *, encoding: str) -> tuple[str, datetime
         raise AcquisitionSecurityError(
             "WINDOW_STREAM_PARTITION", "partition value is not a canonical date"
         )
-    if encoding == "iso_date_prefix" and value != key and _ISO_TIMESTAMP.fullmatch(value) is None:
-        raise AcquisitionSecurityError(
-            "WINDOW_STREAM_PARTITION", "partition value is not canonical"
-        )
     if encoding == "iso_date_prefix" and value != key:
-        try:
-            datetime.datetime.fromisoformat(value)
-        except ValueError:
+        if _ISO_TIMESTAMP.fullmatch(value) is not None:
+            try:
+                datetime.datetime.fromisoformat(value)
+            except ValueError:
+                raise AcquisitionSecurityError(
+                    "WINDOW_STREAM_PARTITION", "partition timestamp is not real"
+                ) from None
+        elif _ISO_SPACE_MINUTE_TIMESTAMP.fullmatch(value) is not None:
+            try:
+                datetime.datetime.strptime(value, "%Y-%m-%d %H:%M")
+            except ValueError:
+                raise AcquisitionSecurityError(
+                    "WINDOW_STREAM_PARTITION", "partition timestamp is not real"
+                ) from None
+        else:
             raise AcquisitionSecurityError(
-                "WINDOW_STREAM_PARTITION", "partition timestamp is not real"
-            ) from None
+                "WINDOW_STREAM_PARTITION", "partition value is not canonical"
+            )
     return (key, parsed_date)
 
 

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/src/mostlyright/data_harness/acquisition/sandbox.py

@@ -769,6 +769,41 @@ class CrawlerSandbox:
         request would have been given. Retrieval therefore spawns cold.
         """
 
+        admitted = self._admitted_warm_pool(size)
+        if admitted is None:
+            return 0
+        pool, launch, admitted_size = admitted
+        return pool.fill(launch, size=admitted_size)
+
+    def top_up_warm_pool(self, *, limit: int) -> int:
+        """Add at most ONE idle child, never taking the pool past ``limit``. Answers what is ready.
+
+        The refill a long-lived owner wants between requests, and the reason it is not
+        ``start_warm_pool`` with a smaller number: a child handed out between the owner's count
+        and its call turns "one more" into "the whole pool", spawned under one hold of the pool's
+        lock, and every request that arrives during that hold waits for all of it. This decides
+        the increment inside the lock, so a call costs one spawn or none.
+
+        The boundary rules are ``start_warm_pool``'s, unchanged and applied here too: an invalid
+        ceiling is refused, a coordinator holding a bounded cgroup root gets zero and a ceiling of
+        zero closes the pool. Believe the count rather than the ceiling, for the same reason.
+        """
+
+        admitted = self._admitted_warm_pool(limit)
+        if admitted is None:
+            return 0
+        pool, launch, admitted_limit = admitted
+        return pool.add_one(launch, limit=admitted_limit)
+
+    def _admitted_warm_pool(self, size: int) -> tuple[_WarmSandboxPool, _WorkerLaunch, int] | None:
+        """Check the requested size against the boundary and open the pool it admits.
+
+        One place, because ``start_warm_pool`` and ``top_up_warm_pool`` must not be able to
+        disagree about what the boundary permits: the size validation, the bounded-cgroup clamp
+        and the zero-closes-the-pool rule are the pool's admission decision, not either caller's.
+        ``None`` means the boundary admits no pool at all and the caller answers zero.
+        """
+
         if type(size) is not int or not 0 <= size <= MAX_WARM_POOL_SIZE:
             raise AcquisitionSecurityError(
                 "SANDBOX_WARM_POOL",
@@ -778,8 +813,7 @@ class CrawlerSandbox:
             size = min(size, BOUNDED_CGROUP_WARM_POOL_SIZE)
         if size == 0:
             self.close_warm_pool()
-            return 0
-        launch = self._worker_launch(prewarm=True)
+            return None
         if self._warm_pool is None:
             self._warm_pool = _WarmSandboxPool(
                 timeout_seconds=self._timeout_seconds,
@@ -788,7 +822,7 @@ class CrawlerSandbox:
                 child_address_space_bytes=self._child_address_space_bytes,
                 networkless_child=self._child_is_networkless(),
             )
-        return self._warm_pool.fill(launch, size=size)
+        return self._warm_pool, self._worker_launch(prewarm=True), size
 
     def warm_pool_ready_count(self) -> int:
         """How many pre-spawned children are idle and unbound right now."""
@@ -6713,23 +6747,49 @@ class _WarmSandboxPool:
 
         with self._lock:
             self._prune(launch)
-            deadline = time.monotonic() + WARM_POOL_FILL_TIMEOUT_SECONDS
-            while len(self._ready) < size:
-                self._ready.append(
-                    _prepare_worker(
-                        launch,
-                        timeout_seconds=self._timeout_seconds,
-                        memory_cgroup_root_fd=self._memory_cgroup_root_fd,
-                        hosted_session_container_memory_bytes=(
-                            self._hosted_session_container_memory_bytes
-                        ),
-                        child_address_space_bytes=self._child_address_space_bytes,
-                        networkless_child=self._networkless_child,
-                        track_deadline=False,
-                        ready_timeout_seconds=max(deadline - time.monotonic(), 0.0),
-                    )
+            return self._fill_locked(launch, size=size)
+
+    def add_one(self, launch: _WorkerLaunch, *, limit: int) -> int:
+        """Add at most one child, never taking the pool past ``limit``. Answers what is ready.
+
+        AN INCREMENT, NOT A TARGET, AND THE DIFFERENCE IS A LOCK HOLD. A caller that reads
+        ``ready_count`` and then asks ``fill`` for "one more than that" is making two calls with
+        a gap between them: a take landing in the gap empties the pool, and the fill then spawns
+        the whole difference under a single hold of this lock -- which is the wait every taker
+        queued behind it inherits, and the opposite of what topping up by one is for. Here the
+        count and the spawn are one critical section, so a pass adds one child or none whatever
+        else is running.
+
+        ``limit`` is a ceiling on the pool, not a number to reach. A pool already at or above it
+        spawns nothing and reports what it holds.
+        """
+
+        with self._lock:
+            self._prune(launch)
+            if len(self._ready) >= limit:
+                return len(self._ready)
+            return self._fill_locked(launch, size=len(self._ready) + 1)
+
+    def _fill_locked(self, launch: _WorkerLaunch, *, size: int) -> int:
+        """Spawn until the pool holds ``size``. Callers hold ``_lock`` and have pruned."""
+
+        deadline = time.monotonic() + WARM_POOL_FILL_TIMEOUT_SECONDS
+        while len(self._ready) < size:
+            self._ready.append(
+                _prepare_worker(
+                    launch,
+                    timeout_seconds=self._timeout_seconds,
+                    memory_cgroup_root_fd=self._memory_cgroup_root_fd,
+                    hosted_session_container_memory_bytes=(
+                        self._hosted_session_container_memory_bytes
+                    ),
+                    child_address_space_bytes=self._child_address_space_bytes,
+                    networkless_child=self._networkless_child,
+                    track_deadline=False,
+                    ready_timeout_seconds=max(deadline - time.monotonic(), 0.0),
                 )
-            return len(self._ready)
+            )
+        return len(self._ready)
 
     def take(self, launch: _WorkerLaunch) -> _PreparedWorker | None:
         """Hand out one live child started from exactly ``launch``, or nothing.

{mostlyright_data-0.16.2 → mostlyright_data-0.17.0}/src/mostlyright/data_harness/bounded_query.py

@@ -118,6 +118,21 @@ MAX_QUERY_ROWS = 1_000_000
 #: reference table selects one or two; an unpruned read of a billion-row table selects three
 #: hundred and eighteen. The ceiling is where the list stops being a query and starts being an
 #: export, and the refusal names ``prune`` as the fix rather than raising the ceiling.
+#:
+#: ⚠ THIS IS THE REPOSITORY'S ONLY COPY OF THE NUMBER, AND IT IS THE ENFORCED ONE.
+#: :func:`bind_parts` refuses over it and ``hosted_session_worker`` returns it as a claim's
+#: ceiling, so a second declaration elsewhere could drift from the wall that actually refuses and
+#: nothing would fail. ``hosted_run_worker`` imports THIS one to decide the severity of an
+#: appending version's part-count report, and ``test_stream_refresh_append`` pins it to Studio's
+#: own ``v4/table_parts.MAX_QUERY_PARTS`` under ``STUDIO_ROOT``.
+#:
+#: ⚠ AND IT IS ALSO STUDIO'S QUERY WALL, WHICH IS THE ONE PRODUCTION MEETS. A read Studio cannot
+#: prune below this many parts is refused ``QUERY_TOO_MANY_PARTS`` -- and an UNPRUNED read prunes
+#: nothing at all (``v4/table_parts.select_parts`` returns every part when the query states no
+#: ``prune`` block), so a plain "show me this table" meets this number and not the format's own
+#: 4,096. It is eight times lower than
+#: :data:`~mostlyright.data_harness.table_manifest.MAX_PARTS_PER_VERSION`, which is why a writer
+#: that watched only the format's ceiling would watch the wall nobody hits first.
 MAX_QUERY_PARTS = 512
 
 #: How many rows are taken from the engine at a time. The running byte budget is checked BETWEEN