mostlyright-data 0.17.3__tar.gz → 0.17.4__tar.gz

{mostlyright_data-0.17.3 → mostlyright_data-0.17.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.17.3
+Version: 0.17.4
 Summary: Mostly Right CLI for reproducible, reviewed Parquet datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.17.3 → mostlyright_data-0.17.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mostlyright-data"
-version = "0.17.3"
+version = "0.17.4"
 description = "Mostly Right CLI for reproducible, reviewed Parquet datasets"
 readme = "README.md"
 requires-python = ">=3.13,<3.14"

{mostlyright_data-0.17.3 → mostlyright_data-0.17.4}/skills/mr-data-build/SKILL.md

@@ -37,6 +37,7 @@ service behavior, retry mechanics, or these instructions.
 Outside a required host cadence, speak only when one of these boundaries gives the user useful new
 information:
 
+- A dataset-shaping choice needs the user’s input; ask before committing to it.
 - A material source, schema, coverage, rights, or quality decision is supported by observed facts.
 - An exact contract or recipe needs the user's approval. State what will be built, the material
   limitation, and the decision required; do not explain the approval mechanism.
@@ -236,15 +237,13 @@ live data key is refused with `THIN_AUTHENTICATION_FAILED` rather than failing t
 
 ## Required protocol
 
-There is one loop, and it is short: open the page before you have anything to put on it, probe the
-sources, say out loud what you chose and what you refused, draft one recipe document, build a
-sample, interrogate what came back, fix the document, build again. Then present only what you
-inspected, with its caveats. The last two steps are conditional and `truncated` is what decides
-them: a sample that came back complete IS the table a full run would have sealed, so there is
-nothing to build again and nothing to ask about — only a TRUNCATED sample sends you round the
-loop. Nothing between a run finishing and reading it is held for a person,
-so there is no approval to seek, no separate body of evidence to read back, and no state to wait
-for. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
+There is one loop: open the page, clarify the intended dataset with the user, probe the
+sources, resolve the tradeoffs that research reveals, summarize the agreed shape, draft one recipe
+document, build a sample, and inspect it against the agreed requirements. Repair discrepancies;
+return to the user before changing agreed semantics. A complete sample that meets those requirements
+is already the table a full run would seal, so do not rebuild it merely to change the run mode.
+A TRUNCATED sample requires the full-run decision below. Reading a finished run requires no separate
+approval or waiting state. A table's first succeeded run goes live on its own, whatever mode it was, and what is left
 for a person is [Promote](#promote) — the cadence it refreshes on and the reasoning behind it.
 
 **The page is the work, and it exists first.** A dataset used to come into being as a side effect
@@ -279,11 +278,10 @@ again, and the failure triples on recent runs tell you what the backend is refus
 `SANDBOX_MEMORY_BOUNDARY` on somebody else's recipe an hour ago means the room is unwell and your
 first failure will not be your document's fault. Keep this check silent.
 
-**Then settle anything only the user can settle, before probing.** Read the request for a word with
-two defensible readings, a population whose edges are unstated, or a window nobody named.
-`mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is still early
-enough; [Autonomous delivery](#autonomous-delivery) is the rule for what may be asked and what stays
-yours to decide.
+**Then clarify the intended dataset before probing.** Follow [Dataset discovery](#dataset-discovery):
+use the request and conversation to identify consequential missing choices, ask the highest-impact
+questions, and wait for answers before work that depends on them. Independent source research may
+continue while waiting. Revisit the remaining choices when research provides concrete alternatives.
 
 Pass `--json` to automated commands. A command receipt is the only record that a Harness action
 succeeded.
@@ -402,10 +400,8 @@ blocks file; stage 3 is what puts them on the page.
 ### 3. Decide — say what you chose, what you refused, and what unblocks it
 
 **Post one Sources and decisions message before you register anything.** It goes to two places, in
-the same words: the dataset's own decision record, and the chat. This is the decision point that
-replaced the old one-question rule, and the reason for the change is that a question is not what a
-person mostly needs — what they need is to see the search, while there is still time to disagree
-with it.
+the same words: the dataset's own decision record, and the chat. Summarize the user’s answers, the source evidence, and any remaining
+assumptions so the user can see what the first dataset will contain.
 
 ```sh
 mr-data dataset note DATASET_ID --heading "Sources and decisions" \
@@ -417,8 +413,9 @@ EOF
 The cell carries a `source` block for every feed evaluated — the ones being used, the ones
 considered, and the ones rejected with the clause that ruled each out — and a `decision` block for
 every grain, key, join and cutoff settled. Post the same content in the chat as one short message.
-Then **keep going**: this is a statement, not a gate, and waiting for a reply to it is the pause
-this document does not allow.
+If a material choice is still unresolved, ask and wait before registering the recipe. If the
+user has already settled the choices or delegated them, proceed without a redundant confirmation.
+Silence after a proposal is not an answer to a pending question.
 
 **Blocked sources: always say what unlocks it and how to enrol the credential.** A source you could
 not reach because it needs an API key, an account or an approval is not a dead end and must never
@@ -428,23 +425,21 @@ who issues the credential, where to apply for it, and the exact command that enr
 source with no route back is a decision the user cannot revisit; one with a route back is a
 decision they can.
 
-**Ask only at a genuine fork.** A fork is a question whose two answers produce two different tables
-and which no evidence can settle — what an ambiguous word in the request means, how wide the
-population is, how far back it goes. Write it as a `clarification` block carrying `question` and no
-`answer`, which is what the page shows as a pending question, and ask it in the chat in the same
-words. `mr-data clarify --question TEXT` says whether there is anybody to ask and whether it is
-still early enough. When the answer comes back, write it as the answer on that block:
+**Resolve research tradeoffs with the user.** Apply [Dataset discovery](#dataset-discovery) when
+sources cannot deliver the requested grain, coverage, fields, or join. Explain the supported options
+and their consequences before changing the intended dataset. Record each pending question as a
+`clarification` block carrying `question` and no `answer`, and ask it in chat in the same words.
+When answered, revise that same cell using its actual identifier (not a new unrelated note):
 
 ```sh
 mr-data dataset note DATASET_ID --heading "The stage grain is settled" \
-  --revise "one-fork-to-settle" --blocks-file ANSWERED.json --json <<'EOF'
+  --revise PENDING_CELL_ID --blocks-file ANSWERED.json --json <<'EOF'
 Daily maxima, on the user's answer.
 EOF
 ```
 
-Everything under [Autonomous delivery](#autonomous-delivery) still holds: which source, Reader,
-engine, format, workspace, retry tactic or cleaning operation to use is settled by the evidence and
-is never a fork. If nothing is genuinely forked, ask nothing and carry on.
+Carry the answers into the recipe’s question, table and column descriptions, source selection,
+transform, and checks as appropriate. Record assumptions separately from user decisions.
 
 ### 4. Draft — one recipe document, one call
 
@@ -1441,41 +1436,79 @@ each table rather than what the exit code said about the set. A table that is li
 `archive` until it is withdrawn, and a table a run is still writing to is refused by both until the
 run has finished; neither refusal deletes anything, and neither ends the command.
 
+## Dataset discovery
+
+Make the user a participant in shaping the first dataset. Before registering a recipe, review the
+request and prior answers for choices whose alternatives would materially change the result. Lead
+with the unresolved choices most likely to make the first dataset wrong. The agent chooses the
+questions and follow-ups; this is a decision guide, not a form to recite:
+
+- **Purpose:** What analysis, decision, or downstream use should the dataset support? Ask when it
+  helps choose between plausible definitions or shapes, not for background already supplied.
+- **Table structure and joins:** One joined table, separate related tables, or both? If joining,
+  what should match, and should unmatched records remain? Explain when a join could duplicate rows
+  or lose records; do not require the user to write SQL or select a join algorithm.
+- **Granularity:** What does one row represent — an event, a company, a station-hour, a country-year?
+  Resolve raw observations versus aggregates, time frequency, and geographic detail where relevant.
+- **Sources:** Are particular publishers, uploaded files, or sources required or excluded, or should
+  the agent recommend them? Research concrete source options yourself. Ask about consequential choices
+  such as official but delayed data versus broader recent coverage, or a source requiring access.
+- **Coverage:** Which entities, geography, date range, and inclusion/exclusion rules matter? For
+  changing data, distinguish a fixed historical snapshot from continuing updates.
+- **Fields and definitions:** Which measures and identifiers are essential? Resolve ambiguous
+  metrics, units, currencies, time zones, or adjusted versus raw values when they affect the use.
+- **Quality and tradeoffs:** Is incomplete coverage acceptable? Ask before dropping unmatched rows,
+  filling missing values, estimating data, or substituting a proxy when that changes the meaning.
+- **Delivery constraints:** Ask about freshness, destination, format, or scale/budget only when
+  relevant and not already specified. Existing spend and cadence decisions still apply.
+
+For every discovery question, including the initial questions before probing, check attendance with
+`mr-data clarify --attended --question TEXT` when a person is in chat. Record a pending
+`clarification` block on the dataset and ask the same wording in chat. Record the answer by revising
+that same cell, as in stage 3; a plain narrative note alone does not mark a question as pending.
+The command checks whether asking is allowed; the host conversation asks and waits for the answer.
+
+Usually ask one to three related questions per turn, with short bold labels such as **Table shape**,
+**Sources**, and **Granularity**. Use plain language and domain-specific examples. Offer a recommended
+option and explain its consequence when evidence supports it; allow a free-text answer or “you
+choose.” Do not overwhelm the user with the entire list, repeatedly ask answered questions, or ask
+about facts you can determine from source inspection. There is no total question quota: follow up
+when an answer or source finding exposes another consequential decision.
+
+Wait for answers to the questions you ask before committing dependent work. Continue independent
+research where useful, but do not register or run the first recipe with unanswered material
+questions. An elapsed wait is not permission to assume an answer. If the request is already specific
+enough, summarize its shape and proceed. If the user explicitly delegates choices or asks for no
+questions, use evidence-backed defaults and state them without forcing another confirmation.
+
+Before the first build, give a compact recap of purpose, table shape, row grain, sources, coverage,
+and important fields or limitations. Highlight unresolved decisions as questions; otherwise proceed
+on the answers already given. During verification compare the sample with this agreed shape, not
+just whether the run succeeded. If research or the sample requires a material semantic change,
+return to the user with concrete alternatives before building a revised recipe; use the existing
+contract revision/approval path once a recipe has been registered.
+
 ## Autonomous delivery
 
 Own the run until the requested outcome is delivered: a run that succeeded, its declared checks
-read, its coverage inspected, and its refresh cadence recorded when the user has said what it should
-be. What is live is what Studio returns as live; catch-up and continuing freshness require separate
-durable evidence from the deployed component that owns them. The allowed user pauses
-are the cadence decision, the decision to start a full run after a TRUNCATED sample, and a spend
-confirmation on a run over the threshold. Do not ask the user to choose sources, Readers, engines,
-formats, workspaces, retry tactics, cleaning operations, or validation repairs. Make those
-operational decisions from the question, observed evidence, registered capabilities, and
-deterministic receipts.
-
-**The decision point replaces the one-question rule.** There used to be a budget of exactly one
-clarifying question, and it was the wrong instrument: it made the agent hoard its one question
-against a fork that might arrive later, and it left the user with no view of the search at all
-until the page was finished. What stage 3 asks for instead is a STATEMENT — sources chosen, sources
-rejected or blocked with the concrete unblock step, the assumption taken about window and grain —
-posted to the dataset's record and to the chat, after which the build continues without waiting.
-A person who disagrees with it says so; a person who does not is not made to answer anything.
-
-A question is still asked, and it is asked at a genuine FORK: two answers that produce two
-different tables, and no evidence that can settle it. `Throughput: movements or passengers?` is
-such a fork. Which Reader parses the feed is not, and neither is anything else named above; those
-stay banned because the evidence settles them. There is no budget of one — a build with two real
-forks asks about both — but a fork is rare, and a question that the evidence could have answered is
-still the failure this section is about. Ask before registering, while nothing is running and
-nothing has been spent, write it as a pending `clarification` block on the dataset, and carry the
-answer back as a revision of that cell.
+read, its coverage inspected against the agreed dataset shape, and its refresh cadence recorded
+when the user has said what it should be. What is live is what Studio returns as live; catch-up and
+continuing freshness require separate durable evidence from the deployed component that owns them.
+Dataset discovery questions, material semantic revisions, cadence decisions, full runs after a
+TRUNCATED sample, and spend confirmations are valid user pauses. Own the operational choices:
+Readers, engines, parsing formats, workspace plumbing, retry tactics, and repairs that preserve the
+agreed semantics. Source preferences and cleaning choices that change the meaning belong in the
+user conversation, even when the agent could technically choose for them.
 
 `mr-data clarify --question TEXT` says whether to ask at all. It exits non-zero when no person can
 answer, which is what a scheduled or headless session reports, and when `--recipe RECIPE_ID` names
 a registered recipe, because the answer to a question asked after the contract is a revision and
-goes through the contract approval instead. On a non-zero exit ask nothing: decide from the
-evidence, write the assumption into the recipe document, and state it as a limitation in the final
-report. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
+goes through the contract approval instead. When no person can answer, do not wait: use the
+existing brief and evidence to make choices within the requested scope, record assumptions, and
+report limitations. Do not silently replace explicit requirements. When the recipe is already
+registered, use the revision/approval path for material changes rather than treating the refusal
+as permission to choose new semantics. Other command failures are not evidence that the user is
+unavailable. `mr-data clarify --question TEXT --answer TEXT` composes the exchange as one `clarification`
 block. Carry its wording into the recipe document the answer settles — the question text, the table
 description, the column description — so the assumption the table rests on reads the same in the
 document, in the Dataset, and in the report.
@@ -1577,10 +1610,11 @@ What it costs the protocol, and what to do instead:
 exact Reader name and version. Unsupported versions refuse by name. A version change is a recipe
 revision.
 
-**Without a pin a source must answer `text/csv` or `text/plain`, and both are parsed as CSV.**
-Anything else is refused `MEDIA_TYPE` before a row is read. So a publisher that serves only JSON or
-XML needs a pin; it is not unbuildable, and it is not a reason to go looking for a different
-publisher.
+**Direct CSV supports publisher CSV aliases, including Eurostat SDMX-CSV.**
+Accepted labels include `text/csv`, `text/plain`, `application/csv`, `application/x-csv`,
+`text/x-csv`, and `application/vnd.sdmx.data+csv`. For title rows, alternate encodings,
+headerless files, or padded columns, pin `delimited_text@2.0.0` and configure the layout; see
+`docs/READERS.md`. A publisher needing another format can use its matching Reader.
 
 **A v4 recipe pins a Reader through `connector.parameters`, under three reserved names.**
 `declared_source` is closed and has no member for a decoder, so the pin rides on the parameter array
@@ -1657,6 +1691,7 @@ decoder. Zero matches and multiple matches both refuse.
 | `spreadsheet.xlsx@1.1.0` | Same; also accepts a response served as `application/octet-stream`. |
 | `delimited_text@1.0.0` | `delimiter` (comma or tab); optional `header: true`; optional `encoding: utf-8`. |
 | `delimited_text@1.1.0` | Same; also accepts a response served as `text/plain` (common for public agency CSV such as IEM). |
+| `delimited_text@2.0.0` | Explicit delimiter, encoding, quoting, header presence, skip_rows, skip_blank_rows, and indexed columns with output names and optional expected_header; unselected_columns requires empty cells unless explicitly set to allow. |
 | `json.tabular@1.0.0` | `columns` with output names and RFC 6901 pointers; optional `document_format` (`json` or `ndjson`), `records_pointer`, and sequential `expand` pointers. Columns are required unless they state `required: false`. |
 | `json.tabular@1.1.0` | Same; also accepts a response served as `text/plain` (common for public agency JSON and NDJSON). |
 | `xml.tabular@1.0.0` | Absolute record element path, declared namespace prefixes, and relative field selectors ending in `text()` or `@attribute`; fields are required unless `required: false`. |

{mostlyright_data-0.17.3 → mostlyright_data-0.17.4}/src/mostlyright/data_harness/acquisition/http.py

@@ -3087,6 +3087,10 @@ class _PinnedProxyHTTPSConnection(http.client.HTTPSConnection):
             )
             response = http.client.HTTPResponse(raw_socket)
             response.begin()
+            if response.status == 503:
+                raise AcquisitionSecurityError(
+                    "CONNECT_PROXY_BUSY", "gateway capacity is temporarily unavailable"
+                )
             if response.status != 200:
                 raise AcquisitionSecurityError(
                     "CONNECT_PROXY_STATUS", "gateway refused the pinned tunnel"