mostlyright-data 0.31.5__tar.gz → 0.31.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/PKG-INFO +3 -2
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/README.md +2 -1
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/SKILL.md +29 -14
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/5-draft-one-recipe-document-one-call.md +7 -6
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/autonomous-delivery.md +3 -2
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/commands.md +4 -4
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/enriching-with-small-models.md +5 -1
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/live-run.md +2 -1
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/making-a-table-incremental.md +9 -8
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/patterns.md +3 -2
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/preparing-table-search.md +7 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/sources.md +33 -26
- mostlyright_data-0.31.7/src/mostlyright/data_harness/thin/delivery.py +261 -0
- mostlyright_data-0.31.7/src/mostlyright/data_harness/thin/description_shape.py +120 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/fast_features.py +122 -65
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe_lint.py +14 -29
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_datasets.py +6 -6
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_runs.py +429 -66
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_stream.py +23 -3
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/vocabulary.py +10 -6
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/render.py +25 -1
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/.gitignore +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/pyproject.toml +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/scripts/hatch_build.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/agents/openai.yaml +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/1-open-the-page-and-the-link-to-it-in-the-first-message.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/2-brief-two-to-four-questions-each-with-a-recommended-answer.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/3-probe-read-a-source-before-committing-to-it.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/4-decide-say-what-you-chose-what-you-refused-and-ask-one-question.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/6-build-one-run-sized-to-acquire-every-measured-source-whole.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/7-interrogate-ask-the-run-what-it-actually-delivered.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/8-fix-revise-the-document-and-register-it-again.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/9-present-only-what-survived-inspection-with-caveats.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/agent-protocol.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/before-the-first-tool-call.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/boundaries.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/cloud-authentication-preflight.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/cross-repository-protocol-reference.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/installation-parity.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/narrating-the-run.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/not-hosted-yet.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/one-install.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/prediction-labels.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/promote.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/readers.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/receipts.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/recipe-test.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/recording-a-stream-venue.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/recovering-an-import-failure.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/reference-pages.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/required-protocol.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/source-credentials.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/the-one-thing-to-say-about-the-skill-itself.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/transforms.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/user-communication-contract.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/writing-a-decision-record.md +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/scripts/write_research_notebook.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/__init__.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/agent_protocol.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/canonical.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/formats.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/hosted_crawler_protocol.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/key_seam.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/page_coverage.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/part_check_evidence.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/session_probes.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/skill_assets.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/table_manifest.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/__init__.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/acquire.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/acquire_cancel.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/activity.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/approvals.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/backoff.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/categories.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/commands.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/dataset-categories-v1.json +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/download.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/enrichment.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/matching_benchmark.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/narrative.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/parity.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/probe.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/progress_rate.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/progress_vocabulary.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/propose.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe_brief.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe_explain.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe_forecast.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/recipe_test.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/refresh_rehearsal.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/release_check.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/research.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/router.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/run_diagnose.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/run_prefix.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/runs.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/session.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/stream.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/stream_venue.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/studio_capabilities.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/support.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/table_search_contract.json +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/transport.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/user_agent.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_artifacts.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_catalog.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_connected.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_connections.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_dataset_covers.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_handoff.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_matching.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_narrative.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_public.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_query.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_reader.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_search.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_search_contract.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_secrets.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/thin/v4_tables.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/__init__.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/attendance.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/clarification.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/cloud_auth.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/commands/__init__.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/commands/auth.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/commands/clarify.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/commands/login.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/commands/whoami.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/credential_native.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/credential_store.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/credentials.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/login.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/path_kind.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/plain_file.py +0 -0
- {mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/src/mostlyright/data_harness/ux/remediation.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: mostlyright-data
|
|
3
|
-
Version: 0.31.
|
|
3
|
+
Version: 0.31.7
|
|
4
4
|
Summary: Mostly Right hosted CLI for reviewed datasets
|
|
5
5
|
Project-URL: Homepage, https://mostlyright.md/
|
|
6
6
|
Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
|
|
@@ -103,7 +103,8 @@ explicit standalone bounded inspection; row ceilings do not impose a five-minute
|
|
|
103
103
|
Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
|
|
104
104
|
including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
|
|
105
105
|
with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
|
|
106
|
-
|
|
106
|
+
resumes a failed or cancelled progressive full from the pages it saved, and can reissue a
|
|
107
|
+
supported platform execution failure of any other run without changing the recipe.
|
|
107
108
|
|
|
108
109
|
Use the identifiers returned by registration and run submission. `mr-data dataset create`
|
|
109
110
|
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
|
|
@@ -88,7 +88,8 @@ explicit standalone bounded inspection; row ceilings do not impose a five-minute
|
|
|
88
88
|
Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
|
|
89
89
|
including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
|
|
90
90
|
with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
|
|
91
|
-
|
|
91
|
+
resumes a failed or cancelled progressive full from the pages it saved, and can reissue a
|
|
92
|
+
supported platform execution failure of any other run without changing the recipe.
|
|
92
93
|
|
|
93
94
|
Use the identifiers returned by registration and run submission. `mr-data dataset create`
|
|
94
95
|
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
|
|
@@ -27,17 +27,21 @@ On a table that is already live, list the missed entries and ask the user first.
|
|
|
27
27
|
the recipe digest, so the next build is a full resync. A paid source is called again for every
|
|
28
28
|
per-row request whose saved answer no longer applies.
|
|
29
29
|
|
|
30
|
-
**Gate 2.
|
|
31
|
-
|
|
32
|
-
No documented limit:
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
documents
|
|
40
|
-
|
|
30
|
+
**Gate 2. Omit an undocumented pace, and leave a low concurrency out.** A page collection that
|
|
31
|
+
states no pace starts at 1000 ms and adapts down to 250 ms. A stated gap is a hard floor that
|
|
32
|
+
never adapts. No documented limit: omit pages.min_interval_milliseconds. State a number only when the
|
|
33
|
+
publisher documents one. Registration refuses a gap under 4 ms, which is 256 requests a second
|
|
34
|
+
to one host from a collection. That is a platform bound, never a target. Where the deployment
|
|
35
|
+
spreads requests over its egress addresses, an omitted pace is held to 4 requests a second per
|
|
36
|
+
address and to max(4, M) a second host-wide when spread over M addresses. A stated gap is kept
|
|
37
|
+
per address, so the publisher can see a multiple of that rate. No command prints how many
|
|
38
|
+
addresses there are. When the publisher's limit is per key or per account, state the gap it
|
|
39
|
+
documents. Leave `pages.concurrency` out, or state it only at 64 or above. A stated width
|
|
40
|
+
strictly under 64 is a ceiling on the width the worker sizes, and `run --full` refuses it
|
|
41
|
+
unless `--accept-slow` says why. `concurrency: 1` stays that miss. Keep a lower ceiling only
|
|
42
|
+
when a run's refusal told you to lower it or the publisher documents a limit on concurrent
|
|
43
|
+
connections, and record the reason in a dataset note so the next session finds it. Then choose
|
|
44
|
+
the source shape that fits.
|
|
41
45
|
|
|
42
46
|
| Source shape | Use | Always state | Recipe |
|
|
43
47
|
| --- | --- | --- | --- |
|
|
@@ -149,7 +153,7 @@ step 6. The stage 3 reference holds its rule. Gate 4 is step 6.
|
|
|
149
153
|
has a date-like column, `batch` for a per-row GET whose API takes a list of ids, and
|
|
150
154
|
`reference` for a bulk dump beside a window
|
|
151
155
|
([fast builds](references/making-a-table-incremental.md#fast-builds-tablepartition-batch-and-reference)).
|
|
152
|
-
|
|
156
|
+
Omit an undocumented gap, drop a stated `pages.concurrency` under 64, page a keyed API in parallel and pin the
|
|
153
157
|
ranged Parquet Reader where the entries `pace`, `concurrency`, `parallel_pagination` and
|
|
154
158
|
`ranged_parquet` say so and the source meets each condition
|
|
155
159
|
([fetching fast](references/making-a-table-incremental.md#fetching-fast-pace-width-parallel-pages-and-ranged-parquet)).
|
|
@@ -237,7 +241,9 @@ These failures look like recipe faults and are not:
|
|
|
237
241
|
| `EXECUTION_CAPACITY_CLAIM_FAILED` during a platform release | Start the run again after the release. Keep the recipe. |
|
|
238
242
|
| "Schedule held" on the dashboard | Read what holds it with `mr-data table show TABLE_ID`. A recipe with no refreshing plan and no `rebuild` stays held. [Promotion and cadence](references/promote.md) has the repair. |
|
|
239
243
|
|
|
240
|
-
Legacy `room_fault: true` can identify a supported `mr-data run --retry RUN_ID` case.
|
|
244
|
+
Legacy `room_fault: true` can identify a supported `mr-data run --retry RUN_ID` case. A failed
|
|
245
|
+
or cancelled progressive full is resumed by the same command from the pages it saved, unless the
|
|
246
|
+
failure says the recipe must change.
|
|
241
247
|
**`room_fault: false` does not prove the recipe is wrong.** Repeating a terminal coordinate may
|
|
242
248
|
only return its stored failure. Retry at most three times when supported by evidence, then report
|
|
243
249
|
what remains unresolved. Never narrow coverage, drop checks, guess encodings or switch sources
|
|
@@ -264,6 +270,14 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
|
|
|
264
270
|
Then run `mr-data models suggest RUN_ID --json`. It reads the run's preview and names where
|
|
265
271
|
the table could use a deterministic derivation or a model call, and whether this deployment
|
|
266
272
|
serves that model. Confirm each lead against the rows before you offer it.
|
|
273
|
+
A succeeded full run's `mr-data status` and `mr-data watch` output carries a `delivery` block.
|
|
274
|
+
It names a missing cover, a description that is missing or out of the skill's shape,
|
|
275
|
+
natural-language search when the run's table has none, and always the enrichment offer.
|
|
276
|
+
Complete every step before you report the build as done. Studio marks no main table, so you
|
|
277
|
+
judge it: skip the search step for a supporting table and say so in one line. A scratch table
|
|
278
|
+
you test and demote owes none of these steps.
|
|
279
|
+
Offering enrichment is mandatory. Accepting it is the person's choice, and a no is a complete
|
|
280
|
+
answer.
|
|
267
281
|
9. Any truncated source means a preview, not the requested complete dataset. Inspect coverage
|
|
268
282
|
even when the run succeeded. Untruncated acquisition does not prove publisher pagination, nested
|
|
269
283
|
records or population coverage is complete. A null coverage window does not mean all history was acquired.
|
|
@@ -279,7 +293,8 @@ merely to obtain a green run. Read [recovery details](references/agent-protocol.
|
|
|
279
293
|
repair, which is why the decision belongs in step 5. A table left not refresh-ready never
|
|
280
294
|
refreshes unless it declares `rebuild`; it does not become ready later.
|
|
281
295
|
Download and verify artifacts when the user requested bytes.
|
|
282
|
-
10.
|
|
296
|
+
10. Search is on by default for the dataset's main table. Supporting tables such as lookups and
|
|
297
|
+
code lists do not need it. Enable it, inspect search readiness and test meaningful queries against
|
|
283
298
|
the observed version as [search preparation](references/preparing-table-search.md) describes.
|
|
284
299
|
Search preparation runs independently: pending or failed indexing never means the data build
|
|
285
300
|
failed, and declared metadata alone never proves observed search quality. Before delivery,
|
|
@@ -33,12 +33,11 @@ the identifier stage 1 printed. With it, the registration attaches to the page s
|
|
|
33
33
|
watching; without it there is no page whose record could carry the brief, and the command refuses
|
|
34
34
|
-- `THIN_RECIPE_DATASET_UNBOUND`, below, whether or not `--no-lint` was passed.
|
|
35
35
|
|
|
36
|
-
**A document that carries `dataset.id`
|
|
37
|
-
empty page's description and never replaces
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
Leave `dataset.name` in: the title is the recipe's own subject, settled below.
|
|
36
|
+
**A document that carries `dataset.id` may also carry `dataset.description`.** Registration FILLS
|
|
37
|
+
an empty page's description and never replaces a stored one. The page's own sentence is still
|
|
38
|
+
written, below, in the `mr-data dataset set` call that carries the title and the tags with it,
|
|
39
|
+
under the version it read. The preflight does not refuse the pair. Leave `dataset.name` in: the
|
|
40
|
+
title is the recipe's own subject, settled below.
|
|
42
41
|
|
|
43
42
|
**Registration refuses a dataset nobody was asked about**, with `THIN_BRIEF_MISSING`. The record
|
|
44
43
|
has to carry either an answered `clarification` block or a delegation `decision` block first. Where
|
|
@@ -301,6 +300,8 @@ uses must follow from the actual fields and coverage, without promising an outco
|
|
|
301
300
|
no bold field labels, em dashes,
|
|
302
301
|
semicolons, fake quotations or unsupported claims
|
|
303
302
|
- every `##` heading in DESCRIPTION.md is one of the five, in that relative order, none repeated
|
|
303
|
+
- Coverage and Columns are present. A succeeded full run's `delivery` block checks the opening,
|
|
304
|
+
the headings and the 4000-character bound, and names the description until it passes
|
|
304
305
|
|
|
305
306
|
**Every dataset gets a cover before its first run is presented.** Once the description is written,
|
|
306
307
|
generate one with `mr-data cover DATASET_ID "what the image should depict" --json`, describing the
|
|
@@ -27,8 +27,9 @@ state what is live and what is not, rather than claiming a promotion or reportin
|
|
|
27
27
|
coordinates.
|
|
28
28
|
|
|
29
29
|
A typed refusal is recovery evidence, not a terminal outcome while a safe alternative remains.
|
|
30
|
-
Record it and continue with the first supported action that fits the evidence: retry a room fault
|
|
31
|
-
with `mr-data run --retry RUN_ID`, select another
|
|
30
|
+
Record it and continue with the first supported action that fits the evidence: retry a room fault,
|
|
31
|
+
or resume a failed or cancelled progressive full, with `mr-data run --retry RUN_ID`, select another
|
|
32
|
+
supported source, Reader, format or capacity
|
|
32
33
|
class, or revise the recipe document. Retry an unchanged transient failure at most three times and
|
|
33
34
|
never repeat a permanent refusal without changing the condition that caused it. Re-run source fitness and feasibility classification whenever the selected
|
|
34
35
|
alternative could change their result. Preserve the requested
|
{mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/commands.md
RENAMED
|
@@ -23,12 +23,12 @@ run in full.
|
|
|
23
23
|
| `mr-data connected` | `connected list` shows the public datasets connected to this workspace and `connected add SLUG` connects one by its slug. It needs `mr-data login`: both ask Cloud, with the device key, over the routes MCP's `list_connected_datasets` and `connect_dataset` use. |
|
|
24
24
|
| `mr-data ask` | `ask "QUESTION" [--run RUN_ID ...] [--failure-code CODE]` asks the Mostly Right support agent one question and waits for the answer; `--no-wait` prints the question id at once and `ask status QUESTION_ID` reads it later. It needs `mr-data login` and a workspace the agent is switched on for: while it is off the answer is `THIN_CLOUD_SUPPORT_AGENT_DISABLED`, which is final. The agent may file a problem for the Mostly Right team and prints its reference (`SR-N`). |
|
|
25
25
|
| `mr-data dataset` | Bring the dataset page into existence before there is anything on it, then fill it in while somebody watches. `dataset create --name TEXT` mints it and prints the `dataset_id`, unless the user's own request already has an empty dataset open, in which case it answers `open_request_reused` with that `dataset_id` and mints nothing (`--separate` mints a new one for a different dataset); `dataset show ID` reads it back; `dataset set ID --name TEXT --topics "a,b,c" --license ID --description-file F` writes the title, the descriptive tags, the SPDX licence and the description under the version it was read at, retrying once if somebody else wrote first, and an empty `--topics` or `--license` takes that value off the page; send `--name` and `--topics` whenever you send `--description-file`, because a description sent on its own can make Studio write a new title and new tags, and read `changed` and `written_by_studio` on the receipt for every field the write moved; a saved write whose public sync fails exits 2 and reports `public_projection_synced: false` — use `dataset sync ID` to retry that sync without rewriting Studio; `dataset note ID --heading H --blocks-file B` writes one cell of the decision record that OUTLIVES every run, and `--list` reads it back; `dataset watch ID` follows the page's own event stream; `dataset activity ID --phase P --message TEXT` says what is happening right now, silently, and is never a chat message and never a cell; `dataset publish ID [--mode public|link|private]` says who can read the dataset — `public` lists it in the public directory and serves it at an address anybody can read, `link` serves it at an unlisted address, `private` takes it back to the workspace — and `dataset publish ID --show` reads that back without changing it; `dataset archive ID --confirm-name TITLE` retires the page and frees its title, deleting nothing. |
|
|
26
|
-
| `mr-data recipe` | Register one recipe document (dataset, question, table plan, sources, transform, checks and units) in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it. `--no-lint` sends it as written instead. A recipe that misses a feature that sets how fast a run fetches (
|
|
26
|
+
| `mr-data recipe` | Register one recipe document (dataset, question, table plan, sources, transform, checks and units) in one call, and print the identifiers the server derived. The document is read first and every fault comes back in one refusal, each with the JSON pointer that names it. `--no-lint` sends it as written instead. A recipe that misses a feature that sets how fast a run fetches (a stated `pages.concurrency` under 64, a per-row request open to `batch`, a page walk open to `pagination.parallel`, or a Parquet file open to a ranged read) is refused `THIN_RECIPE_SLOW`, with each feature's pointer and fix, before anything is sent. Apply the fixes, or pass `--accept-slow "REASON"` with one sentence on why the slow path is right for this source. The receipt prints the reason and the features it covered. Registration refuses `THIN_BRIEF_MISSING` when the dataset's record carries no answered question and no delegation, and `THIN_RECIPE_DATASET_UNBOUND` when the document names no `dataset.id`. `--delegated "their words"` writes the delegation onto the dataset and registers in the same command, and clears the first of those two and never the second. `mr-data recipe show ID` reads one back. `mr-data recipe explain FILE` (or a recipe identifier) lists every reason its table can or cannot refresh, each with a rule code and Studio's suggested repair, and exits 2 when one blocks it. `mr-data recipe forecast RECIPE_ID` says what one run would ask of each source before it runs, and exits 3 when the run would overrun today's platform quota or one attempt. `--digest` is optional there, and without it the digest the recipe was registered with is forecast. `mr-data recipe readiness --table TABLE_ID --current-only` reads the revision one table runs now. The explanation's `fast_features` names each fast feature the recipe uses or misses, with a one-line fix. [Making a table incremental](making-a-table-incremental.md#the-loop-explain-forecast-run-diagnose) says when to use each. |
|
|
27
27
|
| `mr-data cover` | Generate and attach one branded 1200×630 dataset cover. Give it the dataset ID and what the image should depict; Studio fixes the model, single-color style, curated random palette, dimensions and storage. |
|
|
28
|
-
| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID`, with `--digest RECIPE_DIGEST` when you want the run to assert which bytes it means. Neither is positional. The digest is the bare hex the registration receipt printed. Left out, the run reads it back from the registered recipe, which is the same value, because a recipe identifier names one revision. A full of a recipe that misses a fetch-speed feature is refused `THIN_RECIPE_SLOW`, as registration is, unless `--accept-slow "REASON"` is passed. Other modes print the missed features and are not refused. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted source-action plan, and a plan shape it does not refresh returns `RESYNC_REQUIRED` before acquisition. [Which plans refresh](making-a-table-incremental.md#which-plans-refresh) lists them, and `mr-data recipe explain` says which one a table is. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal. Report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. `--max-rows` cuts rows after a source is fetched and decoded, so it does not shorten a collection walk. A sample with no clamp is refused: `CLAMPS_REQUIRED` from Studio, and `THIN_REQUEST_INVALID` from this client first. A table's first succeeded run goes live, truncated or not, so test on a scratch table as [the probe rule](3-probe-read-a-source-before-committing-to-it.md#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table) says. A large run is held for a size confirmation (not a charge), which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full an existing legacy sample-first pair is holding once its preview has succeeded, naming either half of the pair. Where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID`
|
|
29
|
-
| `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. `--diagnose` adds everything else Studio can say about the run: its progress, a per-row source's plan counts, its checkpoints, and how each attempt ended, with a lost attempt's cause marked as an inference. |
|
|
28
|
+
| `mr-data run` | Start one run against a registered recipe, named as `--recipe RECIPE_ID`, with `--digest RECIPE_DIGEST` when you want the run to assert which bytes it means. Neither is positional. The digest is the bare hex the registration receipt printed. Left out, the run reads it back from the registered recipe, which is the same value, because a recipe identifier names one revision. A full of a recipe that misses a fetch-speed feature is refused `THIN_RECIPE_SLOW`, as registration is, unless `--accept-slow "REASON"` is passed. Other modes print the missed features and are not refused. The mode is one of six: `sample`, `full`, `refresh`, `backfill`, `compact` and `replay`. Four have a shorthand flag (`--sample`, `--full`, `--refresh`, `--backfill`) and two do not, so write `--mode MODE`, which is accepted for every one of them. A refresh executes only Studio's persisted source-action plan, and a plan shape it does not refresh returns `RESYNC_REQUIRED` before acquisition. [Which plans refresh](making-a-table-incremental.md#which-plans-refresh) lists them, and `mr-data recipe explain` says which one a table is. Use `mr-data table resync TABLE_ID --request-id UUID` for an explicit full source reread, retaining that UUID if the response is uncertain. `--backfill` states the exact window with `--window START END`. `--mode replay --sources-from RUN_ID` asks Studio to run the registered revision against the RETAINED RAW INPUTS of one named successful run of the same table: nothing is fetched from this computer, nothing is acquired again, the result is compared against that run and never becomes the live version, and it neither asks for nor records an approval. Where Studio has replay switched off, or the named run is not successful, not the same table, or no longer retains its inputs, it answers a typed refusal. Report the code rather than retrying in another mode. `--sources-from` on any other mode is refused `THIN_ARGUMENT_INVALID` before anything is sent. `--max-rows` and `--max-source-bytes` bound ONE SOURCE rather than the finished table. `--max-rows` cuts rows after a source is fetched and decoded, so it does not shorten a collection walk. A sample with no clamp is refused: `CLAMPS_REQUIRED` from Studio, and `THIN_REQUEST_INVALID` from this client first. A table's first succeeded run goes live, truncated or not, so test on a scratch table as [the probe rule](3-probe-read-a-source-before-committing-to-it.md#the-probe-rule-test-on-a-scratch-table-never-on-the-real-table) says. A large run is held for a size confirmation (not a charge), which `--confirm` settles and whose printed `confirm_command` re-states every argument the request carried. `--approve-full RUN_ID` releases the full an existing legacy sample-first pair is holding once its preview has succeeded, naming either half of the pair. Where the deployment still wants a person at a browser it answers `THIN_INTERACTIVE_HUMAN_REQUIRED` and names the run page. `--retry RUN_ID` resumes a failed or cancelled progressive full from the pages it saved (`resume_capture_run_id` and the ceilings it carried; `retry_of_run_id` as well only when Studio admits a named retry). A failure that says the recipe must change is refused before anything is posted. Any other failed run is tried again when its triple carries `room_fault: true`, re-stating that run's own coordinate and ceilings and naming the failed run so Studio starts a new attempt, and is refused with the triple and the next step Studio's recovery category names when it is not. `room_fault` is Studio's `recovery.room_fault` where Studio states one. A source that kept refusing (`SOURCE_RATE_LIMITED`, `SOURCE_HTTP_5XX`, `SOURCE_PROXY_BUSY`) is sent anyway: Studio admits it 120 seconds after the failure and refuses `RUN_RETRY_TOO_SOON` with a Retry-After before then, so wait that long and run `--retry` again. `--cancel RUN` stops one. |
|
|
29
|
+
| `mr-data status` | Report which of the seven states one run is in, what it delivered and whether a ceiling cut it short, and — on a failure — what failed, where, and whether the fault was the execution room's rather than the recipe's. Exits non-zero on a failed run. `--diagnose` adds everything else Studio can say about the run: its progress, a per-row source's plan counts, its checkpoints, and how each attempt ended, with a lost attempt's cause marked as an inference. A succeeded full run carries `delivery`: the steps still owed before the build is done (a description missing or out of the skill's shape, a missing cover, natural-language search when the dataset's main table has none, and always the enrichment offer), each with its command. It never changes the exit code. |
|
|
30
30
|
| `mr-data runs` | Report this workspace's own runs: identifier, mode, state and creation time, plus the failure triple of any that failed and whether that failure was the execution room's. `--dataset DATASET_ID` narrows it to one dataset's runs, so "what have I run for this dataset?" is one scoped read, and a value that is not an identifier is refused `THIN_ARGUMENT_INVALID`. `--status` and `--mode` narrow it too and are refused `THIN_ARGUMENT_INVALID` for a word outside the seven states or the six modes; the three combine. Studio narrows before it cuts a page, so a filtered page holds the newest runs that match however many runs the workspace has. `--limit` says how many, and pages are followed to reach it; a listing that does not end says how many were read and suggests a smaller `--limit` or a narrower filter. The answer prints `next_cursor`, and `--cursor` continues from it. |
|
|
31
|
-
| `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. `mr-data watch RUN_ID RUN_ID ...` follows up to 16 runs at once, prints one line as each ends, and returns once all have finished with one receipt per run (`failed_run_ids`, `refused_run_ids`); it exits 2 when any run failed and 1 when a run could not be watched. `--from-seq` and `--last-event-id` resume one run only. |
|
|
31
|
+
| `mr-data watch` | Stream one run's live progress under the durable event type of each stage, resuming across stream cuts. Exits non-zero when the run failed. `mr-data watch RUN_ID RUN_ID ...` follows up to 16 runs at once, prints one line as each ends, and returns once all have finished with one receipt per run (`failed_run_ids`, `refused_run_ids`); it exits 2 when any run failed and 1 when a run could not be watched. `--from-seq` and `--last-event-id` resume one run only. A full run that succeeds carries the same `delivery` block as `mr-data status`, and the multi-run receipt merges every run's steps. |
|
|
32
32
|
| `mr-data query` | Run one bounded read-only SQL statement over a finished run's own table and print the rows. Name the table `run_table` in the SQL, whatever the recipe called it. Bounded on rows, on answer bytes and on time, and refused by name when a bound is crossed. When no worker can answer, the refusal names `mr-data download RUN_ID` as `fallback_command`, which brings the table here to read locally. The question is asked through Mostly Right Cloud; while its usage paywall is on and the workspace is unsubscribed, each answered query counts toward the person's monthly allowance, and a spent allowance refuses `THIN_USAGE_LIMIT`. A refusal after the question was queued names its `query_id`; running the same command again reads that query back. |
|
|
33
33
|
| `mr-data search` | `search VERB` prepares and tests versioned search for a new table through Studio; no model or index runs on this computer. The verbs are `inspect`, `sample`, `values`, `propose`, `configure`, `validate`, `add-cases`, `explain`, `preview`, `evaluate`, `estimate`, `build`, `status`, `activate`, `cancel`, `rollback`, `locate` and `try`. `inspect`, `propose` and `validate` also take `--recipe FILE` before registration. `validate --spec` checks a spec and its evaluation cases against the pinned contract without sending anything. `locate` resolves evaluation-case rows to locators, and `try` runs one query against the live search. A build returns promptly with job coordinates, and an operation Studio does not serve answers a typed capability error. [Prepare search](preparing-table-search.md) says how it fits a build. |
|
|
34
34
|
| `mr-data peek` | Print the columns, their logical types and the first rows of one run's table. The type comes off the run's own column profile where the preview carries none, so `type` is null only for a run that sealed no profile. |
|
|
@@ -88,7 +88,11 @@ also keep the rows it settled from being sent at all: see [Needs Studio 4.14](#n
|
|
|
88
88
|
5. **After the run (step 8).** Run `mr-data models suggest RUN_ID --json`. It reads the run's
|
|
89
89
|
preview and names, column by column, a deterministic derivation or a model call that would
|
|
90
90
|
improve the table, each with `model_status` for the model it needs. It spends no call. Read
|
|
91
|
-
it as leads: confirm each against the rows before offering it.
|
|
91
|
+
it as leads: confirm each against the rows before offering it. The run's `delivery` block on
|
|
92
|
+
`mr-data status` and `mr-data watch` lists this offer as required, beside a missing
|
|
93
|
+
description, cover or main-table search. Search is its own required step, not part of this
|
|
94
|
+
offer. Make the offer before you report the build as done. Offering is
|
|
95
|
+
mandatory. Accepting is the person's choice. If nothing is worth adding, say so in one line.
|
|
92
96
|
6. **Deliver (step 10).** End the report with what would make the table better: the remaining
|
|
93
97
|
offers, deterministic ones first, each with the column it reads and what it would add. One
|
|
94
98
|
concrete sentence each, not a menu of everything possible.
|
{mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/live-run.md
RENAMED
|
@@ -30,7 +30,8 @@ appends again, and `watch` stops rather than reconnecting.
|
|
|
30
30
|
coordinate replays the stored result rather than making a new attempt — a stored failure included —
|
|
31
31
|
which is why `mr-data run --retry RUN_ID` names the failed run to Studio rather than re-sending the
|
|
32
32
|
request as it was; the ceilings stay the same unless Studio does not admit the named retry, and
|
|
33
|
-
then the byte ceiling is raised as it was before.
|
|
33
|
+
then the byte ceiling is raised as it was before. A failed or cancelled progressive full is
|
|
34
|
+
resumed from the pages it saved, and names the run only when Studio admits that name. It is also why `SANDBOX_MEMORY_BOUNDARY` reads as a source problem and is not one: it names
|
|
34
35
|
the source being fetched, the cause is a clean room with no delegated memory cgroup, it is
|
|
35
36
|
intermittent and per-instance, and `room_fault` is true on that run so nothing has to be inferred
|
|
36
37
|
from the wording. Report the failure once, retry it when supported, then report the replacement
|
|
@@ -359,14 +359,14 @@ too, so change them in the first revision.
|
|
|
359
359
|
|
|
360
360
|
| Entry | It reports | The fix |
|
|
361
361
|
| --- | --- | --- |
|
|
362
|
-
| `pace` | a
|
|
363
|
-
| `concurrency` |
|
|
362
|
+
| `pace` | a stated gap under the registration floor | registration refuses it. An omitted gap starts at 1000 ms and adapts down to 250 ms, and a stated gap is a hard floor that never adapts. Omit the member when the publisher documents no limit |
|
|
363
|
+
| `concurrency` | a stated `pages.concurrency` strictly under 64 | remove it, unless a run's refusal told you to lower it or the publisher documents a limit on concurrent connections. At 64 and above the stated width stands. Record the reason in a dataset note |
|
|
364
364
|
| `parallel_pagination` | a serial `GET` page walk over JSON on `authenticated.https.api_key@2.0.0`, on a source with no window | move to `@2.1.0` with the counted `pagination.parallel`, when the first page states its page count |
|
|
365
365
|
| `ranged_parquet` | a `parquet.tabular@1.0.0` pin on a public file with no request window, in a recipe that states no `execution_policy` | pin `parquet.tabular@1.1.0`, when the publisher answers byte ranges and the limits in [Readers](readers.md) fit |
|
|
366
366
|
|
|
367
|
-
A stated gap
|
|
368
|
-
|
|
369
|
-
is yours to check against the source.
|
|
367
|
+
A stated gap at or above the registration floor is in use, and it never adapts. An omitted gap
|
|
368
|
+
is in use too. A slow gap that is the publisher's limit is correct. The entries read the document
|
|
369
|
+
alone. They cannot know a publisher's limit, so every fix is yours to check against the source.
|
|
370
370
|
On a table that is already live, list the missed entries and ask the user first. Each fix changes
|
|
371
371
|
the recipe digest, so the next build is a full resync. A paid source is called again for every
|
|
372
372
|
per-row request whose saved answer no longer applies.
|
|
@@ -630,9 +630,10 @@ For Binance BTCUSDT one-minute klines, start the backfill at `2017-08-17T00:00:0
|
|
|
630
630
|
}
|
|
631
631
|
```
|
|
632
632
|
|
|
633
|
-
The 1000 above is
|
|
634
|
-
|
|
635
|
-
|
|
633
|
+
The 1000 above is a stated hard floor that never adapts. Omit
|
|
634
|
+
`window.request.min_interval_milliseconds` when the publisher documents no limit: an omitted gap
|
|
635
|
+
starts at 1000 ms and adapts down to 250 ms. State a number only when the publisher documents one.
|
|
636
|
+
`mr-data recipe explain` reports that window under `fast_features` as `pace`, in use.
|
|
636
637
|
|
|
637
638
|
Keep `granularity: day`, `timezone: UTC` and `merge.materialization: partition_replace`.
|
|
638
639
|
For the acquired `open_time_ms` column declare
|
{mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/patterns.md
RENAMED
|
@@ -14,8 +14,9 @@ Before you register a copy, replace every placeholder:
|
|
|
14
14
|
Reader settings, read from the source itself.
|
|
15
15
|
- `rights_claim` holds the basis you found, the digest of the terms you read, and a note.
|
|
16
16
|
- `credential.secret_name` names a key the user enrolled with `mr-data keys set`.
|
|
17
|
-
- A gap of 1000 ms
|
|
18
|
-
|
|
17
|
+
- A gap of 1000 ms in a worked recipe is a hard floor that never adapts. When the publisher
|
|
18
|
+
documents no limit, omit `pages.min_interval_milliseconds` instead of copying 1000. State a
|
|
19
|
+
number only when the publisher documents one, and say which you used and why.
|
|
19
20
|
|
|
20
21
|
Then run `mr-data recipe explain RECIPE.json --json`, and take Gate 3's bounded test on a scratch
|
|
21
22
|
table before any full run.
|
|
@@ -57,6 +57,13 @@ missing search support only when `error` begins with
|
|
|
57
57
|
`THIN_SEARCH_CAPABILITY_UNAVAILABLE` as a capability limit too. Any other refusal is not
|
|
58
58
|
capability evidence. That includes 401, 403, 429, timeouts and invalid responses. Resolve it.
|
|
59
59
|
|
|
60
|
+
Search is on by default for the dataset's main table, whether or not the brief mentions it.
|
|
61
|
+
Supporting tables such as lookups, code lists and dimension tables do not need it. A succeeded
|
|
62
|
+
full run's `delivery` block names the search step whenever the run's table has no search, a
|
|
63
|
+
search whose build failed included. Studio marks no main table, so you judge which one it is.
|
|
64
|
+
Skip the step for a supporting table, while the capability is missing, or when the table has
|
|
65
|
+
nothing searchable, and say why in one line.
|
|
66
|
+
|
|
60
67
|
When the brief does not mention search, search is optional only while the capability is missing.
|
|
61
68
|
Then omit `table.search`, keep the drafted intent beside the recipe, and record the limit in the
|
|
62
69
|
dataset's Sources and decisions note. Never put that limit in the description, title or topics.
|
{mostlyright_data-0.31.5 → mostlyright_data-0.31.7}/skills/mr-data-build/references/sources.md
RENAMED
|
@@ -58,9 +58,10 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
|
|
|
58
58
|
(up to 50 000, or up to 4 000 000 on a per-row request collection), `concurrency` (up to
|
|
59
59
|
10 000), `request_timeout_seconds`, `retries` and a `revisit` rule. `retries` is 0 to 3 and 2
|
|
60
60
|
when left out. Registration refuses 4 or more. Its pace is `min_interval_milliseconds` in whole
|
|
61
|
-
milliseconds,
|
|
62
|
-
`min_interval_seconds
|
|
63
|
-
meaning when it is left out. [Pace and width](#pace-and-width)
|
|
61
|
+
milliseconds. Left out, a collection starts at 1000 ms and adapts down to 250 ms. The legacy
|
|
62
|
+
whole-second `min_interval_seconds` is the other spelling. Only `allowed_origins` is required.
|
|
63
|
+
Every other member has a stated meaning when it is left out. [Pace and width](#pace-and-width)
|
|
64
|
+
says which to state.
|
|
64
65
|
- `limits.max_requests` is REQUIRED and bounds every request the run makes, discovery and detail
|
|
65
66
|
pages together. `limits.max_source_bytes` bounds the fetched bytes; `limits.max_rows` clamps the
|
|
66
67
|
merged relation.
|
|
@@ -71,37 +72,43 @@ Write it as a source whose connector is that coordinate plus a `collection` memb
|
|
|
71
72
|
|
|
72
73
|
**The pace sets the rate. Concurrency does not.**
|
|
73
74
|
|
|
74
|
-
- A collection that states no pace
|
|
75
|
-
about 2.8 hours.
|
|
76
|
-
-
|
|
77
|
-
|
|
78
|
-
- A stated gap is
|
|
79
|
-
|
|
80
|
-
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
75
|
+
- A collection that states no pace starts at 1000 ms and adapts down to 250 ms. A stated gap is
|
|
76
|
+
a hard floor that never adapts. At 1000 ms, 10 000 pages take about 2.8 hours.
|
|
77
|
+
- No documented limit: omit pages.min_interval_milliseconds. State a number only when the
|
|
78
|
+
publisher documents one. Never guess a gap the publisher did not publish.
|
|
79
|
+
- A stated gap at or above the registration floor is in use. `mr-data recipe explain` names a
|
|
80
|
+
gap under that floor, and does not ask you to fill in an omitted one.
|
|
81
|
+
- Registration refuses a gap under 4 ms by name, which is 256 requests a second to one host
|
|
82
|
+
from a collection. That is a platform bound, never a target.
|
|
83
|
+
- Where the deployment spreads a collection's requests over its egress addresses, an omitted
|
|
84
|
+
pace is held to 4 requests a second per address and to max(4, M) a second host-wide when
|
|
85
|
+
spread over M addresses. A stated gap is kept per address, so the publisher can see a multiple
|
|
86
|
+
of that rate. That holds for every per-row request with no platform credential, your own key
|
|
87
|
+
included. No command prints how many addresses there are. When the publisher's limit is per
|
|
88
|
+
key or per account, state the gap it documents.
|
|
85
89
|
- The worker holds the gap between two requests to one host whatever `pages.concurrency` admits.
|
|
86
90
|
A wider window never sends faster than the pace.
|
|
87
|
-
- Leave `pages.concurrency` out. Left out, the worker uses the
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
+
- Leave `pages.concurrency` out, or state it only at 64 or above. Left out, the worker uses the
|
|
92
|
+
capacity it is admitted and picks the width. A stated width strictly under 64 is a ceiling on
|
|
93
|
+
that width, and `run --full` refuses it unless `--accept-slow` says why. `concurrency: 1`
|
|
94
|
+
stays that miss. State a lower ceiling only when a run's refusal told you to lower it or the
|
|
95
|
+
publisher documents a limit on concurrent connections. Record the reason in a dataset note,
|
|
96
|
+
so the next session finds it.
|
|
91
97
|
- Above 10 000 fetches in one run, `pages.max_fetches_per_run` times the gap must stay within
|
|
92
98
|
25 200 seconds. That is the eight-hour run lease less a one-hour reserve. Registration refuses
|
|
93
|
-
a pair that waits longer, by name.
|
|
94
|
-
admits 25 200 fetches.
|
|
99
|
+
a pair that waits longer, by name. An omitted gap is priced at 1000 ms for that check, which
|
|
100
|
+
admits 25 200 fetches. A larger budget states the gap the publisher documents, or fewer
|
|
101
|
+
fetches per run.
|
|
95
102
|
- A platform credential's binding bounds the rate from above. The declared gap still applies
|
|
96
|
-
under it, and an omitted
|
|
97
|
-
prints the binding's rate and `effective_rps` once the recipe
|
|
98
|
-
|
|
103
|
+
under it, and an omitted pace still starts at 1000 ms and adapts down to 250 ms under that
|
|
104
|
+
cap. `mr-data recipe forecast` prints the binding's rate and `effective_rps` once the recipe
|
|
105
|
+
is registered.
|
|
99
106
|
- State one spelling. Remove the legacy `min_interval_seconds` when you state
|
|
100
107
|
`min_interval_milliseconds`.
|
|
101
108
|
|
|
102
|
-
`mr-data recipe explain` reports both under `fast_features`. `pace` names a
|
|
103
|
-
|
|
104
|
-
runtime at the declared ceiling.
|
|
109
|
+
`mr-data recipe explain` reports both under `fast_features`. `pace` names a stated gap under the
|
|
110
|
+
registration floor. An omitted gap is in use. `concurrency` names a stated width strictly under
|
|
111
|
+
64. `mr-data recipe forecast` prints the rate and the runtime at the declared ceiling.
|
|
105
112
|
|
|
106
113
|
**`limits.max_requests` is charged per HTTP request, redirect hops and retries included** — a page
|
|
107
114
|
that redirects once costs two. For a collection that lists pages, registration only checks that
|