mostlyright-data 0.31.11__tar.gz → 0.31.12__tar.gz

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mostlyright-data
-Version: 0.31.11
+Version: 0.31.12
 Summary: Mostly Right hosted CLI for reviewed datasets
 Project-URL: Homepage, https://mostlyright.md/
 Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/skills/mr-data-build/references/preparing-table-search.md

@@ -194,7 +194,46 @@ column names, the descriptions and the planner. Never bind a word that two colum
 such as "growth", "change" or "best", unless the brief names the metric. "Highest growth"
 pointed at a revenue column turns "highest growth last 30 days" into a revenue sort.
 
-Studio caps a spec at 128 filter aliases and 64 sort aliases, and `search validate` refuses more.
+### Declare a floor for every rate
+
+A growth, change, ratio or per-unit column ranks nonsense first when its base is tiny. Revenue
+that grew from $5,000 to $80 million is 1,600,000% growth, and a film with three votes has a
+perfect rating. Sorting by such a column shows shells and placeholders, not the rows the reader
+asked for. For each sortable rate column, declare the column it is a rate of, and the smallest
+base that makes the rate meaningful, in `order_floors` (contract 4.54.0, up to 32 entries):
+
+```json
+"order_floors": [
+  {"column": "revenue_growth_pct", "base_column": "revenue_prior_year_usd", "min": 10000000},
+  {"column": "revenue_cagr_5y_pct", "base_column": "revenue_5y_ago_usd", "min": 10000000},
+  {"column": "vote_average", "base_column": "vote_count", "min": 100}
+]
+```
+
+- `column` is in `sortable_columns` or is a sort alias's column. `base_column` is another numeric
+  column in `filterable_columns`. List each base early in `filterable_columns`: a package keeps 64
+  filters, role and alias columns first and the rest in spec order, and a base past that cut
+  loses its floor. `search validate TABLE --spec` names such a floor in `warnings` under
+  `/order_floors/`; treat that warning as a floor that will not apply.
+- `min` is a whole number in the base column's own unit, written without a decimal point or
+  exponent: `10000000` on a column in dollars, `10` on a column in millions.
+- Floors are declared on a search revision with `mr-data search configure --spec`, not in the
+  recipe's `table.search`, which refuses the member. A revision configured from the recipe's own
+  spec carries none, and a new version of a private table keeps them only when you configure the
+  revision again. A plan made without the planner, such as a keystroke search, applies no floor.
+- A Studio older than contract 4.54.0 refuses the member as unknown. Configure without it there.
+- When a plan orders by `column`, in either direction, Studio adds `base_column >= min` as a
+  filter. The reader sees it in the search summary and can remove it. A query that already filters
+  `base_column` keeps its own filter.
+- Prefer the true base (the earlier period's revenue, the vote count) over a size proxy such as
+  market cap. When the table has no base column, add one in the transform: it is cheaper than a
+  wrong ranking.
+- Choose `min` from the built table with `mr-data query`: the smallest base at which the rate stops
+  being dominated by placeholders. State the floor at the start of the base column's
+  description, since the planner reads only the first 160 characters of a description.
+
+A recipe's spec takes 128 filter aliases and 64 sort aliases, and a search revision 512 and 256.
+`search validate` refuses more.
 Rank the phrases and keep the top of the list. Put first the words that name the kind of row, the
 facets users type most, the popularity sorts and the product's own vocabulary. A phrase you leave
 out maps to nothing. The planner may still recover it from the column names and values, and
@@ -225,8 +264,13 @@ roles. It never sees rows.
 `planner_detail.reason`. `external_context_not_allowed` means the recipe withholds consent.
 `private_source` means a source is not public.
 
-Write each column description as if the planner reads it. Say what the column holds, its unit,
-and the words people use for it.
+Write each column description as if the planner reads it, because it does, and the page shows it
+to the reader beside results ordered by that column. Say what the column holds, the period it
+covers ("trailing twelve months", "latest fiscal year against the year before", "over the last
+five fiscal years"), its unit, and the words people use for it, abbreviations included
+("compound annual growth rate, CAGR"). The planner picks the column a query asks about, named
+bare ("revenue") or in a question ("what is the P/E ratio?"), from these descriptions. It reads
+the first 160 characters of each, so lead with what the column measures and its period.
 
 ### Name and describe columns for search
 

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/src/mostlyright/data_harness/thin/table_search_contract.json

@@ -1,8 +1,14 @@
 {
  "schema_version": "mostlyright-thin-table-search-contract.v1",
- "source_commit": "0dcecc1b8dfdd3f8960497fdfcffeeb72e240abd",
+ "source_commit": "5109da1f92eaaf4fc41ed35a03e5d010268d7c7b",
  "recipe_search_definition": "recipe_search",
  "$defs": {
+  "common.safe_integer": {
+   "description": "An integer exactly representable by both Python JSON integers and TypeScript numbers.",
+   "type": "integer",
+   "minimum": -9007199254740991,
+   "maximum": 9007199254740991
+  },
   "evaluation_case": {
    "type": "object",
    "additionalProperties": false,
@@ -119,6 +125,14 @@
       ]
      },
      "description": "Keel's recipe reader does not admit long_text_columns yet (4.42.0): a search revision declares it."
+    },
+    {
+     "not": {
+      "required": [
+       "order_floors"
+      ]
+     },
+     "description": "The recipe readers do not admit order_floors (4.54.0): a search revision declares it."
     }
    ]
   },
@@ -252,6 +266,35 @@
       "direction"
      ]
     },
+    "order_floors": {
+     "description": "Order floors (4.54.0): for a sortable column whose order means little on rows with a tiny or missing base (a growth rate on almost no revenue, a rating from a handful of votes), the base column and the least value of it at which a row answers that order. When a plan orders by `column`, either direction, and states no predicate on `base_column`, the planner adds `base_column >= min` (origin `parser`), which the search node runs and reports as any predicate. `column` is declared sortable (in `sortable_columns` or a sort alias's, even where a v3 package sorts numbers by default), `base_column` is another declared numeric column in `filterable_columns`, and each `column` has at most one floor. `min` is an integer (a request body carries no fractional or exponent number), in the base column's own unit. Admitted on a search revision; a recipe's `table.search` may not name it yet.",
+     "type": "array",
+     "maxItems": 32,
+     "items": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+       "column": {
+        "type": "string",
+        "minLength": 1,
+        "maxLength": 128
+       },
+       "base_column": {
+        "type": "string",
+        "minLength": 1,
+        "maxLength": 128
+       },
+       "min": {
+        "$ref": "#/$defs/common.safe_integer"
+       }
+      },
+      "required": [
+       "column",
+       "base_column",
+       "min"
+      ]
+     }
+    },
     "planner": {
      "type": "object",
      "additionalProperties": false,

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/src/mostlyright/data_harness/thin/v4.py

@@ -149,8 +149,8 @@ from mostlyright.data_harness.thin.v4_matching import CONTRACT_PATHS as MATCHING
 #: ``JOB_INVALID`` outright, which in production was every refresh a collection epoch was offered
 #: for rather than only the collection runs. This package must not reach a Studio older than the
 #: commit it pins; ``docs/V4-WORKER-PROTOCOL.md`` states it beside the layout's own ordering rule.
-PINNED_V4_OPENAPI_SOURCE_SHA256 = "b8f2ac008a714a12b88516d4c723ef8a7af18100c2678831fbdafd50fc6a5f20"
-PINNED_V4_CONTRACT_VERSION = "4.49.0"
+PINNED_V4_OPENAPI_SOURCE_SHA256 = "e7bd4e994b32d62ae00b4d7fc2f8936e4964cd4160b7aa30ae010528ff57b1a8"
+PINNED_V4_CONTRACT_VERSION = "4.54.0"
 
 #: The ``schema_version`` every command body that predates 4.14.0 stamps: ``dataset activity``,
 #: ``table rename``, ``table archive`` and ``dataset archive``. It is the number the 0.25 line

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/src/mostlyright/data_harness/thin/v4_runs.py

@@ -1494,7 +1494,16 @@ def _retry_refusal_step(run: Mapping[str, Any]) -> str:
 #: A source that kept refusing. Studio admits a person's named retry of these once its wait has
 #: passed and answers ``RUN_RETRY_TOO_SOON`` with a Retry-After before then (planning #1234), so
 #: this command sends them and lets Studio say when.
-PERSON_RETRY_CODES = frozenset({"SOURCE_RATE_LIMITED", "SOURCE_HTTP_5XX", "SOURCE_PROXY_BUSY"})
+PERSON_RETRY_CODES = frozenset(
+    {
+        "SOURCE_RATE_LIMITED",
+        "SOURCE_HTTP_5XX",
+        "SOURCE_PROXY_BUSY",
+        "TRANSPORT_FAILURE",
+        "TOTAL_TIMEOUT",
+        "DNS_TEMPORARY_FAILURE",
+    }
+)
 
 #: Failure codes a named retry may take without a recipe change. A hand copy of Studio's
 #: ``agent_recovery`` retry and platform sets. The server refuses ``retry_of_run_id`` for anything

{mostlyright_data-0.31.11 → mostlyright_data-0.31.12}/src/mostlyright/data_harness/thin/v4_search_contract.py

@@ -54,10 +54,57 @@ def _recipe_search_definition(recipe: dict[str, Any]) -> dict[str, Any]:
     return localize(recipe["$defs"]["table_plan"]["properties"]["search"])
 
 
-def build_bundle(table_search: dict[str, Any], recipe: dict[str, Any], pin: dict[str, Any]) -> dict:
-    """The closure of the definitions the client checks, from the vendored documents."""
+#: A reference into the common schema is carried into the bundle as a local definition named
+#: with this prefix, so the bundle stays one self-contained document.
+COMMON_REF = "common.schema.json#/$defs/"
+COMMON_PREFIX = "common."
 
-    definitions = table_search["$defs"]
+
+def _localize_common(value: Any) -> Any:
+    """``value`` with every common-schema reference pointing at its bundled local copy."""
+
+    if isinstance(value, list):
+        return [_localize_common(item) for item in value]
+    if not isinstance(value, dict):
+        return value
+    result = {}
+    for key, item in value.items():
+        if key == "$ref" and isinstance(item, str) and item.startswith(COMMON_REF):
+            result[key] = "#/$defs/" + COMMON_PREFIX + item[len(COMMON_REF) :]
+        else:
+            result[key] = _localize_common(item)
+    return result
+
+
+def build_bundle(
+    table_search: dict[str, Any],
+    recipe: dict[str, Any],
+    pin: dict[str, Any],
+    common: dict[str, Any] | None = None,
+) -> dict:
+    """The closure of the definitions the client checks, from the vendored documents. A
+    definition the search schema takes from the common schema is copied in under
+    :data:`COMMON_PREFIX`."""
+
+    def within_common(value: Any) -> Any:
+        """A common definition with its own local references pointing at their copies."""
+
+        if isinstance(value, list):
+            return [within_common(item) for item in value]
+        if not isinstance(value, dict):
+            return value
+        return {
+            key: (
+                "#/$defs/" + COMMON_PREFIX + _ref_name(item)
+                if key == "$ref" and isinstance(item, str) and item.startswith("#/$defs/")
+                else within_common(item)
+            )
+            for key, item in value.items()
+        }
+
+    definitions = dict(table_search["$defs"])
+    for name, definition in ((common or {}).get("$defs") or {}).items():
+        definitions[COMMON_PREFIX + name] = within_common(definition)
     recipe_member = "recipe_search"
     wanted = [SPEC, recipe_member, CASES]
     selected: dict[str, Any] = {recipe_member: _recipe_search_definition(recipe)}
@@ -66,6 +113,8 @@ def build_bundle(table_search: dict[str, Any], recipe: dict[str, Any], pin: dict
         if isinstance(value, dict):
             ref = value.get("$ref")
             if isinstance(ref, str):
+                if ref.startswith(COMMON_REF):
+                    ref = "#/$defs/" + COMMON_PREFIX + ref[len(COMMON_REF) :]
                 name = _ref_name(ref)
                 if name not in selected:
                     selected[name] = definitions[name]
@@ -81,6 +130,7 @@ def build_bundle(table_search: dict[str, Any], recipe: dict[str, Any], pin: dict
         if name not in selected:
             selected[name] = definitions[name]
             collect(definitions[name])
+    selected = {name: _localize_common(definition) for name, definition in selected.items()}
     return {
         "schema_version": BUNDLE_SCHEMA_VERSION,
         "source_commit": pin["source_commit"],
@@ -95,6 +145,7 @@ def bundle_from_vendor(vendor: Path = VENDOR) -> dict[str, Any]:
         json.loads((schemas / "table-search.schema.json").read_text(encoding="utf-8")),
         json.loads((schemas / "recipe.schema.json").read_text(encoding="utf-8")),
         json.loads((vendor / "client-v4-pin.json").read_text(encoding="utf-8")),
+        json.loads((schemas / "common.schema.json").read_text(encoding="utf-8")),
     )