ripple-sql 0.1.16__tar.gz → 0.1.18__tar.gz

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/CHANGELOG.md

@@ -6,6 +6,36 @@ Notable changes to Ripple. The format follows
 
 ## [Unreleased]
 
+## [0.1.18] - 2026-10-07
+
+### Added
+- The file check reaches people wherever they work. `ripple.yml` names destinations once, each pointing at an environment variable that holds the webhook URL (a URL in the file is refused). Each feed routes beside its `files:` pattern: `notify: {sender, team, contact, on}`, so the uploader gets the plain message and the data team the report. Slack, Teams Workflows and Discord get their own card shapes; anything else gets one stable JSON object for Zapier, Make, n8n or Power Automate. Old Office 365 connector URLs are refused by name. `ripple check-file --notify` sends; `--markdown` prints a pull request comment body; `--json` gains `schema_version` and `notify`. `ripple notify-test NAME` posts a labeled test card.
+- `ripple watch DIR` is the landing-folder gate as part of the product: incoming/ to ready/, rejected/ (with the sender's message and the report beside the file) or held/, one alert per file, a daily reminder for files left waiting, a failed alert left beside the file with exit 1. One pass for cron or Task Scheduler, `--every` to loop, `--dry-run` to preview. Stdlib only, so it runs on Windows.
+- A file whose every header was retitled in Google Sheets style ("Email Domain" for email_domain) now binds by header style instead of being refused. It stops honestly, since a loader that keeps headers as written lands none of those columns, and tells the sender the names to rename to. `normalize: fivetran` on the source passes it.
+- When `ripple serve` does not read the warehouse by itself, the terminal and the page say why and what to do, for example "warehouse unknown · run with --dialect snowflake, or add profiles.yml beside dbt_project.yml".
+- Warehouse columns fill in by themselves. Ripple reads dbt's `target/catalog.json` (written by `dbt docs generate` or `--write-catalog`) on every run and attaches each source's real columns to the project's own sources: attach only, never a new relation, never over a model's own SQL, never over `.ripple/schemas.json`, and not when the catalog names another physical relation than the project's. When the project is on Snowflake, BigQuery or Databricks and the user's `snow`, `bq` or `databricks` CLI is installed and signed in, `ripple serve` and the MCP server read `information_schema.columns` for the tables that have no columns, in the background, save them to `.ripple/schemas.json` with `origin` and `fetched_at`, and rebuild while the old graph keeps answering; the page shows "Snowflake · prod · reading columns", then the landing with the old figures kept and Undo. It starts on its own only with a known connection (the user's default, the only one, or one picked before, remembered in the user's cache) that signs in without a browser; otherwise the page offers one button, "Fetch from Snowflake · prod", or a connection picker. Names are checked against plain identifier patterns before they reach SQL; results are matched back to the exact database, schema and table; a table whose name differs only by case is skipped; `bq` gets an explicit row cap and a result that reaches it is not saved; a CLI that fails or prints anything but complete JSON saves nothing. Fetched columns replace the old list (a dropped column does not live on), are read again after a week, and a table the connection cannot see is not asked for again for a week; a failed fetch waits 30 minutes before trying on its own. `schemas.json` writes are locked and atomic. New: `ripple fetch-schema`, the MCP tool `fetch_schema`, MCP `instructions` at connect time telling the agent how columns arrive, `lineage_summary.warehouse`, the page's "where columns came from" line (source, tables, age) and a "fully analyzed" count with its gain in the ledger after a landing. `RIPPLE_WAREHOUSE=off` turns the warehouse read off.
+- Drop the warehouse's column list on the page and watch the count move. Two review classes shrink when Ripple knows the columns of tables the project reads but does not own; the page now takes that list itself. Under the reasons it fixes sits one dashed target, "Add warehouse columns · information_schema.columns · csv or json", with "Copy query" (the query names the schemas the gaps live in, in the project's dialect) and "show" to read it first. Drop the export (or pick it; a column list dropped anywhere on the page is offered with one button, and a file that is not one, dropped on the target, offers "Check this file"). The local server reads the whole file, keeps only the tables the project reads (schema-qualified when the export carries `table_schema`), saves them to `.ripple/schemas.json`, and rebuilds in the background while the old graph keeps answering (an answer given meanwhile says "answered before the column import"). The target shows the real count, "Rebuilding · 712 of 1,908 models", from a build callback (`LineageGraph.build(project, on_progress=...)`). When it lands, the numbers change in place with the old figure kept beside them ("2,810 need review · was 3,356", "+4,688" on column links, "12 · was 315" on the rows, a row that reached zero stays as "0 · was 231") and the target becomes the receipt: "cols.csv · 674 tables · 213 models now fully analyzed · Undo". Undo puts the file back as it was, if nothing else touched it since, and rebuilds again. A second drop of the same export says "already included · no changes" and rebuilds nothing; a file with no matching tables saves nothing. Routes: `POST /api/ingest`, `POST /api/undo`, `GET /api/status`, `GET /api/home`. The server now writes exactly one file, `.ripple/schemas.json`, and only from the page it opened. `ripple ingest-schema` reads `table_schema` too, so a real export's `gitlab_dotcom.users` matches the read. Three seats (Codex, Sonnet, Claude) settled the words and the states in a design debate.
+
+### Changed
+- The file check's wording moved from check_file/report.py to check_file/text.py, which was at its 500-line budget.
+- The review count says why. GitLab's data project showed "3356 need review" and 2,706 of those links carried no reason: their house style names a CTE after the model it reads (`with prep_action as (select ... from prep_action)`), the engine marks that as a CTE shadowing a real table, and the mark never reached the link. Every link at review trust now carries a sentence (the engine's own word where it has one: the shadowing CTE, a column a CTE does not name and passes through to its only source, a script transform; "traced at low confidence" otherwise), and the sentences fold into a few classes in one module, each named for the project that produced it. `ripple` prints the classes under the count with a number and the command that shrinks the class (`ripple unresolved`, `ripple ingest-schema cols.csv`, `ripple --json`); `lineage_summary` and the served page carry the same rows as `review_reasons`; on the page the count is a button that opens the list.
+- The home of a large project is a browser, not a listing. GitLab's data project (1,908 models) opened on forty folder rows of chips with "+ 695 more" links. It now opens on the search (⌘K, results most-read first), a folder tree on the left with counts, the hubs (the models most of the project is built on) and the chosen folder as a ranked table in the middle (columns, read by, feeds from, sortable, a status dot per row), and the selected model on the right with its three numbers, its most-read columns (click one to ask) and the way to its lineage. Enter on a row opens its lineage; the last three questions asked in this browser come back as "Continue" chips. A small project gets the same page with a tree of one folder. The typeahead sorts its candidates most-read first before capping matches, so on a big project the hubs come first instead of the first 400 source tables.
+- Ripple's own identity, from the name. A ripple is a drop and its rings. The drop is the one color on the page, a vermilion that marks only the change you asked about (the chosen column, the Ask button); the rings are off-white, like light on dark water, and carry everything the change reaches (wires, the lit chain, the selected row). Wires fade with distance the way rings do: the first hop bright, the second dimmer, the third faint. The mark is rings and the pulse on the chosen column is one ring leaving it. The ground is a warm charcoal; surfaces are flat with one border, no blur, no glow, no gradient, no grid, one small radius. Two open-license faces ship inside the package and are embedded in the page as data URIs, Atkinson Hyperlegible Next for words and JetBrains Mono for names (64 KB together, latin subsets, SIL Open Font License files beside them), so a served page and the offline file look the same on every machine and still load nothing from the network. The top block is one object: the project name in mono beside the rings mark, the counts as a ledger, and the question as a grouped bar with the drop mark inside it, a segmented direction (flipping it re-asks the question on screen) and Ask attached; one drawn ripple sits top right, the only decoration on the page. The copy is cut to labels: the file box says "Check a file · csv, tsv, xlsx or parquet, header only", the legend says built from, dashboard, review, the text bar says Text, Show, Copy, the chip rows carry no captions, and the selected model on the home is one sentence, not tiles.
+- The answer page's look. The headline is now the user's own column and what happens to it, one line of type: `net_arr changes 3 columns in 3 models and 1 dashboard number.` ("at least" when truncated, "changes nothing downstream." when nothing does), with the full name, the deepest hop and the review count under it. The page is dark: a spotlight ground with two soft lights that drift, frosted glass panels and chips with a lit top edge, and one signal red on the chosen column (solid fill, a ring and a glow), the wires that leave it, and the Ask button. The panel floats up when an answer lands, the chosen column pulses once, the first-hop wires draw once, chips lift on hover; all of it is off under reduced motion. The board gets a hop ruler (one hairline across the columns with a tick where each hop starts), model rows with a rule instead of cards, a dashed review outline with no fill, and hover that lights one chain back to the chosen column while only the other wires dim, so chips and review notes stay readable; a click pins the chain and Escape clears it. On an answer the file-check box becomes one quiet row at the foot of the panel. The coverage row says answers work without the warehouse's column list. Every text token clears 4.5:1 on what it sits on. DESIGN.md describes the result; a three-seat design debate produced the headline, the ruler and the hover rules, and the owner picked the material from three rendered directions.
+
+### Fixed
+- A file that fits no table named every checkable table in one line (28,886 characters on GitLab). It now names five, the closest fit first, and how many more.
+- `ripple fetch-schema` no longer calls a project complete when it only had nothing it could ask about.
+- The summary says "1 link needs review".
+- The file check counts the columns the project reads. It diffed a file only against a source's yml-declared columns and refused otherwise, while the lineage graph already held the columns every staging model reads from that table. Found on GitLab's data team project (2021 snapshot), which loads finance spreadsheets through sheetload: 29 of 686 source tables declare yml columns, 0 of the 93 sheetload tables do, and the check refused every one of them with the six column names of the net ARR sheet in hand. The expected header is now the declared columns plus the columns downstream reads, from one helper that the report, the header binding and the table list all use; the refusal is kept for a table nothing declares and nothing reads. `ripple check-file --source sheetload.net_arr_net_iacv_conversion_factors` on that project now reports the dropped and renamed columns with their readers and writes the sender's message.
+- The dialect sniffer no longer reads a regex literal as a psql meta-command. `\d` inside `REGEXP_CONTAINS(app_version, r'^(\d+\.\d+')` matched the psql short-circuit and returned postgres before BigQuery's markers were weighed (14 points to 3), so Mozilla's bigquery-etl parsed as postgres: 1,464 of its 1,826 tables analyzed in full, against 1,551 once read as BigQuery. A meta-command begins a line; the check is anchored there.
+- On Windows, a warehouse CLI installed as a `.cmd` or `.bat` shim (gcloud ships `bq.cmd`) got only the first line of a multi-line query, because cmd.exe ends a batch file's command line at the first newline. Ripple now puts each argument to a shim on one line; a real executable gets its arguments unchanged.
+
+## [0.1.17] - 2026-09-22
+
+### Fixed
+- A dbt node whose artifact row carries no compiled SQL now reads the compiled file dbt wrote on disk (`target/compiled/<package>/<path>`, or the row's own `compiled_path`, in either path spelling) before falling back to the raw jinja. Found running the dbt 2.0 demo on jaffle-shop: dbt-oss 2.0.4 left `compiled_code` empty in every Parquet row while `target/compiled/` held every model, so Ripple silently read raw jinja and a `date_spine` macro became an unresolved table (128 links and one link flagged instead of the manifest's 132). A parse-only manifest gets the same treatment. The file answers only when it is at least as new as the model's own source (found under the package that owns it), the macros and `dbt_project.yml`, so a compile left behind an edit never stands in for the edited SQL. A two-model review of the change found the package source path, the macro edits and the Windows separators; all fixed.
+
 ## [0.1.16] - 2026-09-21
 
 ### Added
@@ -324,7 +354,9 @@ First public release.
 ### Removed
 - The dark whole-graph canvas that `ripple serve` used to open (`static/index.html`). Every surface now draws one answer.
 
-[Unreleased]: https://github.com/bteh/ripple/compare/v0.1.16...HEAD
+[Unreleased]: https://github.com/bteh/ripple/compare/v0.1.18...HEAD
+[0.1.18]: https://github.com/bteh/ripple/compare/v0.1.17...v0.1.18
+[0.1.17]: https://github.com/bteh/ripple/compare/v0.1.16...v0.1.17
 [0.1.16]: https://github.com/bteh/ripple/releases/tag/v0.1.16
 [0.1.15]: https://github.com/bteh/ripple/releases/tag/v0.1.15
 [0.1.14]: https://github.com/bteh/ripple/releases/tag/v0.1.14

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ripple-sql
-Version: 0.1.16
+Version: 0.1.18
 Summary: Offline column-level SQL lineage. See what breaks before you merge.
 Project-URL: Homepage, https://github.com/bteh/ripple
 Project-URL: Repository, https://github.com/bteh/ripple
@@ -232,19 +232,25 @@ Ripple is built to be driven by an agent, so the line between what is parsed and
 is written down and enforced: the engine never guesses, the agent may, and you always know which one
 is talking. [`docs/AI-BOUNDARY.md`](docs/AI-BOUNDARY.md) is the rule and the reasoning.
 
-## Bring your warehouse's schemas
+## Your warehouse's columns, filled in for you
 
-Ripple never connects to your warehouse. But your SQL probably reads tables defined outside the repo, and without their columns those models stop at `SELECT *`. Fix: your agent already has warehouse access, so let it fill the gap. Ask it to "resolve ripple's unresolved tables" and it will call `unresolved_tables`, fetch the columns from `information_schema.columns`, and hand them back via `ingest_schema`. They land in `.ripple/schemas.json`, which you can commit so the whole team gets the coverage.
+Your SQL probably reads tables defined outside the repo, and without their columns those models stop at `SELECT *`. Ripple fills those columns by itself, from two places, with nothing to export:
 
-The credential boundary is the point. Your agent holds the connection; Ripple never does.
+- **dbt's `target/catalog.json`.** `dbt docs generate` (or `--write-catalog` on dbt 2) already wrote every source's real columns there. Ripple reads it on every run.
+- **Your own warehouse CLI.** When the project is on Snowflake, BigQuery or Databricks and `snow`, `bq` or `databricks` is installed and signed in, `ripple serve` and the MCP server read `information_schema.columns` for the missing tables through it, in the background, and the page shows the count move. Only table and column names are read. Columns older than a week are read again.
 
-No agent? Same thing by hand:
+Ripple never stores, prompts for, or sends a credential, and never opens a connection of its own: it runs the CLI you already signed in to, with the connection you marked default (or the only one). It starts on its own only when signing in needs no browser; otherwise the page shows one button, "Fetch from Snowflake · prod". Several connections and no default: the page asks once and remembers the name in your cache, not in the repo. `RIPPLE_WAREHOUSE=off` turns it off.
+
+The columns land in `.ripple/schemas.json` with where they came from and when, which you can commit so the whole team gets the coverage. From a terminal:
 
 ```bash
-ripple unresolved                 # which tables block the most coverage, worst first
-ripple ingest-schema cols.csv     # CSV with a table,column header (or JSON, or - for stdin)
+ripple fetch-schema               # read missing columns through your CLI, print what moved
+ripple unresolved                 # which tables still block coverage, worst first
+ripple ingest-schema cols.csv     # no CLI? an information_schema.columns export works too
 ```
 
+An agent connected over MCP is told all of this when it connects: it calls `fetch_schema`, or, when Ripple cannot read the warehouse and the agent can, queries `information_schema.columns` itself and hands the names to `ingest_schema`.
+
 ## A file arrives from another team
 
 Vendors and business units hand over CSVs and spreadsheets by hand. When a column is renamed or dropped in one of them, the pipeline loads it anyway and dashboards go blank the next morning. Before loading, ask Ripple what this exact file breaks:
@@ -308,6 +314,15 @@ The same check is on the `ripple serve` page: drop the file anywhere on it (or c
 
 When the loader got there first, check the table that landed instead of a file: `ripple check-file --columns -` reads the column names from stdin (an `information_schema` query), `--columns "a,b,c"` takes them inline, and the MCP tool takes `columns=[...]`. Where to put all of this in a pipeline, with a gate script that moves files between `incoming/`, `ready/` and `rejected/`: [docs/GUARDRAIL.md](docs/GUARDRAIL.md).
 
+To tell the person who sent it, and the data team, without anyone forwarding
+anything: name a Slack, Teams or Discord channel (or a Zapier, Make, n8n or
+Power Automate hook for email) in `ripple.yml`, point it at an environment
+variable, and run `ripple watch` on the folder files land in. Power Automate
+hosts receive Teams cards by default; set `kind: json` for a custom flow
+that expects JSON. Setup is in
+[docs/GUARDRAIL.md](docs/GUARDRAIL.md#who-hears-about-it).
+
+
 ## Bring your query history (optional)
 
 Your SQL files say what is supposed to happen. The warehouse's query log says what
@@ -362,10 +377,11 @@ difference; Ripple reads the repo, not the scheduler.
 engine. The manifest is still schema v12; when a project turns it off
 (`--no-write-json`) and writes the Parquet information schema instead
 (`--generate-info-schema`), Ripple reads `target/info_schema/v1/` through
-the same path (`pip install pyarrow`). dbt 2.0's own column-level lineage
-needs the full dbt distribution, a dbt Labs login and a warehouse
-connection; the open-source engine writes `dbt.column_lineage.parquet`
-empty. Ripple needs none of those.
+the same path (`pip install pyarrow`); when a build leaves the compiled
+SQL only under `target/compiled/`, Ripple reads it from there. dbt 2.0's
+own column-level lineage needs the full dbt distribution, a dbt Labs login
+and a warehouse connection; the open-source engine writes
+`dbt.column_lineage.parquet` empty. Ripple needs none of those.
 
 **Dialects:** Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB,
 Spark SQL. Detected per project, override with `--dialect`. Powered by

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/README.md

@@ -191,19 +191,25 @@ Ripple is built to be driven by an agent, so the line between what is parsed and
 is written down and enforced: the engine never guesses, the agent may, and you always know which one
 is talking. [`docs/AI-BOUNDARY.md`](docs/AI-BOUNDARY.md) is the rule and the reasoning.
 
-## Bring your warehouse's schemas
+## Your warehouse's columns, filled in for you
 
-Ripple never connects to your warehouse. But your SQL probably reads tables defined outside the repo, and without their columns those models stop at `SELECT *`. Fix: your agent already has warehouse access, so let it fill the gap. Ask it to "resolve ripple's unresolved tables" and it will call `unresolved_tables`, fetch the columns from `information_schema.columns`, and hand them back via `ingest_schema`. They land in `.ripple/schemas.json`, which you can commit so the whole team gets the coverage.
+Your SQL probably reads tables defined outside the repo, and without their columns those models stop at `SELECT *`. Ripple fills those columns by itself, from two places, with nothing to export:
 
-The credential boundary is the point. Your agent holds the connection; Ripple never does.
+- **dbt's `target/catalog.json`.** `dbt docs generate` (or `--write-catalog` on dbt 2) already wrote every source's real columns there. Ripple reads it on every run.
+- **Your own warehouse CLI.** When the project is on Snowflake, BigQuery or Databricks and `snow`, `bq` or `databricks` is installed and signed in, `ripple serve` and the MCP server read `information_schema.columns` for the missing tables through it, in the background, and the page shows the count move. Only table and column names are read. Columns older than a week are read again.
 
-No agent? Same thing by hand:
+Ripple never stores, prompts for, or sends a credential, and never opens a connection of its own: it runs the CLI you already signed in to, with the connection you marked default (or the only one). It starts on its own only when signing in needs no browser; otherwise the page shows one button, "Fetch from Snowflake · prod". Several connections and no default: the page asks once and remembers the name in your cache, not in the repo. `RIPPLE_WAREHOUSE=off` turns it off.
+
+The columns land in `.ripple/schemas.json` with where they came from and when, which you can commit so the whole team gets the coverage. From a terminal:
 
 ```bash
-ripple unresolved                 # which tables block the most coverage, worst first
-ripple ingest-schema cols.csv     # CSV with a table,column header (or JSON, or - for stdin)
+ripple fetch-schema               # read missing columns through your CLI, print what moved
+ripple unresolved                 # which tables still block coverage, worst first
+ripple ingest-schema cols.csv     # no CLI? an information_schema.columns export works too
 ```
 
+An agent connected over MCP is told all of this when it connects: it calls `fetch_schema`, or, when Ripple cannot read the warehouse and the agent can, queries `information_schema.columns` itself and hands the names to `ingest_schema`.
+
 ## A file arrives from another team
 
 Vendors and business units hand over CSVs and spreadsheets by hand. When a column is renamed or dropped in one of them, the pipeline loads it anyway and dashboards go blank the next morning. Before loading, ask Ripple what this exact file breaks:
@@ -267,6 +273,15 @@ The same check is on the `ripple serve` page: drop the file anywhere on it (or c
 
 When the loader got there first, check the table that landed instead of a file: `ripple check-file --columns -` reads the column names from stdin (an `information_schema` query), `--columns "a,b,c"` takes them inline, and the MCP tool takes `columns=[...]`. Where to put all of this in a pipeline, with a gate script that moves files between `incoming/`, `ready/` and `rejected/`: [docs/GUARDRAIL.md](docs/GUARDRAIL.md).
 
+To tell the person who sent it, and the data team, without anyone forwarding
+anything: name a Slack, Teams or Discord channel (or a Zapier, Make, n8n or
+Power Automate hook for email) in `ripple.yml`, point it at an environment
+variable, and run `ripple watch` on the folder files land in. Power Automate
+hosts receive Teams cards by default; set `kind: json` for a custom flow
+that expects JSON. Setup is in
+[docs/GUARDRAIL.md](docs/GUARDRAIL.md#who-hears-about-it).
+
+
 ## Bring your query history (optional)
 
 Your SQL files say what is supposed to happen. The warehouse's query log says what
@@ -321,10 +336,11 @@ difference; Ripple reads the repo, not the scheduler.
 engine. The manifest is still schema v12; when a project turns it off
 (`--no-write-json`) and writes the Parquet information schema instead
 (`--generate-info-schema`), Ripple reads `target/info_schema/v1/` through
-the same path (`pip install pyarrow`). dbt 2.0's own column-level lineage
-needs the full dbt distribution, a dbt Labs login and a warehouse
-connection; the open-source engine writes `dbt.column_lineage.parquet`
-empty. Ripple needs none of those.
+the same path (`pip install pyarrow`); when a build leaves the compiled
+SQL only under `target/compiled/`, Ripple reads it from there. dbt 2.0's
+own column-level lineage needs the full dbt distribution, a dbt Labs login
+and a warehouse connection; the open-source engine writes
+`dbt.column_lineage.parquet` empty. Ripple needs none of those.
 
 **Dialects:** Snowflake, BigQuery, Databricks, Redshift, Postgres, DuckDB,
 Spark SQL. Detected per project, override with `--dialect`. Powered by

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ripple-sql"
-version = "0.1.16"
+version = "0.1.18"
 description = "Offline column-level SQL lineage. See what breaks before you merge."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -76,5 +76,8 @@ line-length = 100
 target-version = "py310"
 
 [tool.ruff.lint]
-select = ["E", "F", "I", "UP", "B", "SIM"]
+select = ["E", "F", "I", "UP", "B", "SIM", "PLW1514"]
+# PLW1514 is preview-only; explicit keeps other preview rules out of E/F/...
+preview = true
+explicit-preview-rules = true
 ignore = ["E501"]  # line length is the formatter's job, not the linter's

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/src/ripple/answer_page.py

@@ -27,6 +27,39 @@ EXPLORE = Path(__file__).parent / "static" / "explore.js"
 FOCUS = Path(__file__).parent / "static" / "focus.js"
 FIND = Path(__file__).parent / "static" / "find.js"
 CHECK = Path(__file__).parent / "static" / "check.js"
+HOME = Path(__file__).parent / "static" / "home.js"
+INGEST = Path(__file__).parent / "static" / "ingest.js"
+FONTS = Path(__file__).parent / "static" / "fonts"
+# the two faces ship inside the package (SIL Open Font License, see the OFL
+# files beside them) and are embedded in the page, so a served page and the
+# offline file look the same and neither loads anything from the network
+FONT_FACES = (
+    ("Atkinson Hyperlegible Next", "atkinson-hyperlegible-next-latin.woff2", "400 700"),
+    ("JetBrains Mono", "jetbrains-mono-latin.woff2", "400 700"),
+)
+_font_css: str | None = None
+
+
+def font_css() -> str:
+    """@font-face rules with the font files as data URIs, read once."""
+    global _font_css
+    if _font_css is None:
+        import base64
+
+        rules = []
+        for family, name, weights in FONT_FACES:
+            path = FONTS / name
+            if not path.is_file():
+                continue
+            data = base64.b64encode(path.read_bytes()).decode("ascii")
+            rules.append(
+                f'@font-face{{font-family:"{family}";font-style:normal;font-weight:{weights};'
+                f'font-display:swap;src:url(data:font/woff2;base64,{data}) format("woff2")}}'
+            )
+        _font_css = "\n".join(rules)
+    return _font_css
+
+
 EMBED_BUDGET_ENV = "RIPPLE_EMBED_GRAPH_BYTES"
 DEFAULT_EMBED_BUDGET = 5_000_000
 TRUST_CODE = {"verified": "v", "high_confidence": "h", "moderate": "m", "review_required": "r"}
@@ -161,12 +194,15 @@ def _page(data: dict, title: str) -> str:
     blob = json.dumps(data, separators=(",", ":")).replace("</", "<\\/")
     html = (
         TEMPLATE.read_text(encoding="utf-8")
+        .replace("/*FONTS*/", font_css())
         .replace("/*CSS*/", STYLE.read_text(encoding="utf-8"))
         .replace("/*TWIN*/", TWIN.read_text(encoding="utf-8"))
         .replace("/*EXPLORE*/", EXPLORE.read_text(encoding="utf-8"))
         .replace("/*FOCUS*/", FOCUS.read_text(encoding="utf-8"))
         .replace("/*FIND*/", FIND.read_text(encoding="utf-8"))
         .replace("/*CHECK*/", CHECK.read_text(encoding="utf-8"))
+        .replace("/*HOME*/", HOME.read_text(encoding="utf-8"))
+        .replace("/*INGEST*/", INGEST.read_text(encoding="utf-8"))
     )
     # one pass: a title that happens to contain "__DATA__" (a column named
     # so) must not have the data block spliced into it, and the data block

{ripple_sql-0.1.16 → ripple_sql-0.1.18}/src/ripple/check_file/__init__.py

@@ -1,8 +1,9 @@
 """A file arrives from an external team: what breaks if it is loaded.
 
 The file feeds one landing table the graph already knows. Its header is
-diffed against the table's declared columns, and every column that is gone
-gets the blast radius `ripple breaks` would give it. Header only: no data
+diffed against the table's columns, the ones its yml or an ingested schema
+declares and the ones the project's SQL reads, and every column that is
+gone gets the blast radius `ripple breaks` would give it. Header only: no data
 row is read, nothing leaves the machine, and a similar name is a hint,
 never an assumption.
 
@@ -20,16 +21,21 @@ from ripple.check_file.binding import (
     BrokenRippleYml,
     bind,
     bind_by_header,
+    bind_by_style,
     bind_rule,
+    checkable_sources,
     declared_sources,
+    expected_columns,
     file_map,
     file_rules,
     find_source,
     fits,
     normalize_for,
+    source_reads,
     table_choices,
     table_list,
 )
+from ripple.check_file.header_sense import looks_like_data
 from ripple.check_file.naming import RULES, rule_or_refuse
 from ripple.check_file.readers import (
     DELIMITERS,
@@ -40,19 +46,24 @@ from ripple.check_file.readers import (
     XML_PART_CAP,
     CannotCheck,
     names,
+    read_cells,
     read_header,
 )
 from ripple.check_file.report import (
-    COLUMN_LIST,
     COMPATIBLE,
-    HEADER_ONLY,
     LOOKS_LIKE,
     MAX_DEPTH,
-    NO_LAYOUT,
-    VALUES_NOT_CHECKED,
     binding_from,
     check,
     failure_report,
+)
+from ripple.check_file.text import (
+    COLUMN_LIST,
+    HEADER_ONLY,
+    NO_LAYOUT,
+    VALUES_NOT_CHECKED,
+    data_row_lines,
+    data_row_message,
     lines,
     no_binding_lines,
     sender_message,
@@ -78,24 +89,30 @@ __all__ = [
     "CannotCheck",
     "bind",
     "bind_by_header",
+    "bind_by_style",
     "bind_rule",
     "binding_from",
     "check",
+    "checkable_sources",
     "declared_sources",
+    "expected_columns",
     "failure_report",
     "file_map",
     "file_rules",
     "find_source",
     "fits",
     "lines",
+    "looks_like_data",
     "names",
     "no_binding_lines",
     "normalize_for",
+    "read_cells",
     "read_header",
     "rule_or_refuse",
     "run",
     "run_columns",
     "sender_message",
+    "source_reads",
     "strict_lines",
     "table_choices",
     "table_list",
@@ -130,9 +147,11 @@ def run(
     layout = {"sheet": sheet, "header_row": header_row if header_row is not None else 1}
     asked = {"rule": None, "detail": None, "strict": strict}
     roots = [root, *getattr(project, "dbt_roots", [])]
+    style = False
+    rule = None
     try:
         spelling = rule_or_refuse(normalize)
-        rule, how = None, None
+        how = None
         if not source:
             rules = file_rules(roots)
             rule, how = bind_rule(path, root, rules)
@@ -141,13 +160,17 @@ def run(
                     "sheet": sheet if sheet is not None else rule["sheet"],
                     "header_row": header_row if header_row is not None else rule["header_row"],
                 }
-        header = read_header(path, encoding, layout["sheet"], layout["header_row"])
+        cells = read_cells(path, encoding, layout["sheet"], layout["header_row"])
+        # a parquet schema's names are authoritative, even numeric ones
+        if path.suffix.lower() != ".parquet" and looks_like_data(cells):
+            return _data_row(path.name, layout, asked, rule)
+        header = names(cells, path.name)
         if source:
             table, how = source, "--source"
         elif rule:
             table = rule["table"]
         else:
-            found, how, tied = bind_by_header(header, graph)
+            found, how, tied, style = bind_by_style(header, graph)
             if found is None:
                 text = no_binding_lines(path.name, header, graph, tied)
                 return failure_report(path.name, text, table_choices(header, graph), asked, layout)
@@ -165,11 +188,36 @@ def run(
                 if not source:
                     raise
                 spelling = "none"
-        return check(
-            graph, table, header, path.name, how, noun, binding_from(how, strict), layout, spelling
-        )
+        binding = binding_from(how, strict, style)
+        report = check(graph, table, header, path.name, how, noun, binding, layout, spelling)
+        report["pattern"] = rule["pattern"] if rule else None
+        return report
     except (CannotCheck, OSError) as e:
-        return failure_report(path.name, [f"Cannot check: {e}"], binding=asked, layout=layout)
+        return _held(path.name, [f"Cannot check: {e}"], asked, layout, rule)
+
+
+def _held(
+    file_name: str, text: list[str], asked: dict, layout: dict, rule: dict | None, message=None
+) -> dict:
+    """A failure report that keeps the files: pattern it bound by, if any."""
+    return failure_report(
+        file_name,
+        text,
+        binding=asked,
+        layout=layout,
+        table=rule["table"] if rule else None,
+        pattern=rule["pattern"] if rule else None,
+        message=message,
+    )
+
+
+def _data_row(file_name: str, layout: dict, asked: dict, rule: dict | None) -> dict:
+    """The held report for a header row that reads as data, its cells left out."""
+    row = layout["header_row"]
+    text = data_row_lines(file_name, row)
+    report = _held(file_name, text, asked, layout, rule, data_row_message(file_name, row))
+    report["fix_by"] = "sender"
+    return report
 
 
 def run_columns(
@@ -187,13 +235,14 @@ def run_columns(
     file name there is no files: pattern to match."""
     layout = {"sheet": None, "header_row": None}
     asked = {"rule": None, "detail": None, "strict": strict}
+    style = False
     try:
         spelling = rule_or_refuse(normalize)
         header = names(columns, COLUMN_LIST)
         if source:
             table, how = source, "--source"
         else:
-            found, how, tied = bind_by_header(header, graph)
+            found, how, tied, style = bind_by_style(header, graph)
             if found is None:
                 text = no_binding_lines(COLUMN_LIST, header, graph, tied, file=False)
                 return failure_report(
@@ -211,7 +260,7 @@ def run_columns(
             COLUMN_LIST,
             how,
             noun,
-            binding_from(how, strict),
+            binding_from(how, strict, style),
             layout,
             spelling,
         )