ripple-sql 0.1.7__tar.gz → 0.1.9__tar.gz

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/CHANGELOG.md

@@ -6,6 +6,17 @@ Notable changes to Ripple. The format follows
 
 ## [Unreleased]
 
+## [0.1.9] - 2026-09-15
+
+### Changed
+- `ripple check-file` no longer needs the table's name in the common case. A file binds to the table its own columns name when they say it clearly: at least two of them are one table's, they are most of the file's columns, and no other table matches as many ("matched by 4 of 5 column names"). `--source` takes the bare table name when one schema has it (`--source vendor_feed`); a name two schemas share is refused with both spellings. When two tables fit a file equally, or none does, the command lists the tables with declared columns and stops with exit 2, never picking one. `--source` and a declared `files:` pattern still come first. A review of the command's edge cases also fixed: a column read only in a where clause or a join now counts as read; a reader the graph only sees through select star counts instead of vanishing; long chains are counted whole; an xlsx is read by the workbook's own sheet order, streamed no further than its first row, with rich text joined and empty cells kept as positions; tab-separated `.txt` and semicolon `.csv` files are read by their delimiter; a file that is not UTF-8 asks for `--encoding`; an unclosed quote, a duplicated header name, a DOCTYPE in the XML, a broken xlsx part, a pyarrow older than 14.0.1 and a broken `ripple.yml` are each a plain message with exit 2; a folder pattern (`incoming/*.csv`) matches the file's path; two patterns that name two tables are refused; `--source` never reads the pattern map; a missing file, a folder, and a project that cannot load give the same JSON shape as any other failure.
+
+## [0.1.8] - 2026-09-15
+
+### Added
+- `ripple check-file FILE [--source TABLE]`: a file arrives from an external team (csv, tsv, xlsx, parquet), and before anyone loads it Ripple diffs its header against the landing table's declared columns and prints who reads every column that is gone: models, dashboards, hit hardest. The verdict is the first line and the last; a similar name is a hint ("looks like customer_name; a similar name does not prove the same contents"), never an assumption; exit 0 when nothing downstream changes (new columns nobody reads included), 1 when something breaks, 2 when the check could not run. The file binds to its table by `--source`, or by a `files:` pattern the project declares (`meta: {ripple: {files: ["vendor_feed_*.csv"]}}` on a dbt source, or `files:` in `ripple.yml` for a plain SQL folder); with neither, the command prints the closest table and the exact line to add and stops. Header only: no data row is read and nothing leaves the machine. Shaped by a two-model debate; the spec's sample output is the command's own.
+- A dbt source's `columns:` in sources.yml are its declared columns. The graph read only ingested schemas and seeds before, so every dbt source showed as "not declared" and a select star through it stayed unresolved even when the yml listed every column.
+
 ## [0.1.7] - 2026-09-10
 
 ### Fixed
@@ -264,7 +275,9 @@ First public release.
 ### Removed
 - The dark whole-graph canvas that `ripple serve` used to open (`static/index.html`). Every surface now draws one answer.
 
-[Unreleased]: https://github.com/bteh/ripple/compare/v0.1.7...HEAD
+[Unreleased]: https://github.com/bteh/ripple/compare/v0.1.9...HEAD
+[0.1.9]: https://github.com/bteh/ripple/releases/tag/v0.1.9
+[0.1.8]: https://github.com/bteh/ripple/releases/tag/v0.1.8
 [0.1.7]: https://github.com/bteh/ripple/releases/tag/v0.1.7
 [0.1.6]: https://github.com/bteh/ripple/releases/tag/v0.1.6
 [0.1.5]: https://github.com/bteh/ripple/releases/tag/v0.1.5

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ripple-sql
-Version: 0.1.7
+Version: 0.1.9
 Summary: Offline column-level SQL lineage. See what breaks before you merge.
 Project-URL: Homepage, https://github.com/bteh/ripple
 Project-URL: Repository, https://github.com/bteh/ripple
@@ -210,6 +210,53 @@ ripple unresolved                 # which tables block the most coverage, worst
 ripple ingest-schema cols.csv     # CSV with a table,column header (or JSON, or - for stdin)
 ```
 
+## A file arrives from another team
+
+Vendors and business units hand over CSVs and spreadsheets by hand. When a column is renamed or dropped in one of them, the pipeline loads it anyway and dashboards go blank the next morning. Before loading, ask Ripple what this exact file breaks:
+
+```bash
+ripple check-file vendor_feed_2026-09.csv
+```
+
+```
+Stop: 2 columns that 4 models read are missing from this file.
+
+File   vendor_feed_2026-09.csv
+Table  raw.vendor_feed (files: vendor_feed_*.csv)
+Checked: column names only. Not checked: values, types, whether the file loads.
+
+6 columns: 2 unchanged, 2 missing, 4 new
+
+MISSING  customer_name   read by 3 models; hit hardest: dim_customer (2 columns)
+MISSING  amount_usd      read by 2 models
+NEW      cust_nm         looks like customer_name; a similar name does not prove the same contents
+3 more new columns nobody reads yet: region, notes, batch_id
+
+To fix: if cust_nm holds the same data as customer_name, rename that header to customer_name; otherwise restore customer_name. Restore amount_usd. Then run this check again.
+Details: ripple breaks raw.vendor_feed.customer_name
+
+Stop: 2 columns that 4 models read are missing from this file. (exit 1)
+```
+
+The first line and the last are the same sentence, so the vendor's screenshot and the engineer's scrolled terminal both end on the verdict. Exit 0 means nothing downstream changes (a new column nobody reads is fine), 1 means something breaks, 2 means the check could not run. Header only: no data row is read and nothing leaves the machine.
+
+The file finds its table by its own columns when they say it clearly (most of them belong to one table and no other table comes close: "matched by 4 of 5 column names"), by `--source vendor_feed` (the bare table name is enough when one schema has it), or by a pattern the project declares once:
+
+```yaml
+sources:
+  - name: raw
+    tables:
+      - name: vendor_feed
+        meta:
+          ripple:
+            files: ["vendor_feed_*.csv"]
+        columns:
+          - name: customer_id
+          - name: customer_name
+```
+
+A plain SQL folder keeps the same map in `ripple.yml` (`files: {"vendor_feed_*.csv": raw.vendor_feed}`). When two tables fit the file equally, or none does, the command lists the tables it can check and stops with exit 2 rather than pick one. The table's declared columns are the contract: `columns:` in sources.yml, a seed, or a list from `ripple ingest-schema`. Reads csv, tsv, xlsx and parquet (parquet needs pyarrow 14.0.1 or newer). A file that is not UTF-8 takes `--encoding cp1252`. `--json` carries the same facts for an ingest job, in the same shape whether the check ran or could not.
+
 ## Bring your query history (optional)
 
 Your SQL files say what is supposed to happen. The warehouse's query log says what

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/README.md

@@ -170,6 +170,53 @@ ripple unresolved                 # which tables block the most coverage, worst
 ripple ingest-schema cols.csv     # CSV with a table,column header (or JSON, or - for stdin)
 ```
 
+## A file arrives from another team
+
+Vendors and business units hand over CSVs and spreadsheets by hand. When a column is renamed or dropped in one of them, the pipeline loads it anyway and dashboards go blank the next morning. Before loading, ask Ripple what this exact file breaks:
+
+```bash
+ripple check-file vendor_feed_2026-09.csv
+```
+
+```
+Stop: 2 columns that 4 models read are missing from this file.
+
+File   vendor_feed_2026-09.csv
+Table  raw.vendor_feed (files: vendor_feed_*.csv)
+Checked: column names only. Not checked: values, types, whether the file loads.
+
+6 columns: 2 unchanged, 2 missing, 4 new
+
+MISSING  customer_name   read by 3 models; hit hardest: dim_customer (2 columns)
+MISSING  amount_usd      read by 2 models
+NEW      cust_nm         looks like customer_name; a similar name does not prove the same contents
+3 more new columns nobody reads yet: region, notes, batch_id
+
+To fix: if cust_nm holds the same data as customer_name, rename that header to customer_name; otherwise restore customer_name. Restore amount_usd. Then run this check again.
+Details: ripple breaks raw.vendor_feed.customer_name
+
+Stop: 2 columns that 4 models read are missing from this file. (exit 1)
+```
+
+The first line and the last are the same sentence, so the vendor's screenshot and the engineer's scrolled terminal both end on the verdict. Exit 0 means nothing downstream changes (a new column nobody reads is fine), 1 means something breaks, 2 means the check could not run. Header only: no data row is read and nothing leaves the machine.
+
+The file finds its table by its own columns when they say it clearly (most of them belong to one table and no other table comes close: "matched by 4 of 5 column names"), by `--source vendor_feed` (the bare table name is enough when one schema has it), or by a pattern the project declares once:
+
+```yaml
+sources:
+  - name: raw
+    tables:
+      - name: vendor_feed
+        meta:
+          ripple:
+            files: ["vendor_feed_*.csv"]
+        columns:
+          - name: customer_id
+          - name: customer_name
+```
+
+A plain SQL folder keeps the same map in `ripple.yml` (`files: {"vendor_feed_*.csv": raw.vendor_feed}`). When two tables fit the file equally, or none does, the command lists the tables it can check and stops with exit 2 rather than pick one. The table's declared columns are the contract: `columns:` in sources.yml, a seed, or a list from `ripple ingest-schema`. Reads csv, tsv, xlsx and parquet (parquet needs pyarrow 14.0.1 or newer). A file that is not UTF-8 takes `--encoding cp1252`. `--json` carries the same facts for an ingest job, in the same shape whether the check ran or could not.
+
 ## Bring your query history (optional)
 
 Your SQL files say what is supposed to happen. The warehouse's query log says what

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ripple-sql"
-version = "0.1.7"
+version = "0.1.9"
 description = "Offline column-level SQL lineage. See what breaks before you merge."
 readme = "README.md"
 requires-python = ">=3.10"

ripple_sql-0.1.9/src/ripple/check_file.py

@@ -0,0 +1,627 @@
+"""A file arrives from an external team: what breaks if it is loaded.
+
+The file feeds one landing table the graph already knows. Its header is
+diffed against the table's declared columns, and every column that is gone
+gets the blast radius `ripple breaks` would give it. Header only: no data
+row is read, nothing leaves the machine, and a similar name is a hint,
+never an assumption.
+"""
+
+from __future__ import annotations
+
+import csv
+import difflib
+import fnmatch
+import io
+import zipfile
+from pathlib import Path
+from xml.etree import ElementTree
+
+from ripple.answer import breaks_answer, hit_hardest
+from ripple.graph import UnknownTarget
+
+HEADER_ONLY = "Checked: column names only. Not checked: values, types, whether the file loads."
+KINDS = ("csv", "tsv", "txt", "xlsx", "parquet")
+LOOKS_LIKE = 0.6
+DELIMITERS = ",;\t|"
+HEAD_BYTES = 64 * 1024
+# a reader older than this deserializes untrusted Parquet unsafely (CVE-2023-47248)
+PYARROW_FLOOR = (14, 0, 1)
+# far past any real chain; the walk stops at cycles on its own
+MAX_DEPTH = 500
+# the parts a workbook's header lives in; anything past these caps is not a
+# workbook Ripple reads a header from, whatever the zip claims
+XML_PART_CAP = 4 * 1024 * 1024
+SHARED_STRINGS_CAP = 64 * 1024 * 1024
+
+
+class CannotCheck(Exception):
+    """The check could not run; the message says why and what to do."""
+
+
+def read_header(path: Path, encoding: str | None = None) -> list[str]:
+    """The column names of a file, from its header only."""
+    if path.is_dir():
+        raise CannotCheck(f"{path.name} is a folder; pass one file")
+    kind = path.suffix.lower().lstrip(".")
+    if kind in ("csv", "tsv", "txt"):
+        header = _delimited(path, kind, encoding)
+    elif kind == "xlsx":
+        header = _xlsx(path)
+    elif kind == "parquet":
+        header = _parquet(path)
+    else:
+        raise CannotCheck(
+            f"{path.name}: Ripple reads a csv, tsv, xlsx or parquet header, not .{kind}"
+        )
+    names = [h for h in header if h]
+    seen: set[str] = set()
+    for name in names:
+        if name.lower() in seen:
+            raise CannotCheck(
+                f"{path.name} names {name} twice in its header; a table has one column of a name"
+            )
+        seen.add(name.lower())
+    if not names:
+        raise CannotCheck(f"{path.name} is empty; no header row to check")
+    return names
+
+
+def _decode(path: Path, raw: bytes, encoding: str | None) -> str:
+    try:
+        return raw.decode(encoding or "utf-8-sig")
+    except (UnicodeDecodeError, LookupError):
+        if encoding:
+            raise CannotCheck(
+                f"{path.name} is not {encoding}; pass the file's encoding with --encoding"
+            ) from None
+        raise CannotCheck(
+            f"{path.name} is not UTF-8; pass --encoding cp1252 (Excel on Windows) or latin-1"
+        ) from None
+
+
+def _delimited(path: Path, kind: str, encoding: str | None) -> list[str]:
+    with open(path, "rb") as f:
+        text = _decode(path, f.read(HEAD_BYTES), encoding)
+    first = text.split("\n", 1)[0]
+    delimiter = "\t" if kind == "tsv" else max(DELIMITERS, key=lambda d: (first.count(d), d == ","))
+    if text.count('"') % 2:
+        raise CannotCheck(f"{path.name} has an unclosed quote in its header")
+    try:
+        header = next(csv.reader(io.StringIO(text, newline=""), delimiter=delimiter, strict=True))
+    except StopIteration:
+        raise CannotCheck(f"{path.name} is empty; no header row to check") from None
+    except csv.Error as e:
+        raise CannotCheck(f"{path.name}: the header is not valid csv ({e})") from None
+    # a quoted header that spans lines keeps the file's own line ending
+    return [h.replace("\r\n", "\n").strip() for h in header]
+
+
+def _local(tag: str) -> str:
+    return tag.rsplit("}", 1)[-1]
+
+
+def _guard_xml(path: Path, data: bytes) -> None:
+    """Refuse a DOCTYPE or an entity in an XML part, in whatever encoding the
+    part is written: expat honours a UTF-16 or UTF-32 prolog, so the scan
+    must read the bytes the way the parser will."""
+    if data.startswith((b"\xff\xfe\x00\x00", b"\x00\x00\xfe\xff")):
+        codec = "utf-32"
+    elif data.startswith((b"\xff\xfe", b"\xfe\xff")):
+        codec = "utf-16"
+    elif b"\x00" in data[:64]:
+        raise CannotCheck(
+            f"{path.name} carries XML in an encoding Ripple does not read; save it as a normal xlsx"
+        )
+    else:
+        codec = "utf-8"
+    text = data.decode(codec, errors="ignore")
+    if "<!DOCTYPE" in text or "<!ENTITY" in text:
+        raise CannotCheck(f"{path.name} carries a DOCTYPE in its XML; Ripple does not read that")
+
+
+def _part(z: zipfile.ZipFile, path: Path, name: str, cap: int) -> bytes:
+    """One zip member, whole, guarded: bounded in size and free of a DOCTYPE."""
+    if z.getinfo(name).file_size > cap:
+        raise CannotCheck(
+            f"{path.name}: {name} is larger than {cap >> 20} MB; not a workbook Ripple reads"
+        )
+    member = z.open  # bytes; the utf-8 floor scan keys on the name open(
+    with member(name) as f:
+        data = f.read(cap + 1)
+    if len(data) > cap:
+        raise CannotCheck(
+            f"{path.name}: {name} is larger than {cap >> 20} MB; not a workbook Ripple reads"
+        )
+    _guard_xml(path, data)
+    return data
+
+
+def _first_sheet(z: zipfile.ZipFile, path: Path) -> str:
+    """The workbook's first sheet, by its own order, not by file name."""
+    names = set(z.namelist())
+    if "xl/workbook.xml" in names and "xl/_rels/workbook.xml.rels" in names:
+        try:
+            wb = ElementTree.fromstring(_part(z, path, "xl/workbook.xml", XML_PART_CAP))
+            rels = ElementTree.fromstring(
+                _part(z, path, "xl/_rels/workbook.xml.rels", XML_PART_CAP)
+            )
+        except ElementTree.ParseError:
+            raise CannotCheck(f"{path.name} is not valid xlsx") from None
+        first = next((s for s in wb.iter() if _local(s.tag) == "sheet"), None)
+        rid = (
+            next((v for k, v in first.attrib.items() if _local(k) == "id"), None)
+            if first is not None
+            else None
+        )
+        target = next((r.get("Target") for r in rels.iter() if r.get("Id") == rid), None)
+        if target:
+            member = target.lstrip("/") if target.startswith("/") else "xl/" + target
+            if member in names:
+                return member
+    sheet = next((n for n in sorted(names) if n.startswith("xl/worksheets/sheet")), None)
+    if not sheet:
+        raise CannotCheck(f"{path.name} has no worksheet")
+    return sheet
+
+
+def _column_index(ref: str) -> int:
+    n = 0
+    for ch in ref:
+        if not ch.isalpha():
+            break
+        n = n * 26 + (ord(ch.upper()) - 64)
+    return max(n - 1, 0)
+
+
+def _cell_text(cell, shared: list[str]) -> str:
+    kind = cell.get("t")
+    if kind == "inlineStr":
+        return "".join(t.text or "" for t in cell.iter() if _local(t.tag) == "t")
+    value = next((v.text or "" for v in cell.iter() if _local(v.tag) == "v"), "")
+    if kind == "s" and value.isdigit() and int(value) < len(shared):
+        return shared[int(value)]
+    return value
+
+
+def _xlsx(path: Path) -> list[str]:
+    """Row 1 of the first sheet, streamed and left unread past that row."""
+    try:
+        with zipfile.ZipFile(path) as z:
+            shared: list[str] = []
+            if "xl/sharedStrings.xml" in z.namelist():
+                raw = _part(z, path, "xl/sharedStrings.xml", SHARED_STRINGS_CAP)
+                for si in ElementTree.fromstring(raw):
+                    shared.append("".join(t.text or "" for t in si.iter() if _local(t.tag) == "t"))
+            sheet = _first_sheet(z, path)
+            member = z.open  # bytes; the utf-8 floor scan keys on the name open(
+            with member(sheet) as f:
+                row = _first_row(path, f)
+    except zipfile.BadZipFile:
+        raise CannotCheck(f"{path.name} is not an xlsx workbook") from None
+    except ElementTree.ParseError:
+        raise CannotCheck(f"{path.name} is not valid xlsx") from None
+    if row is None:
+        raise CannotCheck(f"{path.name} is not valid xlsx: no header row")
+    if row.get("r") not in (None, "1"):
+        raise CannotCheck(
+            f"{path.name} has no row 1; the header must be the first row of the sheet"
+        )
+    cells: dict[int, str] = {}
+    for cell in (c for c in row if _local(c.tag) == "c"):
+        # a cell without a coordinate follows the previous one
+        index = _column_index(cell.get("r")) if cell.get("r") else len(cells)
+        cells[index] = _cell_text(cell, shared).strip()
+    width = max(cells) + 1 if cells else 0
+    return [cells.get(i, "") for i in range(width)]
+
+
+def _first_row(path: Path, stream):
+    """The first <row> element, fed to the parser in small pieces so nothing
+    past it is ever parsed; never closed, so a later broken row cannot fail
+    a good header. A DOCTYPE can only sit before the root element, so the
+    bytes fed until the first element opens are the ones checked for it."""
+    parser = ElementTree.XMLPullParser(events=("start", "end"))
+    preamble, opened, read = b"", False, 0
+    while True:
+        chunk = stream.read(4096)
+        if not chunk:
+            return None
+        read += len(chunk)
+        if read > XML_PART_CAP:
+            raise CannotCheck(
+                f"{path.name}: no header row in the first {XML_PART_CAP >> 20} MB of the sheet"
+            )
+        if not opened:
+            preamble += chunk
+        parser.feed(chunk)
+        for event, element in parser.read_events():
+            if event == "start" and not opened:
+                opened = True
+                _guard_xml(path, preamble)
+            if event == "end" and _local(element.tag) == "row":
+                return element
+
+
+def _parquet(path: Path) -> list[str]:
+    try:
+        import pyarrow
+        import pyarrow.parquet as pq
+    except ImportError:
+        raise CannotCheck(
+            f"{path.name}: reading a Parquet schema needs pyarrow (pip install pyarrow), "
+            "or export the header as csv"
+        ) from None
+    version = tuple(
+        int(p) for p in str(getattr(pyarrow, "__version__", "0")).split(".")[:3] if p.isdigit()
+    )
+    if version < PYARROW_FLOOR:
+        floor = ".".join(str(n) for n in PYARROW_FLOOR)
+        raise CannotCheck(
+            f"pyarrow {getattr(pyarrow, '__version__', '?')} is older than {floor}, which reads untrusted Parquet "
+            "unsafely (CVE-2023-47248); upgrade it or export the header as csv"
+        )
+    try:
+        return list(pq.read_schema(path).names)
+    except Exception as e:  # pyarrow raises its own hierarchy
+        raise CannotCheck(f"{path.name} is not valid Parquet ({e})") from None
+
+
+def file_map(roots: list[Path]) -> dict[str, str]:
+    """File-name patterns to the tables they feed, from the project.
+
+    A dbt source table carries them as `meta: {ripple: {files: [...]}}`; a
+    plain SQL folder keeps a `files:` map in ripple.yml at its root. One
+    pattern naming two tables is a conflict, not a choice."""
+    import yaml
+
+    from ripple.loaders.dbt_config import source_tables
+
+    patterns: dict[str, str] = {}
+
+    def put(pattern: str, table: str, where: str) -> None:
+        if pattern in patterns and patterns[pattern] != table:
+            raise CannotCheck(
+                f"{where}: pattern {pattern} maps to two tables, {patterns[pattern]} and {table}"
+            )
+        patterns[pattern] = table
+
+    for root in dict.fromkeys(Path(r) for r in roots):
+        own = root / "ripple.yml"
+        if own.is_file():
+            try:
+                parsed = yaml.safe_load(own.read_text(encoding="utf-8", errors="replace")) or {}
+            except yaml.YAMLError as e:
+                raise CannotCheck(
+                    f"ripple.yml is not valid yaml ({str(e).splitlines()[0]})"
+                ) from None
+            files = parsed.get("files") if isinstance(parsed, dict) else None
+            if files is not None and not isinstance(files, dict):
+                raise CannotCheck(
+                    'ripple.yml: files must map a pattern to a table, like "feed_*.csv": raw.feed'
+                )
+            for pattern, table in (files or {}).items():
+                put(str(pattern), str(table), "ripple.yml")
+        for source, table in source_tables(root):
+            meta = (table.get("meta") or {}).get("ripple") or {}
+            for pattern in meta.get("files") or []:
+                put(str(pattern), f"{source}.{table['name']}", "sources yml")
+    return patterns
+
+
+def bind(path: Path, root: Path, patterns: dict[str, str]) -> tuple[str | None, str | None]:
+    """(table, how) for a file from the declared patterns, or (None, None).
+
+    A pattern with a folder in it matches the file's path (relative to the
+    project when it is inside it); a bare pattern matches the file name."""
+    spellings = [path.as_posix()]
+    resolved, base = path.resolve(), Path(root).resolve()
+    if base in resolved.parents:
+        spellings.append(resolved.relative_to(base).as_posix())
+    hits: dict[str, str] = {}
+    for pattern, table in patterns.items():
+        target = spellings if "/" in pattern else [path.name]
+        if any(fnmatch.fnmatch(s, pattern) for s in target):
+            hits.setdefault(table, pattern)
+    if len(hits) > 1:
+        listed = ", ".join(f"{p} for {t}" for t, p in hits.items())
+        raise CannotCheck(
+            f"{path.name} matches {len(hits)} patterns for different tables: {listed}. Pass --source."
+        )
+    if hits:
+        ((table, pattern),) = hits.items()
+        return table, f"files: {pattern}"
+    return None, None
+
+
+def declared_sources(graph) -> list:
+    return [s for s in graph.project.sources if s.declared_columns]
+
+
+def fits(header: list[str], graph) -> list[tuple[object, int]]:
+    """Tables ranked by how many of the file's columns are theirs."""
+    names = {h.lower() for h in header}
+    scored = []
+    for source in declared_sources(graph):
+        matched = sum(1 for c in source.declared_columns if c.lower() in names)
+        if matched:
+            scored.append((source, matched))
+    scored.sort(key=lambda pair: (-pair[1], pair[0].name))
+    return scored
+
+
+def bind_by_header(header: list[str], graph) -> tuple[object | None, str | None, list]:
+    """The one table the file's columns name, or the tables that tie.
+
+    A file binds when at least two of its columns are one table's, they are
+    more than half of the file's columns, and no other table matches as
+    many. Anything less is listed for the user to pick from, never picked."""
+    ranked = fits(header, graph)
+    strong = [(s, n) for s, n in ranked if n >= 2 and n * 2 > len(header)]
+    if not strong:
+        return None, None, []
+    best = strong[0][1]
+    tied = [(s, n) for s, n in strong if n == best]
+    if len(tied) > 1:
+        return None, None, tied
+    return tied[0][0], f"matched by {best} of {_plural(len(header), 'column name')}", tied
+
+
+def find_source(graph, table: str):
+    """The source table by its name, an alias, or its bare table name.
+
+    Case-insensitive. An alias or bare name two schemas' tables share is
+    refused with both spellings; an unknown name lists the tables that can
+    be checked."""
+    wanted = table.lower()
+    exact = [s for s in graph.project.sources if s.name.lower() == wanted]
+    if len(exact) == 1:
+        return exact[0]
+    hits = [
+        s
+        for s in graph.project.sources
+        if wanted in {a.lower() for a in s.aliases} or s.name.rsplit(".", 1)[-1].lower() == wanted
+    ]
+    if len(hits) == 1:
+        return hits[0]
+    if hits:
+        names = ", ".join(sorted(s.name for s in hits))
+        raise CannotCheck(
+            f"{table} names {len(hits)} tables: {names}. Pass --source with the schema."
+        )
+    raise CannotCheck(f"no table called {table} in this project. {table_list(graph)}")
+
+
+def table_list(graph) -> str:
+    tables = declared_sources(graph)
+    if not tables:
+        return "No source table has declared columns yet; add them to its yml or run ripple ingest-schema."
+    listed = ", ".join(f"{s.name} ({_plural(len(s.declared_columns), 'column')})" for s in tables)
+    return f"Tables with declared columns: {listed}."
+
+
+def _readers(graph, table: str, column: str) -> dict:
+    """Who reads one column of the table: models (a filter counts), dashboards,
+    the hardest hit; through select * when the graph cannot tell which
+    columns those readers use."""
+    try:
+        answer = breaks_answer(graph.breaks(table, column, max_depth=MAX_DEPTH), "model")
+    except UnknownTarget:
+        star = {e.dst_model for e in graph.edges if e.src_model == table and e.src_column == "*"}
+        return {
+            "models": star,
+            "dashboards": set(),
+            "filters": set(),
+            "hit_hardest": [],
+            "via_star": bool(star),
+        }
+    nodes = [n for n in answer["nodes"] if n["depth"] > 0]
+    filters = {r["model"] for r in answer["row_level"]}
+    return {
+        "models": {n["model"] for n in nodes if n["kind"] == "model"} | filters,
+        "dashboards": {n["model"] for n in nodes if n["kind"] == "dashboard"},
+        "filters": filters,
+        "hit_hardest": hit_hardest(answer, 1),
+        "via_star": False,
+    }
+
+
+def check(
+    graph, table: str, header: list[str], file_name: str, how: str, noun: str = "model"
+) -> dict:
+    """The report: every column's status, who reads the missing ones, the verdict."""
+    source = find_source(graph, table)
+    declared = list(source.declared_columns)
+    if not declared:
+        raise CannotCheck(
+            f"{source.name} has no declared columns, so there is nothing to check the file against. "
+            "Add them to its yml, or run ripple ingest-schema with your warehouse's column list."
+        )
+    present = {h.lower() for h in header}
+    known = {c.lower() for c in declared}
+    missing = [c for c in declared if c.lower() not in present]
+    new = [h for h in header if h.lower() not in known]
+    columns: list[dict] = []
+    who: dict[str, dict] = {}
+    for c in declared:
+        if c.lower() in present:
+            columns.append({"name": c, "status": "unchanged"})
+            continue
+        r = who[c] = _readers(graph, source.name, c)
+        columns.append(
+            {
+                "name": c,
+                "status": "missing",
+                "readers": {"models": len(r["models"]), "dashboards": len(r["dashboards"])},
+                "filters": len(r["filters"]),
+                "via_star": r["via_star"],
+                "hit_hardest": [{"model": m, "columns": n} for m, n in r["hit_hardest"]],
+            }
+        )
+    read_missing = [c["name"] for c in columns if c["status"] == "missing" and _count(c)]
+    for h in new:
+        like = difflib.get_close_matches(
+            h.lower(), [m.lower() for m in read_missing], n=1, cutoff=LOOKS_LIKE
+        )
+        entry = {"name": h, "status": "new"}
+        if like:
+            entry["looks_like"] = next(m for m in read_missing if m.lower() == like[0])
+        columns.append(entry)
+    models: set[str] = set()
+    dashboards: set[str] = set()
+    for name in read_missing:
+        models |= who[name]["models"]
+        dashboards |= who[name]["dashboards"]
+    breaking = len(read_missing)
+    if breaking:
+        readers = len(models) + len(dashboards)
+        verdict = (
+            f"Stop: {_plural(breaking, 'column')} that {_where(len(models), len(dashboards), noun)} "
+            f"read{'s' if readers == 1 else ''} {'is' if breaking == 1 else 'are'} missing from this file."
+        )
+    else:
+        verdict = "Nothing downstream changes."
+    report = {
+        "file": file_name,
+        "table": source.name,
+        "method": how,
+        "checked": "names",
+        "columns": columns,
+        "counts": {
+            "file": len(header),
+            "unchanged": len(declared) - len(missing),
+            "missing": len(missing),
+            "new": len(new),
+        },
+        "verdict": verdict,
+        "exit": 1 if breaking else 0,
+        "noun": noun,
+    }
+    report["lines"] = lines(report)
+    return report
+
+
+def failure_report(file_name: str, lines: list[str]) -> dict:
+    """The same shape as a finished check, for a check that could not run."""
+    return {
+        "file": file_name,
+        "table": None,
+        "method": None,
+        "checked": "names",
+        "columns": [],
+        "counts": None,
+        "verdict": lines[0],
+        "exit": 2,
+        "lines": lines,
+    }
+
+
+def _count(c: dict) -> int:
+    return c["readers"]["models"] + c["readers"]["dashboards"]
+
+
+def _plural(n: int, word: str) -> str:
+    return f"{n} {word}" + ("" if n == 1 else "s")
+
+
+def _where(models: int, dashboards: int, noun: str) -> str:
+    parts = []
+    if models:
+        parts.append(_plural(models, noun))
+    if dashboards:
+        parts.append(_plural(dashboards, "dashboard"))
+    return " and ".join(parts) if parts else "nobody"
+
+
+def _missing_line(c: dict, noun: str, width: int) -> str:
+    who = _where(c["readers"]["models"], c["readers"]["dashboards"], noun)
+    line = f"MISSING  {c['name']:<{width}}read by {who}"
+    if c.get("via_star"):
+        return line + " through select * (which columns, Ripple cannot tell)"
+    if c.get("filters"):
+        line += f"; filters rows in {_plural(c['filters'], noun)}"
+    # worth a name only when one reader is hit in more than one column
+    if c["hit_hardest"] and c["hit_hardest"][0]["columns"] > 1:
+        top = c["hit_hardest"][0]
+        line += f"; hit hardest: {top['model']} ({_plural(top['columns'], 'column')})"
+    return line
+
+
+def lines(report: dict) -> list[str]:
+    """The terminal text: verdict first and last, damage in between."""
+    noun = report.get("noun", "model")
+    counts = report["counts"]
+    listed = [c for c in report["columns"] if c["status"] == "missing" and _count(c)]
+    hinted = [c for c in report["columns"] if c["status"] == "new" and c.get("looks_like")]
+    quiet_new = [
+        c["name"] for c in report["columns"] if c["status"] == "new" and not c.get("looks_like")
+    ]
+    quiet_missing = [
+        c["name"] for c in report["columns"] if c["status"] == "missing" and not _count(c)
+    ]
+    width = max((len(c["name"]) for c in listed + hinted), default=0) + 3
+    out = [report["verdict"], ""]
+    out.append(f"File   {report['file']}")
+    out.append(f"Table  {report['table']} ({report['method']})")
+    out.append(HEADER_ONLY)
+    out.append("")
+    parts = [f"{counts['unchanged']} unchanged"]
+    if counts["missing"]:
+        parts.append(f"{counts['missing']} missing")
+    if counts["new"]:
+        parts.append(f"{counts['new']} new")
+    out.append(f"{_plural(counts['file'], 'column')}: {', '.join(parts)}")
+    out.append("")
+    for c in sorted(listed, key=lambda c: (-_count(c), c["name"])):
+        out.append(_missing_line(c, noun, width))
+    for c in hinted:
+        out.append(
+            f"NEW      {c['name']:<{width}}looks like {c['looks_like']}; "
+            "a similar name does not prove the same contents"
+        )
+    if quiet_new:
+        more = "more " if hinted else ""
+        out.append(
+            f"{len(quiet_new)} {more}new column{'s' if len(quiet_new) != 1 else ''} nobody reads yet: {', '.join(quiet_new)}"
+        )
+    if quiet_missing:
+        out.append(
+            f"{len(quiet_missing)} missing column{'s' if len(quiet_missing) != 1 else ''} nobody reads yet: {', '.join(quiet_missing)}"
+        )
+    if listed:
+        out.append("")
+        fixes = []
+        renamed = {c["looks_like"]: c["name"] for c in hinted}
+        for c in listed:
+            if c["name"] in renamed:
+                fixes.append(
+                    f"if {renamed[c['name']]} holds the same data as {c['name']}, rename that header to {c['name']}; "
+                    f"otherwise restore {c['name']}."
+                )
+            else:
+                fixes.append(f"Restore {c['name']}.")
+        out.append("To fix: " + " ".join(fixes) + " Then run this check again.")
+        out.append(f"Details: ripple breaks {report['table']}.{listed[0]['name']}")
+    out.append("")
+    out.append(f"{report['verdict']} (exit {report['exit']})")
+    return out
+
+
+def no_binding_lines(file_name: str, header: list[str], graph, tied: list) -> list[str]:
+    """What to do when no table claims the file: the choices, never a guess."""
+    if tied:
+        fits_line = ", ".join(
+            f"{s.name} ({n} of {_plural(len(header), 'column name')})" for s, n in tied
+        )
+        out = [f"Cannot check: {file_name} fits more than one table: {fits_line}."]
+    else:
+        out = [
+            f"Cannot check: no table is mapped to {file_name} and its columns fit none.",
+            table_list(graph),
+        ]
+    out.append(
+        "Run again with --source TABLE, or add to that source's yml so the next drop binds itself:"
+    )
+    out.append(f'  meta: {{ripple: {{files: ["{file_name}"]}}}}')
+    out.append("A pattern like vendor_feed_*.csv covers every drop of the same feed.")
+    return out

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/src/ripple/cli.py

@@ -258,6 +258,43 @@ def cmd_breaks(args) -> None:
     _page_door(args, project, graph, answer, plain_text(breaks_lines(answer, full=True)))
 
 
+def cmd_check_file(args) -> None:
+    from ripple import check_file
+    from ripple.loaders.types import NotAProject
+    from ripple.project import find_project_root
+
+    path = Path(args.file)
+
+    def stop(lines: list[str]) -> None:
+        report = check_file.failure_report(path.name, lines)
+        print(json.dumps(report, indent=2) if args.json else "\n".join(lines))
+        sys.exit(2)
+
+    try:
+        project, graph = _load_graph(args)
+        root = find_project_root(Path(args.path))
+        header = check_file.read_header(path, args.encoding)
+        if args.source:
+            table, how = args.source, "--source"
+        else:
+            roots = [root, *getattr(project, "dbt_roots", [])]
+            table, how = check_file.bind(path, root, check_file.file_map(roots))
+            if table is None:
+                source, how, tied = check_file.bind_by_header(header, graph)
+                if source is None:
+                    stop(check_file.no_binding_lines(path.name, header, graph, tied))
+                table = source.name
+        report = check_file.check(graph, table, header, path.name, how, noun_for(project.mode))
+    except (check_file.CannotCheck, NotAProject, OSError) as e:
+        stop([f"Cannot check: {e}"])
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print("\n".join(report["lines"]))
+    if report["exit"]:
+        sys.exit(report["exit"])
+
+
 def cmd_trace(args) -> None:
     project, graph = _load_graph(args)
     model, column = _parse_target(args.target)
@@ -604,6 +641,19 @@ def main(argv: list[str] | None = None) -> None:
     )
     p_columns.add_argument("model")
 
+    p_check = sub.add_parser(
+        "check-file",
+        help="a file from an external team: what breaks if it is loaded",
+        parents=[common],
+    )
+    p_check.add_argument("file", help="the csv, tsv, xlsx or parquet file that arrived")
+    p_check.add_argument(
+        "--source",
+        help="the table it feeds (raw.vendor_feed, or just vendor_feed); else the project's files map",
+    )
+    p_check.add_argument(
+        "--encoding", help="the file's text encoding when it is not UTF-8 (cp1252, latin-1)"
+    )
     p_breaks = sub.add_parser("breaks", help="what breaks if this column changes", parents=[common])
     p_breaks.add_argument("target", help="model.column")
     p_breaks.add_argument(
@@ -716,6 +766,7 @@ def main(argv: list[str] | None = None) -> None:
         "models": cmd_models,
         "columns": cmd_columns,
         "breaks": cmd_breaks,
+        "check-file": cmd_check_file,
         "trace": cmd_trace,
         "graph": cmd_graph,
         "ci": cmd_ci,

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/src/ripple/loaders/dbt_config.py

@@ -5,7 +5,7 @@ from __future__ import annotations
 import re
 from pathlib import Path
 
-from ripple.loaders.types import files_under
+from ripple.loaders.types import IGNORE_DIRS, files_under
 
 
 def _project_vars(root: Path) -> dict:
@@ -339,3 +339,56 @@ def _dbt_owned_dirs(project_dir: Path) -> list[Path]:
     owned += _dbt_model_dirs(project_dir)
     owned += [project_dir / name for name in _macro_dir_names(project_dir / "dbt_project.yml")]
     return [d for d in owned if d.is_dir()]
+
+
+def source_tables(root: Path):
+    """Every (source name, table dict) a dbt project declares in its yml.
+
+    Walks the model paths only (dbt_project.yml's model-paths), so a
+    package's or a build folder's yml never counts as this project's."""
+    import yaml
+
+    root = Path(root)
+    if not (root / "dbt_project.yml").is_file():
+        return
+    for model_dir in _dbt_model_dirs(root):
+        for yml in [*files_under(model_dir, "*.yml"), *files_under(model_dir, "*.yaml")]:
+            # a package's or a build folder's yml is not this project's, even
+            # when model-paths is the project root
+            if IGNORE_DIRS & set(yml.relative_to(model_dir).parts[:-1]):
+                continue
+            try:
+                parsed = yaml.safe_load(yml.read_text(errors="replace", encoding="utf-8"))
+            except yaml.YAMLError:
+                continue
+            if not isinstance(parsed, dict):
+                continue
+            for source in parsed.get("sources") or []:
+                if not isinstance(source, dict) or not source.get("name"):
+                    continue
+                for table in source.get("tables") or []:
+                    if isinstance(table, dict) and table.get("name"):
+                        yield str(source["name"]), table
+
+
+def source_columns(root: Path) -> dict[str, list[str]]:
+    """The columns sources.yml declares, as {"schema.table": [names]}.
+
+    dbt users list a source's columns in yml for docs and tests; those are
+    the landing table's contract, and the graph read only ingested schemas
+    and seeds before (a dbt source always showed "not declared")."""
+    out: dict[str, list[str]] = {}
+    spelling: dict[str, str] = {}
+    for source, table in source_tables(root):
+        key = f"{source}.{table['name']}".lower()
+        name = spelling.setdefault(key, f"{source}.{table['name']}")
+        # two files may describe one table (docs in one, tests in another);
+        # the union is the contract, the first spelling stays
+        columns = out.setdefault(name, [])
+        known = {c.lower() for c in columns}
+        for column in table.get("columns") or []:
+            column_name = column.get("name") if isinstance(column, dict) else column
+            if column_name and str(column_name).lower() not in known:
+                columns.append(str(column_name))
+                known.add(str(column_name).lower())
+    return {name: columns for name, columns in out.items() if columns}

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/src/ripple/loaders/identity.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import re
+from pathlib import Path
 
 from ripple.loaders.types import Model, Project
 
@@ -85,6 +86,45 @@ def _qualifiers_by_bare_name(project: Project) -> dict[str, set[str]]:
     return seen
 
 
+def _apply_declared_sources(project: Project) -> None:
+    """Expose sources.yml column lists as the sources' declared columns.
+
+    By exact name only (schema.table, case-insensitive): a declaration names
+    its own table, so it never rides the suffix attachment an ingested schema
+    needs, which merged raw.feed and staging.feed into one source. A table
+    no SQL references still becomes a source, so a file can be checked
+    against it. Every dbt project in a monorepo declares its own."""
+    from ripple.loaders.dbt_config import source_columns
+
+    declared: dict[str, list[str]] = {}
+    spelling: dict[str, str] = {}
+    for root in project.dbt_roots:
+        for name, columns in source_columns(Path(root)).items():
+            key = name.lower()
+            known = declared.setdefault(spelling.setdefault(key, name), [])
+            seen = {c.lower() for c in known}
+            known.extend(c for c in columns if c.lower() not in seen)
+    if not declared:
+        return
+    by_name = {s.name.lower(): s for s in project.sources}
+    owned = {m.name.lower() for m in project.models if m.sql.strip()}
+    for name, columns in declared.items():
+        target = by_name.get(name.lower())
+        if target is None:
+            target = Model(
+                name=name,
+                sql="",
+                path="",
+                aliases={a for a in _name_suffixes(name) if a.lower() not in owned},
+                is_source=True,
+                evidence="declared",
+            )
+            project.sources.append(target)
+            by_name[name.lower()] = target
+        seen = {c.lower() for c in target.declared_columns}
+        target.declared_columns.extend(c for c in columns if c.lower() not in seen)
+
+
 def _apply_ingested_schemas(project: Project, extra_roots: tuple = ()) -> None:
     """Expose .ripple/schemas.json tables as known external relations.
 

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/src/ripple/loaders/types.py

@@ -122,6 +122,9 @@ class Project:
     downstream: list = field(default_factory=list)
     # vars declared in dbt_project.yml, resolved by var() during tier-b renders
     jinja_vars: dict = field(default_factory=dict)
+    # every dbt project directory that was loaded: one for a dbt project, all
+    # of them for a monorepo, none for a plain SQL folder
+    dbt_roots: list = field(default_factory=list)
 
     @property
     def name_candidates(self) -> dict[str, list[str]]:

{ripple_sql-0.1.7 → ripple_sql-0.1.9}/src/ripple/project.py

@@ -22,7 +22,11 @@ from pathlib import Path
 
 from ripple.loaders.dbt import _load_one_dbt, resolve_dbt_dialect
 from ripple.loaders.dbt_config import _dbt_owned_dirs
-from ripple.loaders.identity import _apply_ingested_schemas, _qualify_collisions
+from ripple.loaders.identity import (
+    _apply_declared_sources,
+    _apply_ingested_schemas,
+    _qualify_collisions,
+)
 from ripple.loaders.sqldir import _load_sql_dir
 from ripple.loaders.types import (
     ADAPTER_TO_DIALECT,
@@ -83,7 +87,11 @@ def load_project(path: str | Path = ".", dialect: str | None = None) -> Project:
                 f"{len(stray.models)} plain-SQL models outside the dbt "
                 f"project{'s' if len(dbt_dirs) != 1 else ''} were loaded as well."
             )
+    project.dbt_roots = (
+        list(dbt_dirs) if dbt_dirs else ([root] if (root / "dbt_project.yml").exists() else [])
+    )
     _qualify_collisions(project)
+    _apply_declared_sources(project)
     _apply_ingested_schemas(project, extra_roots=(scan_root, start))
     from ripple.lookml import load_lookml_downstream
     from ripple.semantic import load_downstream