coop-data-doc 0.15.0__tar.gz

coop_data_doc-0.15.0/.github/workflows/ci.yml

@@ -0,0 +1,33 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install ruff
+      - run: ruff check src tests
+      - run: ruff format --check src tests
+
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q

coop_data_doc-0.15.0/.github/workflows/publish.yml

@@ -0,0 +1,38 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - name: Smoke-test the wheel
+        run: |
+          python -m venv /tmp/smoke
+          /tmp/smoke/bin/pip install dist/*.whl
+          /tmp/smoke/bin/coop-data-doc --version
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # PyPI trusted publishing
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

coop_data_doc-0.15.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.pyc
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/

coop_data_doc-0.15.0/ARCHITECTURE.md

@@ -0,0 +1,167 @@
+# Architecture
+
+How `coop-data-doc` turns two git repos into lineage documentation. Read this
+before changing code; `CONTRIBUTING.md` has the rules, this file has the *why*
+and the *how*.
+
+## The pipeline
+
+```
+coop-data-doc.yml ──► [crawler] ──► FileInventory (classified files)
+                           │
+             ┌─────────────┴───────────────┐
+             ▼                             ▼
+   [SQL parsers (M2)]            [Power BI parsers (M3)]
+   sql_objects: CREATE           tmdl: tables/columns/measures/partitions
+     TABLE/VIEW + columns        bim:  same, from model.bim JSON
+   sql_procs: proc DML           mcode: partition M ──► SourceRef
+     (INSERT/MERGE/UPDATE/       dax:  measure ──► measure/table refs
+      SELECT INTO/EXEC)          pbir: report/page/visual + field bindings
+                                       (PBIR folders AND legacy report.json)
+                                 pbix: best-effort zip extraction
+             │                             │
+             ▼                             ▼
+   resolve_stub_references()      link_visual_bindings()
+             └──────────────┬───────────────┘
+                 prune_schemas() — drop system/ignored schemas
+                 assign_layers() — bronze/silver/gold (rules + heuristic)
+                            │
+                     [linker (M4)]
+                cache → exact → config rule → fuzzy → interactive prompt
+                answers persist to .lineage-cache.json (commit it)
+                            │
+                            ▼
+                  LineageGraph  ── graph.json + diagnostics.json (artifacts)
+                            │
+             ┌──────────────┴──────────────┐
+             ▼                             ▼
+  [render/markdown (M5)]         [render/site (M5)]
+  per-node .md + index.md        MkDocs Material, dark default,
+  + manifest.json (for agents)   offline search + vendored mermaid
+```
+
+`cli.run_pipeline()` (`src/coop_data_doc/cli.py`) is the executable version of
+this diagram — read it first when tracing behavior.
+
+## The data model (everything flows through one graph)
+
+`src/coop_data_doc/graph/model.py`:
+
+- **Node** — `id`, `node_type`, `name`, `schema_name`, `source_file`,
+  `columns`, free-form `metadata`. Ids are stable slugs:
+  `"{type}:{schema}.{name}"`, lowercased, brackets stripped
+  (`[dbo].[Fact Sales]` → `gold_table:dbo.fact sales`). For Power BI nodes,
+  `schema_name` holds the normalized semantic-model name.
+- **NodeType** — `silver_table, gold_table, view, stored_proc,
+  semantic_model, pbi_table, measure, report, report_page, visual`.
+- **Edge** — `source_id`, `target_id`, `edge_type`, `evidence`
+  (a `"file: snippet"` string proving the edge — every edge is auditable).
+- **Edge direction.** Edges are *authored* in the parser-natural direction,
+  which is not always the data-flow direction. `Edge.flow()` normalizes:
+
+  | edge_type | authored as | data flows |
+  | --- | --- | --- |
+  | `reads` | proc/view → table it reads | target → source |
+  | `writes` | proc → table it writes | source → target |
+  | `feeds` | view → pbi_table; pbi_table → model; visual → page → report | source → target |
+  | `defines` | proc → table it CREATEs | source → target |
+  | `references` | measure → measure/table; proc → proc (EXEC) | target → source |
+  | `visualizes` | visual → pbi_table/measure | target → source |
+
+  All traversal (`upstream()` / `downstream()`) uses `flow()`, so callers
+  never think about authoring direction.
+
+## Key design decisions
+
+**Deterministic by construction.** Every iteration is sorted, ids are
+normalized, serialization sorts keys and edges, and nothing embeds a
+timestamp. Same inputs + same cache ⇒ byte-identical output
+(`tests/test_determinism.py` enforces this). This is what makes
+`coop-data-doc check` a valid CI freshness gate.
+
+**Never guess lineage.** Anything not statically provable becomes a warning
+or an `unresolved` marker, never an invented edge: dynamic SQL
+(`sp_executesql`), opaque .pbix models, unrecognized M partitions.
+
+**AST first, regex fallback second (SQL).** T-SQL proc bodies routinely
+defeat sqlglot (cursors, WHILE, TRY/CATCH). `sql_procs` therefore processes
+*statement by statement*: split the body on `;` (string/comment-aware
+scanner), try sqlglot per chunk, and only for unparseable chunks apply
+documented regex patterns — marking the proc
+`metadata.parse_quality = "regex_fallback"` plus a warning so humans know to
+eyeball it. Two sqlglot subtleties are handled centrally: temp tables lose
+their `#` in the AST (flagged `temporary=True` instead — see
+`is_temp_table`), and `UPDATE alias ... FROM table AS alias` needs alias
+resolution for the write target.
+
+**Layer assignment is a post-pass** (`layering.assign_layers`). Object *type*
+comes from the SQL; the medallion *layer* (bronze/silver/gold) is assigned
+from `config.layers` rules — by schema and/or source-path glob, precedence
+gold → silver → bronze — with a read/write heuristic fallback (a table only
+ever read → silver source; one created here → gold). `display_name` carries
+the original-case name for rendering while ids stay normalized.
+`prune_schemas` first drops system schemas (`sys`/`information_schema`/
+`tempdb`/`db_*`) and any `ignore_schemas`, which would otherwise appear as
+phantom nodes from catalog references.
+
+**Name gaps are a first-class problem.** View schemas and semantic-model
+names are similar but not identical (e.g. schema `sales` feeds the
+"Sales Analytics" model). The linker ladder
+(`linker/resolver.py`) goes: cache → exact id match → `schema_mappings`
+config rule → fuzzy (`difflib`, auto-accept ≥ 0.92, prompt 0.60–0.92) →
+interactive `questionary` prompt. Every interactive answer is written to
+`.lineage-cache.json` immediately (crash-safe), so the second run asks
+nothing.
+
+**Two renderers, one graph.** `render/markdown.py` emits strict fixed-order
+YAML front-matter (`id`, `type`, `name`, `schema`, `source_file`, `path`,
+`upstream_inputs`, `downstream_dependents`, `tags`) so agents can parse pages
+without heuristics; `manifest.json` is the whole serialized graph for
+programmatic consumers. Page filenames come from `slug()` (filesystem-safe,
+length-bounded, hash-suffixed for uniqueness — not derivable from the id), so
+the `path` field is the source of truth for where a node's page lives. `render/site.py` synthesizes a Material config and
+post-processes the built HTML so the portal works over `file://` with zero
+network: vendored `mermaid.min.js` (Material skips its CDN fetch when
+`window.mermaid` exists), vendored iframe-worker shim (URL rewritten in the
+HTML), `font: false`, `use_directory_urls: false`.
+
+**Human content survives regeneration.** Each page has a Business Intent
+block between `<!-- intent:begin/end -->` markers; the renderer carries the
+existing block forward verbatim. `check` copies the committed tree before
+re-rendering for the same reason.
+
+## For agents: answering questions from the output
+
+- *"What breaks if I drop column X from view Y?"* — open the view's page
+  (find it via the `path` field, not by computing a filename), read
+  `downstream_dependents`, follow each page's front-matter transitively
+  (or walk `manifest.json` edges with the
+  flow table above).
+- *"Where does this report number come from?"* — visual page →
+  `visualizes` → measure (DAX shown on the measure page) → `references` →
+  pbi_table → `feeds` → view → `reads` → gold table → `writes` ← proc →
+  `reads` → silver sources.
+- Trust levels: edges carry `evidence`; nodes parsed via fallback carry
+  `metadata.parse_quality = "regex_fallback"`; DAX/measure edges are
+  heuristic (`dax_refs_heuristic`).
+
+## Repo layout
+
+```
+src/coop_data_doc/
+├── cli.py            entrypoints + run_pipeline (the orchestration) + interactive menu
+├── config.py         coop-data-doc.yml model (repos/layers/ignore_schemas) + ParseWarning
+├── crawler.py        repo walk + FileKind classification
+├── graph/            model.py (Node/Edge/LineageGraph, display_name), serialize.py
+├── parsers/          sql_common/sql_objects/sql_procs, tmdl/bim/mcode/dax/pbir/pbix
+├── layering.py       medallion layer assignment + system/ignored-schema pruning
+├── linker/           resolver.py (ladder), cache.py, interactive.py
+├── diagnostics.py    severity-classified warnings → console / JSON / HTML page
+├── progress.py       stderr progress bars + spinner (TTY-only)
+├── wizard.py         interactive `setup` (repos, layers, ignore, mappings)
+├── upgrade.py        `upgrade` — the only networked command (PyPI/git)
+├── render/           markdown.py, mermaid.py, site.py (layer-grouped nav)
+└── templates/assets/ vendored mermaid + iframe-worker + custom.css
+tasks/                original builder briefs — double as interface docs
+tests/                fixtures/repo_sql + fixtures/repo_pbi drive everything
+```

coop_data_doc-0.15.0/CLAUDE.md

@@ -0,0 +1,50 @@
+# coop-data-doc — agent guide
+
+Offline, deterministic data-lineage doc generator for SQL + Power BI estates.
+Start with `ARCHITECTURE.md` (pipeline, data model, design decisions), then
+`CONTRIBUTING.md` (rules). The `tasks/` briefs document each module's
+interface in depth.
+
+## Commands
+
+```bash
+.venv/bin/python -m pytest -q          # full suite (fast, <1s)
+.venv/bin/ruff check src tests        # lint
+.venv/bin/ruff format --check src tests   # formatting (CI enforces this too)
+.venv/bin/coop-data-doc build --non-interactive   # run the tool itself
+```
+
+If `.venv` is missing: `python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"`.
+
+## Hard rules
+
+1. **Determinism** (CI-enforced) — sorted iteration everywhere, no
+   timestamps/randomness in output, and `newline="\n"` on every generated
+   `write_text` (cross-OS byte-identity). `tests/test_determinism.py`
+   byte-compares two full builds.
+2. **Offline pipeline** — no network/DB/LLM anywhere in doc generation;
+   built HTML must work over `file://` (vendored assets in
+   `src/coop_data_doc/templates/assets/`). Sole exception: the explicit
+   `upgrade` command (`upgrade.py`); the pipeline never imports it.
+3. **Pure parsers** (convention, reviewed not CI-enforced) — no print/exit
+   outside `cli.py`, `wizard.py`, `progress.py`, and `linker/interactive.py`;
+   warnings are returned as `ParseWarning` values.
+4. **Never guess lineage** — un-provable things become warnings or
+   `unresolved` markers, not edges.
+
+## Orientation shortcuts
+
+- Orchestration: `cli.run_pipeline()` — the whole pipeline in one function
+  (crawl → SQL parse → PBI parse → prune_schemas → assign_layers → link).
+- Data model + edge-direction semantics: `graph/model.py` (read `Edge.flow()`
+  before touching traversal — `reads`/`references`/`visualizes` are authored
+  opposite to data flow). `id`/`name` are normalized (lowercase) for matching;
+  `display_name` keeps original case for rendering (`Node.qualified_display`).
+- Layers (bronze/silver/gold) come from `config.layers` rules in `layering.py`,
+  not from node type; object type is parser-detected.
+- Diagnostics (`diagnostics.py`) classify every warning by severity and render
+  the console summary + `diagnostics.json` + the HTML Diagnostics page.
+- Tests are fixture-driven: `tests/fixtures/repo_sql` and `repo_pbi` are
+  miniature real repos; most tests assert exact node-id/edge-key sets.
+- When adding a parser case, extend the fixtures rather than writing inline
+  SQL/JSON strings, so the crawler/CLI/determinism suites cover it too.

coop_data_doc-0.15.0/CONTRIBUTING.md

@@ -0,0 +1,67 @@
+# Contributing
+
+## Module map
+
+| Module | Files | Role |
+| --- | --- | --- |
+| M0 core graph | `graph/model.py`, `graph/serialize.py` | `Node`/`Edge`/`LineageGraph`; the only data structure modules share |
+| M1 config + crawler | `config.py`, `crawler.py` | `coop-data-doc.yml` loading, repo walking, `FileKind` classification |
+| M2 SQL parser | `parsers/sql_common.py`, `sql_objects.py`, `sql_procs.py` | sqlglot (tsql) AST lineage with a regex fallback ladder |
+| M3 Power BI extractor | `parsers/tmdl.py`, `bim.py`, `mcode.py`, `dax.py`, `pbir.py`, `pbix.py` | semantic models, measures, reports, best-effort pbix |
+| M4 linker | `linker/resolver.py`, `cache.py`, `interactive.py` | joins SQL ↔ PBI: cache → exact → config rule → fuzzy → prompt |
+| M4½ layering | `layering.py` | medallion layer (bronze/silver/gold) from `config.layers` rules (schema/path), heuristic fallback; object *type* stays parser-detected |
+| M4¾ diagnostics | `diagnostics.py` | severity-classified warnings/unresolved → console summary, `diagnostics.json`, and the HTML Diagnostics page |
+| M5 renderers | `render/markdown.py`, `mermaid.py`, `site.py` | agent Markdown + offline MkDocs Material portal; nav grouped by layer→type; `schema.Object` (original-case `display_name`) titles |
+| M6 CLI | `cli.py`, `wizard.py`, `upgrade.py`, `progress.py` | interactive menu (bare invocation), `setup` / `init` / `scan` / `build` / `update` / `check` / `help` / `upgrade`; stderr progress bars + spinner |
+
+The original builder briefs live in `tasks/` and double as interface documentation.
+
+## Non-negotiable rules
+
+1. **Deterministic output.** Iterate everything in sorted order; never embed
+   timestamps or randomness; pass `newline="\n"` to every `write_text` of a
+   generated artifact (Windows would otherwise emit CRLF and break
+   cross-platform byte-identity). `tests/test_determinism.py` builds twice
+   and byte-compares.
+2. **Offline at runtime.** No network, no DB connections, no LLM calls
+   anywhere in the documentation pipeline. The built HTML must work over
+   `file://` (vendored assets live in `src/coop_data_doc/templates/assets/`).
+   The single sanctioned exception is the explicit `upgrade` command
+   (`upgrade.py`), which checks PyPI/git for tool and dependency updates —
+   nothing in the pipeline may import it.
+3. **Parsers are pure.** No printing or exiting outside `cli.py`,
+   `wizard.py`, and `linker/interactive.py`; warnings are returned as
+   `ParseWarning` values. Parsers/renderers may accept an optional
+   `on_file`/`on_node` callback for progress reporting (the CLI supplies
+   it) — that's a reporting hook, not printing; the parser never renders.
+4. **Page filenames go through `slug()`** (`render/mermaid.py`): always
+   filesystem-safe (Windows-illegal chars stripped), length-bounded, and
+   uniquified with a short id-hash. Never build a page path by hand; agents
+   read the `path` front-matter field.
+5. **Never guess lineage.** Dynamic SQL, opaque pbix models, and
+   unrecognized partition sources produce warnings/unresolved markers, not
+   invented edges.
+
+## Adding a parser
+
+Implement `(entries: list[FileEntry], graph: LineageGraph) -> list[ParseWarning]`,
+register the file kind in `crawler.py`, wire it into `cli.run_pipeline`, and add
+fixtures under `tests/fixtures/` with node/edge-set assertions.
+
+## Developing
+
+```bash
+pip install -e ".[dev]"
+pytest
+ruff check src tests
+```
+
+## Releasing
+
+Bump the version in **both** `pyproject.toml` and `src/coop_data_doc/__init__.py`
+on every release. This is mandatory: installs from the git URL (pipx/uv/pip) are
+**version-gated** — `pipx upgrade` re-clones the branch but only installs when the
+version number increased. Ship code changes under the same version and users'
+`coop-data-doc upgrade` will silently report "already at latest" and skip them.
+Use semver: patch for fixes, minor for new commands/features, major for breaking
+config/output changes.

coop_data_doc-0.15.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aaron Jennings
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.