coltrace 0.1.0.dev7__tar.gz → 0.1.0.dev9__tar.gz

{coltrace-0.1.0.dev7/src/coltrace.egg-info → coltrace-0.1.0.dev9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev7
+Version: 0.1.0.dev9
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -32,7 +32,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev7`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev9`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -58,10 +58,16 @@ separate release step; these commands install the local project.
 
 The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
 extractions. Literal SQL projections, aliases and expressions describe source
-fields before their first Python use. Query filters are control metadata rather
-than value dependencies of every output. The supplied
-`spark.sql(build_query(table)).toPandas()` pattern works with a resolved Spark
-session or an explicit source contract for an injected session.
+fields before their first Python use. Every database column the SQL reads is a
+selectable lineage node (`origin="table"`), so a column used only inside
+`SUM(...)` or a `WHERE` clause can be traced to the DataFrame columns it feeds.
+Query filters are control metadata rather than value dependencies of every
+output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
+an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
+and joins keeps its known text, with unknown parts shown as `{expression}`.
+`SELECT *`, `UNION`, tables read by name (`spark.table`, Snowpark
+`session.table`, `dbt.ref`) and SQL files read inside the project are modeled
+too; see [extraction architectures](docs/extraction-architectures.md).
 
 `report.readers` inventories resolved `pd.read_*` calls with file, line, context,
 return shape and resulting DataFrame identity. File schemas are unknown unless
@@ -221,7 +227,7 @@ the shortest eligible distance, so a node reached both ways appears as direct.
 | --- | --- |
 | Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
 | Resolved `pd.read_*` calls | Located inventory; supported DataFrame readers plus explicit/observed source schemas; iterators/collections remain distinct |
-| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts |
+| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts; database columns as nodes; dialect fallback |
 | MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
 | Literal dictionary `DataFrame` construction | String column keys and literal values |
 | `frame["column"]`, `frame.loc[:, "column"]` | Literal or statically known labels (`for c in COLS`, f-strings) |
@@ -308,6 +314,8 @@ tox/nox and pytest/mypy/Ruff caches, `build`, and `dist`. Linked files/directori
 and Windows reparse points are skipped. A `.coltraceignore` file at the analyzed
 root uses gitignore syntax (root-level subset, including `!` negation). The root
 `.gitignore` is also applied unless `respect_gitignore = false` is set.
+`sql_dialect = "bigquery"` names the dialect of SQL text whose reader does not;
+text that does not parse is still retried in common dialects and reported.
 `observers = ["notify_team", "client.*"]` declares project helpers that only read
 the DataFrames they receive, so they do not interrupt lineage.
 `ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.

{coltrace-0.1.0.dev7 → coltrace-0.1.0.dev9}/README.md

@@ -6,7 +6,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev7`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev9`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -32,10 +32,16 @@ separate release step; these commands install the local project.
 
 The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
 extractions. Literal SQL projections, aliases and expressions describe source
-fields before their first Python use. Query filters are control metadata rather
-than value dependencies of every output. The supplied
-`spark.sql(build_query(table)).toPandas()` pattern works with a resolved Spark
-session or an explicit source contract for an injected session.
+fields before their first Python use. Every database column the SQL reads is a
+selectable lineage node (`origin="table"`), so a column used only inside
+`SUM(...)` or a `WHERE` clause can be traced to the DataFrame columns it feeds.
+Query filters are control metadata rather than value dependencies of every
+output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
+an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
+and joins keeps its known text, with unknown parts shown as `{expression}`.
+`SELECT *`, `UNION`, tables read by name (`spark.table`, Snowpark
+`session.table`, `dbt.ref`) and SQL files read inside the project are modeled
+too; see [extraction architectures](docs/extraction-architectures.md).
 
 `report.readers` inventories resolved `pd.read_*` calls with file, line, context,
 return shape and resulting DataFrame identity. File schemas are unknown unless
@@ -195,7 +201,7 @@ the shortest eligible distance, so a node reached both ways appears as direct.
 | --- | --- |
 | Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
 | Resolved `pd.read_*` calls | Located inventory; supported DataFrame readers plus explicit/observed source schemas; iterators/collections remain distinct |
-| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts |
+| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts; database columns as nodes; dialect fallback |
 | MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
 | Literal dictionary `DataFrame` construction | String column keys and literal values |
 | `frame["column"]`, `frame.loc[:, "column"]` | Literal or statically known labels (`for c in COLS`, f-strings) |
@@ -282,6 +288,8 @@ tox/nox and pytest/mypy/Ruff caches, `build`, and `dist`. Linked files/directori
 and Windows reparse points are skipped. A `.coltraceignore` file at the analyzed
 root uses gitignore syntax (root-level subset, including `!` negation). The root
 `.gitignore` is also applied unless `respect_gitignore = false` is set.
+`sql_dialect = "bigquery"` names the dialect of SQL text whose reader does not;
+text that does not parse is still retried in common dialects and reported.
 `observers = ["notify_team", "client.*"]` declares project helpers that only read
 the DataFrames they receive, so they do not interrupt lineage.
 `ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.

{coltrace-0.1.0.dev7 → coltrace-0.1.0.dev9}/docs/backlog.md

@@ -64,11 +64,23 @@ not a capability of the published release.
 | CT-098 | Reshaping (pivot, melt, get_dummies) | Implemented dev7 with open derived schemas; stack/unstack/crosstab pending |
 | CT-099 | Observing libraries and declared observers | Implemented dev7 |
 | CT-100 | Native `.ipynb` input | Proposed: notebooks were converted outside ColTrace for these tests |
+| CT-101 | Database columns as nodes | Implemented dev8 (schema 0.6): `table` frames, pass/rename/derive from SQL fields, controls as filter consumption |
+| CT-102 | Runtime-assembled SQL | Implemented dev8: labeled placeholders, `%`/join folding, comment-led SQL, fragment recovery |
+| CT-103 | SQL dialect fallback and `sql_dialect` | Implemented dev8 |
+| CT-104 | Conversion keeps receiver SQL (`spark.sql(...).toPandas()`) | Implemented dev8 |
+| CT-105 | Ranked candidates for ambiguous names | Implemented dev8: `LineageGraph.candidates`, notebook `ON_AMBIGUOUS` |
+| CT-106 | Cursor `execute` then `fetch*` on separate statements | Proposed: remember the last SQL text executed on a resolved cursor name |
+| CT-107 | Jinja/dbt templates (`{{ ref() }}`) | Proposed: out of scope until a real project needs it |
+| CT-108 | Dict entries, exact unrolling, accumulators | Implemented dev9 |
+| CT-109 | Filters along the call chain | Implemented dev9 |
+| CT-110 | Open `SELECT *` schemas, UNION, tables by name, SQL files | Implemented dev9 |
+| CT-111 | Extraction census tool | Done in the session scratchpad; promote to `tools/` if it becomes a regular check |
+| CT-112 | `%%bigquery df` cell magic as a SQL source | Proposed (Jupyter/Colab users) |
 | CT-095 | Lazy materialization | Proposed: avoid eager pass copies of every known column on each row operation for very wide frames |
 
 ## Next checkpoint
 
-1. Rerun the internal compatibility report with dev7 and classify the first remaining identity break for the target column.
+1. Rerun the internal compatibility report with dev8: select the target column (now a database column if it is only read inside SQL) and classify the first remaining break.
 2. Inspect the generated HTML's pan/zoom/selection behavior in a normal browser.
 3. Run ColTrace against a small real pipeline with a manually checked expected graph.
 4. Classify missing/incorrect relations by semantic rule, using [accuracy gates](accuracy.md).

{coltrace-0.1.0.dev7 → coltrace-0.1.0.dev9}/docs/compatibility-checkpoint.md

@@ -1,159 +1,254 @@
-# Project compatibility checkpoint
-
-Local checkpoint: `0.1.0.dev7`, 2026-11-03. Publication is a separate step.
-This work responds to an anonymized compatibility report produced inside the
-user's company. The private project was not accessed, copied or executed; every
-fixture below is a new synthetic reproduction of a reported technical shape.
-
-## What the report showed (dev6)
-
-On a 58-file Databricks-style project, dev6 parsed 57 files and produced 32
-frames, 11 column versions, 0 sources, 4 edges and 1,612 diagnostics. The
-requested business column was absent. The leading causes were:
-
-| Report finding | Root cause in dev6 |
-| --- | --- |
-| `CT_SYNTAX_ERROR` on the entrypoint | Raw `%pip` magics made the orchestrator notebook unparseable; `%run` chains were invisible |
-| 0 sources | Data entered through an injected `run_query(sql)` closure; nested functions were never resolved |
-| 478 `CT_UNSUPPORTED_EXPRESSION` | `&`/`|`, `and`/`or`, ternaries, comprehensions and slices invalidated reachable frames |
-| 270 `CT_CONDITIONAL_FLOW`, 36 `CT_CONDITIONAL_RETURN` | Every frame mutated in a branch became a boundary; `if df.empty: return df` erased the result identity |
-| 258 `CT_ANALYSIS_LIMIT` | Path and call budgets were counted for the whole project, not per function or module |
-| 72 `CT_OPAQUE_CALL`, 287 `CT_UNRESOLVED_RECEIVER` | Common methods (`fillna`, `astype`, `sort_values`, `np.where`, …) invalidated the frame; dict writes were reported as DataFrame gaps |
-| 21 `CT_UNSUPPORTED_CONTROL_FLOW`, 17 `CT_UNSUPPORTED_FUNCTION` | `class`, `+=`, `assert`, nested and decorated functions invalidated state |
-| 9 `CT_DYNAMIC_COLUMN`, 36 `CT_UNSUPPORTED_TARGET` | `for c in COLS: df[c] = …` and tuple unpacking were not modeled |
-
-## Corrected behavior (dev7)
-
-- **Notebook source.** Databricks `.py` exports and raw IPython magics are
-  prepared in place: magic lines become inert statements with unchanged line
-  numbers, magic-looking text inside strings is preserved, and `# MAGIC %run ./x`
-  or raw `%run` includes the target notebook statically in the caller's
-  namespace. Included notebooks are not analyzed a second time standalone.
-- **Per-unit budgets.** Each module and each uncalled-function template has its
-  own path and call budget.
-- **Joins.** After `if`/`try`/loops, a column whose version differs between
-  alternatives receives an inferred merge version reading every possible value.
-  Untouched columns keep their identity. Early returns are joined at the function
-  exit; different frame alternatives produce a new inferred `merge` frame whose
-  columns may come from any alternative. Invalidated alternatives remain
-  boundaries. Merged facts are never `confirmed`.
-- **Expressions.** Bitwise masks, boolean operators, ternaries, comparisons,
-  slices, comprehensions, walrus and starred expressions are interpreted. An
-  operand of unknown provenance makes the result `inferred`/`partial` instead of
-  cutting the dependency on tracked columns. Constant folding covers literal
-  arithmetic, known dicts (`{**DEFAULTS, **params}`), known builtins and string
-  methods.
-- **Pandas/NumPy catalog.** Documented Series methods (masks, aggregates,
-  elementwise transforms, row selection), `.str`/`.dt`/`.cat` accessors,
-  NumPy elementwise functions and `pd.to_numeric`-style functions derive values.
-  DataFrame `sort_values`, `head`, `reset_index`, `drop(columns=…)`, `query`,
-  `set_index`, `fillna`/`astype` (including per-column dicts), `insert`, `pop`,
-  `merge`/`join`/`concat` (inferred), partial `.loc[mask, col] = …` updates and
-  more exports are modeled. Unlisted DataFrame summaries such as `groupby`
-  leave the receiver intact and report `CT_UNMODELED_RESULT`.
-- **Python structures.** Classes (methods, instance attributes, dataclass-like
-  fields), nested functions with closures, decorated functions (assumed
-  transparent, inferred), `*args/**kwargs`, `+=`, `assert`, `del`, `match`,
-  fixed-width unpacking and loops over literal or configured label collections
-  (unrolled exactly) are interpreted.
-- **Injected sources.** Nested closures such as `run_query` are followed into
-  `client.query(sql).to_dataframe()`. When an unresolved call result is used
-  with a DataFrame-only API, it becomes an inferred frame of unresolved origin;
-  if its first argument is SQL text, the SELECT list seeds its columns.
-  Interpolations that cannot be resolved become placeholders and are excluded
-  from fields. Source contracts may now name an injected parameter explicitly.
-- **Noise.** Writes to mappings such as `os.environ` or config dicts are no
-  longer reported as DataFrame gaps; parameters and DataFrame-used names still
-  are. Inferences and assumptions are `info` diagnostics, distinct from warnings.
-- **Inputs.** `.coltraceignore` and the root `.gitignore` are applied, and the
-  notebook cell writes artifacts outside the analyzed tree by default.
-
-## Measured on the synthetic reproduction
-
-[`examples/databricks_project`](../examples/databricks_project/README.md)
-reproduces the report's structure in 13 files. Results on the same input:
-
-| Metric | dev6 | dev7 |
-| --- | ---: | ---: |
-| Files parsed | 12 of 13 | 13 of 13 |
-| DataFrames / column versions | 0 / 0 | 10 / 88 |
-| Sources | 0 | 1 SQL (6 fields) |
-| Warnings / errors | 32 / 1 | 0 / 0 |
-| `n_contacts → priority_score → published_score` | not found | found, with exports |
-
-Its acceptance test lists expected relations and forbidden ones (row-filter and
-sort-key columns must not become value inputs). A generated 60-module
-project with ~240 functions, branches, loops, classes and merges is analyzed in
-about 5 seconds locally. These numbers are engineering checks on synthetic
-input, not measured accuracy on the private project.
-
-## Validation
-
-941 tests pass (67 more than dev6, with dev6 join tests rewritten to require
-inferred-only paths); Ruff lint/format and strict mypy pass. Lineage snapshots are schema `0.5` (adds the `info` severity); `0.4`
-snapshots load unchanged.
-
-```sh
-python -m pytest -q
-python -m ruff check src tests examples
-python -m ruff format --check src tests examples
-python -m mypy
-python -m coltrace scan examples/databricks_project --out .coltrace/databricks
-python -m coltrace trace n_contacts --from .coltrace/databricks/scan.json
-```
-
-## Second round: public projects
-
-Before publishing, ColTrace was run statically (nothing executed) over 14
-public projects, about 1,200 files: the Kedro spaceflights starter, the
-DataTalksClub MLOps and Data Engineering Zoomcamps, four Databricks solution
-accelerators, the recommenders library, the owid/covid-19-data production ETL
-scripts, and two data science books whose notebooks were converted to `.py`
-outside ColTrace. Every report had to survive a JSON snapshot round trip.
-
-| Project | Files | Seconds | Column versions | Warnings | Errors |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| Kedro spaceflights | 14 | crash → 0.0 | → 39 | → 2 | → 0 |
-| MLOps Zoomcamp | 88 | 2.9 → 1.2 | 633 → 2,613 | 1,637 → 41 | 0 → 0 |
-| Data Engineering Zoomcamp | 72 | crash → 0.6 | → 32 | → 25 | → 0 |
-| Databricks CLV | 2 | 0.2 → 0.1 | 11 → 40 | 28 → 8 | 0 → 0 |
-| Databricks demand forecasting | 4 | 0.5 → 0.4 | 35 → 46 | 55 → 8 | 0 → 0 |
-| Databricks propensity | 10 | 1.8 → 3.6 | 32 → 7 | 72 → 25 | 0 → 0 |
-| Databricks safety stock | 3 | 0.1 → 0.0 | 59 → 59 | 19 → 0 | 0 → 0 |
-| Hands-on ML notebooks | 27 | 8.5 → 6.3 | 203 → 344 | 5,173 → 158 | 1 → 0 |
-| Data Science Handbook notebooks | 136 | 5.0 → 3.0 | 350 → 606 | 898 → 209 | 3 → 2 |
-| MLOps notebooks | 26 | 0.8 → 0.3 | 282 → 827 | 375 → 9 | 1 → 0 |
-| recommenders notebooks | 62 | 2.6 → 1.0 | 396 → 569 | 564 → 214 | 3 → 2 |
-| owid/covid-19-data scripts | 450 | 148.4 → 39.8 | 8,119 → 66,334 | 8,931 → 8,129 | 0 → 0 |
-| recommenders library | 249 | 52.0 → 11.9 | 1,654 → 3,394 | 6,075 → 1,561 | 0 → 0 |
-
-The first column of each pair is the first dev7 build, the second the final
-one. Column counts in the propensity notebooks fell because PySpark DataFrames
-were previously mistaken for Pandas frames. Remaining errors are genuine syntax
-errors in the source (a missing comma, deliberately invalid book examples).
-The owid scripts now resolve their `src/` package, so far more of their code is
-interpreted, which explains both the larger counts and the remaining warnings.
-
-Hand-written expectations for the Kedro nodes and two MLOps scripts: 14 of 14
-expected relations found and 0 of 9 forbidden relations reported. Defects fixed
-in this round included an analysis crash on Windows paths longer than 260
-characters, duplicated frame/reader/column identities that made most real
-snapshots unloadable, project-wide state copying on every branch, and analyzed
-code's `SyntaxWarning` output. The covered patterns are listed in
-[supported patterns](supported-patterns.md#public-project-patterns-dev7-second-round).
-
-## Remaining boundaries
-
-This is still bounded abstract interpretation, not a general control-flow solver.
-Loops over unknown collections are summarized by one possible iteration joined
-with zero iterations. `groupby(...).apply`, `stack`/`unstack`, `crosstab` and
-row-wise callbacks with side effects remain unmodeled results. Calls to unknown
-helpers that receive a DataFrame still interrupt its provenance unless declared
-as `observers`. Merge suffixes and
-row multiplicity are not resolved. Async functions, reflection, `global`/`nonlocal`
-writes and properties remain unresolved. `.ipynb` JSON files, live cells and
-`dbutils.notebook.run` (a separate namespace) are not inputs. Opaque calls that
-receive a DataFrame still interrupt its provenance.
-
-The next acceptance input is a rerun of the internal compatibility report on
-dev7, identifying the first remaining identity break for the target column.
+# Project compatibility checkpoint
+
+Local checkpoint: `0.1.0.dev9`, 2026-11-04 (dev7 and dev8 sections below are
+kept as history). Publication is a separate step.
+This work responds to an anonymized compatibility report produced inside the
+user's company. The private project was not accessed, copied or executed; every
+fixture below is a new synthetic reproduction of a reported technical shape.
+
+## What the report showed (dev6)
+
+On a 58-file Databricks-style project, dev6 parsed 57 files and produced 32
+frames, 11 column versions, 0 sources, 4 edges and 1,612 diagnostics. The
+requested business column was absent. The leading causes were:
+
+| Report finding | Root cause in dev6 |
+| --- | --- |
+| `CT_SYNTAX_ERROR` on the entrypoint | Raw `%pip` magics made the orchestrator notebook unparseable; `%run` chains were invisible |
+| 0 sources | Data entered through an injected `run_query(sql)` closure; nested functions were never resolved |
+| 478 `CT_UNSUPPORTED_EXPRESSION` | `&`/`|`, `and`/`or`, ternaries, comprehensions and slices invalidated reachable frames |
+| 270 `CT_CONDITIONAL_FLOW`, 36 `CT_CONDITIONAL_RETURN` | Every frame mutated in a branch became a boundary; `if df.empty: return df` erased the result identity |
+| 258 `CT_ANALYSIS_LIMIT` | Path and call budgets were counted for the whole project, not per function or module |
+| 72 `CT_OPAQUE_CALL`, 287 `CT_UNRESOLVED_RECEIVER` | Common methods (`fillna`, `astype`, `sort_values`, `np.where`, …) invalidated the frame; dict writes were reported as DataFrame gaps |
+| 21 `CT_UNSUPPORTED_CONTROL_FLOW`, 17 `CT_UNSUPPORTED_FUNCTION` | `class`, `+=`, `assert`, nested and decorated functions invalidated state |
+| 9 `CT_DYNAMIC_COLUMN`, 36 `CT_UNSUPPORTED_TARGET` | `for c in COLS: df[c] = …` and tuple unpacking were not modeled |
+
+## Corrected behavior (dev7)
+
+- **Notebook source.** Databricks `.py` exports and raw IPython magics are
+  prepared in place: magic lines become inert statements with unchanged line
+  numbers, magic-looking text inside strings is preserved, and `# MAGIC %run ./x`
+  or raw `%run` includes the target notebook statically in the caller's
+  namespace. Included notebooks are not analyzed a second time standalone.
+- **Per-unit budgets.** Each module and each uncalled-function template has its
+  own path and call budget.
+- **Joins.** After `if`/`try`/loops, a column whose version differs between
+  alternatives receives an inferred merge version reading every possible value.
+  Untouched columns keep their identity. Early returns are joined at the function
+  exit; different frame alternatives produce a new inferred `merge` frame whose
+  columns may come from any alternative. Invalidated alternatives remain
+  boundaries. Merged facts are never `confirmed`.
+- **Expressions.** Bitwise masks, boolean operators, ternaries, comparisons,
+  slices, comprehensions, walrus and starred expressions are interpreted. An
+  operand of unknown provenance makes the result `inferred`/`partial` instead of
+  cutting the dependency on tracked columns. Constant folding covers literal
+  arithmetic, known dicts (`{**DEFAULTS, **params}`), known builtins and string
+  methods.
+- **Pandas/NumPy catalog.** Documented Series methods (masks, aggregates,
+  elementwise transforms, row selection), `.str`/`.dt`/`.cat` accessors,
+  NumPy elementwise functions and `pd.to_numeric`-style functions derive values.
+  DataFrame `sort_values`, `head`, `reset_index`, `drop(columns=…)`, `query`,
+  `set_index`, `fillna`/`astype` (including per-column dicts), `insert`, `pop`,
+  `merge`/`join`/`concat` (inferred), partial `.loc[mask, col] = …` updates and
+  more exports are modeled. Unlisted DataFrame summaries such as `groupby`
+  leave the receiver intact and report `CT_UNMODELED_RESULT`.
+- **Python structures.** Classes (methods, instance attributes, dataclass-like
+  fields), nested functions with closures, decorated functions (assumed
+  transparent, inferred), `*args/**kwargs`, `+=`, `assert`, `del`, `match`,
+  fixed-width unpacking and loops over literal or configured label collections
+  (unrolled exactly) are interpreted.
+- **Injected sources.** Nested closures such as `run_query` are followed into
+  `client.query(sql).to_dataframe()`. When an unresolved call result is used
+  with a DataFrame-only API, it becomes an inferred frame of unresolved origin;
+  if its first argument is SQL text, the SELECT list seeds its columns.
+  Interpolations that cannot be resolved become placeholders and are excluded
+  from fields. Source contracts may now name an injected parameter explicitly.
+- **Noise.** Writes to mappings such as `os.environ` or config dicts are no
+  longer reported as DataFrame gaps; parameters and DataFrame-used names still
+  are. Inferences and assumptions are `info` diagnostics, distinct from warnings.
+- **Inputs.** `.coltraceignore` and the root `.gitignore` are applied, and the
+  notebook cell writes artifacts outside the analyzed tree by default.
+
+## Measured on the synthetic reproduction
+
+[`examples/databricks_project`](../examples/databricks_project/README.md)
+reproduces the report's structure in 13 files. Results on the same input:
+
+| Metric | dev6 | dev7 |
+| --- | ---: | ---: |
+| Files parsed | 12 of 13 | 13 of 13 |
+| DataFrames / column versions | 0 / 0 | 10 / 88 |
+| Sources | 0 | 1 SQL (6 fields) |
+| Warnings / errors | 32 / 1 | 0 / 0 |
+| `n_contacts → priority_score → published_score` | not found | found, with exports |
+
+Its acceptance test lists expected relations and forbidden ones (row-filter and
+sort-key columns must not become value inputs). A generated 60-module
+project with ~240 functions, branches, loops, classes and merges is analyzed in
+about 5 seconds locally. These numbers are engineering checks on synthetic
+input, not measured accuracy on the private project.
+
+## Validation
+
+dev9: 1,024 tests pass (984 in dev8; dev9 adds container, accumulator, filter
+and extraction-shape tests). dev8: 984 (941 in dev7, plus the SQL boundary
+tests; SQL control tests require database-table columns consumed as row
+controls).
+Ruff lint/format and strict mypy pass. Lineage snapshots are schema `0.6`
+(adds `dataframes[].kind`); `0.4` and `0.5` snapshots load as DataFrame frames.
+
+```sh
+python -m pytest -q
+python -m ruff check src tests examples
+python -m ruff format --check src tests examples
+python -m mypy
+python -m coltrace scan examples/databricks_project --out .coltrace/databricks
+python -m coltrace trace n_contacts --from .coltrace/databricks/scan.json
+python -m coltrace trace contact_attempts --from .coltrace/databricks/scan.json
+```
+
+## Second round: public projects
+
+Before publishing, ColTrace was run statically (nothing executed) over 14
+public projects, about 1,200 files: the Kedro spaceflights starter, the
+DataTalksClub MLOps and Data Engineering Zoomcamps, four Databricks solution
+accelerators, the recommenders library, the owid/covid-19-data production ETL
+scripts, and two data science books whose notebooks were converted to `.py`
+outside ColTrace. Every report had to survive a JSON snapshot round trip.
+
+| Project | Files | Seconds | Column versions | Warnings | Errors |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Kedro spaceflights | 14 | crash → 0.0 | → 39 | → 2 | → 0 |
+| MLOps Zoomcamp | 88 | 2.9 → 1.2 | 633 → 2,613 | 1,637 → 41 | 0 → 0 |
+| Data Engineering Zoomcamp | 72 | crash → 0.6 | → 32 | → 25 | → 0 |
+| Databricks CLV | 2 | 0.2 → 0.1 | 11 → 40 | 28 → 8 | 0 → 0 |
+| Databricks demand forecasting | 4 | 0.5 → 0.4 | 35 → 46 | 55 → 8 | 0 → 0 |
+| Databricks propensity | 10 | 1.8 → 3.6 | 32 → 7 | 72 → 25 | 0 → 0 |
+| Databricks safety stock | 3 | 0.1 → 0.0 | 59 → 59 | 19 → 0 | 0 → 0 |
+| Hands-on ML notebooks | 27 | 8.5 → 6.3 | 203 → 344 | 5,173 → 158 | 1 → 0 |
+| Data Science Handbook notebooks | 136 | 5.0 → 3.0 | 350 → 606 | 898 → 209 | 3 → 2 |
+| MLOps notebooks | 26 | 0.8 → 0.3 | 282 → 827 | 375 → 9 | 1 → 0 |
+| recommenders notebooks | 62 | 2.6 → 1.0 | 396 → 569 | 564 → 214 | 3 → 2 |
+| owid/covid-19-data scripts | 450 | 148.4 → 39.8 | 8,119 → 66,334 | 8,931 → 8,129 | 0 → 0 |
+| recommenders library | 249 | 52.0 → 11.9 | 1,654 → 3,394 | 6,075 → 1,561 | 0 → 0 |
+
+The first column of each pair is the first dev7 build, the second the final
+one. Column counts in the propensity notebooks fell because PySpark DataFrames
+were previously mistaken for Pandas frames. Remaining errors are genuine syntax
+errors in the source (a missing comma, deliberately invalid book examples).
+The owid scripts now resolve their `src/` package, so far more of their code is
+interpreted, which explains both the larger counts and the remaining warnings.
+
+Hand-written expectations for the Kedro nodes and two MLOps scripts: 14 of 14
+expected relations found and 0 of 9 forbidden relations reported. Defects fixed
+in this round included an analysis crash on Windows paths longer than 260
+characters, duplicated frame/reader/column identities that made most real
+snapshots unloadable, project-wide state copying on every branch, and analyzed
+code's `SyntaxWarning` output. The covered patterns are listed in
+[supported patterns](supported-patterns.md#public-project-patterns-dev7-second-round).
+
+## Third round: the dev7 report (dev8)
+
+The user ran dev7 on the private project and returned a second anonymized
+report. Structure was fixed (0 errors; 58/58 files; 3,017 column versions, 388
+frames and 23 sources instead of 11, 32 and 0; 2,053 of 2,300 diagnostics were
+`info`), but `graph.select(<target>)` still failed, so the notebook cell
+stopped before exporting the trace. The report narrowed the cause to the SQL
+boundary:
+
+| Report finding | Root cause in dev7 |
+| --- | --- |
+| `CT_PARTIAL_SQL ... ParseError` on an extractor | SQL reached through an unresolved `run_query` parameter was parsed with no dialect; BigQuery backticks do not parse in generic SQL |
+| `CT_PARTIAL_SQL` on interpolations | Unknown interpolated fragments (a dynamic `AND ...` clause) made the statement unparsable |
+| 63 `CT_INFERRED_FRAME` on `spark.sql(build_query()).toPandas()` | The conversion's own call has no SQL argument; the receiver's SQL text was not carried over |
+| Target is "a source-table field consumed inside the SQL" | Physical columns were text in `sources[].fields[].inputs`, never lineage nodes, so no selection could find them |
+
+The synthetic reproduction gained `quality_extract.py` (f-string SQL with
+backticks, a helper fragment, an unknown metric column, an uncalled extractor)
+and `topic_changes_extract.py` (`spark.sql(...).toPandas()` with an injected
+session). On dev7 it reproduced every finding; on dev8:
+
+| Check | dev7 | dev8 |
+| --- | --- | --- |
+| `select("contact_attempts")` (only inside `SUM(...)`) | not found | database column → `n_attempts` → `resolution_rate` |
+| Template extractor SQL with backticks and a dynamic clause | no fields (comment-led text not recognized; backticks alone: ParseError) | 5 fields; `SUM({metric})` partial; recovery notes |
+| `spark.sql(build_query(days)).toPandas()` | inferred frame, unknown columns | SQL source, 4 fields, complete schema |
+| `select("n_attempts")` (called path and template) | ambiguous, 7 identities | ambiguous, 5 identities, called path ranked first |
+| Warnings | 0 | 0 |
+
+Changes, with ADR-059 to ADR-064:
+
+- **Database columns are nodes** (schema 0.6, `dataframes[].kind == "table"`).
+  SQL fields read them as `pass`, `rename` or derivations; WHERE/JOIN/GROUP/ORDER
+  columns are `filter` consumptions only. One fully named relation is one
+  identity across extractions.
+- **Runtime-assembled SQL.** Unknown interpolations are labeled with their
+  Python expression (`{table}`); `%` formatting, `str.format`, concatenation
+  with unknown strings and `join` over known generator expressions keep or fold
+  their text; SQL may start with comments.
+- **Parsing recovery.** Dialect hints from the call shape, a project
+  `sql_dialect`, then fallback dialects; unplaceable fragments are dropped only
+  as a last resort. Every recovery is an `info` note, and the scan suggests the
+  dialect to declare.
+- **Conversions keep receiver SQL**: `spark.sql(text).toPandas()`,
+  `sdf.toPandas()`, `client.query(text).to_dataframe()`; a Spark transformation
+  in between drops the query. A name whose `.query/.sql/.execute` receives SQL
+  text is a SQL client, not a DataFrame.
+- **Selection.** `LineageGraph.candidates` ranks identities (called code before
+  templates, earliest origin first). A database column and its same-name SQL
+  output are one selectable family. The notebook cell continues with the first
+  candidate (`ON_AMBIGUOUS = "first"`) and lists the others; a missing name
+  that appears in unresolved SQL text is pointed out with its location.
+
+The public-project harness was rerun on dev8: no crashes, every snapshot
+round-trips, and frames, columns and warnings are unchanged in 11 of 13
+projects (none of them reads SQL into Pandas through the shapes above). In owid,
+`%` formatting now folds constant column names such as
+`"days_since_%sth_case" % THRESHOLD`, so more code is modeled and some existing
+gaps after it are now reported (+74 `CT_OPAQUE_CALL`); disabling the new text
+rules reproduces the dev7 numbers exactly. owid analysis time is unchanged
+(about 39 s). Hand-written precision checks: 14/14 expected, 0/9 forbidden.
+
+## Fourth round: the dev8 run and an extraction census (dev9)
+
+The user ran dev8 on the private project. Extraction now worked (3,140 column
+versions, 0 errors; physical table fields appeared among modeled names), but
+the run used a pre-dev7 copy of the notebook cell, which printed every
+informational note; the platform truncated the output, and the cell stopped at
+"Column not found with these filters". The visible diagnostics after extraction
+pointed at results kept in dicts and combined with `acc = None` accumulators
+(`tot.merge(...)` on an unresolved receiver, `CT_UNRESOLVED_RECEIVER` runs,
+`pd.concat` and `groupby` on lost frames).
+
+Synthetic reproductions confirmed three defects: a dict filled by subscript was
+still the empty literal, so loops over it were skipped silently; iterating
+`frames.items()` invalidated the frames; and a loop inspected once never saw the
+accumulator's `merge` branch. A fourth issue was selection: a database column
+belonged only to the function of its first reading site (often the injected
+`run_query` closure), so a `function=` filter on the extractor excluded it.
+All four are fixed; the synthetic example now includes a dict of results
+combined by an accumulator (`combine_sources.py`).
+
+The user also asked for a map of extraction architectures. A static census of
+public projects classified every extraction call shape and compared it with
+ColTrace's output; the result is [extraction architectures](extraction-architectures.md).
+Shapes fixed in this round: `SELECT *` open schemas, `UNION`, tables read by
+name, `.result()` and row-only steps before conversion, literal projections,
+keyword options on conversions, SQL files, and SQL clients whose name looked
+like a DataFrame.
+
+## Remaining boundaries
+
+This is still bounded abstract interpretation, not a general control-flow solver.
+Loops over unknown collections are summarized by one possible iteration joined
+with zero iterations. `groupby(...).apply`, `stack`/`unstack`, `crosstab` and
+row-wise callbacks with side effects remain unmodeled results. Calls to unknown
+helpers that receive a DataFrame still interrupt its provenance unless declared
+as `observers`. Merge suffixes and
+row multiplicity are not resolved. Async functions, reflection, `global`/`nonlocal`
+writes and properties remain unresolved. `.ipynb` JSON files, live cells and
+`dbutils.notebook.run` (a separate namespace) are not inputs. Opaque calls that
+receive a DataFrame still interrupt its provenance.
+
+The next acceptance input is a rerun of the internal compatibility report on
+dev7, identifying the first remaining identity break for the target column.

{coltrace-0.1.0.dev7 → coltrace-0.1.0.dev9}/docs/decisions.md

@@ -63,6 +63,19 @@ choices made within the user's authorization.
 | ADR-056 | User-declared observers | `[tool.coltrace].observers` lets users assert that project helpers only read their arguments, like source contracts for extraction |
 | ADR-057 | Evidence strength for inferred frames | DataFrame-only use promotes any unresolved result; weaker use (string subscripts, shared names) only supports documented conversions; Spark-only use rules frames out |
 | ADR-058 | Model reshaping with open derived schemas | Labels created from data derive from identified input columns instead of being unmodeled |
+| ADR-059 | Database columns are lineage nodes (schema 0.6) | The report's target column existed only inside SQL; `table` frames make physical columns selectable while SQL controls stay row controls |
+| ADR-060 | One identity per fully named relation | The same table read by several extractions is one node; relations named by unknown interpolations stay local to their site |
+| ADR-061 | Dialect fallback with explicit notes | Text that does not parse in its known or configured dialect is retried in common dialects; success is reported, never silent |
+| ADR-062 | Labeled placeholders and last-resort fragment removal | Unknown interpolations are shown as their Python expression; unparsable dynamic fragments are dropped only after every dialect fails, and the schema becomes incomplete |
+| ADR-063 | Documented conversions keep their receiver's SQL | `spark.sql(text).toPandas()` describes the SELECT rows; a transformation in between drops the query |
+| ADR-064 | Ranked, never merged, ambiguous candidates | Called code before templates and the earliest origin first; a database column and its same-name SQL output form one selectable family |
+| ADR-065 | Track dict entries; unroll known-length collections | Results kept in dicts were lost silently; exact iteration keeps each frame's identity |
+| ADR-066 | Inspect a second loop iteration for `None` accumulators | `acc = None` patterns need the state one iteration produces; bounded to loops that carry a `None` |
+| ADR-067 | Filters match database columns along the call chain | A table column is used by every extractor that reads it, not only by the closure where SQL runs |
+| ADR-068 | `SELECT *` is an open table schema | Used columns become table columns; lookups through merges of different frames never add columns to a table |
+| ADR-069 | Read SQL files as data | Only inside the analyzed root, SQL/text suffixes, bounded size; never executed |
+| ADR-070 | Keep ADR-057 for conversions | A census suggested promoting every `toPandas()` after SQL; precision tests and ADR-057 keep usage, annotations or contracts as the evidence |
+| ADR-071 | One record per distinct diagnostic | Unrolled iterations and repeated calling contexts produced identical records (the private run showed one line repeated dozens of times); identical records carry no extra information |
 
 The initial codename was FeatureTrace. The early architecture proposed reusable
 function summaries; direct contextual interpretation is used first because it