coltrace 0.1.0.dev8__tar.gz → 0.1.0.dev10__tar.gz

{coltrace-0.1.0.dev8/src/coltrace.egg-info → coltrace-0.1.0.dev10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev8
+Version: 0.1.0.dev10
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -32,7 +32,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev8`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev10`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -65,6 +65,9 @@ Query filters are control metadata rather than value dependencies of every
 output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
 an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
 and joins keeps its known text, with unknown parts shown as `{expression}`.
+`SELECT *`, `UNION`, tables read by name (`spark.table`, Snowpark
+`session.table`, `dbt.ref`) and SQL files read inside the project are modeled
+too; see [extraction architectures](docs/extraction-architectures.md).
 
 `report.readers` inventories resolved `pd.read_*` calls with file, line, context,
 return shape and resulting DataFrame identity. File schemas are unknown unless

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/README.md

@@ -6,7 +6,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev8`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev10`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -39,6 +39,9 @@ Query filters are control metadata rather than value dependencies of every
 output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
 an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
 and joins keeps its known text, with unknown parts shown as `{expression}`.
+`SELECT *`, `UNION`, tables read by name (`spark.table`, Snowpark
+`session.table`, `dbt.ref`) and SQL files read inside the project are modeled
+too; see [extraction architectures](docs/extraction-architectures.md).
 
 `report.readers` inventories resolved `pd.read_*` calls with file, line, context,
 return shape and resulting DataFrame identity. File schemas are unknown unless

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/backlog.md

@@ -71,11 +71,24 @@ not a capability of the published release.
 | CT-105 | Ranked candidates for ambiguous names | Implemented dev8: `LineageGraph.candidates`, notebook `ON_AMBIGUOUS` |
 | CT-106 | Cursor `execute` then `fetch*` on separate statements | Proposed: remember the last SQL text executed on a resolved cursor name |
 | CT-107 | Jinja/dbt templates (`{{ ref() }}`) | Proposed: out of scope until a real project needs it |
+| CT-108 | Dict entries, exact unrolling, accumulators | Implemented dev9 |
+| CT-109 | Filters along the call chain | Implemented dev9 |
+| CT-110 | Open `SELECT *` schemas, UNION, tables by name, SQL files | Implemented dev9 |
+| CT-111 | Extraction census tool | Done in the session scratchpad; promote to `tools/` if it becomes a regular check |
+| CT-112 | `%%bigquery df` cell magic as a SQL source | Proposed (Jupyter/Colab users) |
+| CT-113 | `global`/`nonlocal`, `globals()` reads, per-statement budgets | Implemented dev10 |
+| CT-114 | Row and group iteration, records, scalar value flow | Implemented dev10 |
+| CT-115 | Labels chosen among known strings (read, write, `get`) | Implemented dev10 |
+| CT-116 | Row-wise apply with labeled Series, tuple joins, record exports, introspection builtins | Implemented dev10 |
+| CT-117 | BigQuery `UNNEST`, STRUCT paths, `JOIN USING` | Implemented dev10 |
+| CT-118 | Executor fan-out (`ThreadPoolExecutor.submit`, `as_completed`, `map`) and `tqdm` wrappers | Proposed: seen in LLM enrichment steps of the unified pipeline |
+| CT-119 | Control dependencies of dict values built under guards | Proposed |
+| CT-120 | Series index operations (`s.index[mask]`, `Series.get(label)`) after aggregation | Proposed: seen in a Pareto topic selection |
 | CT-095 | Lazy materialization | Proposed: avoid eager pass copies of every known column on each row operation for very wide frames |
 
 ## Next checkpoint
 
-1. Rerun the internal compatibility report with dev8: select the target column (now a database column if it is only read inside SQL) and classify the first remaining break.
+1. Rerun the private pipeline with dev10 and the current notebook cell: select the target database column and confirm that the prioritization outputs and the published exports appear downstream; classify any remaining break.
 2. Inspect the generated HTML's pan/zoom/selection behavior in a normal browser.
 3. Run ColTrace against a small real pipeline with a manually checked expected graph.
 4. Classify missing/incorrect relations by semantic rule, using [accuracy gates](accuracy.md).

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/compatibility-checkpoint.md

@@ -1,7 +1,7 @@
 # Project compatibility checkpoint
 
-Local checkpoint: `0.1.0.dev8`, 2026-11-04 (dev7 sections below are kept as
-history). Publication is a separate step.
+Local checkpoint: `0.1.0.dev10`, 2026-11-05 (dev7 to dev9 sections below are
+kept as history). Publication is a separate step.
 This work responds to an anonymized compatibility report produced inside the
 user's company. The private project was not accessed, copied or executed; every
 fixture below is a new synthetic reproduction of a reported technical shape.
@@ -90,8 +90,10 @@ input, not measured accuracy on the private project.
 
 ## Validation
 
-dev8: 984 tests pass (941 in dev7, plus the SQL boundary tests below; SQL
-control tests now require database-table columns consumed as row controls).
+dev9: 1,024 tests pass (984 in dev8; dev9 adds container, accumulator, filter
+and extraction-shape tests). dev8: 984 (941 in dev7, plus the SQL boundary
+tests; SQL control tests require database-table columns consumed as row
+controls).
 Ruff lint/format and strict mypy pass. Lineage snapshots are schema `0.6`
 (adds `dataframes[].kind`); `0.4` and `0.5` snapshots load as DataFrame frames.
 
@@ -207,6 +209,129 @@ gaps after it are now reported (+74 `CT_OPAQUE_CALL`); disabling the new text
 rules reproduces the dev7 numbers exactly. owid analysis time is unchanged
 (about 39 s). Hand-written precision checks: 14/14 expected, 0/9 forbidden.
 
+## Fourth round: the dev8 run and an extraction census (dev9)
+
+The user ran dev8 on the private project. Extraction now worked (3,140 column
+versions, 0 errors; physical table fields appeared among modeled names), but
+the run used a pre-dev7 copy of the notebook cell, which printed every
+informational note; the platform truncated the output, and the cell stopped at
+"Column not found with these filters". The visible diagnostics after extraction
+pointed at results kept in dicts and combined with `acc = None` accumulators
+(`tot.merge(...)` on an unresolved receiver, `CT_UNRESOLVED_RECEIVER` runs,
+`pd.concat` and `groupby` on lost frames).
+
+Synthetic reproductions confirmed three defects: a dict filled by subscript was
+still the empty literal, so loops over it were skipped silently; iterating
+`frames.items()` invalidated the frames; and a loop inspected once never saw the
+accumulator's `merge` branch. A fourth issue was selection: a database column
+belonged only to the function of its first reading site (often the injected
+`run_query` closure), so a `function=` filter on the extractor excluded it.
+All four are fixed; the synthetic example now includes a dict of results
+combined by an accumulator (`combine_sources.py`).
+
+The user also asked for a map of extraction architectures. A static census of
+public projects classified every extraction call shape and compared it with
+ColTrace's output; the result is [extraction architectures](extraction-architectures.md).
+Shapes fixed in this round: `SELECT *` open schemas, `UNION`, tables read by
+name, `.result()` and row-only steps before conversion, literal projections,
+keyword options on conversions, SQL files, and SQL clients whose name looked
+like a DataFrame.
+
+Measured on the census projects, extraction sites with SQL fields went from 43
+to 79 and inferred frames of unresolved origin from 68 to 35; conversions whose
+result is never used as a DataFrame stay unmodeled by design (ADR-057, ADR-070).
+On the earlier public harness nothing crashed, every snapshot round-trips, and
+the hand-written precision checks stay at 14/14 expected and 0/9 forbidden.
+From dev9 identical diagnostics are reported once, so warning counts are not
+comparable with earlier rounds (owid: 858 distinct warnings; recommenders: 538).
+owid models more code (16,317 frames and 67,450 column versions, from 15,416
+and 66,426) because paths built from `__file__`, `os.path.join` and known
+`**kwargs` now resolve and known-length loops are unrolled.
+
+## Fifth round: the anonymized unified pipeline (dev10)
+
+The user shared an anonymized copy of the whole pipeline with the output of the
+last run, because the target column still did not flow end to end. It is a
+notebook orchestrator that runs two KPI pipelines with `%run`; each extracts
+from BigQuery through injected `run_query` closures, builds weekly facts, trends
+(rank-based trend tests and a change-point library) and a prioritization score,
+then publishes JSON-in-script files and an LLM-assisted reading. The copy was
+analyzed statically outside the repository; nothing from it is part of the
+repository, and the fixtures below are synthetic.
+
+Measured with the same script on that copy:
+
+| | dev9 (published wheel) | dev10 |
+| --- | --- | --- |
+| Frames / column versions | 404 / 3,189 | 599 / 6,419 |
+| Extraction sources / table frames | 24 / 7 | 40 / 12 |
+| Target database column | not found | 19 downstream columns, through the score to the published export |
+| A database field of the second KPI | not found | 45 downstream columns |
+| `CT_ANALYSIS_LIMIT` | 30 | 0 |
+| `CT_UNRESOLVED_RECEIVER` | 42 | 8 |
+| `CT_UNMODELED_RESULT` | 20 | 1 |
+| `CT_OPAQUE_CALL` | 10 | 6 |
+| `CT_CONDITIONAL_RETURN` | 9 | 2 |
+| `CT_UNRESOLVED_CALL` | 7 | 1 |
+| `CT_DYNAMIC_COLUMN` | 2 | 1 |
+| `CT_PARTIAL_SQL` | 10 | 15 (more SQL is reached; 39 before `UNNEST`/`USING` support) |
+
+Causes, in pipeline order (each is a test in
+`tests/integration/test_unified_pipeline_shapes.py`):
+
+1. The first KPI's extract cached its client in a module global
+   (`global _client`). Functions declaring `global` were rejected, so the
+   warehouse source was never created and the target column did not exist.
+2. Early notebook cells spent the conditional-path budget
+   (30 `CT_ANALYSIS_LIMIT`), which reset bindings in later cells.
+3. The orchestrator read `%run` results through `globals().get(...)`.
+4. Trends iterate groups and rows, collect dicts, build frames from records,
+   and compute scalars with `list(...values)`, `float`, `round`, `max`, `zip`
+   and helpers that return labels under guards.
+5. The main break: a mask helper picks the period column by a parameter
+   (`"month" if kind == ... else "quarter"`). The dynamic read invalidated the
+   facts frame just before the ranking, so every ranked value had an opaque
+   origin.
+6. The score is computed by a row-wise `apply` returning `pd.Series({...})`,
+   attached with `pd.concat(axis=1)`.
+7. The ranking returns `(dict, flag)` on several paths, builds the dict with a
+   comprehension over `to_dict(orient="records")`, and the caller reads it with
+   `.get`.
+8. `getattr(frame, "columns", [])` and `isinstance(frame, ...)` were opaque
+   calls that invalidated the trend frames.
+9. The second KPI passed `run_query(sql)` straight to a post-processing
+   function, so its SQL was never attached to the frame.
+10. `np.asarray(values)` passed to a change-point library invalidated the trend
+    series, and a guarded `return {"flag": False}` was turned into a scalar.
+11. The second KPI's SQL uses `FROM t AS s, UNNEST(s.items) AS c`, STRUCT fields
+    and `LEFT JOIN ... USING (key)`.
+
+What remains in that copy: LLM judge steps (thread-pool fan-out, `tqdm`, the LLM
+calls themselves), a topic selection that indexes an aggregated Series by its
+index, and SQL names that anonymization collapsed into duplicates.
+
+### Validation
+
+1,044 tests pass (15 new integration tests in `test_unified_pipeline_shapes.py`
+and 5 new SQL unit tests); Ruff and strict mypy pass. On the public harness
+(13 projects, round 14) nothing crashes, every snapshot round-trips, and the
+hand-written precision checks stay at 14/14 expected and 0/9 forbidden.
+Distinct warnings went from 1,859 to 1,849 in total while more code is modeled
+(owid: 67,450 to 83,918 column versions and 858 to 909 warnings; recommenders:
+3,495 to 4,179 column versions and 538 to 496 warnings). Analysis time is
+unchanged when projects run alone (owid 54 s). Under memory pressure, owid's
+572 MB snapshot round-trip in the harness took hours; it is a harness cost, not
+an analysis one.
+
+The harness caught one regression before release: scalars computed from cells
+made a library call that also received Series look like a scalar function,
+which hid an opaque call and the frame built from its result. It is fixed and
+covered by `test_unknown_call_receiving_column_views_stays_opaque`. On the
+extraction census, warnings fell in 9 of 18 projects (for example 67 to 39 and
+41 to 15) and rose by one in two, where newly modeled frames reach unknown
+library calls; one of those calls was `load_table_from_dataframe`, which is now
+a recorded export.
+
 ## Remaining boundaries
 
 This is still bounded abstract interpretation, not a general control-flow solver.
@@ -215,10 +340,13 @@ with zero iterations. `groupby(...).apply`, `stack`/`unstack`, `crosstab` and
 row-wise callbacks with side effects remain unmodeled results. Calls to unknown
 helpers that receive a DataFrame still interrupt its provenance unless declared
 as `observers`. Merge suffixes and
-row multiplicity are not resolved. Async functions, reflection, `global`/`nonlocal`
-writes and properties remain unresolved. `.ipynb` JSON files, live cells and
+row multiplicity are not resolved. Async functions, reflection and properties
+remain unresolved (`global` and `nonlocal` are analyzed from dev10). Executor
+fan-out (`ThreadPoolExecutor.submit`, `as_completed`) and `tqdm` wrappers are
+opaque. `.ipynb` JSON files, live cells and
 `dbutils.notebook.run` (a separate namespace) are not inputs. Opaque calls that
 receive a DataFrame still interrupt its provenance.
 
-The next acceptance input is a rerun of the internal compatibility report on
-dev7, identifying the first remaining identity break for the target column.
+The next acceptance input is a rerun of the private pipeline with dev10 and
+the current notebook cell, selecting the target database column and confirming
+the prioritization outputs and published exports downstream.

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/decisions.md

@@ -69,6 +69,24 @@ choices made within the user's authorization.
 | ADR-062 | Labeled placeholders and last-resort fragment removal | Unknown interpolations are shown as their Python expression; unparsable dynamic fragments are dropped only after every dialect fails, and the schema becomes incomplete |
 | ADR-063 | Documented conversions keep their receiver's SQL | `spark.sql(text).toPandas()` describes the SELECT rows; a transformation in between drops the query |
 | ADR-064 | Ranked, never merged, ambiguous candidates | Called code before templates and the earliest origin first; a database column and its same-name SQL output form one selectable family |
+| ADR-065 | Track dict entries; unroll known-length collections | Results kept in dicts were lost silently; exact iteration keeps each frame's identity |
+| ADR-066 | Inspect a second loop iteration for `None` accumulators | `acc = None` patterns need the state one iteration produces; bounded to loops that carry a `None` |
+| ADR-067 | Filters match database columns along the call chain | A table column is used by every extractor that reads it, not only by the closure where SQL runs |
+| ADR-068 | `SELECT *` is an open table schema | Used columns become table columns; lookups through merges of different frames never add columns to a table |
+| ADR-069 | Read SQL files as data | Only inside the analyzed root, SQL/text suffixes, bounded size; never executed |
+| ADR-070 | Keep ADR-057 for conversions | A census suggested promoting every `toPandas()` after SQL; precision tests and ADR-057 keep usage, annotations or contracts as the evidence |
+| ADR-071 | One record per distinct diagnostic | Unrolled iterations and repeated calling contexts produced identical records (the private run showed one line repeated dozens of times); identical records carry no extra information |
+| ADR-072 | Analyze `global` and `nonlocal` | A cached module-level warehouse client was lost because functions declaring `global` were rejected; declared names read the module namespace, resolved calls write back, templates never do |
+| ADR-073 | Budgets per top-level statement | Long notebooks spent the path budget in early cells and cut later ones; each top-level module statement gets the full budget |
+| ADR-074 | Values computed from cells are derived scalars | Builtins, NumPy, `math` and subscripts over cells carry dependencies and never alias a frame; guarded scalar returns depend on the guard's columns, containers keep their structure |
+| ADR-075 | Records are frames; record exports yield records | List-of-dict records build open-schema frames; `to_dict(orient="records")` stays an export and also yields row dicts whose values are cells |
+| ADR-076 | Labels chosen among known strings | A column picked by a parameter invalidated the whole frame; reading every possible column (inferred) is sound and bounded to 8 strings; writes behave like branch joins |
+| ADR-077 | Row-wise apply with labeled Series builds columns | Pandas returns a DataFrame whose columns are the labels; each column reads the row fields of its value |
+| ADR-078 | Join fixed-width tuples by position | `return {}, False` and `return result, ok` lost both positions when merged as unordered members |
+| ADR-079 | Introspection builtins are pure | `isinstance`, `hasattr`, `callable` and literal `getattr` were opaque calls that invalidated frames |
+| ADR-080 | Promote unresolved results at DataFrame parameters | Same evidence as ADR-057, observed in the callee; one frame per result object, so later uses share its identity |
+| ADR-081 | BigQuery `UNNEST`, STRUCT paths and `USING` | Nested fields are columns named by their path; `USING` keys are controls of both sides and an unqualified key is their `COALESCE`; physical tables are assumed to hold a key only as the single remaining candidate |
+| ADR-082 | Warehouse writers are exports | `load_table_from_dataframe`, `write_pandas` and `to_gbq` read the frame they receive; treating them as opaque invalidated it |
 
 The initial codename was FeatureTrace. The early architecture proposed reusable
 function summaries; direct contextual interpretation is used first because it

coltrace-0.1.0.dev10/docs/extraction-architectures.md

@@ -0,0 +1,66 @@
+# Extraction architectures
+
+How data enters a Pandas pipeline varies more than any other part of it. This
+page maps the extraction shapes ColTrace has met in a real Databricks project
+and in a static census of public projects, and states what the dev9 analysis
+does with each one. Nothing is executed: SQL text, SQL files and notebook
+source are read as data.
+
+The census (November 2026) covered Airflow/Astro SDK DAGs, Prefect recipes,
+Dagster and Hamilton examples, the BigQuery Python samples, Snowpark demos, dbt
+Python models, Databricks talks, Kaggle SQL course notebooks, Feast examples, a
+Google Cloud data-engineering course and a small personal ETL repository.
+
+## What a recognized extraction produces
+
+- A **source** record with the SQL text (unknown interpolations shown as
+  `{expression}`), relations, output fields and row controls.
+- A **DataFrame** whose columns are written by the source.
+- A **database table** frame for every physical relation the SQL reads, with one
+  column node per physical column. Selecting such a column (for example a field
+  used only inside `SUM(...)` or a `WHERE`) traces it into the DataFrame.
+
+## Map
+
+| Shape | Example | dev9 treatment |
+| --- | --- | --- |
+| Pandas SQL readers | `pd.read_sql(sql, con)`, `read_sql_query`, `read_gbq` | Source with fields; dialect from reader or fallback |
+| Connector chains | `bigquery.Client().query(sql).to_dataframe()`, `snowflake ... cursor().execute(sql).fetch_pandas_all()`, `duckdb.sql(sql).df()` | Source with fields when the connector object is resolved |
+| Injected callables | `run_query(sql)` closure or parameter | Followed through closures; otherwise inferred from DataFrame use, SQL fields seeded |
+| Unresolved session or client | `spark.sql(sql).toPandas()`, `client.query(sql).result().to_dataframe(...)`, `conn.execute(sql).fetchdf()` | SQL kept through `.result()` and the conversion; a frame once used as one, returned under a `pd.DataFrame` annotation, or declared by a contract |
+| Tables read by name | `spark.table("db.t")`, `spark.read.table(...)`, Snowpark `session.table("T")`, dbt `dbt.ref("m")`/`dbt.source("s", "t")`, BigQuery `client.list_rows(t)` | Open schema of that table (as `SELECT *`) |
+| Row-only steps before conversion | `.filter(...)`, `.where`, `.limit(n)`, `.distinct()`, `.orderBy`, `.sample` | SQL or table kept |
+| Literal projections before conversion | `.select("A", "B")`, `.drop("SECRET")` | Field list narrowed; a literal `select` makes the schema complete |
+| Spark column expressions | `.withColumn(...)`, `.groupBy(...).agg(...)`, `select(F.col(...))` | SQL dropped: the SELECT list no longer describes the result |
+| `SELECT *` / `t.*` / `* EXCEPT (...)` over one table | `pd.read_sql("SELECT * FROM t", con)` | Open schema: columns the code uses become columns of `t`; merges with other frames never add columns to `t` |
+| `UNION [ALL]` | `SELECT a FROM t UNION SELECT b FROM u` | Labels from the first branch, inputs from every branch by position |
+| f-strings, `.format`, `%`, `+`, `join` | `f"... FROM {TABLE} WHERE d >= '{start}'"` | Known constants, config values and parameter defaults folded; unknown parts shown as `{expression}` |
+| Dynamic fragments | `f"... WHERE x = 1 {extra}"` | Unplaceable fragment ignored as a last resort, with a note |
+| Vendor dialects | BigQuery backticks, Databricks, Snowflake | Reader or call-shape dialect, `[tool.coltrace].sql_dialect`, then fallbacks, with a note |
+| SQL files | `open("sql/q.sql").read()`, `(Path(__file__).parent / "sql" / "q.sql").read_text().format(**params)` | Read as data when inside the analyzed root, SQL/text suffix, at most 200 KB |
+| DB-API parameters | `%s`, `%(name)s`, `?`, `:name`, `@name` | Read as parameters, never as columns |
+| Results kept in containers | `results = {}`; `results["a"] = df`; `for k, d in results.items()` | Dict entries tracked; loops over known-length collections unrolled exactly |
+| Accumulators | `acc = None`; `acc = d if acc is None else acc.merge(d)` | A second iteration is inspected; `None` dropped where the name is used as a frame |
+
+## Boundaries
+
+| Shape | Why | What to do |
+| --- | --- | --- |
+| Jinja/dbt SQL templates (`{{ ref() }}`, `{% if %}`) | Template languages are not SQL | Keep a rendered example, or declare a source contract/schema |
+| `cursor.execute(sql)` and `cursor.fetchall()` in separate statements | The query is not attached to the later fetch | Use `pd.read_sql`, or a contract |
+| `%%bigquery df` / `%sql` cell magics | Neutralized as notebook magics | Use the client API in Python cells |
+| Feature stores (`store.get_historical_features(...).to_df()`) | Feature references are not SQL | Declare a source schema |
+| Unknown SDK returning Pandas | No SQL text or table name | `[[tool.coltrace.sources]]` contract with `pandas_method` if needed |
+| Ibis, Polars, Dask, bigframes transformations | Different libraries | Lineage starts at the Pandas conversion |
+
+Spelling alone never creates a DataFrame (ADR-057): `x.toPandas()` on an
+unknown `x`, or a conversion whose result is never used as a frame, stays
+unresolved. A source contract makes any of these explicit:
+
+```toml
+[[tool.coltrace.sources]]
+callable = "spark.sql"
+kind = "sql"
+dialect = "databricks"
+pandas_method = "toPandas"
+```

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/lineage-json.md

@@ -243,6 +243,12 @@ fully named relation is one identity across the whole snapshot.
 `load_report` upgrades 0.4 and 0.5 snapshots by adding `kind: "dataframe"` to
 their frames; 0.2/0.3 migrations are unchanged.
 
+From dev10 (still schema 0.6), a nested database field is a `table` column
+named by its path: `items.score` for `UNNEST(s.items) AS c` with `c.score`, and
+`info.flag` for the STRUCT access `s.info.flag`. The diagnostic code
+`CT_LABEL_CHOICE` (info) marks a column read or written through a label chosen
+among known strings.
+
 ## Changes in 0.5
 
 Schema 0.5 adds the diagnostic severity `"info"` for recorded inferences and

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/supported-patterns.md

@@ -419,6 +419,117 @@ raises `AmbiguousColumnError` for distinct lineages (its `candidates` use the
 same order); ranking never merges them. The notebook cell continues with the
 first candidate by default (`ON_AMBIGUOUS = "first"`).
 
+## Values, labels and records across a notebook pipeline (dev10)
+
+The dev9 run on an anonymized two-KPI notebook pipeline lost the target column
+between the warehouse extract and the prioritization score. Each cause below has
+a synthetic reproduction in `tests/integration/test_unified_pipeline_shapes.py`.
+
+- **Cached clients and globals.** Functions that declare `global` or `nonlocal`
+  are analyzed: a module-level singleton (`_client = None` filled on first use)
+  keeps its connector identity, and resolved calls write declared globals back
+  to the module (symbolic templates never do). `globals().get("NAME")` and
+  `globals()["NAME"]` read the namespace, as notebooks do after `%run`.
+- **Rows and groups.** `for _, row in df.iterrows()`, `itertuples()` and
+  `for key, group in df.groupby(...)` bind row cells and group frames;
+  `row["c"]`, `row.c` and `row.get("c")` read the column.
+- **Records.** `pd.DataFrame(list_of_dicts)` with known string keys builds a
+  frame whose columns are the possible values of each key (inferred, open
+  schema). `df.to_dict(orient="records")` is still recorded as an export and
+  now also yields row dicts whose values are cells of each column, so
+  `{key: row for row in records}` followed by `rows.get(key)["c"]` keeps
+  lineage. A dict comprehension is a tracked dict: one inspected entry stands
+  for every entry, and its unknown key may match any lookup.
+- **Scalar value flow.** `float`, `int`, `round`, `sum`, `min`, `max`, `abs`,
+  `list`, `tuple`, `sorted`, `set`, `.values`, `.tolist()`, `.item()`,
+  subscripts and `math.*` carry the dependencies of the cells they read
+  (inferred). NumPy and scalar Pandas functions over cells or lists of cells
+  (`np.asarray(values)`, `np.mean(x[:i])`, `pd.isna(v)`) return derived values,
+  not views of a frame, so passing them to an unresolved library no longer
+  invalidates the source frame. A scalar returned under `if slope < 0:` depends
+  on the guard's columns (control dependency, inferred); constant containers
+  returned under a guard keep their structure.
+- **Pairs and shared dicts.** `zip` and `enumerate` pair known members (an
+  unknown list pairs with unknown values); `record["q"] = q` on a dict collected
+  by `append` updates every alias of that dict.
+- **Labels chosen at run time.** A value that is one of a few known strings
+  (`"a" if flag else "b"`, `if/else` assignments, `LABELS[kind]` over a constant
+  mapping or list; at most 8 strings) is a label choice. `df[col]`,
+  `df.loc[rows, col]` and `df.get(col)` read each possible column (inferred,
+  `CT_LABEL_CHOICE`); a label absent from a closed schema cannot be the chosen
+  one. `df[col] = v` gives each candidate column an inferred version that reads
+  its previous value and `v`, like a branch join. Before dev10 such a read
+  invalidated the whole frame (`CT_DYNAMIC_COLUMN`).
+- **Constant Series.** `pd.Series([False] * len(df), index=df.index)` is a
+  Series without column inputs, aligned with `df`; joined with a mask (the
+  fallback of a mask helper), it is used as a mask.
+- **Row-wise apply with labeled values.** `df.apply(f, axis=1)` where `f`
+  returns `pd.Series({"a": x, "b": y})` builds a frame with one column per
+  label, each derived from the row fields its value reads;
+  `pd.concat([df, that], axis=1)` attaches the columns.
+- **Tuple returns.** Fixed-width tuples returned on alternative paths
+  (`return {}, False` and `return result, ok`) join position by position.
+- **Introspection.** `isinstance(df, ...)`, `hasattr(df, "x")`, `callable(f)`
+  and `getattr(obj, "name"[, default])` with a literal name neither read nor
+  change column data. `getattr(df, "columns", [])` is `df.columns`; module,
+  class and instance attributes resolve as attribute reads.
+- **Unresolved results as arguments.** An unresolved conversion result
+  (`client.query(sql).to_dataframe()` on an unknown client) passed straight to a
+  parameter used as a DataFrame (`postprocess(run_query(sql))`) is promoted with
+  its SQL there (ADR-057 evidence: the callee's DataFrame use), once per result.
+- **Warehouse writers.** `client.load_table_from_dataframe(df, table)`,
+  `write_pandas(conn, df, table)` and `pandas_gbq.to_gbq(df, table)` are
+  recorded exports, like `spark.createDataFrame(df)`.
+- **Budgets.** The conditional-path and call budgets apply per top-level module
+  statement, so a long notebook is not cut by its earlier cells.
+- **BigQuery SQL.** In `FROM t AS s, UNNEST(s.items) AS c`, `c.score` is the
+  nested column `items.score` of `t` and `c` alone is the array column;
+  `UNNEST(arr) AS u(x)` makes `x` the element. `s.info.flag` is the nested
+  column `info.flag`. In `JOIN x USING (k)`, `k` is a control of both sides: a
+  CTE or derived table holds `k` when it outputs it, and a physical table when it
+  is the only remaining candidate; an unqualified `k` reads both sides
+  (`COALESCE`). Before dev10 every `USING` made the summary partial.
+
+Still outside after dev10: executor fan-out (`ThreadPoolExecutor.submit`,
+`as_completed`, `executor.map`) and `tqdm` wrappers are opaque; control
+dependencies of dict values built under guards (`{"flag": True}` versus
+`{"flag": False}`) are not recorded; LLM calls are opaque by nature.
+
+## Containers, accumulators and extraction shapes (dev9)
+
+The dev8 run on the private project showed lineage lost right after extraction,
+where results are kept in dicts and combined. A census of public projects then
+mapped extraction shapes; see [extraction architectures](extraction-architectures.md).
+
+- **Dicts of frames** are tracked entry by entry: literals, `d[k] = df`,
+  `d[k]`, `d.get(k)`, `items()`, `values()`, `keys()`, `update`, `setdefault`,
+  `pop`, and `f(**known_dict)`. A key that is not statically known may select
+  any entry (inferred). Before dev9 a dict filled by subscript stayed the empty
+  literal, so loops over it were skipped without a diagnostic.
+- **Loops** over a list, tuple or dict of known length (up to 64 items) are
+  unrolled exactly, binding each frame in turn. A literal empty iterable is
+  skipped; a name bound to an empty list or dict is still inspected, because it
+  may have been filled through an untracked path.
+- **Accumulators**: when a loop assigns a name that is `None` before it, a
+  second iteration is inspected so `acc.merge(part)` is seen; a join of `None`
+  and frames is the frame wherever the name is used as one.
+- **Selection filters**: `function=`/`module=` match a database column from any
+  function that reads it, including the callers on the analysis call chain (an
+  extractor that calls an injected `run_query`).
+- **`SELECT *`** over one table is an open schema of that table; **UNION**
+  branches are combined by position; **tables read by name**
+  (`spark.table`, `session.table`, `dbt.ref`, `dbt.source`, `list_rows`) are
+  `SELECT *` of that table; `.result()` and row-only Spark/Snowpark steps keep the
+  SQL; literal `select`/`drop` narrow it; conversions accept keyword options.
+- **SQL files** inside the analyzed root are read as text (`open(...).read()`,
+  `Path.read_text()`, `Path(__file__).parent / "sql"`, `os.path.join`), never
+  executed, and reported with `CT_SQL_FILE`. `__file__` is the module path
+  relative to the analyzed root, so snapshots never record local paths.
+- Identical diagnostics (same code, message, location and scope) are reported
+  once.
+- `client.query(SQL_CONSTANT)` on a name that looked like a DataFrame is a SQL
+  client call: `DataFrame.query` never takes SQL text.
+
 ## Public project patterns (dev7, second round)
 
 These rules come from static runs over public pipelines (Kedro starters, MLOps
@@ -497,8 +608,9 @@ invalidate frames they receive.
 
 ## Remaining unknown behavior
 
-`groupby(...).apply`, `stack`/`unstack`, `crosstab`, unlisted Series methods
-and other unsupported calls are opaque or unmodeled. Explicit column
+`groupby(...).apply`, `stack`/`unstack`, `crosstab`, unlisted Series methods,
+executor fan-out (`ThreadPoolExecutor.submit`) and other unsupported calls are
+opaque or unmodeled. Explicit column
 arguments can still be recorded as consumption, but unknown returned values do
 not acquire imaginary dependencies. Unknown calls can mutate arguments; affected
 state is invalidated.

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/docs/troubleshooting.md

@@ -114,6 +114,16 @@ unresolved facts. Reusing one after upgrading will not show newly supported edge
 - Keep helper scripts and outputs out of the analysis with `.coltraceignore`
   (gitignore syntax at the analyzed root) or `[tool.coltrace].exclude`.
 
+## Notebook output is truncated or still says "Observed names"
+
+Use the current `examples/notebook_cell.py`. Cells copied before dev7 print every
+informational note (thousands of `CT_CONDITIONAL_FLOW` lines), which notebook
+platforms truncate, and label modeled names as "Observed names". The current
+cell prints counts, grouped warnings, ranked candidates and, when `FUNCTION` or
+`MODULE` filters exclude a name, the identities that exist without them. Keep
+helper scripts (for example a one-off patch script) out of the analysis with
+`.coltraceignore`.
+
 ## A column used inside SQL is not found
 
 Since dev8, columns read by SQL text are `table` columns and can be selected
@@ -133,6 +143,23 @@ directly. If one is still missing:
   candidate is the most likely origin. Pass `--column-id` (CLI) or `COLUMN_ID`
   (notebook cell) to choose another.
 
+## A column stops at a helper that picks the column by a parameter
+
+Before dev10, `col = "month" if kind == "m" else "quarter"` followed by
+`df[col]` invalidated the frame (`CT_DYNAMIC_COLUMN`, then `CT_OPAQUE_CALL` on
+the next method). From dev10 the read is a label choice: each possible column
+is read with inferred evidence and the note `CT_LABEL_CHOICE` names the site. A
+label built from unknown text (user input, an f-string with unknown parts) is
+still dynamic; replace it with a constant mapping or a conditional over known
+labels.
+
+## A score computed in a row-wise `apply` has no inputs
+
+`df.apply(f, axis=1)` is modeled when `f` resolves and returns a value or
+`pd.Series({...})` with literal labels. Labels built at run time, callbacks that
+write to the row, and `groupby(...).apply` stay unmodeled
+(`CT_UNMODELED_RESULT`).
+
 ## A helper call interrupts lineage
 
 `CT_OPAQUE_CALL` marks a call ColTrace cannot resolve that receives a DataFrame

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/examples/databricks_project/README.md

@@ -12,6 +12,7 @@ business rules.
 | `shared/config.py` | Configured column lists and default parameters |
 | `pipeline_a/extract/primary_extract.py` | f-string SQL builder, `if df.empty: return df` guard, loop over configured columns |
 | `pipeline_a/extract/quality_extract.py` | SQL assembled from f-strings with BigQuery backticks, a helper fragment, an unknown metric column and an uncalled extractor (analyzed as a template) |
+| `pipeline_a/transform/combine_sources.py` | Results kept in a dict and combined with an `acc = None` accumulator and `merge` |
 | `shared/topic_changes_extract.py` | `spark.sql(build_query(days)).toPandas()` and `sdf = spark.sql(...); sdf.toPandas()` with an injected session |
 | `pipeline_a/transform/priority_rank.py` | Scalar parameters, `np.where`, `isin` masks, filters, `sort_values().head(n)` |
 | `pipeline_a/publish/serialize_webapp.py` | Projection, `to_dict(orient="records")`, a writer class |
@@ -29,7 +30,7 @@ python -m coltrace trace contact_attempts --from .coltrace/databricks/scan.json
 python -m coltrace graph priority_score --from .coltrace/databricks/scan.json --output .coltrace/databricks/priority_score.html
 ```
 
-Expected (dev8 also covers the follow-up report): all 15 files parse, the SQL SELECT list becomes the source of six
+Expected (dev8 and dev9 also cover the follow-up reports): all 16 files parse, the SQL SELECT list becomes the source of six
 columns, and `n_contacts` flows into `volume_share`, `priority_score` and
 `published_score`, which reach the Parquet export and the serialized view. Row
 filter and sort-key columns are recorded as control, not as value inputs.

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/examples/databricks_project/pipeline_a/main_orchestrator.py

@@ -12,6 +12,7 @@ from shared.warehouse import make_run_query
 from pipeline_a.extract.primary_extract import extract_primary
 from pipeline_a.extract.quality_extract import extract_quality
 from pipeline_a.publish.serialize_webapp import serialize
+from pipeline_a.transform.combine_sources import combine
 from pipeline_a.transform.priority_rank import prioritize
 
 # COMMAND ----------
@@ -32,3 +33,12 @@ payload_a = serialize(ranked_a, "/dbfs/tmp/pipeline_a.json")
 quality_a = extract_quality(run_query)
 topics_a = extract_topic_changes(spark, days=7)
 quality_a.merge(topics_a, on="session_id", how="left").to_parquet("/dbfs/tmp/quality_a.parquet")
+
+# COMMAND ----------
+
+results = {}
+results["handled"] = quality_a[["session_id", "n_attempts"]]
+results["waiting"] = extract_quality(run_query, metric="wait_seconds")[["session_id", "n_attempts"]]
+combined = combine(results)
+combined["attempts_total"] = combined["attempts_handled"] + combined["attempts_waiting"]
+combined.to_parquet("/dbfs/tmp/attempts.parquet")

coltrace-0.1.0.dev10/examples/databricks_project/pipeline_a/transform/combine_sources.py

@@ -0,0 +1,9 @@
+"""Results collected in a dict and combined with an ``acc = None`` accumulator."""
+
+
+def combine(results):
+    total = None
+    for name, part in results.items():
+        part = part.rename(columns={"n_attempts": f"attempts_{name}"})
+        total = part if total is None else total.merge(part, on="session_id", how="outer")
+    return total

{coltrace-0.1.0.dev8 → coltrace-0.1.0.dev10}/examples/notebook_cell.py

@@ -121,6 +121,15 @@ def test_column():
         print(f"Continuing with the first identity: {selected.id}")
     except ColumnNotFoundError:
         print("Column not found: " + missing_column_message(report, COLUMN_ID or COLUMN))
+        if not COLUMN_ID and (FUNCTION or MODULE):
+            unfiltered = graph.candidates(COLUMN)
+            if unfiltered:
+                print("Without FUNCTION/MODULE filters these identities exist:")
+                for item in unfiltered[:10]:
+                    print(
+                        f"  {item.id} | {item.origin} | {item.module}::{item.scope} | "
+                        f"{item.location.path}:{item.location.line}"
+                    )
         print("Database columns read by SQL text are selectable too (origin=table).")
         names = sorted({item.name for item in report.columns})
         print("Modeled static column names (not runtime df.columns):", ", ".join(names[:30]))