coltrace 0.1.0.dev10__tar.gz → 0.1.0.dev11__tar.gz

{coltrace-0.1.0.dev10/src/coltrace.egg-info → coltrace-0.1.0.dev11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev10
+Version: 0.1.0.dev11
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -32,7 +32,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev10`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev11`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/README.md

@@ -6,7 +6,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev10`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev11`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/backlog.md

@@ -84,6 +84,8 @@ not a capability of the published release.
 | CT-118 | Executor fan-out (`ThreadPoolExecutor.submit`, `as_completed`, `map`) and `tqdm` wrappers | Proposed: seen in LLM enrichment steps of the unified pipeline |
 | CT-119 | Control dependencies of dict values built under guards | Proposed |
 | CT-120 | Series index operations (`s.index[mask]`, `Series.get(label)`) after aggregation | Proposed: seen in a Pareto topic selection |
+| CT-121 | Dict and JSON outputs as destinations (`json.dump`, `f.write(json.dumps(...))` of values from cells) | Proposed: a co-occurrence extraction reaches a JSON config only through dicts |
+| CT-122 | Journey reach, logical columns, grouped steps, compact notebook/CLI output | Implemented dev11 |
 | CT-095 | Lazy materialization | Proposed: avoid eager pass copies of every known column on each row operation for very wide frames |
 
 ## Next checkpoint

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/compatibility-checkpoint.md

@@ -1,6 +1,6 @@
 # Project compatibility checkpoint
 
-Local checkpoint: `0.1.0.dev10`, 2026-11-05 (dev7 to dev9 sections below are
+Local checkpoint: `0.1.0.dev11`, 2026-11-05 (dev7 to dev10 sections below are
 kept as history). Publication is a separate step.
 This work responds to an anonymized compatibility report produced inside the
 user's company. The private project was not accessed, copied or executed; every
@@ -332,6 +332,45 @@ extraction census, warnings fell in 9 of 18 projects (for example 67 to 39 and
 library calls; one of those calls was `load_table_from_dataframe`, which is now
 a recorded export.
 
+## Sixth round: the dev10 run on the private pipeline (dev11)
+
+The user ran dev10 on the private pipeline and sent the cell output and the run
+artifacts. The analysis matched the anonymized copy (6,413 column versions,
+598 frames, 40 sources; the target database column had the same 19 downstream
+columns). Two complaints remained, both about how results were presented and
+selected:
+
+1. The target database column's journey "only showed references". Every step
+   sat on the extraction line (the SQL fields computed from the column and its
+   row controls), repeated once per calling context, and the outcome read "Last
+   recorded use; no later destination was established", although its derived
+   columns reach the published exports.
+2. Selecting the SQL output by name "gave an error". The cell in use was an
+   older copy that stops at an ambiguous name and prints every informational
+   note; its list held 85 identities. With the current cell, the first identity
+   was a secondary extraction of the same name (a totals query), not the one the
+   transform uses.
+
+Fixes: `journey.reach` and `journey.outcomes` follow the values into derived
+columns up to their exports (ADR-083); `journey.step_groups` counts repeated
+steps per place (ADR-085); `LineageGraph.logical_columns` groups identities into
+same-name lineages, and ranking starts from where a frame's story of the column
+begins, then prefers the family the pipeline transforms (ADR-084). Export events
+name their target instead of an option such as `orient="records"`. The notebook
+cell and `coltrace trace` print the grouped journey, where the values go and
+each derived column's definition, transformations and exports.
+
+On the anonymized copy the name now resolves to 10 logical columns (from 85
+identities); the first is the extraction the transform uses, and the database
+column's journey ends with "Values reach 2 recorded export(s)". The cell prints
+about 150 lines for either selection.
+
+1,051 tests pass (7 new in `test_journey_reach.py`); precision checks stay at
+14/14 expected and 0/9 forbidden. Measuring the new ranking on owid exposed a
+slowdown introduced during the change (an undirected view rebuilt per lookup:
+41 s to rank 6,629 identities of one name); it was fixed before release
+(0.15 s). Trace time on owid is unchanged (10 to 13 s per column, as before).
+
 ## Remaining boundaries
 
 This is still bounded abstract interpretation, not a general control-flow solver.

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/decisions.md

@@ -87,6 +87,9 @@ choices made within the user's authorization.
 | ADR-080 | Promote unresolved results at DataFrame parameters | Same evidence as ADR-057, observed in the callee; one frame per result object, so later uses share its identity |
 | ADR-081 | BigQuery `UNNEST`, STRUCT paths and `USING` | Nested fields are columns named by their path; `USING` keys are controls of both sides and an unqualified key is their `COALESCE`; physical tables are assumed to hold a key only as the single remaining candidate |
 | ADR-082 | Warehouse writers are exports | `load_table_from_dataframe`, `write_pandas` and `to_gbq` read the frame they receive; treating them as opaque invalidated it |
+| ADR-083 | Journeys follow values into derived columns | A database column's journey ended at the SQL that read it, so users saw only references; `reach` and `outcomes` show where its values go while `fate` keeps the column's own outcome |
+| ADR-084 | Logical columns for ambiguous names | 85 identities of one name were unusable as a choice; same-name lineages group them, and ranking starts from where a frame's story of the column begins, then prefers the family the pipeline transforms |
+| ADR-085 | Count repeated steps per place | An extractor called per segment repeated each step per calling context; grouped steps keep every record in JSON while readers see each action once |
 
 The initial codename was FeatureTrace. The early architecture proposed reusable
 function summaries; direct contextual interpretation is used first because it

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/lineage-json.md

@@ -204,6 +204,20 @@ The last recorded terminal outcome is a branch-local observation, not a claim
 that no other frame branch retains or consumes the column.
 These journey additions do not change the persisted snapshot structure.
 
+From dev11 the journey also answers where a column's values go when the column
+itself ends early (a database column read by SQL ends at the extraction):
+
+| Field | Content |
+| --- | --- |
+| `step_groups` | One entry per `relationship`, `kind`, `label`, location and scope, with `count` (records from repeated calls or loop iterations), distinct `contexts`, `step_ids`, the weakest `evidence`, `explanation` and the first `expression` |
+| `outcomes` | Derived columns by name, nearest first: `name`, a representative `column_id`, all `column_ids`, `families`, `distance`, `relationship`, `definitions`, `actions` (applications, writes, derivations), `destinations` (exports) and the most informative `fate` among its families |
+| `reach` | `status` (`exported`, `boundary`, `derived`, or the column's own fate status), a `label`, `derived_names`, and `destinations`, each export site with the column names (`via`) that carry the values there |
+
+`fate` keeps describing the selected column's own family. Export events are
+labeled by their target (`Export DataFrame to "out.csv"`) or, without one, by the
+method (`Export DataFrame with to_json`); options such as `orient` are never
+reported as a destination.
+
 ## Validation and evolution
 
 `load_report` validates schema kind/version, record fields and types, evidence

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/supported-patterns.md

@@ -414,10 +414,23 @@ SQL text is a SQL client, not a Pandas DataFrame using `DataFrame.query`.
 
 `LineageGraph.candidates(name)` lists every identity a name matches, most likely
 origin first: analyses of called code before uncalled-function templates, then
-database column, source field, write, and stronger evidence. `select` still
-raises `AmbiguousColumnError` for distinct lineages (its `candidates` use the
-same order); ranking never merges them. The notebook cell continues with the
-first candidate by default (`ON_AMBIGUOUS = "first"`).
+where the frame's story of that column starts (database column, source field,
+write; from dev11 a source overwritten in place, such as
+`df["x"] = pd.to_numeric(df["x"])`, still counts as a source), and stronger
+evidence. Among equally likely first origins (one extraction issued by several
+extractors; at most 64 tied families), the family whose values reach more
+derived columns comes first. `select` still raises `AmbiguousColumnError` for
+distinct lineages (its `candidates` use the same order); ranking never merges
+them.
+
+`LineageGraph.logical_columns(candidates)` (dev11) groups those identities into
+logical columns: every version of the name joined by same-name relations
+(copies, in-place cleanups, named aggregations, the `concat` of per-segment
+frames). Two separate extractions of one name stay separate. Each entry gives
+the first version (`column`), the function that issued the extraction (`via`),
+the number of versions, the names derived from its values and the exports they
+reach. The notebook cell lists logical columns instead of every identity and
+continues with the first one by default (`ON_AMBIGUOUS = "first"`).
 
 ## Values, labels and records across a notebook pipeline (dev10)
 

{coltrace-0.1.0.dev10 → coltrace-0.1.0.dev11}/docs/troubleshooting.md

@@ -118,12 +118,28 @@ unresolved facts. Reusing one after upgrading will not show newly supported edge
 
 Use the current `examples/notebook_cell.py`. Cells copied before dev7 print every
 informational note (thousands of `CT_CONDITIONAL_FLOW` lines), which notebook
-platforms truncate, and label modeled names as "Observed names". The current
-cell prints counts, grouped warnings, ranked candidates and, when `FUNCTION` or
-`MODULE` filters exclude a name, the identities that exist without them. Keep
-helper scripts (for example a one-off patch script) out of the analysis with
+platforms truncate, and label modeled names as "Observed names". Older cells also
+stop at an ambiguous name ("Multiple identities. Set COLUMN_ID ...") and list
+every identity. The current cell prints counts, grouped warnings, logical
+columns for an ambiguous name (continuing with the first), the journey with
+repeated steps counted, where the values go, and each derived column's
+definition, transformations and exports. When `FUNCTION` or `MODULE` filters
+exclude a name it lists the identities that exist without them. Keep helper
+scripts (for example a one-off patch script) out of the analysis with
 `.coltraceignore`.
 
+## A database column's journey only shows SQL references
+
+A database column read by SQL text ends at the extraction: its own steps are the
+SQL fields computed from it (`derivation`) and its uses in `WHERE`, `JOIN` or
+`GROUP BY` (row controls), and its own outcome is "last recorded use". From
+dev11 the journey follows its values into those derived columns: `reach` names
+the exports they reach and `outcomes` lists each derived column with where it is
+defined, transformed and exported. To follow one derived column in detail,
+select it by name; for a name produced by several extractions, the first logical
+column is the one the pipeline goes on to transform, and `FUNCTION` set to the
+extractor narrows it explicitly.
+
 ## A column used inside SQL is not found
 
 Since dev8, columns read by SQL text are `table` columns and can be selected