coltrace 0.1.0.dev6__tar.gz → 0.1.0.dev8__tar.gz

{coltrace-0.1.0.dev6 → coltrace-0.1.0.dev8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev6
+Version: 0.1.0.dev8
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -32,7 +32,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev6`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev8`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -58,10 +58,13 @@ separate release step; these commands install the local project.
 
 The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
 extractions. Literal SQL projections, aliases and expressions describe source
-fields before their first Python use. Query filters are control metadata rather
-than value dependencies of every output. The supplied
-`spark.sql(build_query(table)).toPandas()` pattern works with a resolved Spark
-session or an explicit source contract for an injected session.
+fields before their first Python use. Every database column the SQL reads is a
+selectable lineage node (`origin="table"`), so a column used only inside
+`SUM(...)` or a `WHERE` clause can be traced to the DataFrame columns it feeds.
+Query filters are control metadata rather than value dependencies of every
+output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
+an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
+and joins keeps its known text, with unknown parts shown as `{expression}`.
 
 `report.readers` inventories resolved `pd.read_*` calls with file, line, context,
 return shape and resulting DataFrame identity. File schemas are unknown unless
@@ -91,22 +94,37 @@ See [database source usage](docs/database-sources.md), the
 [executable capture example](examples/read_capture/README.md).
 The [source checkpoint](docs/source-checkpoint.md) records validation and limits.
 
-## Common Python scaffolding
-
-The dev6 checkpoint preserves independent DataFrame lineage across ordinary
-type annotations and unrelated configuration expressions. It supports bounded
-`copy(deep=True)`, `head(n)` and immutable `shape` observations.
-
-Selected `if`, `try`, `with` and loop paths are inspected independently. Facts
-from uncertain paths have inferred evidence. Mutated pre-existing frames and
-incompatible bindings remain explicit join boundaries; general path merging is
-not claimed. This makes previously skipped reads visible without treating every
-alternative as confirmed execution. Static scanning still executes no project
-code. See the [compatibility checkpoint](docs/compatibility-checkpoint.md) and
+## Notebook projects and common Python structure
+
+Databricks notebook source (`# Databricks notebook source`, `# COMMAND ----------`,
+`# MAGIC`) and raw magics such as `%pip` are parsed with the magics neutralized
+in place. `%run ./child` includes the child notebook in the caller's namespace,
+statically. Nothing is executed. `.ipynb` files and live cells are not inputs.
+
+Branches, `try` blocks, loops and early returns are joined: a column that may
+hold different values after alternative paths gets an inferred merge version
+reading each possibility, so lineage continues without being claimed as
+confirmed. Loops over configured label collections (`for c in COLS`) are
+unrolled exactly. Classes, nested functions and closures, decorators,
+`*args/**kwargs`, `+=`, unpacking, comprehensions and ternaries are interpreted
+rather than erasing state.
+
+A documented catalog of Pandas/NumPy semantics covers common Series methods and
+accessors, `np.where`/`pd.to_numeric`-style functions, row-selecting DataFrame
+methods (`sort_values`, `head`, `reset_index`, `query`, …), whole-frame value
+methods (`fillna`, `astype`, …), partial `.loc[mask, col]` updates and
+`merge`/`concat` (inferred). When data enters through an injected helper such as
+`run_query(sql)`, ColTrace follows the closure when possible; otherwise, a result
+used as a DataFrame becomes an inferred frame whose SQL SELECT list seeds its
+columns, and the scan suggests a source contract to confirm it.
+
+See the [compatibility checkpoint](docs/compatibility-checkpoint.md), the
+[Databricks-style example](examples/databricks_project/README.md) and the
 [multi-module example](examples/robustness_pipeline/README.md).
 
 Notebook output separates discovered files from modeled frames/columns, shows
-configuration and versions, prints errors first, and saves full diagnostics
+configuration and versions, prints errors first, separates informational notes
+(inferences) from warnings, and saves full diagnostics outside the analyzed tree
 before attempting column selection. Listed static labels are not runtime
 `df.columns`. Capture remains opt-in.
 
@@ -206,20 +224,25 @@ the shortest eligible distance, so a node reached both ways appears as direct.
 | --- | --- |
 | Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
 | Resolved `pd.read_*` calls | Located inventory; supported DataFrame readers plus explicit/observed source schemas; iterators/collections remain distinct |
-| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts |
+| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts; database columns as nodes; dialect fallback |
 | MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
 | Literal dictionary `DataFrame` construction | String column keys and literal values |
-| `frame["column"]`, `frame.loc[:, "column"]` | Single literal string labels, all rows |
-| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, scalar constants, temporary values, versions |
+| `frame["column"]`, `frame.loc[:, "column"]` | Literal or statically known labels (`for c in COLS`, f-strings) |
+| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, masks, ternaries, scalar constants, temporary values, versions |
 | `alias = frame` | Same abstract object and shared mutation |
 | `copy(deep=True)`, `head(n)`, `shape` | Bounded independent copies, row previews and immutable metadata observations |
-| Selected conditional/exception/context blocks | Separate inferred paths; explicit uncertainty at joins |
+| Conditional/exception/context/loop blocks, early returns | Separate inferred paths joined with inferred merge versions |
+| Databricks notebook source, `%pip`, `%run` | Magics neutralized; `%run` included statically in the caller namespace |
+| Series/DataFrame/NumPy catalog | Documented derivations, row selections, value methods, exports; see the subset |
+| `merge`, `join`, `concat` | Inferred frame whose columns may come from any input |
+| Classes, closures, decorators, unpacking | Interpreted; decorators assumed transparent (inferred) |
+| Injected `run_query(sql)` helpers | Followed through closures, or inferred from DataFrame use with SQL columns |
 | `rename(columns={"old": "new"})` | Literal, non-colliding map; new DataFrame, or shared mutation with `inplace=True` |
 | `assign(b=lambda x: x["a"] * 2)` | Sequential simple lambdas; eager arguments use their original scope |
 | `Series.apply(clean_text)` | Resolved scalar callbacks; bounded summaries and inferred evidence |
 | `frame[["a", "b"]] = DataFrame(results.tolist(), index=frame.index)` | Fixed callback output width and matching source index |
 | `dropna`, literal projection, boolean row filters | Preserve value lineage; predicate consumption tracked separately |
-| `to_parquet` / `to_csv` | Located export consumption with literal options |
+| `to_parquet`, `to_csv`, `to_excel`, `to_dict`, … | Located export/serialization consumption |
 | Direct synchronous function calls | Explicit arguments/returns, bounded acyclic contexts and explicit imports |
 | Annotated, uncalled functions | Separate inferred templates, not evidence of pipeline connections |
 
@@ -268,7 +291,7 @@ print(report.to_json())
 
 The earlier `scan_project` API remains available for raw occurrences and receiver
 evidence. Its discovery report (`0.1`) is a different format from the lineage
-snapshot (`0.4`). See [discovery JSON](docs/discovery-json.md) and
+snapshot (`0.5`). See [discovery JSON](docs/discovery-json.md) and
 [lineage JSON](docs/lineage-json.md). Public facts are dataclasses; their serialized
 form is independent of NetworkX and contains no AST or executable objects.
 
@@ -283,8 +306,15 @@ exclude = ["generated", "legacy/*.py", "*_test.py"]
 
 Glob patterns match case-sensitive, project-relative POSIX paths. Patterns with
 no `/` also match individual path components. Built-in exclusions cover `.venv`,
-`venv`, `.git`, `__pycache__`, `.coltrace`, pytest/mypy/Ruff caches, `build`, and
-`dist`. Linked files/directories and Windows reparse points are skipped.
+`venv`, `.git`, `__pycache__`, `.coltrace`, `.ipynb_checkpoints`, `node_modules`,
+tox/nox and pytest/mypy/Ruff caches, `build`, and `dist`. Linked files/directories
+and Windows reparse points are skipped. A `.coltraceignore` file at the analyzed
+root uses gitignore syntax (root-level subset, including `!` negation). The root
+`.gitignore` is also applied unless `respect_gitignore = false` is set.
+`sql_dialect = "bigquery"` names the dialect of SQL text whose reader does not;
+text that does not parse is still retried in common dialects and reported.
+`observers = ["notify_team", "client.*"]` declares project helpers that only read
+the DataFrames they receive, so they do not interrupt lineage.
 `ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.
 
 ## Development