coltrace 0.1.0.dev5__tar.gz → 0.1.0.dev7__tar.gz

{coltrace-0.1.0.dev5/src/coltrace.egg-info → coltrace-0.1.0.dev7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev5
+Version: 0.1.0.dev7
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -32,7 +32,7 @@ column versions across direct function calls. Static scanning never imports or
 executes the project or reads its input data. Optional read capture observes
 column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev5`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev7`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -56,7 +56,7 @@ separate release step; these commands install the local project.
 
 ## Start at the data source
 
-The local dev5 checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
+The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
 extractions. Literal SQL projections, aliases and expressions describe source
 fields before their first Python use. Query filters are control metadata rather
 than value dependencies of every output. The supplied
@@ -91,6 +91,40 @@ See [database source usage](docs/database-sources.md), the
 [executable capture example](examples/read_capture/README.md).
 The [source checkpoint](docs/source-checkpoint.md) records validation and limits.
 
+## Notebook projects and common Python structure
+
+Databricks notebook source (`# Databricks notebook source`, `# COMMAND ----------`,
+`# MAGIC`) and raw magics such as `%pip` are parsed with the magics neutralized
+in place. `%run ./child` includes the child notebook in the caller's namespace,
+statically. Nothing is executed. `.ipynb` files and live cells are not inputs.
+
+Branches, `try` blocks, loops and early returns are joined: a column that may
+hold different values after alternative paths gets an inferred merge version
+reading each possibility, so lineage continues without being claimed as
+confirmed. Loops over configured label collections (`for c in COLS`) are
+unrolled exactly. Classes, nested functions and closures, decorators,
+`*args/**kwargs`, `+=`, unpacking, comprehensions and ternaries are interpreted
+rather than erasing state.
+
+A documented catalog of Pandas/NumPy semantics covers common Series methods and
+accessors, `np.where`/`pd.to_numeric`-style functions, row-selecting DataFrame
+methods (`sort_values`, `head`, `reset_index`, `query`, …), whole-frame value
+methods (`fillna`, `astype`, …), partial `.loc[mask, col]` updates and
+`merge`/`concat` (inferred). When data enters through an injected helper such as
+`run_query(sql)`, ColTrace follows the closure when possible; otherwise, a result
+used as a DataFrame becomes an inferred frame whose SQL SELECT list seeds its
+columns, and the scan suggests a source contract to confirm it.
+
+See the [compatibility checkpoint](docs/compatibility-checkpoint.md), the
+[Databricks-style example](examples/databricks_project/README.md) and the
+[multi-module example](examples/robustness_pipeline/README.md).
+
+Notebook output separates discovered files from modeled frames/columns, shows
+configuration and versions, prints errors first, separates informational notes
+(inferences) from warnings, and saves full diagnostics outside the analyzed tree
+before attempting column selection. Listed static labels are not runtime
+`df.columns`. Capture remains opt-in.
+
 ## Try the complete example
 
 ```sh
@@ -190,15 +224,22 @@ the shortest eligible distance, so a node reached both ways appears as direct.
 | SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts |
 | MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
 | Literal dictionary `DataFrame` construction | String column keys and literal values |
-| `frame["column"]`, `frame.loc[:, "column"]` | Single literal string labels, all rows |
-| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, scalar constants, temporary values, versions |
+| `frame["column"]`, `frame.loc[:, "column"]` | Literal or statically known labels (`for c in COLS`, f-strings) |
+| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, masks, ternaries, scalar constants, temporary values, versions |
 | `alias = frame` | Same abstract object and shared mutation |
+| `copy(deep=True)`, `head(n)`, `shape` | Bounded independent copies, row previews and immutable metadata observations |
+| Conditional/exception/context/loop blocks, early returns | Separate inferred paths joined with inferred merge versions |
+| Databricks notebook source, `%pip`, `%run` | Magics neutralized; `%run` included statically in the caller namespace |
+| Series/DataFrame/NumPy catalog | Documented derivations, row selections, value methods, exports; see the subset |
+| `merge`, `join`, `concat` | Inferred frame whose columns may come from any input |
+| Classes, closures, decorators, unpacking | Interpreted; decorators assumed transparent (inferred) |
+| Injected `run_query(sql)` helpers | Followed through closures, or inferred from DataFrame use with SQL columns |
 | `rename(columns={"old": "new"})` | Literal, non-colliding map; new DataFrame, or shared mutation with `inplace=True` |
 | `assign(b=lambda x: x["a"] * 2)` | Sequential simple lambdas; eager arguments use their original scope |
 | `Series.apply(clean_text)` | Resolved scalar callbacks; bounded summaries and inferred evidence |
 | `frame[["a", "b"]] = DataFrame(results.tolist(), index=frame.index)` | Fixed callback output width and matching source index |
 | `dropna`, literal projection, boolean row filters | Preserve value lineage; predicate consumption tracked separately |
-| `to_parquet` / `to_csv` | Located export consumption with literal options |
+| `to_parquet`, `to_csv`, `to_excel`, `to_dict`, … | Located export/serialization consumption |
 | Direct synchronous function calls | Explicit arguments/returns, bounded acyclic contexts and explicit imports |
 | Annotated, uncalled functions | Separate inferred templates, not evidence of pipeline connections |
 
@@ -247,7 +288,7 @@ print(report.to_json())
 
 The earlier `scan_project` API remains available for raw occurrences and receiver
 evidence. Its discovery report (`0.1`) is a different format from the lineage
-snapshot (`0.4`). See [discovery JSON](docs/discovery-json.md) and
+snapshot (`0.5`). See [discovery JSON](docs/discovery-json.md) and
 [lineage JSON](docs/lineage-json.md). Public facts are dataclasses; their serialized
 form is independent of NetworkX and contains no AST or executable objects.
 
@@ -262,8 +303,13 @@ exclude = ["generated", "legacy/*.py", "*_test.py"]
 
 Glob patterns match case-sensitive, project-relative POSIX paths. Patterns with
 no `/` also match individual path components. Built-in exclusions cover `.venv`,
-`venv`, `.git`, `__pycache__`, `.coltrace`, pytest/mypy/Ruff caches, `build`, and
-`dist`. Linked files/directories and Windows reparse points are skipped.
+`venv`, `.git`, `__pycache__`, `.coltrace`, `.ipynb_checkpoints`, `node_modules`,
+tox/nox and pytest/mypy/Ruff caches, `build`, and `dist`. Linked files/directories
+and Windows reparse points are skipped. A `.coltraceignore` file at the analyzed
+root uses gitignore syntax (root-level subset, including `!` negation). The root
+`.gitignore` is also applied unless `respect_gitignore = false` is set.
+`observers = ["notify_team", "client.*"]` declares project helpers that only read
+the DataFrames they receive, so they do not interrupt lineage.
 `ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.
 
 ## Development