coltrace 0.1.0.dev4__tar.gz → 0.1.0.dev6__tar.gz

{coltrace-0.1.0.dev4 → coltrace-0.1.0.dev6}/MANIFEST.in

@@ -1,4 +1,5 @@
 recursive-include docs *.md
-recursive-include examples *.py *.md
+recursive-include examples *.py *.md *.toml
+include examples/read_capture/project/input.csv
 recursive-include tests *.py
 recursive-include src/coltrace/templates *.html

{coltrace-0.1.0.dev4 → coltrace-0.1.0.dev6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: coltrace
-Version: 0.1.0.dev4
+Version: 0.1.0.dev6
 Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
 Author: Cristiano Diniz
 License-Expression: MIT
@@ -15,6 +15,11 @@ Requires-Dist: ruff>=0.9; extra == "dev"
 Requires-Dist: mypy>=1.14; extra == "dev"
 Requires-Dist: build>=1.2; extra == "dev"
 Requires-Dist: types-networkx>=3.2; extra == "dev"
+Requires-Dist: sqlglot<31,>=30.24; extra == "dev"
+Provides-Extra: sql
+Requires-Dist: sqlglot<31,>=30.24; extra == "sql"
+Provides-Extra: capture
+Requires-Dist: pandas<4,>=2; extra == "capture"
 Provides-Extra: viz
 Requires-Dist: pyvis<0.4,>=0.3.2; extra == "viz"
 Dynamic: license-file
@@ -23,10 +28,11 @@ Dynamic: license-file
 
 Static Pandas column lineage and potential impact analysis for Python projects.
 ColTrace reads source files, identifies supported transformations, and connects
-column versions across direct function calls. It never imports or executes the
-project or reads the project's input data.
+column versions across direct function calls. Static scanning never imports or
+executes the project or reads its input data. Optional read capture observes
+column labels while you explicitly execute your own pipeline.
 
-**Development preview: `0.1.0.dev4`.** The supported subset is deliberately small.
+**Development preview: `0.1.0.dev6`.** The supported subset is deliberately small.
 Unsupported operations produce diagnostics and may interrupt lineage. A graph
 with no edge is not proof that a dependency does not exist.
 
@@ -37,15 +43,73 @@ Requires Python 3.11+. From this checkout:
 ```sh
 python -m venv .venv
 # Activate .venv using your shell, then:
-python -m pip install -e ".[dev,viz]"
+python -m pip install -e ".[dev,viz,sql,capture]"
 coltrace --help
 ```
 
 On Windows without activation, use `.\.venv\Scripts\python.exe -m coltrace`.
 The `viz` extra supplies PyVis for HTML export; basic analysis and JSON need only
-NetworkX and Typer. Pandas is not a runtime dependency. Publication to PyPI is a
+NetworkX and Typer. `sql` adds SQLGlot for bounded SQL parsing; `capture` adds
+Pandas for optional schema observation. Pandas is not a core dependency.
+Publication to PyPI is a
 separate release step; these commands install the local project.
 
+## Start at the data source
+
+The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
+extractions. Literal SQL projections, aliases and expressions describe source
+fields before their first Python use. Query filters are control metadata rather
+than value dependencies of every output. The supplied
+`spark.sql(build_query(table)).toPandas()` pattern works with a resolved Spark
+session or an explicit source contract for an injected session.
+
+`report.readers` inventories resolved `pd.read_*` calls with file, line, context,
+return shape and resulting DataFrame identity. File schemas are unknown unless
+declared or observed. To enumerate actual columns, including those never
+explicitly referenced by the script:
+
+```python
+from coltrace import analyze_project, capture_reads
+
+with capture_reads(project_root) as capture:
+    run_pipeline()  # You explicitly execute your existing pipeline.
+
+report = analyze_project(project_root, source_schemas=capture.schemas)
+```
+
+Capture retains column names and source locations, not rows or credentials.
+Use `capture_dataframe_schema` for an already-loaded Pandas object, including a
+SQL-to-Pandas conversion. Observations must attach to the corresponding analyzed
+`.py` extraction site; arbitrary notebook-cell lineage remains unsupported.
+Unknown SQL, wildcard schemas, dynamic calls and unsupported transformations
+still produce gaps. Schema observation does not resolve hidden value lineage.
+
+See [database source usage](docs/database-sources.md), the
+[researched extraction matrix](docs/database-patterns.md), and
+[read capture](docs/read-capture.md). Try the
+[static SQL example](examples/database_pipeline/README.md) and
+[executable capture example](examples/read_capture/README.md).
+The [source checkpoint](docs/source-checkpoint.md) records validation and limits.
+
+## Common Python scaffolding
+
+The dev6 checkpoint preserves independent DataFrame lineage across ordinary
+type annotations and unrelated configuration expressions. It supports bounded
+`copy(deep=True)`, `head(n)` and immutable `shape` observations.
+
+Selected `if`, `try`, `with` and loop paths are inspected independently. Facts
+from uncertain paths have inferred evidence. Mutated pre-existing frames and
+incompatible bindings remain explicit join boundaries; general path merging is
+not claimed. This makes previously skipped reads visible without treating every
+alternative as confirmed execution. Static scanning still executes no project
+code. See the [compatibility checkpoint](docs/compatibility-checkpoint.md) and
+[multi-module example](examples/robustness_pipeline/README.md).
+
+Notebook output separates discovered files from modeled frames/columns, shows
+configuration and versions, prints errors first, and saves full diagnostics
+before attempting column selection. Listed static labels are not runtime
+`df.columns`. Capture remains opt-in.
+
 ## Try the complete example
 
 ```sh
@@ -141,11 +205,15 @@ the shortest eligible distance, so a node reached both ways appears as direct.
 | Pattern | Support |
 | --- | --- |
 | Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
-| `read_csv`, `read_parquet` | External schema is unknown; CSV chunk/iterator forms are excluded |
+| Resolved `pd.read_*` calls | Located inventory; supported DataFrame readers plus explicit/observed source schemas; iterators/collections remain distinct |
+| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts |
+| MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
 | Literal dictionary `DataFrame` construction | String column keys and literal values |
 | `frame["column"]`, `frame.loc[:, "column"]` | Single literal string labels, all rows |
 | `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, scalar constants, temporary values, versions |
 | `alias = frame` | Same abstract object and shared mutation |
+| `copy(deep=True)`, `head(n)`, `shape` | Bounded independent copies, row previews and immutable metadata observations |
+| Selected conditional/exception/context blocks | Separate inferred paths; explicit uncertainty at joins |
 | `rename(columns={"old": "new"})` | Literal, non-colliding map; new DataFrame, or shared mutation with `inplace=True` |
 | `assign(b=lambda x: x["a"] * 2)` | Sequential simple lambdas; eager arguments use their original scope |
 | `Series.apply(clean_text)` | Resolved scalar callbacks; bounded summaries and inferred evidence |
@@ -200,7 +268,7 @@ print(report.to_json())
 
 The earlier `scan_project` API remains available for raw occurrences and receiver
 evidence. Its discovery report (`0.1`) is a different format from the lineage
-snapshot (`0.3`). See [discovery JSON](docs/discovery-json.md) and
+snapshot (`0.4`). See [discovery JSON](docs/discovery-json.md) and
 [lineage JSON](docs/lineage-json.md). Public facts are dataclasses; their serialized
 form is independent of NetworkX and contains no AST or executable objects.