tracepipe 0.3.3__tar.gz → 0.3.5__tar.gz

{tracepipe-0.3.3 → tracepipe-0.3.5}/CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to TracePipe will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.3.5 - 2026-02-03
+
+### Fixed
+- **DataFrame.fillna double-logging**: `df.fillna({"col": 0})` now logs exactly 1 event
+  - Previously logged both `DataFrame.fillna` and internal `__setitem__` for same change
+  - Added `wrap_pandas_transform_method` with `_in_transform_op` flag to suppress nested setitem
+  - Works for both `fillna` and `replace` operations, including `inplace=True`
+
+### Added
+- Known Limitations section in README documenting concat/dedup tracking gaps
+- Test for `DataFrame.fillna` single-event logging
+
+## 0.3.4 - 2026-02-03
+
+### Fixed
+- **Event deduplication**: Identical events from parallel pipelines are now deduplicated
+  - When multiple DataFrames share row IDs (e.g., from `df.copy()`), same changes are recorded once
+  - Events deduplicated by `(col, old_val, new_val, operation)` signature
+  - Prevents "4 events" when only 1 logical change occurred
+
+### Added
+- `_stable_repr()` helper for robust value comparison in deduplication
+- Tests for cross-pipeline event deduplication behavior
+
 ## 0.3.3 - 2026-02-03
 
 ### Fixed

{tracepipe-0.3.3 → tracepipe-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tracepipe
-Version: 0.3.3
+Version: 0.3.5
 Summary: Row-level data lineage tracking for pandas pipelines
 Project-URL: Homepage, https://github.com/tracepipe/tracepipe
 Project-URL: Documentation, https://tracepipe.github.io/tracepipe/
@@ -276,6 +276,26 @@ tp.enable(mode="debug")  # Full lineage
 
 ---
 
+## Known Limitations
+
+TracePipe tracks **cell mutations** (fillna, replace, loc assignment) and **merge provenance** reliably. However, some patterns are not yet fully supported:
+
+| Pattern | Status | Notes |
+|---------|--------|-------|
+| `df["col"] = df["col"].fillna(0)` | ✅ Tracked | Series + assignment |
+| `df = df.fillna({"col": 0})` | ✅ Tracked | DataFrame-level fillna |
+| `df.loc[mask, "col"] = val` | ✅ Tracked | Conditional assignment |
+| `df.merge(other, on="key")` | ✅ Tracked | Full provenance in debug mode |
+| `pd.concat([df1, df2])` | ⚠️ Partial | Row IDs preserved, but no "source DataFrame" tracking |
+| `df.drop_duplicates(keep='last')` | ⚠️ Partial | Which row was kept is not tracked |
+| Sort + dedup patterns | ⚠️ Partial | "Latest record wins" logic not traced |
+
+**Why?** TracePipe tracks value changes within rows, not row-selection operations. When `drop_duplicates` picks one row over another, that's a provenance decision (not a cell mutation) that isn't currently instrumented.
+
+**Planned for 0.4**: Full row-provenance tracking for concat, drop_duplicates, and sort operations.
+
+---
+
 ## Contributing
 
 ```bash

{tracepipe-0.3.3 → tracepipe-0.3.5}/README.md

@@ -207,6 +207,26 @@ tp.enable(mode="debug")  # Full lineage
 
 ---
 
+## Known Limitations
+
+TracePipe tracks **cell mutations** (fillna, replace, loc assignment) and **merge provenance** reliably. However, some patterns are not yet fully supported:
+
+| Pattern | Status | Notes |
+|---------|--------|-------|
+| `df["col"] = df["col"].fillna(0)` | ✅ Tracked | Series + assignment |
+| `df = df.fillna({"col": 0})` | ✅ Tracked | DataFrame-level fillna |
+| `df.loc[mask, "col"] = val` | ✅ Tracked | Conditional assignment |
+| `df.merge(other, on="key")` | ✅ Tracked | Full provenance in debug mode |
+| `pd.concat([df1, df2])` | ⚠️ Partial | Row IDs preserved, but no "source DataFrame" tracking |
+| `df.drop_duplicates(keep='last')` | ⚠️ Partial | Which row was kept is not tracked |
+| Sort + dedup patterns | ⚠️ Partial | "Latest record wins" logic not traced |
+
+**Why?** TracePipe tracks value changes within rows, not row-selection operations. When `drop_duplicates` picks one row over another, that's a provenance decision (not a cell mutation) that isn't currently instrumented.
+
+**Planned for 0.4**: Full row-provenance tracking for concat, drop_duplicates, and sort operations.
+
+---
+
 ## Contributing
 
 ```bash

{tracepipe-0.3.3 → tracepipe-0.3.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tracepipe"
-version = "0.3.3"
+version = "0.3.5"
 description = "Row-level data lineage tracking for pandas pipelines"
 readme = "README.md"
 license = {file = "LICENSE"}

{tracepipe-0.3.3 → tracepipe-0.3.5}/tests/test_lineage_through_merge.py

@@ -249,6 +249,23 @@ class TestFillnaTrackingVerification:
             result.n_changes >= 1
         ), f"DataFrame.fillna should be tracked, got {result.n_changes} changes"
 
+    def test_dataframe_fillna_logs_once_not_twice(self):
+        """df.fillna({'col': val}) should log exactly 1 event, not 2.
+
+        Previously, both DataFrame.fillna and the internal __setitem__ were
+        recording the same change, causing double-logging.
+        """
+        tp.enable(mode="debug", watch=["a"])
+
+        df = pd.DataFrame({"a": [1.0, None, 3.0]})
+        df = df.fillna({"a": 0})
+
+        result = tp.why(df, col="a", row=1)
+        assert result.n_changes == 1, (
+            f"DataFrame.fillna should log exactly 1 event, got {result.n_changes}. "
+            f"Double-logging bug if > 1. History: {result.history}"
+        )
+
     def test_loc_assignment_tracked(self):
         """df.loc[mask, col] = val should be tracked."""
         tp.enable(mode="debug", watch=["a"])
@@ -350,6 +367,57 @@ class TestNoDoubleLogging:
             f"History: {result.history}"
         )
 
+    def test_cross_pipeline_identical_change_deduplication(self):
+        """Identical changes from parallel pipelines should be deduplicated.
+
+        When multiple pipelines from the same source do the SAME transformation
+        (e.g., both do fillna(0)), the event should only appear once.
+        """
+        tp.enable(mode="debug", watch=["income"])
+
+        # Source data with a row that has None income
+        customers = pd.DataFrame({"id": ["A", "B"], "income": [None, 100.0]})
+
+        # Two parallel pipelines doing the SAME transformation
+        df1 = customers.copy()
+        df1["income"] = df1["income"].fillna(0)  # Records None -> 0
+
+        df2 = customers.copy()
+        df2["income"] = df2["income"].fillna(0)  # Records SAME None -> 0
+
+        # Query df1 - should deduplicate identical events
+        result1 = tp.why(df1, col="income", row=0)
+        assert result1.n_changes == 1, (
+            f"Identical events should be deduplicated. Got {result1.n_changes}. "
+            f"History: {result1.history}"
+        )
+
+    def test_cross_pipeline_different_changes_preserved(self):
+        """Different changes from parallel pipelines should NOT be deduplicated.
+
+        When pipelines do DIFFERENT transformations on the same row,
+        both events should be visible (this is expected behavior - row IDs
+        are shared, so history includes all changes to that row ID).
+        """
+        tp.enable(mode="debug", watch=["income"])
+
+        customers = pd.DataFrame({"id": ["A", "B"], "income": [None, 100.0]})
+
+        # Two pipelines doing DIFFERENT transformations
+        df1 = customers.copy()
+        df1["income"] = df1["income"].fillna(0)  # None -> 0
+
+        df2 = customers.copy()
+        df2["income"] = df2["income"].fillna(99)  # None -> 99
+
+        # Query df1 - since row ID is shared, both changes are visible
+        # This is expected: deduplication only removes IDENTICAL events
+        result1 = tp.why(df1, col="income", row=0)
+        assert result1.n_changes == 2, (
+            f"Different changes should both be visible. Got {result1.n_changes}. "
+            f"History: {result1.history}"
+        )
+
 
 class TestMergeWarningScoping:
     """Tests for merge warnings being scoped to df's lineage."""

{tracepipe-0.3.3 → tracepipe-0.3.5}/tracepipe/__init__.py

@@ -81,7 +81,7 @@ from .core import TracePipeConfig, TracePipeMode
 from .snapshot import DiffResult, Snapshot, diff, snapshot
 
 # === VERSION ===
-__version__ = "0.3.3"
+__version__ = "0.3.5"
 
 # === MINIMAL __all__ ===
 __all__ = [

{tracepipe-0.3.3 → tracepipe-0.3.5}/tracepipe/context.py

@@ -63,6 +63,10 @@ class TracePipeContext:
         # When > 0, __getitem__[mask] skips capture (parent op will capture)
         self._filter_op_depth: int = 0
 
+        # Transform operation tracking (prevents double-counting fillna/replace)
+        # When > 0, __setitem__ skips capture (transform op will capture)
+        self._in_transform_op: int = 0
+
         # GroupBy state stack (supports nesting)
         self._groupby_stack: list[dict] = []
 

{tracepipe-0.3.3 → tracepipe-0.3.5}/tracepipe/instrumentation/pandas_inst.py

@@ -22,6 +22,7 @@ from ..safety import (
     get_caller_info,
     wrap_pandas_method,
     wrap_pandas_method_inplace,
+    wrap_pandas_transform_method,
 )
 from ..utils.value_capture import find_changed_indices_vectorized
 from .apply_capture import instrument_apply_pipe, uninstrument_apply_pipe
@@ -554,6 +555,9 @@ def _wrap_setitem(original):
 
     Captures BEFORE state for existing columns, then executes assignment,
     then records the diff with actual old/new values.
+
+    Skips recording when inside a transform operation (fillna, replace) to
+    avoid double-counting cell changes - the transform wrapper will capture.
     """
 
     @wraps(original)
@@ -565,7 +569,9 @@ def _wrap_setitem(original):
         is_new_column = False
         should_track = False
 
-        if ctx.enabled and isinstance(key, str):
+        # Skip tracking if we're inside a transform operation (fillna, replace)
+        # Those operations will capture the change themselves
+        if ctx.enabled and isinstance(key, str) and ctx._in_transform_op == 0:
             if key in ctx.watched_columns:
                 should_track = True
                 if key in self.columns:
@@ -771,13 +777,15 @@ def instrument_pandas():
             wrapped = wrap_filter_method(method_name, original)
             setattr(pd.DataFrame, method_name, wrapped)
 
-    # === DataFrame transform methods (with inplace support) ===
+    # === DataFrame transform methods (fillna, replace) ===
+    # These use wrap_pandas_transform_method to suppress __setitem__ recording
+    # during the transform, avoiding double-counting cell changes
     transform_methods = ["fillna", "replace"]
     for method_name in transform_methods:
         if hasattr(pd.DataFrame, method_name):
             original = getattr(pd.DataFrame, method_name)
             _originals[f"DataFrame.{method_name}"] = original
-            wrapped = wrap_pandas_method_inplace(method_name, original, _capture_transform)
+            wrapped = wrap_pandas_transform_method(method_name, original, _capture_transform)
             setattr(pd.DataFrame, method_name, wrapped)
 
     # === astype (no inplace) ===

{tracepipe-0.3.3 → tracepipe-0.3.5}/tracepipe/safety.py

@@ -103,7 +103,7 @@ def _make_wrapper(
         method_name: Name for error messages
         original_method: The original pandas method
         capture_func: func(self, args, kwargs, result, ctx, method_name)
-        mode: "standard", "filter", or "inplace"
+        mode: "standard", "filter", "inplace", or "transform"
     """
 
     @wraps(original_method)
@@ -112,10 +112,21 @@ def _make_wrapper(
 
         # === PRE-EXECUTION SETUP ===
         before_snapshot = None
+        is_inplace = kwargs.get("inplace", False)
 
         if mode == "filter" and ctx.enabled:
             ctx._filter_op_depth += 1
-        elif mode == "inplace" and ctx.enabled and kwargs.get("inplace", False):
+        elif mode == "transform" and ctx.enabled:
+            # Suppress __setitem__ recording during transform ops (fillna, replace)
+            # to avoid double-counting the same cell change
+            ctx._in_transform_op += 1
+            # Also handle inplace for transform operations
+            if is_inplace:
+                try:
+                    before_snapshot = self.copy()
+                except Exception:
+                    pass
+        elif mode == "inplace" and ctx.enabled and is_inplace:
             try:
                 before_snapshot = self.copy()
             except Exception:
@@ -127,15 +138,18 @@ def _make_wrapper(
         finally:
             if mode == "filter" and ctx.enabled:
                 ctx._filter_op_depth -= 1
+            elif mode == "transform" and ctx.enabled:
+                ctx._in_transform_op -= 1
 
         # === CAPTURE LINEAGE (SIDE EFFECT) ===
         # Skip capture if we're inside a filter operation (prevents recursion during export)
         if ctx.enabled and ctx._filter_op_depth == 0:
             try:
-                if mode == "inplace" and kwargs.get("inplace", False):
+                # Handle inplace for both "inplace" and "transform" modes
+                if (mode == "inplace" or mode == "transform") and is_inplace:
                     if before_snapshot is not None:
                         capture_func(before_snapshot, args, kwargs, self, ctx, method_name)
-                elif mode == "inplace" and result is not None:
+                elif (mode == "inplace" or mode == "transform") and result is not None:
                     capture_func(self, args, kwargs, result, ctx, method_name)
                 else:
                     capture_func(self, args, kwargs, result, ctx, method_name)
@@ -176,3 +190,14 @@ def wrap_pandas_method_inplace(
 ) -> Callable:
     """Wrap a pandas method that supports inplace=True."""
     return _make_wrapper(method_name, original_method, capture_func, mode="inplace")
+
+
+def wrap_pandas_transform_method(
+    method_name: str, original_method: Callable, capture_func: Callable
+) -> Callable:
+    """Wrap a pandas transform method (fillna, replace) that may trigger internal setitem.
+
+    These methods modify column values and pandas internally uses setitem.
+    We suppress setitem recording during these ops to avoid double-counting.
+    """
+    return _make_wrapper(method_name, original_method, capture_func, mode="transform")

{tracepipe-0.3.3 → tracepipe-0.3.5}/tracepipe/storage/lineage_store.py

@@ -32,6 +32,22 @@ from ..core import (
 from ..utils.value_capture import capture_typed_value
 
 
+def _stable_repr(val) -> str:
+    """Create a stable string representation for deduplication.
+
+    Handles NaN, None, and other values that don't compare equal to themselves.
+    """
+    if val is None:
+        return "None"
+    # Handle NaN (which doesn't equal itself)
+    try:
+        if isinstance(val, float) and val != val:  # NaN check
+            return "NaN"
+    except (TypeError, ValueError):
+        pass
+    return repr(val)
+
+
 class InMemoryLineageStore:
     """
     Columnar storage for lineage data using Structure of Arrays (SoA).
@@ -556,12 +572,15 @@ class InMemoryLineageStore:
         Follows merge lineage recursively to build complete cell provenance.
         This is essential for tracking changes that happened before merge operations.
 
+        Deduplicates events by (col, old_val, new_val, operation) signature to prevent
+        cross-pipeline contamination when multiple DataFrames share row IDs.
+
         Args:
             row_id: Row ID to trace
             max_depth: Maximum merge depth to follow (prevents infinite loops)
 
         Returns:
-            List of events in chronological order, including parent row events.
+            List of UNIQUE events in chronological order, including parent row events.
         """
         visited: set[int] = set()
 
@@ -589,7 +608,23 @@ class InMemoryLineageStore:
         # Sort by step_id to ensure chronological order across lineage
         all_events.sort(key=lambda e: e["step_id"])
 
-        return all_events
+        # Deduplicate by (col, old_val, new_val, operation) signature
+        # This prevents cross-pipeline contamination when multiple DataFrames
+        # share the same row IDs (e.g., df.copy() followed by parallel transforms)
+        seen_signatures: set[tuple] = set()
+        unique_events = []
+        for event in all_events:
+            sig = (
+                event.get("col"),
+                _stable_repr(event.get("old_val")),
+                _stable_repr(event.get("new_val")),
+                event.get("operation"),
+            )
+            if sig not in seen_signatures:
+                seen_signatures.add(sig)
+                unique_events.append(event)
+
+        return unique_events
 
     def get_cell_history_with_lineage(
         self, row_id: int, column: str, max_depth: int = 10

{tracepipe-0.3.3 → tracepipe-0.3.5}/uv.lock

@@ -2051,7 +2051,7 @@ wheels = [
 
 [[package]]
 name = "tracepipe"
-version = "0.3.2"
+version = "0.3.4"
 source = { editable = "." }
 dependencies = [
     { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.10'" },