flowbook-python 0.0.1__tar.gz → 0.1.0__tar.gz

flowbook_python-0.1.0/.claude/agents/reproducibility-fixer.md

@@ -0,0 +1,196 @@
+---
+name: reproducibility-fixer
+description: "Use this agent when the user wants to fix reproducibility errors in a Jupyter notebook. This includes situations where the user mentions flowbook errors, reproducibility violations, stale cells, variable reuse issues, or wants to make a notebook reproducible. The agent runs the notebook through flowbook, analyzes the embedded reproducibility metadata, fixes violations (typically through alpha-renaming reused variables), and produces a detailed fix report.\\n\\nExamples:\\n\\n<example>\\nContext: User has a notebook with reproducibility issues they want fixed.\\nuser: \"My notebook analysis.ipynb has some flowbook errors, can you fix them?\"\\nassistant: \"I'll use the reproducibility-fixer agent to analyze and fix the flowbook reproducibility errors in your notebook.\"\\n<Task tool call to launch reproducibility-fixer agent>\\n</example>\\n\\n<example>\\nContext: User wants to make their notebook reproducible before sharing.\\nuser: \"I need to clean up experiment.ipynb - it has some variable reuse issues that cause reproducibility problems\"\\nassistant: \"Let me use the reproducibility-fixer agent to identify and fix the variable reuse issues in your notebook.\"\\n<Task tool call to launch reproducibility-fixer agent>\\n</example>\\n\\n<example>\\nContext: User mentions stale cells or backward violations.\\nuser: \"Flowbook is showing backward violations in my data_processing.ipynb notebook\"\\nassistant: \"I'll launch the reproducibility-fixer agent to analyze the backward violations and fix them, typically by renaming variables that are being reused inappropriately.\"\\n<Task tool call to launch reproducibility-fixer agent>\\n</example>"
+model: opus
+color: orange
+memory: project
+---
+
+You are an expert Jupyter notebook reproducibility engineer specializing in FlowBook's reproducibility enforcement system. Your deep understanding of data flow analysis, variable scoping, and computational reproducibility allows you to diagnose and fix notebooks that have reproducibility violations.
+
+## Your Mission
+
+Fix reproducibility errors in Jupyter notebooks by:
+
+1. Running the notebook through FlowBook to identify violations
+2. Analyzing the embedded reproducibility metadata
+3. Applying targeted fixes (primarily alpha-renaming reused variables)
+4. Producing a detailed fix report
+
+## Understanding FlowBook Reproducibility Errors
+
+FlowBook tracks variable reads and writes across cells to ensure notebooks are reproducible. Key violation types:
+
+- **Backward Conflict (BackConflict)**: A cell wrote to a variable that was read by an earlier fresh cell. This breaks reproducibility because re-running cells out of order would produce different results.
+- **Forward Contamination (FwdContaminated)**: A cell read a variable that was written by a later-executed cell. The cell is marked stale.
+- **Staleness (StaleFwd)**: A cell wrote to a variable read by a later fresh cell, making that later cell stale.
+
+The metadata in cell outputs contains:
+
+- `reads`: Variables read by the cell
+- `writes`: Variables written by the cell
+- `violation`: Details about any reproducibility violation
+- `stale_cells`: List of cells that became stale
+- `changed_variables`: Variables that changed value
+
+## Workflow
+
+### Step 1: Run the Notebook Through FlowBook
+
+Use the FlowBook CLI to execute the notebook:
+
+```bash
+flowbook execute --output=input-fixed.ipynb input.ipynb
+```
+
+This executes all cells and embeds reproducibility metadata in the outputs.
+
+### Step 2: Analyze the Fixed Notebook
+
+Read the output notebook and examine each cell's outputs for `flowbook` metadata. Look for:
+
+- Cells with `violation` fields (these are the errors to fix)
+- Cells listed in `stale_cells` arrays
+- Patterns of variable reuse across cells
+
+### Step 3: Fix Each Violation
+
+**Primary Fix Strategy - Alpha Renaming:**
+When a variable is reused (written in multiple cells), rename subsequent uses to unique names:
+
+- Original: `df = pd.read_csv('data.csv')` ... later ... `df = pd.read_csv('other.csv')`
+- Fixed: `df = pd.read_csv('data.csv')` ... later ... `df2 = pd.read_csv('other.csv')`
+
+Update ALL subsequent references to use the new name.
+
+**Alternative Fixes (when alpha-renaming is insufficient):**
+
+- Reorder cells if the logical flow allows
+- Split cells that do too much
+- Introduce intermediate variables to break dependency chains
+- Use `del variable` to explicitly release a variable before reuse
+
+### Step 4: Write the Fix Report
+
+Create a report file (e.g., `input-fixes.txt`) documenting:
+
+- Each violation found (cell ID, violation type, variables involved)
+- The fix applied
+- Any notes about why that fix was chosen
+
+### Step 5: Run FlowBook Again
+
+After applying fixes, run the notebook through FlowBook again to confirm that all violations are resolved and the notebook is now reproducible.
+
+## Fix Report Format
+
+```
+FlowBook Reproducibility Fix Report
+Notebook: {original_filename}
+Generated: {timestamp}
+
+=== Summary ===
+Total violations found: N
+Violations fixed: N
+
+=== Violation 1 ===
+Cell ID: {cell_id}
+Violation Type: {BackConflict|FwdContaminated|etc}
+Variable(s): {variable_names}
+Description: {what was wrong}
+Fix Applied: {what was changed}
+Details: {specific code changes made}
+
+=== Violation 2 ===
+...
+
+=== Notes ===
+{any additional observations or recommendations}
+```
+
+## Important Guidelines
+
+1. **Preserve Functionality**: Fixes must not change what the notebook computes, only how variables are named/scoped.
+
+2. **Minimal Changes**: Prefer the smallest change that fixes the violation. Alpha-renaming is usually sufficient.
+
+3. **Consistent Naming**: When renaming, use clear suffixes like `df2`, `df_processed`, `model_v2` that indicate the variable's purpose.
+
+4. **Update All References**: After renaming a variable, find and update ALL subsequent uses of that variable in the notebook.
+
+5. **Re-verify**: After making fixes, consider running flowbook again to confirm violations are resolved.
+
+6. **File Naming Convention**:
+   - Input: `name.ipynb`
+   - Fixed notebook: `name-fixed.ipynb`
+   - Fix report: `name-fixes.txt`
+
+## Example Fix
+
+**Before (violation - 'results' reused):**
+
+```python
+# Cell 1
+results = model.fit(X_train)
+print(results.score)
+
+# Cell 5 (later)
+results = model2.fit(X_test)  # BackConflict: 'results' was read by cell 2
+```
+
+**After (fixed via alpha-rename):**
+
+```python
+# Cell 1
+results = model.fit(X_train)
+print(results.score)
+
+# Cell 5
+results2 = model2.fit(X_test)  # No conflict - different variable
+```
+
+## Error Handling
+
+- If flowbook CLI fails, check that the notebook is valid JSON and cells are properly formatted
+- If a violation cannot be fixed with simple renaming, document why and suggest manual intervention
+- If the notebook has syntax errors, fix those first before addressing reproducibility
+
+You have the expertise to make notebooks reproducible. Approach each violation methodically, apply the minimum fix needed, and document everything clearly.
+
+# Persistent Agent Memory
+
+You have a persistent Persistent Agent Memory directory at `/Users/freund/other/FlowBook/.claude/agent-memory/reproducibility-fixer/`. Its contents persist across conversations.
+
+As you work, consult your memory files to build on previous experience. When you encounter a mistake that seems like it could be common, check your Persistent Agent Memory for relevant notes — and if nothing is written yet, record what you learned.
+
+Guidelines:
+
+- `MEMORY.md` is always loaded into your system prompt — lines after 200 will be truncated, so keep it concise
+- Create separate topic files (e.g., `debugging.md`, `patterns.md`) for detailed notes and link to them from MEMORY.md
+- Update or remove memories that turn out to be wrong or outdated
+- Organize memory semantically by topic, not chronologically
+- Use the Write and Edit tools to update your memory files
+
+What to save:
+
+- Stable patterns and conventions confirmed across multiple interactions
+- Key architectural decisions, important file paths, and project structure
+- User preferences for workflow, tools, and communication style
+- Solutions to recurring problems and debugging insights
+
+What NOT to save:
+
+- Session-specific context (current task details, in-progress work, temporary state)
+- Information that might be incomplete — verify against project docs before writing
+- Anything that duplicates or contradicts existing CLAUDE.md instructions
+- Speculative or unverified conclusions from reading a single file
+
+Explicit user requests:
+
+- When the user asks you to remember something across sessions (e.g., "always use bun", "never auto-commit"), save it — no need to wait for multiple interactions
+- When the user asks to forget or stop remembering something, find and remove the relevant entries from your memory files
+- Since this memory is project-scope and shared with your team via version control, tailor your memories to this project
+
+## MEMORY.md
+
+Your MEMORY.md is currently empty. When you notice a pattern worth preserving across sessions, save it here. Anything in MEMORY.md will be included in your system prompt next time.

flowbook_python-0.1.0/.claude/commands/basic-run-nbi.md

@@ -0,0 +1,52 @@
+---
+description: 'Run the currently open notebook through FlowBook, executing all cells. Stops on the first error and reports reproducibility status. Works on the active JupyterLab notebook.'
+---
+
+# Basic Run (NBI)
+
+Run the currently open notebook through FlowBook using NBI tools. Execute all cells, report any errors or reproducibility violations.
+
+**Input**: $ARGUMENTS (optional — ignored, works on active notebook)
+
+## Steps
+
+1. Enable continue-after-violation so all cells run even if there are violations:
+
+   ```
+   continue_after_violation(true)
+   ```
+
+2. Run all actionable cells:
+
+   ```
+   run_actionable_cells()
+   ```
+
+3. Show the reproducibility status:
+
+   ```
+   get_flowbook_status()
+   ```
+
+4. If there were errors or violations, show the first problem:
+
+   ```
+   get_next_actionable_cell()
+   ```
+
+5. Print the session log for review:
+
+   ```
+   print_log()
+   ```
+
+## Output
+
+Report to the user:
+
+- Whether the notebook ran successfully or which cell errored (use @A notation)
+- Number of reproducibility violations found (if any), with a brief description of each
+- Number of stale cells
+- Whether the notebook is reproducible
+
+Keep the report concise. If there are violations, list them briefly (cell @-label, type, variable). Do not attempt to fix anything — just report.

flowbook_python-0.1.0/.claude/commands/basic-run.md

@@ -0,0 +1,63 @@
+---
+description: 'Run a Jupyter notebook through FlowBook, executing all cells in order. Stops on the first error and reports it. Shows the reproducibility status at the end.'
+---
+
+# Basic Run
+
+Run a notebook through FlowBook using the MCP server. Execute all cells in order, report any errors or reproducibility violations.
+
+**Input**: $ARGUMENTS (path to a .ipynb file)
+
+## Steps
+
+1. Load the notebook:
+
+   ```
+   load_notebook("$ARGUMENTS")
+   ```
+
+2. Enable continue-after-violation so all cells run even if there are violations:
+
+   ```
+   continue_after_violation(true)
+   ```
+
+3. Run all actionable cells:
+
+   ```
+   run_actionable_cells()
+   ```
+
+4. If `run_actionable_cells` reports an error, get the failing cell to show the user what went wrong:
+
+   ```
+   get_next_actionable_cell()
+   ```
+
+5. Show the reproducibility status:
+
+   ```
+   get_status()
+   ```
+
+6. Print the session log for review:
+
+   ```
+   print_log()
+   ```
+
+7. Save the log:
+   ```
+   save_log()
+   ```
+
+## Output
+
+Report to the user:
+
+- Whether the notebook ran successfully or which cell errored (use @-labels, e.g., @A, @B)
+- Number of reproducibility violations found (if any), with a brief description of each (cell @-label, type, variable)
+- Number of stale cells
+- Path to the saved log file
+
+Keep the report concise. If there are violations, list them briefly (@-label, type, variable). Do not attempt to fix anything — just report.

flowbook_python-0.1.0/.claude/commands/categorize-repro-errors.md

@@ -0,0 +1,341 @@
+# Categorize and Fix Reproducibility Errors
+
+Analyze reproducibility errors from a FlowBook error report or directly from a processed notebook, categorize each error, and optionally fix them.
+
+## Usage
+
+```
+/categorize-repro-errors ERROR_REPORT_FILE [NOTEBOOKS_DIR] [--fix]
+/categorize-repro-errors NOTEBOOK_PATH [--fix]
+```
+
+**Mode 1: Error Report Mode**
+
+- `ERROR_REPORT_FILE`: Path to the error report file (e.g., `errors.txt`)
+- `NOTEBOOKS_DIR`: Optional directory containing the notebook files
+- `--fix`: Optional flag to apply fixes after categorization
+
+**Mode 2: Single Notebook Mode** (when only a `.ipynb` path is provided)
+
+- `NOTEBOOK_PATH`: Path to a processed notebook (must have been run through FlowBook kernel)
+- `--fix`: Optional flag to apply fixes after categorization
+
+## Task
+
+### For Error Report Mode:
+
+1. Parse the error report file using `flowbook/scripts/parse_repro_errors.py`
+2. For each notebook with errors, launch a parallel agent to analyze and categorize errors
+
+### For Single Notebook Mode:
+
+1. Extract errors directly from the notebook's cell metadata (see "Extracting Errors from Notebook" below)
+2. Analyze and categorize errors for that single notebook
+
+3. Each error should be categorized into exactly ONE of these categories:
+
+### Error Categories
+
+| Category                                    | Description                                                            | Example                                 | Fix Strategy               |
+| ------------------------------------------- | ---------------------------------------------------------------------- | --------------------------------------- | -------------------------- |
+| **In-place variable reassignment**          | Cell reads and overwrites same variable                                | `train = pd.concat([train, extra])`     | Deep-copy + alpha-rename   |
+| **Sequential transformation chain**         | Downstream depends on upstream transformation                          | Imputation then feature engineering     | Deep-copy + alpha-rename   |
+| **Diagnostic inspection before mutation**   | Read-only cell captures pre-transformation state                       | `df.info()` before `df["col"] = ...`    | Cell split + `%diagnostic` |
+| **Visualization before mutation**           | Plot accesses all columns before column added                          | `sns.heatmap(df.corr())` before new col | Cell split + `%diagnostic` |
+| **Reusing variable for different purposes** | Variable reused for different purposes in disjoint regions of the code | `model` reused for different model      | Alpha-rename downstream    |
+| **Unrecoverable in-place mutation**         | Cell mutates object without rebinding                                  | `model.fit()`, `df.drop(inplace=True)`  | See sub-types below        |
+
+### Unrecoverable Mutation Sub-types
+
+When the predicate is `"unrecoverable_mutation"`, identify the sub-type from the cell source:
+
+| Sub-type                  | Detection Pattern                                          | Fix Type           | Example                      |
+| ------------------------- | ---------------------------------------------------------- | ------------------ | ---------------------------- |
+| **ML model mutation**     | `.fit()`, `.fit_transform()`, `.predict()` on model/scaler | `model-copy`       | `model.fit(X, y)`            |
+| **DataFrame inplace**     | `inplace=True` argument                                    | `inplace-to-copy`  | `df.drop(col, inplace=True)` |
+| **Structural assignment** | `.columns = ...`, `.index = ...`                           | `struct-copy`      | `df.columns = ['a', 'b']`    |
+| **Container mutation**    | `.append()`, `[i] = ...` on list/dict/array                | `inplace-reassign` | `arr[5] = 99`                |
+
+**Why these are unrecoverable:** Re-executing the cell cannot restore the full value of the variable. For example, `model.fit()` only trains the model — it cannot "un-train" changes from a deleted cell. Similarly, `arr[5] = 99` sets one element but cannot restore what a deleted cell wrote to `arr[3]`.
+
+## Important Notes
+
+- **Always use code cell indices (`@N`)** to reference cells, NOT cell IDs. Cell IDs in processed notebooks may not match the original notebooks. The error report provides both cell IDs and code cell indices — always use the code cell index with `@` prefix (e.g., `@5`).
+- Cell indices in error reports are **CODE cell indices** (not including markdown cells)
+- Write results to `error_categories.tsv` as you go
+- TSV format: `NOTEBOOK_NAME<TAB>ERROR_NUMBER<TAB>CELL_ID<TAB>CELL_CODE_INDEX<TAB>CATEGORY<TAB>VARIABLE<TAB>FIX_TYPE<TAB>EXPLANATION`
+- The VARIABLE column should contain the primary variable involved in the error
+- The EXPLANATION column should contain the rationale for the categorization
+
+## Fix Script Usage
+
+After categorization, apply fixes using `flowbook/scripts/fix_repro_errors.py`.
+
+**IMPORTANT:** Always use `@N` notation (code cell index) when calling the fix script. Never use cell IDs.
+
+### High-level fix types (single command):
+
+```bash
+# For in-place reassignment or sequential chain:
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @CODE_INDEX --fix-type inplace-reassign --variable VAR
+
+# For variable reuse:
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @CODE_INDEX --fix-type variable-reuse --variable VAR
+
+# For ML model mutation (unrecoverable):
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @CODE_INDEX --fix-type model-copy --variable VAR
+
+# For DataFrame inplace=True (unrecoverable):
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @CODE_INDEX --fix-type inplace-to-copy --variable VAR
+
+# For structural assignment (unrecoverable):
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @CODE_INDEX --fix-type struct-copy --variable VAR
+```
+
+### Diagnostic/Visualization fixes (agent-driven cell splitting):
+
+For diagnostic and visualization errors, the agent must analyze the cell and split it using primitive operations. **Do NOT blindly add `%diagnostic` to cells that contain mutations.**
+
+The fix script provides three primitive operations:
+
+```bash
+# Replace a cell's source code:
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @N --fix-type set-source --source-file PATH
+
+# Insert a new code cell after @N:
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @N --fix-type insert-cell-after --source-file PATH
+
+# Prepend %diagnostic magic to a cell:
+python flowbook/scripts/fix_repro_errors.py NOTEBOOK @N --fix-type add-diagnostic
+```
+
+**Index safety:** `insert-cell-after` shifts all subsequent code cell indices by 1. When applying multiple fixes to the same notebook, process diagnostic/visualization fixes from **highest code cell index to lowest** to avoid index invalidation. Other fix types (inplace-reassign, model-copy, etc.) don't insert cells and are safe in any order.
+
+#### How to fix diagnostic/visualization errors:
+
+1. **Read the cell source** at the error's code cell index
+2. **Classify each line** as either mutation (writes variables, assigns, calls .fit(), etc.) or diagnostic (print, display, .info(), .head(), plotting, etc.)
+3. **Decide the fix**:
+   - **If the cell is purely diagnostic** (no mutations at all): Just use `add-diagnostic`
+   - **If the cell mixes mutation and diagnostic code**: Split it:
+     a. Write the mutation lines to a temp file
+     b. Write the diagnostic lines to a temp file
+     c. Use `set-source` to replace the original cell with mutation-only code
+     d. Use `insert-cell-after` to add the diagnostic code as a new cell
+     e. Use `add-diagnostic` on the new cell (which is now at @N+1)
+   - **If the cell is purely mutation** (no diagnostic code): This was miscategorized. Do NOT add `%diagnostic`. Instead, recategorize as `inplace-reassign` or `sequential-chain` and apply the appropriate deep-copy fix.
+4. **Add `# [FLOWBOOK FIX]` comments** to both the mutation and diagnostic cells explaining what was done
+
+The script creates `<notebook>-fixed.ipynb` with:
+
+- Comments marked `# [FLOWBOOK FIX]` explaining the original error and fix
+- Deep copies with `_flow_XXXX` suffix for renamed variables
+- Split cells with `%diagnostic` magic on the read-only part
+- For `model-copy`: Uses `safe_model_copy()` which handles sklearn, PyTorch, XGBoost, etc.
+- For `inplace-to-copy`: Converts `df.method(inplace=True)` to `df = df.method()`
+
+## Extracting Errors from Notebook
+
+When using Single Notebook Mode, extract errors from the notebook's cell outputs. FlowBook stores violation information in `display_data` outputs with special metadata keys.
+
+**Python code to extract errors from a notebook:**
+
+```python
+import json
+from pathlib import Path
+
+def extract_errors_from_notebook(notebook_path: str) -> dict:
+    """Extract reproducibility errors from a processed FlowBook notebook.
+
+    Returns dict in same format as parse_repro_errors.py output:
+    {
+        "notebook.ipynb": {
+            "notebook_path": "/path/to/notebook.ipynb",
+            "error_count": N,
+            "errors": [
+                {
+                    "error_num": 1,
+                    "cell_id": "abcd",
+                    "cell_index": 5,  # CODE cell index
+                    "summary": "Cell @A reads and writes the same locations: var",
+                    "predicate": "no_read_and_write",
+                    "locations": ["var"],
+                    "accepted": True
+                },
+                ...
+            ]
+        }
+    }
+    """
+    path = Path(notebook_path)
+    with open(path) as f:
+        nb = json.load(f)
+
+    errors = []
+    code_cell_index = 0
+    error_num = 0
+
+    for cell in nb.get('cells', []):
+        if cell.get('cell_type') != 'code':
+            continue
+
+        cell_id = cell.get('id', '')
+
+        # Check outputs for predicate_violation metadata
+        for output in cell.get('outputs', []):
+            if output.get('output_type') != 'display_data':
+                continue
+
+            metadata = output.get('metadata', {})
+
+            # Check for predicate_violation (NoReadAndWrite, BackwardStale, etc.)
+            if 'predicate_violation' in metadata:
+                violation = metadata['predicate_violation']
+                error_num += 1
+                errors.append({
+                    'error_num': error_num,
+                    'cell_id': violation.get('cell_id', cell_id),
+                    'cell_index': code_cell_index,
+                    'summary': violation.get('message', ''),
+                    'predicate': violation.get('predicate', ''),
+                    'locations': violation.get('locations', []),
+                    'accepted': violation.get('accepted', True)
+                })
+
+            # Also check flowbook metadata for stale cells info
+            if 'flowbook' in metadata:
+                fb = metadata['flowbook']
+                # Stale cells indicate forward contamination (not an error per se,
+                # but useful context for categorization)
+
+        code_cell_index += 1
+
+    notebook_name = path.name
+    return {
+        notebook_name: {
+            'notebook_path': str(path.absolute()),
+            'error_count': len(errors),
+            'errors': errors
+        }
+    }
+```
+
+**Key metadata locations in notebook JSON:**
+
+- `cell.outputs[].metadata.predicate_violation` - Violation details:
+  - `predicate`: Type of violation (`"no_read_and_write"`, `"backward_stale"`, etc.)
+  - `cell_id`: Cell that triggered the violation
+  - `locations`: List of variable names involved
+  - `message`: Human-readable error message
+  - `accepted`: Whether execution continued despite violation
+- `cell.outputs[].metadata.flowbook` - Execution metadata:
+  - `reads`: Variables read by the cell
+  - `writes`: Variables written by the cell
+  - `stale_cells`: List of cell IDs that became stale
+  - `staleness_reasons`: Dict mapping cell_id to reason objects
+
+## Instructions
+
+When the user invokes this command:
+
+### Detect Mode
+
+First, determine which mode to use:
+
+- If the first argument ends with `.ipynb`, use **Single Notebook Mode**
+- Otherwise, use **Error Report Mode**
+
+### Error Report Mode
+
+1. Run the parsing script to get structured error data:
+
+   ```bash
+   python flowbook/scripts/parse_repro_errors.py $ERROR_REPORT_FILE $NOTEBOOKS_DIR --json
+   ```
+
+2. Create/initialize the output file `error_categories.tsv` with header
+
+3. For each notebook with errors, launch a parallel **opus** agent to:
+   - Read the relevant section from the error report (use line_range from parsed data)
+   - Read the notebook to understand context
+   - Categorize each error according to the taxonomy
+   - Identify the primary variable involved
+   - **Always reference cells by code cell index (`@N`), never by cell ID**
+   - For diagnostic/visualization errors: read the cell source, determine which lines are mutation vs diagnostic, and note the split plan
+   - Produce a short, coherent explanation for the categorization
+   - Output TSV lines
+
+### Single Notebook Mode
+
+1. Read the notebook JSON and extract errors using the logic above (check each cell's outputs for `predicate_violation` metadata)
+
+2. If no errors found, report that the notebook has no reproducibility violations
+
+3. For each error found:
+   - Read the cell source code to understand context
+   - Look at surrounding cells for the full picture
+   - Categorize the error according to the taxonomy above
+   - Identify the primary variable involved (from `locations` or by analyzing the code)
+   - **Always reference cells by code cell index (`@N`)**
+
+### Final Steps (both modes)
+
+4. Collect results and write to `error_categories.tsv` (for single notebook mode, also print to console)
+
+5. Print a summary of categories found
+
+6. If `--fix` flag is provided:
+   - For each notebook, initialize the fixed copy with `--init --force`
+   - Apply non-inserting fixes (inplace-reassign, sequential-chain, model-copy, inplace-to-copy, struct-copy, variable-reuse) in any order
+   - Apply diagnostic/visualization splits from **highest code cell index to lowest** (to avoid index shifts from cell insertions):
+     - Read the cell source
+     - Determine mutation vs diagnostic lines
+     - If purely diagnostic: use `add-diagnostic`
+     - If mixed: write temp files and use `set-source` + `insert-cell-after` + `add-diagnostic`
+     - If purely mutation: skip %diagnostic, apply appropriate rename fix instead
+   - Report which notebooks were fixed and where the `-fixed.ipynb` files are
+
+## Progress Reporting
+
+**Report progress as you work.** For each notebook being processed, output status like:
+
+```
+[1/23] backpack-pred-baseline-ensemble-eda.ipynb (4 errors)
+  - Error 1 (@3): Diagnostic inspection before mutation → train_data [split: 4 mutation + 2 diagnostic lines]
+  - Error 2 (@5): Sequential transformation chain → test_data
+  - Error 3 (@8): Visualization before mutation → train_data [pure diagnostic, add %diagnostic]
+  - Error 4 (@12): Sequential transformation chain → train_data
+```
+
+When applying fixes (with `--fix`):
+
+```
+[1/23] backpack-pred-baseline-ensemble-eda.ipynb
+  Initializing: backpack-pred-baseline-ensemble-eda-fixed.ipynb
+  - Fixing @5: sequential-chain --variable test_data
+  - Fixing @12: sequential-chain --variable train_data
+  - Fixing @8: add-diagnostic (pure diagnostic cell)
+  - Fixing @3: set-source + insert-cell-after + add-diagnostic (split mixed cell)
+  ✓ Fixed 4 errors → backpack-pred-baseline-ensemble-eda-fixed.ipynb
+```
+
+At the end, print a summary:
+
+```
+=== Summary ===
+Notebooks processed: 23
+Total errors categorized: 116
+  - In-place variable reassignment: 41
+  - Sequential transformation chain: 51
+  - Diagnostic inspection before mutation: 17
+  - Visualization before mutation: 2
+  - Reusing variable for different purposes: 5
+  - Unrecoverable mutation (ML model): 12
+  - Unrecoverable mutation (inplace): 8
+  - Unrecoverable mutation (structural): 3
+
+Fixed notebooks saved to:
+  - .../backpack-pred-baseline-ensemble-eda-fixed.ipynb
+  - .../forecasting-sticker-sales-fixed.ipynb
+  - ...
+```