@fuzzle/opencode-accountant 0.4.6-next.1 → 0.4.6

package/docs/tools/import-pipeline.md

@@ -1,611 +0,0 @@
-# import-pipeline Tool
-
-The `import-pipeline` tool orchestrates the complete bank statement import workflow, from classification through reconciliation. It coordinates all steps in sequence, ensuring data flows correctly between each stage via import contexts.
-
-This tool is **restricted to the accountant agent only**.
-
-## Overview
-
-The pipeline automates five sequential steps:
-
-1. **Classify** - Detect provider/currency, create contexts, organize files
-2. **Account Declarations** - Ensure all accounts exist in year journals
-3. **Dry Run** - Validate transactions, check for unknown accounts
-4. **Import** - Add transactions to journals, move files to done
-5. **Reconcile** - Verify balances match expectations
-
-**Key behavior**: The pipeline processes files via **import contexts**. Each classified CSV gets a unique context ID, and subsequent steps operate on these contexts **sequentially** with **fail-fast** error handling.
-
-## Arguments
-
-| Argument         | Type    | Default | Description                                                           |
-| ---------------- | ------- | ------- | --------------------------------------------------------------------- |
-| `closingBalance` | string  | -       | Manual closing balance override (e.g., `"CHF 2324.79"`) for all files |
-| `account`        | string  | -       | Manual hledger account override (e.g., `"assets:bank:ubs:checking"`)  |
-| `skipClassify`   | boolean | `false` | If true, skip classification (assumes files already in pending)       |
-
-**Note**: These parameters apply to ALL contexts processed by the pipeline. For per-file overrides, use individual tools manually.
-
-## Output Format
-
-### Success - All Steps Complete
-
-When all steps complete successfully:
-
-```json
-{
-  "success": true,
-  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479", "8b3e9c21-1a4f-4d89-b123-9f8e7d6c5b4a"],
-  "steps": {
-    "classify": {
-      "success": true,
-      "message": "Classification complete (2 context(s) created)"
-    },
-    "accountDeclarations": {
-      "success": true,
-      "message": "Account declarations complete"
-    },
-    "dryRun": {
-      "success": true,
-      "message": "Dry run complete - all transactions validated"
-    },
-    "import": {
-      "success": true,
-      "message": "Imported 42 transactions"
-    },
-    "reconcile": {
-      "success": true,
-      "message": "Balance reconciled: CHF 5432.10"
-    }
-  },
-  "summary": "Successfully imported 42 transaction(s) from 2 file(s)"
-}
-```
-
-**Note**: The `contexts` array contains the context IDs created during classification. These can be used to inspect individual import contexts in `.memory/{uuid}.json`.
-
-### Success - No Files to Import
-
-When the incoming directory is empty:
-
-```json
-{
-  "success": true,
-  "contexts": [],
-  "steps": {
-    "classify": {
-      "success": true,
-      "message": "Classification complete (0 context(s) created)"
-    }
-  },
-  "summary": "No files to import"
-}
-```
-
-### Failure - Unknown Accounts in Dry Run
-
-When dry run detects unknown accounts:
-
-```json
-{
-  "success": false,
-  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479"],
-  "steps": {
-    "classify": {
-      "success": true,
-      "message": "Classification complete (1 context(s) created)"
-    },
-    "accountDeclarations": {
-      "success": true,
-      "message": "Account declarations complete"
-    },
-    "dryRun": {
-      "success": false,
-      "message": "Dry run failed: 3 transactions with unknown accounts"
-    }
-  },
-  "error": "Dry run validation failed: 3 transactions with unknown accounts",
-  "hint": "Update rules file to categorize unknown transactions"
-}
-```
-
-### Failure - Balance Mismatch in Reconcile
-
-When reconciliation fails:
-
-```json
-{
-  "success": false,
-  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479"],
-  "steps": {
-    "classify": { "success": true },
-    "accountDeclarations": { "success": true },
-    "dryRun": { "success": true },
-    "import": { "success": true, "message": "Imported 42 transactions" },
-    "reconcile": {
-      "success": false,
-      "message": "Balance mismatch: expected CHF 5432.10, got CHF 5422.10"
-    }
-  },
-  "error": "Reconciliation failed: Balance mismatch",
-  "hint": "Check for missing transactions or incorrect rules"
-}
-```
-
-## Pipeline Steps
-
-### Step 1: Classify Statements
-
-**Purpose**: Detect provider/currency, create import contexts, organize files
-
-**What happens**:
-
-1. Scans `import/incoming/` for CSV files
-2. Detects provider/currency from each CSV
-3. Creates a context (`.memory/{uuid}.json`) for each file
-4. Extracts metadata (account number, dates, balances)
-5. Moves files to `import/pending/{provider}/{currency}/`
-
-**Output**: Array of context IDs
-
-**Failure scenarios**:
-
-- File collision (target file already exists)
-- Configuration missing
-- Unrecognized CSV format → moved to `import/unrecognized/`
-
-See [classify-statements](classify-statements.md) for details.
-
-### Step 2: Account Declarations
-
-**Purpose**: Ensure all accounts referenced in rules files are declared in year journals
-
-**What happens** (per context):
-
-1. Finds CSV file in `pending/` directory
-2. Matches CSV to rules file
-3. Extracts all account names from rules file
-4. Determines transaction year
-5. Ensures accounts declared in year journal (e.g., `ledger/2026.journal`)
-
-**Failure scenarios**:
-
-- Rules file not found for CSV
-- Multiple years in single CSV
-- Cannot create year journal file
-
-### Step 3: Dry Run
-
-**Purpose**: Validate all transactions can be categorized before importing
-
-**What happens** (per context):
-
-1. Loads context to find CSV file
-2. Runs `hledger print` with rules file
-3. Checks for `income:unknown` or `expenses:unknown` accounts
-4. Reports unknown postings if found
-
-**Failure scenarios**:
-
-- Unknown accounts detected → **Pipeline stops here**
-- CSV parsing errors
-- Rules file syntax errors
-
-**User action required**: Update rules file to categorize unknown transactions, then re-run pipeline.
-
-### Step 4: Import
-
-**Purpose**: Add transactions to journal and mark files as processed
-
-**What happens** (per context):
-
-1. Loads context to find CSV file
-2. Imports transactions using `hledger import`
-3. Moves CSV from `pending/` to `done/`
-4. Updates context with:
-   - `rulesFile` path
-   - `yearJournal` path
-   - `transactionCount`
-   - Updated `filePath` (now in `done/`)
-
-**Failure scenarios**:
-
-- Duplicate transactions detected
-- Journal file write error
-- File move operation fails
-
-See [import-statements](import-statements.md) for details.
-
-### Step 5: Reconcile
-
-**Purpose**: Verify imported transactions result in correct closing balance
-
-**What happens** (per context):
-
-1. Loads context to get account number and closing balance
-2. Finds CSV using account number (critical for multi-account)
-3. Queries hledger for actual balance
-4. Compares expected vs actual balance
-5. Updates context with:
-   - `reconciledAccount`
-   - `actualBalance`
-   - `lastTransactionDate`
-   - `reconciled` (true/false)
-
-**Failure scenarios**:
-
-- Balance mismatch → **Pipeline fails**
-- Missing closing balance (requires manual input)
-- Account not determined from rules file
-
-See [reconcile-statement](reconcile-statement.md) for details.
-
-## Context Flow Through Pipeline
-
-### Visual Flow
-
-```
-┌────────────────────────────────────────────────────────────────┐
-│  USER: Drop CSV files into import/incoming/                   │
-└────────────────────────────────────────────────────────────────┘
-                            │
-                            ▼
-┌────────────────────────────────────────────────────────────────┐
-│  STEP 1: classify-statements                                   │
-│  ┌──────────────────────────────────────────────────────────┐ │
-│  │ For each CSV:                                            │ │
-│  │  • Detect provider/currency                              │ │
-│  │  • Extract metadata (account#, dates, balances)          │ │
-│  │  • Generate UUID                                         │ │
-│  │  • CREATE: .memory/{uuid}.json                          │ │
-│  │  • Move: incoming/ → pending/{provider}/{currency}/     │ │
-│  └──────────────────────────────────────────────────────────┘ │
-│                                                                 │
-│  OUTPUT: ["uuid-1", "uuid-2"]                                 │
-└────────────────────────────────────────────────────────────────┘
-                            │
-                            ▼
-        ┌───────────────────┴───────────────────┐
-        │                                       │
-        ▼                                       ▼
-  ┌──────────┐                          ┌──────────┐
-  │ uuid-1   │                          │ uuid-2   │
-  │ checking │                          │ savings  │
-  └──────────┘                          └──────────┘
-        │                                       │
-        ▼                                       │
-┌────────────────────────────────────────┐      │
-│ STEPS 2-5 (for uuid-1):                │      │
-│  • Account Declarations                 │      │
-│  • Dry Run                              │      │
-│  • Import                               │      │
-│  • Reconcile                            │      │
-│  • UPDATE: .memory/uuid-1.json         │      │
-└────────────────────────────────────────┘      │
-        │ ✓ Success                              │
-        │                                       │
-        │                                       ▼
-        │                          ┌────────────────────────────────────────┐
-        │                          │ STEPS 2-5 (for uuid-2):                │
-        │                          │  • Account Declarations                 │
-        │                          │  • Dry Run                              │
-        │                          │  • Import                               │
-        │                          │  • Reconcile                            │
-        │                          │  • UPDATE: .memory/uuid-2.json         │
-        │                          └────────────────────────────────────────┘
-        │                                       │ ✓ Success
-        │                                       │
-        ▼                                       ▼
-  ┌──────────┐                          ┌──────────┐
-  │ Complete │                          │ Complete │
-  └──────────┘                          └──────────┘
-```
-
-### Sequential Processing (Fail-Fast)
-
-The pipeline processes contexts **one at a time**:
-
-```
-for each contextId:
-  1. Load context
-  2. Run steps 2-5 for this context
-  3. If ANY step fails → STOP PIPELINE
-  4. Update context with results
-  5. Move to next context
-```
-
-**Example**: If you have 2 CSVs (checking and savings):
-
-- Process checking account (uuid-1) through all steps
-- If checking succeeds → Process savings account (uuid-2)
-- If checking fails → **Stop immediately**, savings not processed
-
-**Why fail-fast?**
-
-- Easier debugging (focus on one failure at a time)
-- Prevents cascading errors
-- Maintains data consistency
-
-## Typical Usage
-
-### Scenario 1: Basic Import
-
-```bash
-# Step 1: Drop CSV files in import/incoming/
-cp bank-statement.csv import/incoming/
-
-# Step 2: Run pipeline
-import-pipeline
-
-# Step 3: Check results
-# - Files moved to import/done/
-# - Transactions in ledger journals
-# - Contexts in .memory/
-```
-
-### Scenario 2: Multi-Account Import
-
-```bash
-# Drop multiple CSVs
-cp ubs-checking-2026-02.csv import/incoming/
-cp ubs-savings-2026-02.csv import/incoming/
-
-# Run pipeline (processes sequentially)
-import-pipeline
-
-# Result: Both accounts imported and reconciled independently
-```
-
-### Scenario 3: Manual Balance Override
-
-When CSV doesn't include closing balance (e.g., Revolut):
-
-```bash
-import-pipeline --closingBalance "EUR 1234.56"
-```
-
-**Note**: This applies the same balance to ALL contexts. For different balances per file, use individual tools:
-
-```bash
-# Classify and get contextIds
-classify-statements
-# Output: { "contexts": ["uuid-1", "uuid-2"] }
-
-# Import each manually
-import-statements --contextId "uuid-1" --checkOnly false
-import-statements --contextId "uuid-2" --checkOnly false
-
-# Reconcile with different balances
-reconcile-statement --contextId "uuid-1" --closingBalance "EUR 1000.00"
-reconcile-statement --contextId "uuid-2" --closingBalance "EUR 2000.00"
-```
-
-### Scenario 4: Skip Classification
-
-When files are already in `import/pending/`:
-
-```bash
-import-pipeline --skipClassify
-```
-
-**Warning**: When `skipClassify` is true, the pipeline skips context creation entirely. If no contexts exist from a prior `classify-statements` run, the pipeline returns immediately with "No files to import". Only use this flag after running `classify-statements` separately.
-
-### Scenario 5: Handling Unknown Accounts
-
-```bash
-# First run: dry run fails
-import-pipeline
-# Output: "Dry run failed: 3 transactions with unknown accounts"
-
-# Check which transactions are unknown
-import-statements --contextId {from-output} --checkOnly true
-# Shows unknown postings with suggestions
-
-# Update rules file
-echo "if %description COFFEE SHOP" >> ledger/rules/revolut-eur.rules
-echo "  account2 expenses:food:coffee" >> ledger/rules/revolut-eur.rules
-
-# Re-run pipeline (will pick up where it left off)
-import-pipeline --skipClassify
-```
-
-## Error Handling
-
-### Common Error Patterns
-
-| Error                   | Step                | Cause                         | Resolution                                          |
-| ----------------------- | ------------------- | ----------------------------- | --------------------------------------------------- |
-| File collision          | Classify            | Target file already exists    | Move existing file to done or delete                |
-| Unrecognized CSV        | Classify            | Unknown provider              | Add provider config or move to pending manually     |
-| Unknown accounts        | Dry Run             | Transactions without rules    | Update rules file to categorize                     |
-| Balance mismatch        | Reconcile           | Missing transactions          | Check skip rules, verify all transactions imported  |
-| Context not found       | Import/Reconcile    | Invalid context ID            | Re-run classify-statements                          |
-| Multi-year CSV          | Account Declaration | CSV spans multiple years      | Split CSV by year or import to single year manually |
-| Missing closing balance | Reconcile           | CSV metadata incomplete       | Provide `--closingBalance` parameter                |
-| Account not determined  | Reconcile           | Rules file missing `account1` | Add `account1` directive to rules file              |
-
-### Debugging Failed Pipelines
-
-**Step 1: Check pipeline output**
-
-The output shows which step failed:
-
-```json
-{
-  "success": false,
-  "steps": {
-    "classify": { "success": true },
-    "accountDeclarations": { "success": true },
-    "dryRun": { "success": false } // ← Failed here
-  },
-  "error": "Dry run validation failed..."
-}
-```
-
-**Step 2: Run failed step manually for details**
-
-```bash
-# Get contextId from pipeline output
-# contexts: ["abc123-..."]
-
-# Run the failed step individually
-import-statements --contextId "abc123-..." --checkOnly true
-```
-
-**Step 3: Fix issue and resume**
-
-```bash
-# Fix the issue (e.g., update rules file)
-# Then re-run pipeline with skipClassify
-import-pipeline --skipClassify
-```
-
-**Step 4: Inspect contexts**
-
-```bash
-# View context file
-jq . .memory/abc123-....json
-
-# Check what was successfully completed
-jq '{transactionCount, reconciledAccount, reconciled}' .memory/abc123-....json
-```
-
-## Context Inspection
-
-After pipeline completion, inspect contexts for audit:
-
-```bash
-# List all contexts
-ls -lh .memory/
-
-# Pretty-print a context
-jq . .memory/f47ac10b-58cc-4372-a567-0e02b2c3d479.json
-
-# Get reconciliation status
-jq '{account: .reconciledAccount, reconciled, actualBalance}' .memory/*.json
-
-# Find failed reconciliations
-jq 'select(.reconciled == false)' .memory/*.json
-
-# Get transaction counts
-jq '{file: .filename, count: .transactionCount}' .memory/*.json
-```
-
-## Sequential vs Parallel Processing
-
-### Current Design: Sequential (Fail-Fast)
-
-```
-Context 1: classify → import → reconcile ✓
-  ↓
-Context 2: classify → import → reconcile ✓
-  ↓
-Context 3: classify → import → reconcile ✓
-```
-
-**Advantages**:
-
-- Simple error handling
-- Clear failure point
-- No race conditions
-- Easy to debug
-
-**Disadvantage**:
-
-- Slower for many files
-
-### Why Not Parallel?
-
-Parallel processing was considered but rejected:
-
-```
-Context 1: classify → import → reconcile ✓
-Context 2: classify → import → reconcile ❌ (balance mismatch)
-Context 3: classify → import → reconcile ✓
-```
-
-**Problems with parallel**:
-
-- Complex error handling (partial success)
-- Potential journal corruption
-- Harder to debug
-- Need rollback mechanism
-- Unclear which failed first
-
-**Decision**: Sequential processing is simpler and more reliable. Performance is adequate for typical import volumes (1-10 files).
-
-## Integration with Other Tools
-
-### Called by Pipeline
-
-The pipeline invokes these tools internally:
-
-1. `classify-statements` → Returns context IDs
-2. `import-statements` (per context) → Updates contexts
-3. `reconcile-statement` (per context) → Updates contexts
-
-### Manual Tool Usage
-
-You can run tools independently for more control:
-
-```bash
-# Fine-grained control
-classify-statements
-# → Get contextIds
-
-import-statements --contextId {uuid} --checkOnly true
-# → Validate first
-
-import-statements --contextId {uuid} --checkOnly false
-# → Import
-
-reconcile-statement --contextId {uuid}
-# → Reconcile
-```
-
-### When to Use Pipeline vs Individual Tools
-
-**Use import-pipeline when**:
-
-- Standard import workflow
-- Multiple files with same parameters
-- Want automatic end-to-end processing
-
-**Use individual tools when**:
-
-- Need different parameters per file
-- Debugging specific step
-- Want to inspect results between steps
-- Handling complex edge cases
-
-## Performance
-
-### Timing
-
-For typical import (1-2 CSVs, 50-100 transactions each):
-
-- **Classify**: <1 second
-- **Account Declarations**: <1 second
-- **Dry Run**: 1-2 seconds (hledger processing)
-- **Import**: 1-2 seconds (hledger processing)
-- **Reconcile**: 1-2 seconds (hledger queries)
-
-**Total**: ~5-10 seconds per context
-
-### Scalability
-
-- **10 CSVs**: ~60-100 seconds (sequential)
-- **100 CSVs**: ~10-15 minutes (sequential)
-
-If you regularly import 50+ CSVs, consider:
-
-- Batching by provider/currency
-- Running pipeline multiple times for different subsets
-- Using manual tool invocation for parallel control
-
-## See Also
-
-- [Import Context Architecture](../architecture/import-context.md) - Deep dive into context system
-- [classify-statements Tool](classify-statements.md) - Step 1: Classification
-- [import-statements Tool](import-statements.md) - Step 4: Import
-- [reconcile-statement Tool](reconcile-statement.md) - Step 5: Reconciliation