@fuzzle/opencode-accountant 0.4.6 → 0.5.0

package/docs/tools/import-pipeline.md

@@ -0,0 +1,611 @@
+# import-pipeline Tool
+
+The `import-pipeline` tool orchestrates the complete bank statement import workflow, from classification through reconciliation. It coordinates all steps in sequence, ensuring data flows correctly between each stage via import contexts.
+
+This tool is **restricted to the accountant agent only**.
+
+## Overview
+
+The pipeline automates five sequential steps:
+
+1. **Classify** - Detect provider/currency, create contexts, organize files
+2. **Account Declarations** - Ensure all accounts exist in year journals
+3. **Dry Run** - Validate transactions, check for unknown accounts
+4. **Import** - Add transactions to journals, move files to done
+5. **Reconcile** - Verify balances match expectations
+
+**Key behavior**: The pipeline processes files via **import contexts**. Each classified CSV gets a unique context ID, and subsequent steps operate on these contexts **sequentially** with **fail-fast** error handling.
+
+## Arguments
+
+| Argument         | Type    | Default | Description                                                           |
+| ---------------- | ------- | ------- | --------------------------------------------------------------------- |
+| `closingBalance` | string  | -       | Manual closing balance override (e.g., `"CHF 2324.79"`) for all files |
+| `account`        | string  | -       | Manual hledger account override (e.g., `"assets:bank:ubs:checking"`)  |
+| `skipClassify`   | boolean | `false` | If true, skip classification (assumes files already in pending)       |
+
+**Note**: These parameters apply to ALL contexts processed by the pipeline. For per-file overrides, use individual tools manually.
+
+## Output Format
+
+### Success - All Steps Complete
+
+When all steps complete successfully:
+
+```json
+{
+  "success": true,
+  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479", "8b3e9c21-1a4f-4d89-b123-9f8e7d6c5b4a"],
+  "steps": {
+    "classify": {
+      "success": true,
+      "message": "Classification complete (2 context(s) created)"
+    },
+    "accountDeclarations": {
+      "success": true,
+      "message": "Account declarations complete"
+    },
+    "dryRun": {
+      "success": true,
+      "message": "Dry run complete - all transactions validated"
+    },
+    "import": {
+      "success": true,
+      "message": "Imported 42 transactions"
+    },
+    "reconcile": {
+      "success": true,
+      "message": "Balance reconciled: CHF 5432.10"
+    }
+  },
+  "summary": "Successfully imported 42 transaction(s) from 2 file(s)"
+}
+```
+
+**Note**: The `contexts` array contains the context IDs created during classification. These can be used to inspect individual import contexts in `.memory/{uuid}.json`.
+
+### Success - No Files to Import
+
+When the incoming directory is empty:
+
+```json
+{
+  "success": true,
+  "contexts": [],
+  "steps": {
+    "classify": {
+      "success": true,
+      "message": "Classification complete (0 context(s) created)"
+    }
+  },
+  "summary": "No files to import"
+}
+```
+
+### Failure - Unknown Accounts in Dry Run
+
+When dry run detects unknown accounts:
+
+```json
+{
+  "success": false,
+  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479"],
+  "steps": {
+    "classify": {
+      "success": true,
+      "message": "Classification complete (1 context(s) created)"
+    },
+    "accountDeclarations": {
+      "success": true,
+      "message": "Account declarations complete"
+    },
+    "dryRun": {
+      "success": false,
+      "message": "Dry run failed: 3 transactions with unknown accounts"
+    }
+  },
+  "error": "Dry run validation failed: 3 transactions with unknown accounts",
+  "hint": "Update rules file to categorize unknown transactions"
+}
+```
+
+### Failure - Balance Mismatch in Reconcile
+
+When reconciliation fails:
+
+```json
+{
+  "success": false,
+  "contexts": ["f47ac10b-58cc-4372-a567-0e02b2c3d479"],
+  "steps": {
+    "classify": { "success": true },
+    "accountDeclarations": { "success": true },
+    "dryRun": { "success": true },
+    "import": { "success": true, "message": "Imported 42 transactions" },
+    "reconcile": {
+      "success": false,
+      "message": "Balance mismatch: expected CHF 5432.10, got CHF 5422.10"
+    }
+  },
+  "error": "Reconciliation failed: Balance mismatch",
+  "hint": "Check for missing transactions or incorrect rules"
+}
+```
+
+## Pipeline Steps
+
+### Step 1: Classify Statements
+
+**Purpose**: Detect provider/currency, create import contexts, organize files
+
+**What happens**:
+
+1. Scans `import/incoming/` for CSV files
+2. Detects provider/currency from each CSV
+3. Creates a context (`.memory/{uuid}.json`) for each file
+4. Extracts metadata (account number, dates, balances)
+5. Moves files to `import/pending/{provider}/{currency}/`
+
+**Output**: Array of context IDs
+
+**Failure scenarios**:
+
+- File collision (target file already exists)
+- Configuration missing
+- Unrecognized CSV format → moved to `import/unrecognized/`
+
+See [classify-statements](classify-statements.md) for details.
+
+### Step 2: Account Declarations
+
+**Purpose**: Ensure all accounts referenced in rules files are declared in year journals
+
+**What happens** (per context):
+
+1. Finds CSV file in `pending/` directory
+2. Matches CSV to rules file
+3. Extracts all account names from rules file
+4. Determines transaction year
+5. Ensures accounts declared in year journal (e.g., `ledger/2026.journal`)
+
+**Failure scenarios**:
+
+- Rules file not found for CSV
+- Multiple years in single CSV
+- Cannot create year journal file
+
+### Step 3: Dry Run
+
+**Purpose**: Validate all transactions can be categorized before importing
+
+**What happens** (per context):
+
+1. Loads context to find CSV file
+2. Runs `hledger print` with rules file
+3. Checks for `income:unknown` or `expenses:unknown` accounts
+4. Reports unknown postings if found
+
+**Failure scenarios**:
+
+- Unknown accounts detected → **Pipeline stops here**
+- CSV parsing errors
+- Rules file syntax errors
+
+**User action required**: Update rules file to categorize unknown transactions, then re-run pipeline.
+
+### Step 4: Import
+
+**Purpose**: Add transactions to journal and mark files as processed
+
+**What happens** (per context):
+
+1. Loads context to find CSV file
+2. Imports transactions using `hledger import`
+3. Moves CSV from `pending/` to `done/`
+4. Updates context with:
+   - `rulesFile` path
+   - `yearJournal` path
+   - `transactionCount`
+   - Updated `filePath` (now in `done/`)
+
+**Failure scenarios**:
+
+- Duplicate transactions detected
+- Journal file write error
+- File move operation fails
+
+See [import-statements](import-statements.md) for details.
+
+### Step 5: Reconcile
+
+**Purpose**: Verify imported transactions result in correct closing balance
+
+**What happens** (per context):
+
+1. Loads context to get account number and closing balance
+2. Finds CSV using account number (critical for multi-account)
+3. Queries hledger for actual balance
+4. Compares expected vs actual balance
+5. Updates context with:
+   - `reconciledAccount`
+   - `actualBalance`
+   - `lastTransactionDate`
+   - `reconciled` (true/false)
+
+**Failure scenarios**:
+
+- Balance mismatch → **Pipeline fails**
+- Missing closing balance (requires manual input)
+- Account not determined from rules file
+
+See [reconcile-statement](reconcile-statement.md) for details.
+
+## Context Flow Through Pipeline
+
+### Visual Flow
+
+```
+┌────────────────────────────────────────────────────────────────┐
+│  USER: Drop CSV files into import/incoming/                   │
+└────────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+┌────────────────────────────────────────────────────────────────┐
+│  STEP 1: classify-statements                                   │
+│  ┌──────────────────────────────────────────────────────────┐ │
+│  │ For each CSV:                                            │ │
+│  │  • Detect provider/currency                              │ │
+│  │  • Extract metadata (account#, dates, balances)          │ │
+│  │  • Generate UUID                                         │ │
+│  │  • CREATE: .memory/{uuid}.json                          │ │
+│  │  • Move: incoming/ → pending/{provider}/{currency}/     │ │
+│  └──────────────────────────────────────────────────────────┘ │
+│                                                                 │
+│  OUTPUT: ["uuid-1", "uuid-2"]                                 │
+└────────────────────────────────────────────────────────────────┘
+                            │
+                            ▼
+        ┌───────────────────┴───────────────────┐
+        │                                       │
+        ▼                                       ▼
+  ┌──────────┐                          ┌──────────┐
+  │ uuid-1   │                          │ uuid-2   │
+  │ checking │                          │ savings  │
+  └──────────┘                          └──────────┘
+        │                                       │
+        ▼                                       │
+┌────────────────────────────────────────┐      │
+│ STEPS 2-5 (for uuid-1):                │      │
+│  • Account Declarations                 │      │
+│  • Dry Run                              │      │
+│  • Import                               │      │
+│  • Reconcile                            │      │
+│  • UPDATE: .memory/uuid-1.json         │      │
+└────────────────────────────────────────┘      │
+        │ ✓ Success                              │
+        │                                       │
+        │                                       ▼
+        │                          ┌────────────────────────────────────────┐
+        │                          │ STEPS 2-5 (for uuid-2):                │
+        │                          │  • Account Declarations                 │
+        │                          │  • Dry Run                              │
+        │                          │  • Import                               │
+        │                          │  • Reconcile                            │
+        │                          │  • UPDATE: .memory/uuid-2.json         │
+        │                          └────────────────────────────────────────┘
+        │                                       │ ✓ Success
+        │                                       │
+        ▼                                       ▼
+  ┌──────────┐                          ┌──────────┐
+  │ Complete │                          │ Complete │
+  └──────────┘                          └──────────┘
+```
+
+### Sequential Processing (Fail-Fast)
+
+The pipeline processes contexts **one at a time**:
+
+```
+for each contextId:
+  1. Load context
+  2. Run steps 2-5 for this context
+  3. If ANY step fails → STOP PIPELINE
+  4. Update context with results
+  5. Move to next context
+```
+
+**Example**: If you have 2 CSVs (checking and savings):
+
+- Process checking account (uuid-1) through all steps
+- If checking succeeds → Process savings account (uuid-2)
+- If checking fails → **Stop immediately**, savings not processed
+
+**Why fail-fast?**
+
+- Easier debugging (focus on one failure at a time)
+- Prevents cascading errors
+- Maintains data consistency
+
+## Typical Usage
+
+### Scenario 1: Basic Import
+
+```bash
+# Step 1: Drop CSV files in import/incoming/
+cp bank-statement.csv import/incoming/
+
+# Step 2: Run pipeline
+import-pipeline
+
+# Step 3: Check results
+# - Files moved to import/done/
+# - Transactions in ledger journals
+# - Contexts in .memory/
+```
+
+### Scenario 2: Multi-Account Import
+
+```bash
+# Drop multiple CSVs
+cp ubs-checking-2026-02.csv import/incoming/
+cp ubs-savings-2026-02.csv import/incoming/
+
+# Run pipeline (processes sequentially)
+import-pipeline
+
+# Result: Both accounts imported and reconciled independently
+```
+
+### Scenario 3: Manual Balance Override
+
+When CSV doesn't include closing balance (e.g., Revolut):
+
+```bash
+import-pipeline --closingBalance "EUR 1234.56"
+```
+
+**Note**: This applies the same balance to ALL contexts. For different balances per file, use individual tools:
+
+```bash
+# Classify and get contextIds
+classify-statements
+# Output: { "contexts": ["uuid-1", "uuid-2"] }
+
+# Import each manually
+import-statements --contextId "uuid-1" --checkOnly false
+import-statements --contextId "uuid-2" --checkOnly false
+
+# Reconcile with different balances
+reconcile-statement --contextId "uuid-1" --closingBalance "EUR 1000.00"
+reconcile-statement --contextId "uuid-2" --closingBalance "EUR 2000.00"
+```
+
+### Scenario 4: Skip Classification
+
+When files are already in `import/pending/`:
+
+```bash
+import-pipeline --skipClassify
+```
+
+**Warning**: When `skipClassify` is true, the pipeline skips context creation entirely. If no contexts exist from a prior `classify-statements` run, the pipeline returns immediately with "No files to import". Only use this flag after running `classify-statements` separately.
+
+### Scenario 5: Handling Unknown Accounts
+
+```bash
+# First run: dry run fails
+import-pipeline
+# Output: "Dry run failed: 3 transactions with unknown accounts"
+
+# Check which transactions are unknown
+import-statements --contextId {from-output} --checkOnly true
+# Shows unknown postings with suggestions
+
+# Update rules file
+echo "if %description COFFEE SHOP" >> ledger/rules/revolut-eur.rules
+echo "  account2 expenses:food:coffee" >> ledger/rules/revolut-eur.rules
+
+# Re-run pipeline (will pick up where it left off)
+import-pipeline --skipClassify
+```
+
+## Error Handling
+
+### Common Error Patterns
+
+| Error                   | Step                | Cause                         | Resolution                                          |
+| ----------------------- | ------------------- | ----------------------------- | --------------------------------------------------- |
+| File collision          | Classify            | Target file already exists    | Move existing file to done or delete                |
+| Unrecognized CSV        | Classify            | Unknown provider              | Add provider config or move to pending manually     |
+| Unknown accounts        | Dry Run             | Transactions without rules    | Update rules file to categorize                     |
+| Balance mismatch        | Reconcile           | Missing transactions          | Check skip rules, verify all transactions imported  |
+| Context not found       | Import/Reconcile    | Invalid context ID            | Re-run classify-statements                          |
+| Multi-year CSV          | Account Declaration | CSV spans multiple years      | Split CSV by year or import to single year manually |
+| Missing closing balance | Reconcile           | CSV metadata incomplete       | Provide `--closingBalance` parameter                |
+| Account not determined  | Reconcile           | Rules file missing `account1` | Add `account1` directive to rules file              |
+
+### Debugging Failed Pipelines
+
+**Step 1: Check pipeline output**
+
+The output shows which step failed:
+
+```json
+{
+  "success": false,
+  "steps": {
+    "classify": { "success": true },
+    "accountDeclarations": { "success": true },
+    "dryRun": { "success": false } // ← Failed here
+  },
+  "error": "Dry run validation failed..."
+}
+```
+
+**Step 2: Run failed step manually for details**
+
+```bash
+# Get contextId from pipeline output
+# contexts: ["abc123-..."]
+
+# Run the failed step individually
+import-statements --contextId "abc123-..." --checkOnly true
+```
+
+**Step 3: Fix issue and resume**
+
+```bash
+# Fix the issue (e.g., update rules file)
+# Then re-run pipeline with skipClassify
+import-pipeline --skipClassify
+```
+
+**Step 4: Inspect contexts**
+
+```bash
+# View context file
+jq . .memory/abc123-....json
+
+# Check what was successfully completed
+jq '{transactionCount, reconciledAccount, reconciled}' .memory/abc123-....json
+```
+
+## Context Inspection
+
+After pipeline completion, inspect contexts for audit:
+
+```bash
+# List all contexts
+ls -lh .memory/
+
+# Pretty-print a context
+jq . .memory/f47ac10b-58cc-4372-a567-0e02b2c3d479.json
+
+# Get reconciliation status
+jq '{account: .reconciledAccount, reconciled, actualBalance}' .memory/*.json
+
+# Find failed reconciliations
+jq 'select(.reconciled == false)' .memory/*.json
+
+# Get transaction counts
+jq '{file: .filename, count: .transactionCount}' .memory/*.json
+```
+
+## Sequential vs Parallel Processing
+
+### Current Design: Sequential (Fail-Fast)
+
+```
+Context 1: classify → import → reconcile ✓
+  ↓
+Context 2: classify → import → reconcile ✓
+  ↓
+Context 3: classify → import → reconcile ✓
+```
+
+**Advantages**:
+
+- Simple error handling
+- Clear failure point
+- No race conditions
+- Easy to debug
+
+**Disadvantage**:
+
+- Slower for many files
+
+### Why Not Parallel?
+
+Parallel processing was considered but rejected:
+
+```
+Context 1: classify → import → reconcile ✓
+Context 2: classify → import → reconcile ❌ (balance mismatch)
+Context 3: classify → import → reconcile ✓
+```
+
+**Problems with parallel**:
+
+- Complex error handling (partial success)
+- Potential journal corruption
+- Harder to debug
+- Need rollback mechanism
+- Unclear which failed first
+
+**Decision**: Sequential processing is simpler and more reliable. Performance is adequate for typical import volumes (1-10 files).
+
+## Integration with Other Tools
+
+### Called by Pipeline
+
+The pipeline invokes these tools internally:
+
+1. `classify-statements` → Returns context IDs
+2. `import-statements` (per context) → Updates contexts
+3. `reconcile-statement` (per context) → Updates contexts
+
+### Manual Tool Usage
+
+You can run tools independently for more control:
+
+```bash
+# Fine-grained control
+classify-statements
+# → Get contextIds
+
+import-statements --contextId {uuid} --checkOnly true
+# → Validate first
+
+import-statements --contextId {uuid} --checkOnly false
+# → Import
+
+reconcile-statement --contextId {uuid}
+# → Reconcile
+```
+
+### When to Use Pipeline vs Individual Tools
+
+**Use import-pipeline when**:
+
+- Standard import workflow
+- Multiple files with same parameters
+- Want automatic end-to-end processing
+
+**Use individual tools when**:
+
+- Need different parameters per file
+- Debugging specific step
+- Want to inspect results between steps
+- Handling complex edge cases
+
+## Performance
+
+### Timing
+
+For typical import (1-2 CSVs, 50-100 transactions each):
+
+- **Classify**: <1 second
+- **Account Declarations**: <1 second
+- **Dry Run**: 1-2 seconds (hledger processing)
+- **Import**: 1-2 seconds (hledger processing)
+- **Reconcile**: 1-2 seconds (hledger queries)
+
+**Total**: ~5-10 seconds per context
+
+### Scalability
+
+- **10 CSVs**: ~60-100 seconds (sequential)
+- **100 CSVs**: ~10-15 minutes (sequential)
+
+If you regularly import 50+ CSVs, consider:
+
+- Batching by provider/currency
+- Running pipeline multiple times for different subsets
+- Using manual tool invocation for parallel control
+
+## See Also
+
+- [Import Context Architecture](../architecture/import-context.md) - Deep dive into context system
+- [classify-statements Tool](classify-statements.md) - Step 1: Classification
+- [import-statements Tool](import-statements.md) - Step 4: Import
+- [reconcile-statement Tool](reconcile-statement.md) - Step 5: Reconciliation

package/docs/tools/import-statements.md

@@ -5,6 +5,8 @@ The `import-statements` tool imports classified CSV bank statements into hledger
 - **Check mode** (`checkOnly: true`, default): Validates transactions and reports any that cannot be categorized
 - **Import mode** (`checkOnly: false`): Imports validated transactions and moves processed files to the done directory
 
+**Important**: This tool requires a `contextId` from a prior `classify-statements` run. The context provides the file path and metadata needed for import. See [Import Context Architecture](../architecture/import-context.md) for details.
+
 ## Year-Based Journal Routing
 
 Transactions are automatically routed to year-specific journal files based on transaction dates:
@@ -19,13 +21,47 @@ Transactions are automatically routed to year-specific journal files based on tr
 
 **Constraint:** Each CSV file must contain transactions from a single year. CSVs with transactions spanning multiple years are rejected during check mode with an error message listing the years found.
 
+## Using Context IDs
+
+The tool uses import contexts to locate CSV files and access metadata:
+
+### How It Works
+
+1. **classify-statements** creates a context for each CSV with a unique ID
+2. **import-statements** receives the contextId (via import-pipeline or manual invocation)
+3. Tool loads the context from `.memory/{contextId}.json`
+4. Context provides the file path to the CSV (no need to search by provider/currency)
+5. After import, tool updates context with results (rules file, year journal, transaction count)
+
+### Manual Invocation Example
+
+```bash
+# Step 1: Classify statements
+classify-statements
+# Output: { "classified": [{ "contextId": "abc123-...", ... }] }
+
+# Step 2: Import using contextId
+import-statements --contextId "abc123-..." --checkOnly false
+```
+
+### Automatic Invocation via Pipeline
+
+When using `import-pipeline`, context IDs are passed automatically:
+
+```bash
+# Pipeline handles everything
+import-pipeline
+# Internally: classify → get contextIds → import each context → reconcile each context
+```
+
 ## Arguments
 
-| Argument    | Type    | Default | Description                                 |
-| ----------- | ------- | ------- | ------------------------------------------- |
-| `provider`  | string  | -       | Filter by provider (e.g., `revolut`, `ubs`) |
-| `currency`  | string  | -       | Filter by currency (e.g., `chf`, `eur`)     |
-| `checkOnly` | boolean | `true`  | If true, only validate without importing    |
+| Argument    | Type    | Required | Default | Description                                        |
+| ----------- | ------- | -------- | ------- | -------------------------------------------------- |
+| `contextId` | string  | Yes      | -       | Context ID from `classify-statements` (e.g., UUID) |
+| `checkOnly` | boolean | No       | `true`  | If true, only validate without importing           |
+
+**Note**: When called via `import-pipeline`, the `contextId` is passed automatically. For manual invocation, get the `contextId` from `classify-statements` output.
 
 ## Output Format
 
@@ -61,6 +97,8 @@ When all transactions have matching rules:
 }
 ```
 
+**Note**: When invoked via `import-pipeline`, the CSV file path comes from the import context (loaded via `contextId`). The context also provides metadata like account number and closing balance.
+
 ### Check Mode - Unknown Postings Found
 
 When transactions don't match any `if` pattern in the rules file, the tool returns the full CSV row data for each unknown posting to provide context for classification: