@fuzzle/opencode-accountant 0.4.6 → 0.5.0

package/docs/architecture/import-context.md

@@ -0,0 +1,674 @@
+# Import Context Architecture
+
+The **Import Context** is a shared state system that tracks bank statement CSV files throughout the import pipeline. Each context is a JSON file stored in `.memory/{uuid}.json` that accumulates metadata and results as the file progresses through classification, import, and reconciliation.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [The Multi-Account Problem](#the-multi-account-problem)
+- [Context Schema](#context-schema)
+- [Context Lifecycle](#context-lifecycle)
+- [Context Flow](#context-flow)
+- [File Storage](#file-storage)
+- [Design Decisions](#design-decisions)
+- [Real-World Examples](#real-world-examples)
+- [Troubleshooting](#troubleshooting)
+
+## Overview
+
+### What is an Import Context?
+
+An import context is a persistent JSON document that:
+
+- **Tracks a single CSV file** through the entire import pipeline
+- **Stores all discovered metadata** (provider, currency, account number, dates, balances)
+- **Records progress** (which steps completed, results from each step)
+- **Enables step coordination** (later steps access data from earlier steps)
+- **Provides audit trail** (complete history of what happened to each file)
+
+### Why Context Files?
+
+Before contexts, the pipeline had a critical limitation: when you had multiple accounts from the same provider/currency (e.g., UBS checking account `.0` and UBS savings account `.1`), the reconcile step couldn't distinguish between them. It would select CSVs by modification time, often picking the wrong file.
+
+**Context files solve this by**:
+
+- Giving each CSV a unique identifier (UUID)
+- Storing the account number extracted during classification
+- Allowing reconcile to find the exact right CSV using the account number
+
+## The Multi-Account Problem
+
+### Before Import Context
+
+```
+Scenario: You have two UBS CHF accounts
+  - Checking: 0235-90250546.0
+  - Savings:  0235-90250546.1
+
+Both CSVs end up in: import/done/ubs/chf/
+
+reconcile-statement tool behavior:
+  ❌ Searches for: provider=ubs, currency=chf
+  ❌ Finds: BOTH CSVs
+  ❌ Selects: Most recently modified file
+  ❌ Result: Wrong CSV reconciled against wrong account
+```
+
+### After Import Context
+
+```
+Scenario: Same two UBS CHF accounts
+
+classify-statements creates:
+  - Context UUID-A for checking (accountNumber: "0235-90250546.0")
+  - Context UUID-B for savings  (accountNumber: "0235-90250546.1")
+
+reconcile-statement behavior:
+  ✓ Receives: contextId (e.g., UUID-A)
+  ✓ Loads: Context with accountNumber "0235-90250546.0"
+  ✓ Filters: CSVs by account number in filename
+  ✓ Finds: Exact CSV for checking account
+  ✓ Result: Correct CSV reconciled against correct account
+```
+
+**Key insight**: The context's `accountNumber` field is the critical piece of data that makes multi-account imports work reliably.
+
+## Context Schema
+
+### ImportContext Interface
+
+```typescript
+interface ImportContext {
+  // === Identity ===
+  id: string; // UUID (e.g., "f47ac10b-58cc-4372-a567-0e02b2c3d479")
+  createdAt: string; // ISO 8601 timestamp
+  updatedAt: string; // ISO 8601 timestamp (changes with each update)
+
+  // === File Tracking ===
+  filename: string; // Current filename
+  filePath: string; // Current path (updates as file moves through pipeline)
+  originalFilename?: string; // Original name before any renaming
+
+  // === Provider Detection (set by classify-statements) ===
+  provider: string; // e.g., "ubs", "revolut"
+  currency: string; // e.g., "chf", "eur"
+  accountNumber?: string; // e.g., "0235-90250546.1" (critical for multi-account)
+
+  // === CSV Metadata (extracted by classify-statements) ===
+  fromDate?: string; // Transaction date range start
+  untilDate?: string; // Transaction date range end
+  openingBalance?: string; // Opening balance value
+  closingBalance?: string; // Closing balance value
+
+  // === User Overrides (optional manual corrections) ===
+  account?: string; // Manual hledger account override
+
+  // === Import Results (set by import-statements) ===
+  rulesFile?: string; // Path to rules file used
+  yearJournal?: string; // Year journal file written to
+  transactionCount?: number; // Number of transactions imported
+
+  // === Reconciliation (set by reconcile-statement) ===
+  reconciledAccount?: string; // Account that was reconciled
+  actualBalance?: string; // Actual balance from hledger
+  lastTransactionDate?: string; // Date of last transaction
+  reconciled?: boolean; // true if balances matched
+}
+```
+
+### Design Principles
+
+1. **Flat Structure**: No nested objects, no arrays - easy to read and update
+2. **No Duplication**: Each piece of data appears exactly once
+3. **Progressive Enhancement**: Fields populate as pipeline progresses
+4. **Optional Fields**: Only required fields are id, createdAt, updatedAt, filename, filePath, provider, currency
+
+## Context Lifecycle
+
+### Stage 1: Creation (classify-statements)
+
+When a CSV is classified, a new context is created:
+
+```json
+{
+  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
+  "createdAt": "2026-02-28T10:30:00.000Z",
+  "updatedAt": "2026-02-28T10:30:00.000Z",
+
+  "filename": "ubs-0235-90250546.1-transactions-2026-02-22-to-2026-02-24.csv",
+  "filePath": "import/pending/ubs/chf/ubs-0235-90250546.1-transactions-2026-02-22-to-2026-02-24.csv",
+  "originalFilename": "transactions-0235-90250546.1-2026-02.csv",
+
+  "provider": "ubs",
+  "currency": "chf",
+  "accountNumber": "0235-90250546.1",
+
+  "fromDate": "2026-02-22",
+  "untilDate": "2026-02-24",
+  "openingBalance": "4001.55",
+  "closingBalance": "9001.55"
+}
+```
+
+**What happens**:
+
+- UUID generated for unique identification
+- File tracked at current location (`import/pending/...`)
+- Provider/currency detected from CSV
+- Metadata extracted from CSV headers
+- Context saved to `.memory/{uuid}.json`
+
+### Stage 2: Import (import-statements)
+
+After transactions are imported, context is updated:
+
+```json
+{
+  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
+  "createdAt": "2026-02-28T10:30:00.000Z",
+  "updatedAt": "2026-02-28T10:31:15.000Z", // ← updated
+
+  // ... previous fields unchanged ...
+
+  "filePath": "import/done/ubs/chf/ubs-0235-90250546.1-transactions-2026-02-22-to-2026-02-24.csv", // ← moved
+
+  "rulesFile": "ledger/rules/ubs-0235-90250546.1.rules", // ← added
+  "yearJournal": "ledger/2026.journal", // ← added
+  "transactionCount": 2 // ← added
+}
+```
+
+**What happens**:
+
+- `filePath` updated to reflect move from `pending/` to `done/`
+- `rulesFile` records which rules file was used
+- `yearJournal` records which journal received the transactions
+- `transactionCount` records how many transactions were imported
+- `updatedAt` timestamp refreshed
+
+### Stage 3: Reconciliation (reconcile-statement)
+
+After balance reconciliation, context is updated:
+
+```json
+{
+  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
+  "createdAt": "2026-02-28T10:30:00.000Z",
+  "updatedAt": "2026-02-28T10:31:45.000Z", // ← updated again
+
+  // ... all previous fields unchanged ...
+
+  "reconciledAccount": "assets:bank:ubs:savings", // ← added
+  "actualBalance": "CHF 9001.55", // ← added
+  "lastTransactionDate": "2026-02-24", // ← added
+  "reconciled": true // ← added
+}
+```
+
+**What happens**:
+
+- `reconciledAccount` records which hledger account was checked
+- `actualBalance` records the balance hledger calculated
+- `lastTransactionDate` records the cutoff date for balance calculation
+- `reconciled` flag indicates success/failure
+- `updatedAt` timestamp refreshed
+
+### Stage 4: Persistence
+
+**Context files are kept permanently** in `.memory/` for:
+
+- Audit trail (what happened to each import)
+- Debugging (inspect context when issues occur)
+- Re-running steps (can pass contextId to tools manually)
+
+## Context Flow
+
+### Pipeline Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    USER DROPS CSV FILES                         │
+│                   into import/incoming/                          │
+└─────────────────────────────────────────────────────────────────┘
+                                │
+                                ▼
+┌─────────────────────────────────────────────────────────────────┐
+│            STEP 1: classify-statements                          │
+│  ┌──────────────────────────────────────────────────────────┐  │
+│  │ For each CSV file:                                       │  │
+│  │  1. Detect provider/currency from header                 │  │
+│  │  2. Extract metadata (account#, dates, balances)         │  │
+│  │  3. Generate UUID                                        │  │
+│  │  4. Create .memory/{uuid}.json context file             │  │
+│  │  5. Move CSV to import/pending/{provider}/{currency}/   │  │
+│  └──────────────────────────────────────────────────────────┘  │
+│                                                                  │
+│  OUTPUT: Array of contextIds                                   │
+│    ["uuid-1", "uuid-2", ...]                                   │
+└─────────────────────────────────────────────────────────────────┘
+                                │
+                                ▼
+┌─────────────────────────────────────────────────────────────────┐
+│              STEP 2-5: Process Each Context                     │
+│                    (sequential, fail-fast)                       │
+└─────────────────────────────────────────────────────────────────┘
+                                │
+                                ▼
+        ┌───────────────────────┴───────────────────────┐
+        │                                               │
+        ▼                                               ▼
+┌──────────────────┐                          ┌──────────────────┐
+│  Context UUID-1  │                          │  Context UUID-2  │
+│  (checking .0)   │                          │  (savings .1)    │
+└──────────────────┘                          └──────────────────┘
+        │                                               │
+        ▼                                               ▼
+┌─────────────────────────────────────────────────────────────────┐
+│    STEP 2: Account Declarations (per context)                  │
+│     - Load context to find CSV file                            │
+│     - Find matching rules file                                  │
+│     - Ensure accounts declared in year journal                  │
+└─────────────────────────────────────────────────────────────────┘
+        │                                               │
+        ▼                                               ▼
+┌─────────────────────────────────────────────────────────────────┐
+│    STEP 3: Dry Run (per context)                               │
+│     - Load context to find CSV file                            │
+│     - Run hledger print to preview transactions                 │
+│     - Check for unknown accounts                                │
+│     - Abort if unknowns found                                   │
+└─────────────────────────────────────────────────────────────────┘
+        │                                               │
+        ▼                                               ▼
+┌─────────────────────────────────────────────────────────────────┐
+│    STEP 4: Import (per context)                                │
+│     - Load context to find CSV file                            │
+│     - Import transactions to year journal                       │
+│     - Move CSV from pending/ to done/                          │
+│     - Update context: rulesFile, yearJournal, count            │
+└─────────────────────────────────────────────────────────────────┘
+        │                                               │
+        ▼                                               ▼
+┌─────────────────────────────────────────────────────────────────┐
+│    STEP 5: Reconcile (per context)                             │
+│     - Load context to get accountNumber                        │
+│     - Find CSV by accountNumber (not by timestamp!)            │
+│     - Compare expected vs actual balance                        │
+│     - Update context: reconciliation results                    │
+└─────────────────────────────────────────────────────────────────┘
+        │                                               │
+        ▼                                               ▼
+┌──────────────────┐                          ┌──────────────────┐
+│  ✓ Complete      │                          │  ✓ Complete      │
+│  Context in      │                          │  Context in      │
+│  .memory/uuid-1  │                          │  .memory/uuid-2  │
+└──────────────────┘                          └──────────────────┘
+```
+
+### Key Concepts
+
+1. **One Context Per File**: Each CSV gets its own UUID and context
+2. **Sequential Processing**: Contexts processed one at a time (fail-fast on error)
+3. **Progressive Updates**: Each step adds data to the context
+4. **Isolated State**: Each context is independent (checking vs savings don't interfere)
+
+## File Storage
+
+### Directory Structure
+
+```
+your-project/
+├── .memory/                                 # Import contexts
+│   ├── f47ac10b-58cc-4372-a567-0e02b2c3d479.json  # Checking account context
+│   ├── 8b3e9c21-1a4f-4d89-b123-9f8e7d6c5b4a.json  # Savings account context
+│   └── a1b2c3d4-5e6f-7g8h-9i0j-k1l2m3n4o5p6.json  # Another import
+│
+├── import/
+│   ├── incoming/                            # Drop CSV files here
+│   ├── pending/                             # Classified, awaiting import
+│   │   └── ubs/
+│   │       └── chf/
+│   │           ├── ubs-0235-90250546.0-transactions-2026-02-22-to-2026-02-24.csv
+│   │           └── ubs-0235-90250546.1-transactions-2026-02-22-to-2026-02-24.csv
+│   └── done/                                # Imported successfully
+│       └── ubs/
+│           └── chf/
+│               ├── ubs-0235-90250546.0-transactions-2026-02-22-to-2026-02-24.csv
+│               └── ubs-0235-90250546.1-transactions-2026-02-22-to-2026-02-24.csv
+│
+└── ledger/
+    ├── .hledger.journal                     # Main journal
+    ├── 2026.journal                         # Year journal
+    └── rules/
+        ├── ubs-0235-90250546.0.rules        # Checking account rules
+        └── ubs-0235-90250546.1.rules        # Savings account rules
+```
+
+### Context File Naming
+
+**Format**: `.memory/{uuid}.json`
+
+**Example**: `.memory/f47ac10b-58cc-4372-a567-0e02b2c3d479.json`
+
+**Why UUIDs?**
+
+- Guaranteed unique (no collisions)
+- No dependency on filename (which may change)
+- Portable (can reference from any tool/script)
+- Time-independent (not based on timestamps)
+
+### Gitignore
+
+The `.memory/` directory is **gitignored** by default. Context files are:
+
+- Temporary/transient data
+- Specific to local import runs
+- Not part of the accounting data (which is in `ledger/`)
+
+However, they persist locally for audit/debugging purposes.
+
+## Design Decisions
+
+### 1. One Context Per File (Not Per Import Batch)
+
+**Decision**: Each CSV gets its own context, even if multiple CSVs are imported together.
+
+**Rationale**:
+
+- Enables independent tracking of each account
+- Supports sequential processing (fail on one doesn't corrupt others)
+- Simplifies state management (no arrays, no nested structures)
+
+**Alternative considered**: Single context for entire batch
+
+- **Rejected**: Complicates multi-account scenarios, harder to debug partial failures
+
+### 2. UUID-Based Naming
+
+**Decision**: Use random UUIDs for context IDs, not timestamps or sequential numbers.
+
+**Rationale**:
+
+- No collisions (multiple imports can run concurrently in theory)
+- No ordering assumptions (contexts are independent)
+- Portable (can share context IDs across systems)
+
+**Alternative considered**: Timestamp-based IDs (e.g., `2026-02-28-103000.json`)
+
+- **Rejected**: Collisions possible, implies temporal ordering
+
+### 3. Flat Schema (No Nesting)
+
+**Decision**: All fields at top level, no nested objects or arrays.
+
+**Rationale**:
+
+- Easy to read/debug (inspect any field with `jq .fieldName`)
+- Simple updates (no need to preserve nested structure)
+- No data duplication (currency appears once, not in multiple places)
+
+**Alternative considered**: Nested structure (e.g., `{ classification: {...}, import: {...}, reconcile: {...} }`)
+
+- **Rejected**: More complex to update, harder to query individual fields
+
+### 4. Sequential Processing (Fail-Fast)
+
+**Decision**: Process contexts one at a time. If one fails, stop immediately.
+
+**Rationale**:
+
+- Easier to debug (failure isolated to specific context)
+- Prevents partial imports (all-or-nothing per context)
+- Simpler error handling (don't need to track multiple failures)
+
+**Alternative considered**: Parallel processing with rollback
+
+- **Rejected**: Complex error handling, risk of data corruption, unnecessary optimization
+
+### 5. Persistent Context Files
+
+**Decision**: Keep context files in `.memory/` indefinitely (don't auto-delete).
+
+**Rationale**:
+
+- Audit trail (what happened during each import)
+- Debugging aid (inspect context when issues occur)
+- Enable re-running steps (can manually pass contextId to tools)
+
+**Alternative considered**: Delete after successful pipeline completion
+
+- **Rejected**: Loses audit trail, harder to debug issues after the fact
+
+## Real-World Examples
+
+### Example 1: Single Account Import
+
+**Scenario**: Import one UBS checking account statement
+
+**Step 1**: Drop CSV in `import/incoming/`
+
+```
+transactions-export.csv
+```
+
+**Step 2**: Run `classify-statements`
+
+```json
+{
+  "classified": [
+    {
+      "filename": "ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+      "contextId": "abc123-...",
+      "provider": "ubs",
+      "currency": "chf"
+    }
+  ]
+}
+```
+
+**Context created**: `.memory/abc123-....json`
+
+```json
+{
+  "id": "abc123-...",
+  "filename": "ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+  "filePath": "import/pending/ubs/chf/ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+  "provider": "ubs",
+  "currency": "chf",
+  "accountNumber": "0235-90250546.0",
+  "fromDate": "2026-02-01",
+  "untilDate": "2026-02-28",
+  "closingBalance": "5432.10"
+}
+```
+
+**Step 3**: Run `import-pipeline` (or individual steps)
+
+- Pipeline receives `["abc123-..."]`
+- Processes context abc123
+- Updates context with import/reconcile results
+
+**Final context**: `.memory/abc123-....json`
+
+```json
+{
+  "id": "abc123-...",
+  "filename": "ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+  "filePath": "import/done/ubs/chf/ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+  "provider": "ubs",
+  "currency": "chf",
+  "accountNumber": "0235-90250546.0",
+  "fromDate": "2026-02-01",
+  "untilDate": "2026-02-28",
+  "closingBalance": "5432.10",
+  "rulesFile": "ledger/rules/ubs-0235-90250546.0.rules",
+  "yearJournal": "ledger/2026.journal",
+  "transactionCount": 42,
+  "reconciledAccount": "assets:bank:ubs:checking",
+  "actualBalance": "CHF 5432.10",
+  "lastTransactionDate": "2026-02-28",
+  "reconciled": true
+}
+```
+
+### Example 2: Multi-Account Import (The Critical Use Case)
+
+**Scenario**: Import both UBS checking (`.0`) and savings (`.1`) for the same month
+
+**Step 1**: Drop both CSVs in `import/incoming/`
+
+```
+transactions-checking-2026-02.csv
+transactions-savings-2026-02.csv
+```
+
+**Step 2**: Run `classify-statements`
+
+```json
+{
+  "classified": [
+    {
+      "filename": "ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+      "contextId": "checking-uuid",
+      "accountNumber": "0235-90250546.0"
+    },
+    {
+      "filename": "ubs-0235-90250546.1-transactions-2026-02-01-to-2026-02-28.csv",
+      "contextId": "savings-uuid",
+      "accountNumber": "0235-90250546.1"
+    }
+  ]
+}
+```
+
+**Two contexts created**:
+
+`.memory/checking-uuid.json`:
+
+```json
+{
+  "id": "checking-uuid",
+  "accountNumber": "0235-90250546.0",
+  "filePath": "import/pending/ubs/chf/ubs-0235-90250546.0-transactions-2026-02-01-to-2026-02-28.csv",
+  "closingBalance": "5432.10"
+}
+```
+
+`.memory/savings-uuid.json`:
+
+```json
+{
+  "id": "savings-uuid",
+  "accountNumber": "0235-90250546.1",
+  "filePath": "import/pending/ubs/chf/ubs-0235-90250546.1-transactions-2026-02-01-to-2026-02-28.csv",
+  "closingBalance": "12500.00"
+}
+```
+
+**Step 3**: Run `import-pipeline`
+
+- Pipeline receives `["checking-uuid", "savings-uuid"]`
+- **Processes checking-uuid first**:
+  - import-statements loads context, finds CSV by filePath
+  - reconcile-statement loads context, uses accountNumber to find CSV
+  - ✓ Reconciles checking account
+- **Then processes savings-uuid**:
+  - import-statements loads context, finds CSV by filePath
+  - reconcile-statement loads context, uses accountNumber to find CSV
+  - ✓ Reconciles savings account
+
+**Critical point**: Because each context has its own `accountNumber`, reconcile-statement can filter CSVs correctly:
+
+```typescript
+// reconcile-statement for checking-uuid
+const csvFiles = findCsvFiles(doneDir, 'ubs', 'chf').filter((file) =>
+  file.includes('0235-90250546.0')
+); // ← from context
+
+// reconcile-statement for savings-uuid
+const csvFiles = findCsvFiles(doneDir, 'ubs', 'chf').filter((file) =>
+  file.includes('0235-90250546.1')
+); // ← from context
+```
+
+**Without contexts**: Both would just pick the most recent CSV → **wrong account reconciled** ❌
+
+**With contexts**: Each picks the correct CSV → **right account reconciled** ✓
+
+## Troubleshooting
+
+### Context Not Found
+
+**Error**: `Failed to load import context: abc123-...`
+
+**Cause**: Context file doesn't exist in `.memory/`
+
+**Solutions**:
+
+1. Check if context file exists: `ls .memory/abc123-*.json`
+2. Verify contextId is correct (check classify-statements output)
+3. Re-run classify-statements if context was accidentally deleted
+
+### Wrong CSV Reconciled
+
+**Error**: `Balance mismatch` or `Reconciled wrong account`
+
+**Diagnosis**:
+
+1. Check context file: `cat .memory/{contextId}.json`
+2. Verify `accountNumber` field is set correctly
+3. Check CSV filename contains account number: `ls import/done/ubs/chf/`
+
+**Cause**: Usually means `accountNumber` wasn't extracted during classification
+
+**Solutions**:
+
+1. Update provider config to extract account number via `renamePattern`
+2. Check CSV metadata extraction in `config/import/providers.yaml`
+
+### Multiple Contexts for Same File
+
+**Symptom**: Multiple UUIDs for what seems like the same CSV
+
+**Cause**: This is normal if you've re-run classify-statements multiple times
+
+**Impact**: No problem - each context is independent. The pipeline will process each one separately.
+
+**Cleanup**: You can manually delete old context files if desired: `rm .memory/old-uuid.json`
+
+### Context Out of Sync with File Location
+
+**Error**: `CSV file not found: import/pending/...` (but file is in `import/done/`)
+
+**Cause**: Context's `filePath` is stale (file moved manually without updating context)
+
+**Solutions**:
+
+1. Don't move CSV files manually - let the tools do it
+2. If already moved, update context: `jq '.filePath = "new/path"' .memory/{uuid}.json > temp && mv temp .memory/{uuid}.json`
+3. Or delete context and re-classify
+
+### Inspect Context Easily
+
+**Tip**: Use `jq` to pretty-print and query contexts
+
+```bash
+# Pretty-print entire context
+jq . .memory/abc123-....json
+
+# Get specific field
+jq .accountNumber .memory/abc123-....json
+
+# Find all contexts for UBS
+jq 'select(.provider == "ubs")' .memory/*.json
+
+# Find contexts with failed reconciliation
+jq 'select(.reconciled == false)' .memory/*.json
+```
+
+## See Also
+
+- [classify-statements Tool](../tools/classify-statements.md) - Creates contexts
+- [import-statements Tool](../tools/import-statements.md) - Updates contexts with import results
+- [reconcile-statement Tool](../tools/reconcile-statement.md) - Updates contexts with reconciliation results
+- [import-pipeline Tool](../tools/import-pipeline.md) - Orchestrates context flow through all steps