bankstatementparser 0.0.8__tar.gz → 0.0.9__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/PKG-INFO +429 -101
  2. bankstatementparser-0.0.9/README.md +767 -0
  3. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/__init__.py +14 -16
  4. bankstatementparser-0.0.9/bankstatementparser/_amounts.py +54 -0
  5. bankstatementparser-0.0.9/bankstatementparser/_llm_common.py +169 -0
  6. bankstatementparser-0.0.9/bankstatementparser/additional_parsers.py +634 -0
  7. bankstatementparser-0.0.9/bankstatementparser/api.py +323 -0
  8. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/bank_statement_parsers.py +44 -61
  9. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/base_parser.py +13 -26
  10. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/camt_parser.py +162 -161
  11. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/cli.py +220 -275
  12. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/enrichment/account_mapper.py +8 -5
  13. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/enrichment/categorizer.py +33 -63
  14. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/exceptions.py +4 -0
  15. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/export/ledger.py +3 -9
  16. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/__init__.py +39 -3
  17. bankstatementparser-0.0.9/bankstatementparser/hybrid/evaluation.py +366 -0
  18. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/llm_extractor.py +33 -72
  19. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/ollama_direct.py +26 -35
  20. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/orchestrator.py +66 -27
  21. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/pdf_text.py +36 -14
  22. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/prompts.py +1 -0
  23. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/scanner.py +80 -13
  24. bankstatementparser-0.0.9/bankstatementparser/hybrid/verification.py +400 -0
  25. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/hybrid/vision.py +19 -10
  26. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/input_validator.py +25 -48
  27. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/pain001_parser.py +82 -168
  28. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/parallel.py +5 -11
  29. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/record_types.py +22 -8
  30. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/transaction_deduplicator.py +61 -62
  31. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/transaction_models.py +18 -17
  32. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/zip_security.py +6 -18
  33. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/pyproject.toml +40 -8
  34. bankstatementparser-0.0.8/README.md +0 -440
  35. bankstatementparser-0.0.8/bankstatementparser/additional_parsers.py +0 -376
  36. bankstatementparser-0.0.8/bankstatementparser/api.py +0 -185
  37. bankstatementparser-0.0.8/bankstatementparser/hybrid/verification.py +0 -184
  38. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/LICENSE +0 -0
  39. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/enrichment/__init__.py +1 -1
  40. {bankstatementparser-0.0.8 → bankstatementparser-0.0.9}/bankstatementparser/export/__init__.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: bankstatementparser
3
- Version: 0.0.8
3
+ Version: 0.0.9
4
4
  Summary: BankStatementParser is your essential tool for easy bank statement management. Designed with finance and treasury experts in mind, it offers a simple way to handle CAMT (ISO 20022) formats and more. Get quick, accurate insights from your financial data and spend less time on processing. It's the smart, hassle-free way to stay on top of your transactions.
5
5
  License: Apache Software License
6
6
  License-File: LICENSE
@@ -16,15 +16,16 @@ Classifier: Programming Language :: Python :: 3.13
16
16
  Classifier: Programming Language :: Python :: 3.14
17
17
  Provides-Extra: api
18
18
  Provides-Extra: enrichment
19
+ Provides-Extra: excel
19
20
  Provides-Extra: hybrid
20
21
  Provides-Extra: hybrid-plus
21
22
  Provides-Extra: hybrid-vision
22
23
  Provides-Extra: polars
23
24
  Requires-Dist: defusedxml (>=0.7.1,<0.8.0)
24
25
  Requires-Dist: fastapi (>=0.115.0) ; extra == "api"
25
- Requires-Dist: litellm (>=1.83.0) ; extra == "hybrid" or extra == "hybrid-plus" or extra == "hybrid-vision" or extra == "enrichment"
26
- Requires-Dist: lxml (>=4.9.3)
27
- Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
26
+ Requires-Dist: litellm (>=1.83.10) ; (python_version < "3.14") and (extra == "hybrid" or extra == "hybrid-plus" or extra == "hybrid-vision" or extra == "enrichment")
27
+ Requires-Dist: lxml (>=5.0)
28
+ Requires-Dist: openpyxl (>=3.1.5,<4.0.0) ; extra == "excel"
28
29
  Requires-Dist: pandas (>=2.3.3,<3.0.0)
29
30
  Requires-Dist: pdfplumber (>=0.11.0) ; extra == "hybrid-plus"
30
31
  Requires-Dist: polars (>=1.32.0,<2.0.0) ; extra == "polars"
@@ -36,71 +37,77 @@ Project-URL: Homepage, https://bankstatementparser.com
36
37
  Project-URL: Repository, https://github.com/sebastienrousseau/bankstatementparser
37
38
  Description-Content-Type: text/markdown
38
39
 
39
- # Bank Statement Parser
40
+ <!-- SPDX-License-Identifier: Apache-2.0 -->
40
41
 
41
- Parse bank statements across **six structured formats** (CAMT, PAIN.001, CSV, OFX/QFX, MT940) **and PDFs** — both digital and scanned — into a single unified `Transaction` model. ISO 20022 files take the deterministic path; PDFs fall through to a configurable LLM (Ollama by default, any LiteLLM-supported provider) and finally to a multimodal vision model for scanned/photocopied statements.
42
+ <h1 align="center">Bank Statement Parser</h1>
42
43
 
43
- Built for finance teams, treasury analysts, and fintech developers who need reliable, auditable extraction across the full spectrum of bank statement formats — without sending data to external services unless they explicitly opt in.
44
+ <p align="center">
45
+ Bank statement parsing for Python — six structured formats and
46
+ PDFs, digital or scanned, into one auditable
47
+ <code>Transaction</code> model.
48
+ </p>
44
49
 
45
- [![PyPI](https://img.shields.io/pypi/pyversions/bankstatementparser.svg?style=for-the-badge&v=0.0.8)](https://pypi.org/project/bankstatementparser/)
46
- [![PyPI Downloads](https://img.shields.io/pypi/dm/bankstatementparser.svg?style=for-the-badge)](https://pypi.org/project/bankstatementparser/)
47
- [![Codecov](https://img.shields.io/codecov/c/github/sebastienrousseau/bankstatementparser?style=for-the-badge)](https://codecov.io/github/sebastienrousseau/bankstatementparser?branch=main)
48
- [![License](https://img.shields.io/github/license/sebastienrousseau/bankstatementparser?style=for-the-badge)](LICENSE)
50
+ <p align="center">
51
+ <a href="https://github.com/sebastienrousseau/bankstatementparser/actions"><img src="https://img.shields.io/github/actions/workflow/status/sebastienrousseau/bankstatementparser/quality-gates.yml?style=for-the-badge&logo=github" alt="Build" /></a>
52
+ <a href="https://pypi.org/project/bankstatementparser/"><img src="https://img.shields.io/pypi/pyversions/bankstatementparser.svg?style=for-the-badge&v=0.0.9" alt="PyPI" /></a>
53
+ <a href="https://pypi.org/project/bankstatementparser/"><img src="https://img.shields.io/pypi/dm/bankstatementparser.svg?style=for-the-badge" alt="PyPI Downloads" /></a>
54
+ <a href="https://codecov.io/github/sebastienrousseau/bankstatementparser?branch=main"><img src="https://img.shields.io/codecov/c/github/sebastienrousseau/bankstatementparser?style=for-the-badge" alt="Codecov" /></a>
55
+ <a href="LICENSE"><img src="https://img.shields.io/github/license/sebastienrousseau/bankstatementparser?style=for-the-badge" alt="License" /></a>
56
+ </p>
49
57
 
50
- ## How it works
58
+ ---
51
59
 
52
- `smart_ingest()` routes any input file through the cheapest viable extraction path. Deterministic parsers always run first ($0 cost). Text and vision LLMs are fallbacks for unstandardized PDFs — both are opt-in via separate install extras and can be swapped between any LiteLLM-supported provider (Ollama, Anthropic, OpenAI, Gemini, …).
60
+ Parse bank statements across **six structured formats** (CAMT,
61
+ PAIN.001, CSV, OFX/QFX, MT940) **and PDFs** — both digital and
62
+ scanned — into a single unified `Transaction` model. ISO 20022
63
+ files take the deterministic path; PDFs fall through to a
64
+ configurable LLM (Ollama by default, any LiteLLM-supported
65
+ provider) and finally to a multimodal vision model for
66
+ scanned/photocopied statements.
53
67
 
54
- ```mermaid
55
- flowchart TD
56
- A[smart_ingest&lpar;path&rpar;] --> B{detect_statement_format}
57
- B -- CAMT/PAIN/OFX/MT940/CSV --> C[Path A: deterministic parser<br/>$0, fastest]
58
- C --> Z[IngestResult<br/>source_method='deterministic']
68
+ Built for finance teams, treasury analysts, and fintech developers
69
+ who need reliable, auditable extraction across the full spectrum
70
+ of bank statement formats — without sending data to external
71
+ services unless they explicitly opt in.
59
72
 
60
- B -- pdf or unknown --> D[pypdf extract_text]
61
- D --> E{text len &gt;= 50?}
73
+ ---
62
74
 
63
- E -- yes --> F[Path B: text-LLM<br/>default ollama/llama3]
64
- F --> Y[IngestResult<br/>source_method='llm']
75
+ ## Contents
65
76
 
66
- E -- no --> G[Path C: vision-LLM<br/>opt-in via BSP_HYBRID_VISION_MODEL]
67
- G --> X[IngestResult<br/>source_method='vision']
77
+ **Getting started**
68
78
 
69
- Z --> V[verify_balance<br/>Golden Rule]
70
- Y --> V
71
- X --> V
72
- V --> R[VERIFIED / DISCREPANCY / FAILED]
73
- ```
79
+ - [Install](#install) — pip extras, requirements, local development
80
+ - [Quick Start](#quick-start) — parse a statement in three lines
74
81
 
75
- Every extracted row carries an immutable `transaction_hash`, an audit-trail `source_method` tag, and (for LLM rows) a `confidence` score — see [Hybrid extraction](#hybrid-extraction-pdfs-included-v005) below for the full surface.
82
+ **The pipeline**
76
83
 
77
- ## Key Features
84
+ - [How it works](#how-it-works) — routing diagram: deterministic first, LLM and vision as opt-in fallbacks
85
+ - [Key Features](#key-features) — parsing, hybrid PDF pipeline, verification, enrichment, security
78
86
 
79
- | Feature | Description |
80
- |---|---|
81
- | **6 structured formats** | CAMT.053, PAIN.001, CSV, OFX, QFX, MT940 |
82
- | **Hybrid PDF pipeline** *(v0.0.5)* | `smart_ingest()` routes digital PDFs through a text-LLM and scanned PDFs through a multimodal vision model. Deterministic parsers always tried first ($0 cost). |
83
- | **Local-first LLM** *(v0.0.5)* | Ollama is the default backend; switch to Anthropic, OpenAI, or any LiteLLM provider via `BSP_HYBRID_MODEL`. Vision is opt-in via `BSP_HYBRID_VISION_MODEL` — no surprise downloads. |
84
- | **Golden Rule verification** *(v0.0.5)* | Every result carries `opening + credits − debits == closing` status: `VERIFIED`, `DISCREPANCY`, or `FAILED`. |
85
- | **Idempotent dedup** *(v0.0.5)* | Every `Transaction` carries a stable `transaction_hash` (MD5 of date + normalized description + amount). `Deduplicator.dedupe_by_hash()` makes incremental ingestion safe to re-run. |
86
- | **Categorization** *(v0.0.6)* | `bankstatementparser.enrichment.Categorizer` tags transactions with a pluggable category schema (Plaid 13-category default) and an optional `is_business_expense` flag. Wrapper model never mutates the original `Transaction`. |
87
- | **Interactive review** *(v0.0.6)* | `--type review` CLI walks through discrepancies with accept/edit/skip/delete/quit. `IngestResult.to_json()` / `.from_json()` for stable round-trip with embedded audit trail. |
88
- | **Bounding boxes** *(v0.0.6)* | `Transaction.source_bbox` carries per-row normalized coordinates from the vision path for downstream review UIs. |
89
- | **Direct Ollama bridge** *(v0.0.7)* | Auto-bypasses the upstream LiteLLM ↔ Ollama hang on long vision prompts. `ollama/minicpm-v` recommended over `ollama/llava` for document OCR. |
90
- | **Strip mode** *(v0.0.7)* | `VisionExtractor(strip_rows=True)` splits dense pages into overlapping bands for small local models — fixes sign-flip errors and improves accuracy on 15+ row statements. |
91
- | **Auto-detection** | `detect_statement_format()` identifies the format; `create_parser()` returns the right parser |
92
- | **PII redaction** | Names, IBANs, and addresses masked by default — opt in with `--show-pii` |
93
- | **Streaming** | `parse_streaming()` at 27,000+ tx/s (CAMT) and 52,000+ tx/s (PAIN.001) with bounded memory |
94
- | **Parallel** | `parse_files_parallel()` for multi-file batch processing across CPU cores |
95
- | **Secure ZIP** | `iter_secure_xml_entries()` rejects zip bombs, encrypted entries, and suspicious compression ratios |
96
- | **In-memory parsing** | `from_string()` and `from_bytes()` parse XML without touching disk |
97
- | **Export** | CSV, JSON, Excel (`.xlsx`), and optional Polars DataFrames |
98
- | **100% coverage** | 718 tests, 100% branch coverage, property-based fuzzing with Hypothesis |
87
+ **Library reference**
88
+
89
+ - [PII Redaction](#pii-redaction) masked by default, opt in to see full data
90
+ - [Streaming](#streaming) bounded memory at 50,000 transactions
91
+ - [Performance](#performance) throughput and latency, CI-enforced
92
+ - [Parallel Parsing](#parallel-parsing) multi-file batches across CPU cores
93
+ - [Command Line](#command-line) console script and REST API
94
+ - [Deduplication](#deduplication) — exact hashes + suspected matches
95
+ - [Export](#export) CSV, JSON, Excel, Polars, hledger, beancount
96
+ - [Examples](#examples) 22 runnable scripts (14 deterministic + 8 hybrid)
97
+ - [XML Tag Mapping](#xml-tag-mapping) ISO 20022 tags to DataFrame columns
99
98
 
100
- ## Requirements
99
+ **Operational**
101
100
 
102
- - Python **3.10** through **3.14** (Python 3.9 was dropped in v0.0.6 — pin to v0.0.5 if you cannot upgrade your interpreter)
103
- - Poetry (for local development)
101
+ - [Project Layout](#project-layout)
102
+ - [Security](#security) XXE, ZIP-bomb, and path-traversal protection
103
+ - [Verify the Repository](#verify-the-repository) — the five local gates
104
+ - [Versioning](#versioning) — SemVer policy for the 0.y.z series
105
+ - [Contributing](#contributing)
106
+ - [FAQ](#faq)
107
+ - [Documentation](#documentation) — all reference docs
108
+ - [License](#license)
109
+
110
+ ---
104
111
 
105
112
  ## Install
106
113
 
@@ -108,6 +115,9 @@ Every extracted row carries an immutable `transaction_hash`, an audit-trail `sou
108
115
  # Core install — deterministic parsers only (CAMT, PAIN.001, CSV, OFX, QFX, MT940)
109
116
  pip install bankstatementparser
110
117
 
118
+ # Add Excel (.xlsx) export support (openpyxl)
119
+ pip install 'bankstatementparser[excel]'
120
+
111
121
  # Add the text-LLM path for digital PDFs (litellm + pypdf)
112
122
  pip install 'bankstatementparser[hybrid]'
113
123
 
@@ -116,11 +126,33 @@ pip install 'bankstatementparser[hybrid-plus]'
116
126
 
117
127
  # Add the multimodal vision path for scanned/photocopied PDFs (adds pypdfium2)
118
128
  pip install 'bankstatementparser[hybrid-vision]'
129
+
130
+ # Add LLM-powered transaction categorization
131
+ pip install 'bankstatementparser[enrichment]'
132
+
133
+ # Add the REST API microservice (FastAPI + uvicorn)
134
+ pip install 'bankstatementparser[api]'
119
135
  ```
120
136
 
121
- The core install has zero AI dependencies. Every `[hybrid*]` extra is opt-in and pure-Python — no `poppler`, no system libraries, no GPU required.
137
+ The core install has zero AI dependencies. Every extra is opt-in
138
+ and pure-Python — no `poppler`, no system libraries, no GPU
139
+ required.
140
+
141
+ ### Requirements
122
142
 
123
- ### Local Development
143
+ - Python **3.10** through **3.14** (Python 3.9 was dropped in
144
+ v0.0.6 — pin to v0.0.5 if you cannot upgrade your interpreter).
145
+ - Poetry (for local development).
146
+
147
+ > **Python version note.** The deterministic core (CAMT, PAIN.001,
148
+ > CSV, OFX, QFX, MT940) supports Python **3.10–3.14**. The optional
149
+ > `[hybrid]`, `[hybrid-plus]`, `[hybrid-vision]`, and
150
+ > `[enrichment]` extras pull in `litellm`, which currently requires
151
+ > **Python <3.14** upstream — on Python 3.14 the LLM extras will
152
+ > not install. Pin `Python 3.13` for the full hybrid pipeline;
153
+ > upgrade to 3.14 freely if you only need deterministic parsing.
154
+
155
+ ### Local development
124
156
 
125
157
  Clone and install on **macOS, Linux, or WSL**:
126
158
 
@@ -130,10 +162,12 @@ cd bankstatementparser
130
162
  python3 -m venv .venv
131
163
  source .venv/bin/activate
132
164
  pip install poetry
133
- poetry install --with dev
165
+ poetry install --with dev -E excel
134
166
  make install-hooks # pre-commit hook runs `make verify` before every commit
135
167
  ```
136
168
 
169
+ ---
170
+
137
171
  ## Quick Start
138
172
 
139
173
  ### Parse a CAMT statement
@@ -183,7 +217,8 @@ Works with `.xml`, `.csv`, `.ofx`, `.qfx`, and `.mt940` files.
183
217
 
184
218
  ### Hybrid extraction (PDFs included) *(v0.0.5)*
185
219
 
186
- `smart_ingest()` is the single entry point that routes any file through the cheapest viable extraction path:
220
+ `smart_ingest()` is the single entry point that routes any file
221
+ through the cheapest viable extraction path:
187
222
 
188
223
  ```python
189
224
  from bankstatementparser.hybrid import smart_ingest
@@ -205,12 +240,19 @@ print(result.source_method) # "vision"
205
240
 
206
241
  Every row carries:
207
242
 
208
- - `source_method` — `"deterministic"`, `"llm"`, or `"vision"` for full audit provenance
209
- - `transaction_hash` — MD5 fingerprint of `date | normalized_description | amount`, ready for idempotent re-ingestion
210
- - `confidence` — float between 0 and 1 for LLM rows, `None` for deterministic
211
- - `raw_source_text` best-effort source-text slice for the v0.0.6 review-mode UI
243
+ - `source_method` — `"deterministic"`, `"llm"`, or `"vision"` for
244
+ full audit provenance
245
+ - `transaction_hash` — MD5 fingerprint of
246
+ `date | normalized_description | amount`, ready for idempotent
247
+ re-ingestion
248
+ - `confidence` — float between 0 and 1 for LLM rows, `None` for
249
+ deterministic
250
+ - `raw_source_text` — best-effort source-text slice for the v0.0.6
251
+ review-mode UI
212
252
 
213
- A complete walkthrough with synthetic UK-bank PDFs, mock vs. live mode, and a Mermaid flow diagram lives in [`examples/hybrid/README.md`](examples/hybrid/README.md).
253
+ A complete walkthrough with synthetic UK-bank PDFs, mock vs. live
254
+ mode, and a Mermaid flow diagram lives in
255
+ [`examples/hybrid/README.md`](examples/hybrid/README.md).
214
256
 
215
257
  ### Parse from memory (no disk I/O)
216
258
 
@@ -222,7 +264,8 @@ parser = CamtParser.from_bytes(xml_bytes, source_name="daily.xml")
222
264
  transactions = parser.parse()
223
265
  ```
224
266
 
225
- Pass only decompressed XML to `from_string()` or `from_bytes()`. For ZIP archives, use `iter_secure_xml_entries()`.
267
+ Pass only decompressed XML to `from_string()` or `from_bytes()`.
268
+ For ZIP archives, use `iter_secure_xml_entries()`.
226
269
 
227
270
  ### Parse XML files inside a ZIP archive
228
271
 
@@ -235,11 +278,106 @@ for entry in iter_secure_xml_entries("statements.zip"):
235
278
  print(entry.source_name, len(transactions), "transactions")
236
279
  ```
237
280
 
238
- The iterator enforces size limits, blocks encrypted entries, and rejects suspicious compression ratios before any XML parsing occurs.
281
+ The iterator enforces size limits, blocks encrypted entries, and
282
+ rejects suspicious compression ratios before any XML parsing
283
+ occurs.
284
+
285
+ ---
286
+
287
+ ## How it works
288
+
289
+ `smart_ingest()` routes any input file through the cheapest viable
290
+ extraction path. Deterministic parsers always run first ($0 cost).
291
+ Text and vision LLMs are fallbacks for unstandardized PDFs — both
292
+ are opt-in via separate install extras and can be swapped between
293
+ any LiteLLM-supported provider (Ollama, Anthropic, OpenAI,
294
+ Gemini, …).
295
+
296
+ ```mermaid
297
+ flowchart TD
298
+ A[smart_ingest&lpar;path&rpar;] --> B{detect_statement_format}
299
+ B -- CAMT/PAIN/OFX/MT940/CSV --> C[Path A: deterministic parser<br/>$0, fastest]
300
+ C --> Z[IngestResult<br/>source_method='deterministic']
301
+
302
+ B -- pdf or unknown --> D[pypdf extract_text]
303
+ D --> E{text len &gt;= 50?}
304
+
305
+ E -- yes --> F[Path B: text-LLM<br/>default ollama/llama3]
306
+ F --> Y[IngestResult<br/>source_method='llm']
307
+
308
+ E -- no --> G[Path C: vision-LLM<br/>opt-in via BSP_HYBRID_VISION_MODEL]
309
+ G --> X[IngestResult<br/>source_method='vision']
310
+
311
+ Z --> V[verify_balance<br/>Golden Rule]
312
+ Y --> V
313
+ X --> V
314
+ V --> R[VERIFIED / DISCREPANCY / FAILED]
315
+ ```
316
+
317
+ Every extracted row carries an immutable `transaction_hash`, an
318
+ audit-trail `source_method` tag, and (for LLM rows) a `confidence`
319
+ score — see
320
+ [Hybrid extraction](#hybrid-extraction-pdfs-included-v005) above
321
+ for the full surface.
322
+
323
+ ---
324
+
325
+ ## Key Features
326
+
327
+ ### Parsing
328
+
329
+ | Feature | Description |
330
+ |---|---|
331
+ | **6 structured formats** | CAMT.053, PAIN.001, CSV, OFX, QFX, MT940 |
332
+ | **Auto-detection** | `detect_statement_format()` identifies the format; `create_parser()` returns the right parser |
333
+ | **Streaming** | `parse_streaming()` at 27,000+ tx/s (CAMT) and 52,000+ tx/s (PAIN.001) with bounded memory |
334
+ | **Parallel** | `parse_files_parallel()` for multi-file batch processing across CPU cores |
335
+ | **In-memory parsing** | `from_string()` and `from_bytes()` parse XML without touching disk |
336
+
337
+ ### Hybrid PDF pipeline (LLM-assisted)
338
+
339
+ | Feature | Description |
340
+ |---|---|
341
+ | **Hybrid PDF pipeline** *(v0.0.5)* | `smart_ingest()` routes digital PDFs through a text-LLM and scanned PDFs through a multimodal vision model. Deterministic parsers always tried first ($0 cost). |
342
+ | **Local-first LLM** *(v0.0.5)* | Ollama is the default backend; switch to Anthropic, OpenAI, or any LiteLLM provider via `BSP_HYBRID_MODEL`. Vision is opt-in via `BSP_HYBRID_VISION_MODEL` — no surprise downloads. |
343
+ | **Direct Ollama bridge** *(v0.0.7)* | Auto-bypasses the upstream LiteLLM ↔ Ollama hang on long vision prompts. `ollama/minicpm-v` recommended over `ollama/llava` for document OCR. |
344
+ | **Strip mode** *(v0.0.7)* | `VisionExtractor(strip_rows=True)` splits dense pages into overlapping bands for small local models — fixes sign-flip errors and improves accuracy on 15+ row statements. |
345
+ | **Bounding boxes** *(v0.0.6)* | `Transaction.source_bbox` carries per-row normalized coordinates from the vision path for downstream review UIs. |
346
+ | **Bulk directory scanner** *(v0.0.8)* | `scan_and_ingest(directory, pattern="**/*.pdf")` scans a folder tree, runs `smart_ingest` on every match, deduplicates across the entire batch. |
347
+
348
+ ### Data quality & verification
349
+
350
+ | Feature | Description |
351
+ |---|---|
352
+ | **Golden Rule verification** *(v0.0.5)* | Every result carries `opening + credits − debits == closing` status: `VERIFIED`, `DISCREPANCY`, or `FAILED`. |
353
+ | **Multi-currency verification** *(v0.0.8)* | `verify_balance_multi_currency()` groups transactions by currency and runs the Golden Rule independently per group — no more false `DISCREPANCY` on multi-currency statements. |
354
+ | **Idempotent dedup** *(v0.0.5)* | Every `Transaction` carries a stable `transaction_hash` (MD5 of date + normalized description + amount). `Deduplicator.dedupe_by_hash()` makes incremental ingestion safe to re-run. |
355
+ | **Interactive review** *(v0.0.6)* | `--type review` CLI walks through discrepancies with accept/edit/skip/delete/quit. `IngestResult.to_json()` / `.from_json()` for stable round-trip with embedded audit trail. |
356
+
357
+ ### Enrichment & export
358
+
359
+ | Feature | Description |
360
+ |---|---|
361
+ | **Categorization** *(v0.0.6)* | `bankstatementparser.enrichment.Categorizer` tags transactions with a pluggable category schema (Plaid 13-category default) and an optional `is_business_expense` flag. Wrapper model — never mutates the original `Transaction`. |
362
+ | **Account mapping** *(v0.0.8)* | `AccountMapper` with ordered regex rules loaded from JSON config. First match wins. Pairs with the ledger exporter for end-to-end plaintext-accounting workflows. |
363
+ | **hledger + beancount export** *(v0.0.8)* | `to_hledger()` and `to_beancount()` produce journal strings for plaintext-accounting workflows. Uses `Transaction.category` as the contra-account when set. |
364
+ | **Export** | CSV, JSON, Excel (`.xlsx`), and optional Polars DataFrames |
365
+ | **REST API** *(v0.0.8)* | FastAPI microservice: `POST /ingest` a file, get JSON back. `GET /health` for monitoring. `pip install 'bankstatementparser[api]'`. |
366
+
367
+ ### Security & quality
368
+
369
+ | Feature | Description |
370
+ |---|---|
371
+ | **PII redaction** | Names, IBANs, and addresses masked by default — opt in with `--show-pii` |
372
+ | **Secure ZIP** | `iter_secure_xml_entries()` rejects zip bombs, encrypted entries, and suspicious compression ratios |
373
+ | **Tested** | 818 tests, coverage gated at 100% in CI, property-based fuzzing with Hypothesis |
374
+
375
+ ---
239
376
 
240
377
  ## PII Redaction
241
378
 
242
- PII (names, IBANs, addresses) is **redacted by default** in console output and streaming mode.
379
+ PII (names, IBANs, addresses) is **redacted by default** in
380
+ console output and streaming mode.
243
381
 
244
382
  ```python
245
383
  # Redacted by default
@@ -251,11 +389,16 @@ for tx in parser.parse_streaming(redact_pii=False):
251
389
  print(tx)
252
390
  ```
253
391
 
254
- File exports (CSV, JSON, Excel) always contain the full unredacted data.
392
+ File exports (CSV, JSON, Excel) always contain the full unredacted
393
+ data.
394
+
395
+ ---
255
396
 
256
397
  ## Streaming
257
398
 
258
- Process large files incrementally. Memory stays bounded regardless of file size — tested at 50,000 transactions with sub-2x memory scaling.
399
+ Process large files incrementally. Memory stays bounded regardless
400
+ of file size — tested at 50,000 transactions with sub-2x memory
401
+ scaling.
259
402
 
260
403
  ```python
261
404
  from bankstatementparser import CamtParser
@@ -265,7 +408,11 @@ for transaction in parser.parse_streaming():
265
408
  process(transaction) # each transaction is a dict
266
409
  ```
267
410
 
268
- Works with both `CamtParser` and `Pain001Parser`. PAIN.001 files over 50 MB use chunk-based namespace stripping via a temporary file — the full document is never loaded into memory.
411
+ Works with both `CamtParser` and `Pain001Parser`. PAIN.001 files
412
+ over 50 MB use chunk-based namespace stripping via a temporary
413
+ file — the full document is never loaded into memory.
414
+
415
+ ---
269
416
 
270
417
  ## Performance
271
418
 
@@ -276,7 +423,12 @@ Works with both `CamtParser` and `Pain001Parser`. PAIN.001 files over 50 MB use
276
423
  | **Time to first result** | < 1 ms | < 2 ms |
277
424
  | **Memory scaling** | Constant (1K–50K) | Constant (1K–50K) |
278
425
 
279
- Performance is flat from 1,000 to 50,000 transactions. CI enforces minimum TPS and latency thresholds.
426
+ Performance is flat from 1,000 to 50,000 transactions. CI enforces
427
+ minimum TPS and latency thresholds, and a non-blocking benchmark
428
+ job compares every run against the last known-good `main`
429
+ baseline.
430
+
431
+ ---
280
432
 
281
433
  ## Parallel Parsing
282
434
 
@@ -295,11 +447,16 @@ for r in results:
295
447
  print(r.path, r.status, len(r.transactions), "rows")
296
448
  ```
297
449
 
298
- Uses `ProcessPoolExecutor` to bypass the GIL. Each file is parsed in its own worker process. Auto-detects format per file, or force with `format_name="camt"`.
450
+ Uses `ProcessPoolExecutor` to bypass the GIL. Each file is parsed
451
+ in its own worker process. Auto-detects format per file, or force
452
+ with `format_name="camt"`.
453
+
454
+ ---
299
455
 
300
456
  ## Command Line
301
457
 
302
- After installation a `bankstatementparser` console script is available on `PATH`:
458
+ After installation a `bankstatementparser` console script is
459
+ available on `PATH`:
303
460
 
304
461
  ```bash
305
462
  # Parse and display
@@ -318,9 +475,53 @@ bankstatementparser --type ingest --input statement.pdf --output ledger.csv
318
475
  # v0.0.6 — interactive review of saved IngestResult JSON
319
476
  bankstatementparser --type review --input result.json
320
477
  bankstatementparser --type review --input result.json --output reviewed.json
478
+
479
+ # v0.0.9 — also review rows the LLM was unsure about (confidence < 0.8)
480
+ bankstatementparser --type review --input result.json --review-below 0.8
481
+ ```
482
+
483
+ Supports `--type camt`, `--type pain001`, `--type ingest`
484
+ (v0.0.5), and `--type review` (v0.0.6). The
485
+ `python -m bankstatementparser.cli ...` invocation form continues
486
+ to work for parity with older releases.
487
+
488
+ ### REST API *(v0.0.8)*
489
+
490
+ ```bash
491
+ pip install 'bankstatementparser[api]'
492
+ bankstatementparser-api --port 8000
493
+
494
+ # POST a file, get JSON back
495
+ curl -F file=@statement.pdf http://localhost:8000/ingest
496
+
497
+ # Health check
498
+ curl http://localhost:8000/health
321
499
  ```
322
500
 
323
- Supports `--type camt`, `--type pain001`, `--type ingest` (v0.0.5), and `--type review` (v0.0.6). The `python -m bankstatementparser.cli ...` invocation form continues to work for parity with older releases.
501
+ Default bind is `127.0.0.1` (localhost-only). Use
502
+ `--host 0.0.0.0` for container deployments.
503
+
504
+ #### Security defaults
505
+
506
+ - Uploads are streamed in chunks; the request is rejected with
507
+ **413 Payload Too Large** once the cumulative size exceeds
508
+ `BSP_API_MAX_UPLOAD_BYTES` (default **25 MB**).
509
+ - The uploaded filename is reduced to its basename —
510
+ caller-supplied path components are dropped — and the suffix
511
+ must match one of the allowed input extensions (`.xml`, `.csv`,
512
+ `.ofx`, `.qfx`, `.mt940`, `.sta`, `.pdf`, `.json`). Anything
513
+ else returns **400 Bad Request**.
514
+ - On parse failure the response carries a UUID `correlation_id`;
515
+ the raw exception is logged server-side only. Status
516
+ **422 Unprocessable Entity**.
517
+ - **Authentication, authorization, and rate limiting are out of
518
+ scope** for this microservice — wire them in your reverse proxy
519
+ (nginx `auth_basic` + `limit_req`, a WAF, or an API gateway).
520
+ The default `127.0.0.1` bind means a fresh
521
+ `bankstatementparser-api` is never publicly reachable unless you
522
+ explicitly opt in via `--host 0.0.0.0`.
523
+
524
+ ---
324
525
 
325
526
  ## Deduplication
326
527
 
@@ -338,7 +539,12 @@ print(f"Exact duplicates: {len(result.exact_duplicates)}")
338
539
  print(f"Suspected matches: {len(result.suspected_matches)}")
339
540
  ```
340
541
 
341
- The `Deduplicator` uses deterministic hashing for exact matches and configurable similarity thresholds for suspected matches. Each match group includes a confidence score and reason for auditability.
542
+ The `Deduplicator` uses deterministic hashing for exact matches
543
+ and configurable similarity thresholds for suspected matches. Each
544
+ match group includes a confidence score and reason for
545
+ auditability.
546
+
547
+ ---
342
548
 
343
549
  ## Export
344
550
 
@@ -367,9 +573,66 @@ lazy_df = parser.to_polars_lazy()
367
573
 
368
574
  Install with `pip install bankstatementparser[polars]`.
369
575
 
576
+ ### hledger + beancount *(v0.0.8)*
577
+
578
+ Export transactions to plaintext-accounting journal formats:
579
+
580
+ ```python
581
+ from bankstatementparser.export import to_hledger, to_beancount
582
+
583
+ journal = to_hledger(transactions, account="Assets:Bank:Checking")
584
+ Path("journal.ledger").write_text(journal)
585
+
586
+ # Or beancount format:
587
+ journal = to_beancount(transactions, account="Assets:Bank:Checking")
588
+ ```
589
+
590
+ Uses `Transaction.category` as the contra-account when set by the
591
+ enrichment module.
592
+
593
+ ### Bulk directory scanner *(v0.0.8)*
594
+
595
+ Scan a folder tree and ingest every statement, deduplicating
596
+ across the batch:
597
+
598
+ ```python
599
+ from bankstatementparser.hybrid import scan_and_ingest
600
+
601
+ batch = scan_and_ingest("statements/2026/", pattern="**/*.pdf")
602
+ print(f"{batch.file_count} files, {batch.total_unique} unique transactions")
603
+ ```
604
+
605
+ ### Account mapping *(v0.0.8)*
606
+
607
+ Map transactions to ledger accounts via configurable regex rules:
608
+
609
+ ```python
610
+ from bankstatementparser.enrichment import AccountMapper
611
+
612
+ mapper = AccountMapper.from_json("mapping.json")
613
+ for tx, account in zip(transactions, mapper.map_batch(transactions)):
614
+ print(f"{tx.description} -> {account}")
615
+ ```
616
+
617
+ ### Multi-currency verification *(v0.0.8)*
618
+
619
+ ```python
620
+ from bankstatementparser.hybrid import verify_balance_multi_currency
621
+
622
+ results = verify_balance_multi_currency(
623
+ transactions,
624
+ balances={"GBP": (opening, closing), "EUR": (opening, closing)},
625
+ )
626
+ for currency, v in results.items():
627
+ print(f"{currency}: {v.status.value}")
628
+ ```
629
+
630
+ ---
631
+
370
632
  ## Examples
371
633
 
372
- See [`examples/`](examples/README.md) for 22 runnable scripts (14 deterministic + 8 hybrid):
634
+ See [`examples/`](examples/README.md) for 22 runnable scripts
635
+ (14 deterministic + 8 hybrid):
373
636
 
374
637
  ### Deterministic parsers
375
638
 
@@ -402,37 +665,62 @@ See [`examples/`](examples/README.md) for 22 runnable scripts (14 deterministic
402
665
  | `hybrid/05_dedupe_recurring.py` | `normalize_description()` + `dedupe_by_hash()` for idempotent batching |
403
666
  | `hybrid/06_cli_walkthrough.sh` | Four flavours of the new `--type ingest` CLI subcommand |
404
667
 
405
- See [`examples/hybrid/README.md`](examples/hybrid/README.md) for the full walkthrough including a Mermaid flow diagram, the cross-platform verification matrix, and the Ollama smoke-test results.
668
+ See [`examples/hybrid/README.md`](examples/hybrid/README.md) for
669
+ the full walkthrough including a Mermaid flow diagram, the
670
+ cross-platform verification matrix, and the Ollama smoke-test
671
+ results.
672
+
673
+ ---
406
674
 
407
675
  ## XML Tag Mapping
408
676
 
409
- See [`docs/MAPPING.md`](docs/MAPPING.md) for a complete reference of ISO 20022 XML tags to DataFrame columns across all six formats. Use this when integrating with ERP systems or building reconciliation pipelines.
677
+ See [`docs/MAPPING.md`](docs/MAPPING.md) for a complete reference
678
+ of ISO 20022 XML tags to DataFrame columns across all six formats.
679
+ Use this when integrating with ERP systems or building
680
+ reconciliation pipelines.
681
+
682
+ ---
410
683
 
411
684
  ## Project Layout
412
685
 
413
686
  ```text
414
- bankstatementparser/ Source code (29 modules: deterministic core + hybrid + enrichment subpackages, 100% branch coverage)
415
- bankstatementparser/hybrid/ PDF pipeline: orchestrator, llm_extractor, vision, pdf_text, prompts, verification, ollama_direct
416
- docs/compliance/ ISO 13485 validation, risk register, traceability matrix
417
- examples/ 14 deterministic + 8 hybrid runnable example scripts
418
- scripts/ SBOM generation, checksums, signature verification
419
- tests/ 718 tests (unit, integration, property-based, security, hybrid mocks)
687
+ bankstatementparser/ Source code (32 modules)
688
+ bankstatementparser/hybrid/ PDF pipeline: orchestrator, llm_extractor, vision, scanner, ollama_direct, verification
689
+ bankstatementparser/enrichment/ Categorizer, AccountMapper, EnrichedTransaction
690
+ bankstatementparser/export/ hledger + beancount journal export
691
+ bankstatementparser/api.py REST API microservice (FastAPI)
692
+ docs/compliance/ ISO 13485 validation, risk register, traceability matrix
693
+ examples/ 14 deterministic + 8 hybrid runnable example scripts
694
+ scripts/ SBOM generation, checksums, signature verification
695
+ tests/ 818 tests (unit, integration, property-based, security, hybrid mocks)
420
696
  ```
421
697
 
422
- ## Security
698
+ ---
423
699
 
424
- Bank statement files contain sensitive financial and personal data. This library is designed with security as a primary constraint:
700
+ ## Security
425
701
 
426
- - **XXE protection** `resolve_entities=False`, `no_network=True`, `load_dtd=False`
427
- - **ZIP bomb protection** compression ratio limits, entry size caps, encrypted entry rejection
428
- - **Path traversal prevention** — dangerous pattern blocklist, symlink resolution
429
- - **PII redaction** — default masking of names, IBANs, and addresses
702
+ Bank statement files contain sensitive financial and personal
703
+ data. This library is designed with security as a primary
704
+ constraint:
705
+
706
+ - **XXE protection** — `resolve_entities=False`,
707
+ `no_network=True`, `load_dtd=False`
708
+ - **ZIP bomb protection** — compression ratio limits, entry size
709
+ caps, encrypted entry rejection
710
+ - **Path traversal prevention** — dangerous pattern blocklist,
711
+ symlink resolution
712
+ - **PII redaction** — default masking of names, IBANs, and
713
+ addresses
430
714
  - **Signed commits** — enforced in CI via GitHub API verification
431
- - **Supply chain** — SHA-256 hash-locked dependencies, CycloneDX SBOM, build provenance attestation
715
+ - **Supply chain** — SHA-256 hash-locked dependencies, CycloneDX
716
+ SBOM, build provenance attestation
432
717
 
433
718
  For vulnerability reports, see [SECURITY.md](.github/SECURITY.md).
434
719
 
435
- For the full compliance suite, see [`docs/compliance/`](docs/compliance/).
720
+ For the full compliance suite, see
721
+ [`docs/compliance/`](docs/compliance/).
722
+
723
+ ---
436
724
 
437
725
  ## Verify the Repository
438
726
 
@@ -440,18 +728,31 @@ Run the full validation suite locally:
440
728
 
441
729
  ```bash
442
730
  ruff check bankstatementparser tests examples scripts
731
+ ruff format --check bankstatementparser tests examples scripts
443
732
  python -m mypy bankstatementparser
444
733
  python -m pytest
445
734
  bandit -r bankstatementparser examples scripts -q
446
735
  ```
447
736
 
737
+ ---
738
+
739
+ ## Versioning
740
+
741
+ This project follows [Semantic Versioning](https://semver.org).
742
+ While the version is `0.y.z`, any release may contain breaking
743
+ changes; they are always listed under a **Changed — BREAKING**
744
+ heading in the [CHANGELOG](CHANGELOG.md) with migration notes.
745
+ From `1.0.0`, breaking changes will require a major release.
746
+ Deprecations emit `DeprecationWarning` for at least one minor
747
+ release before removal.
748
+
749
+ ---
750
+
448
751
  ## Contributing
449
752
 
450
753
  Signed commits required. See [CONTRIBUTING.md](CONTRIBUTING.md).
451
754
 
452
- ## License
453
-
454
- Apache License 2.0. See [LICENSE](LICENSE).
755
+ ---
455
756
 
456
757
  ## FAQ
457
758
 
@@ -459,21 +760,48 @@ Apache License 2.0. See [LICENSE](LICENSE).
459
760
  CAMT.053, PAIN.001, CSV, OFX, QFX, and MT940.
460
761
 
461
762
  **Does any data leave my infrastructure?**
462
- No. Zero network calls. XML parsers enforce `no_network=True`. No cloud, no telemetry.
763
+ No. Zero network calls. XML parsers enforce `no_network=True`. No
764
+ cloud, no telemetry.
463
765
 
464
766
  **Is PII redacted automatically?**
465
- Yes. Names, IBANs, and addresses are masked by default in console output and streaming. File exports retain full data.
767
+ Yes. Names, IBANs, and addresses are masked by default in console
768
+ output and streaming. File exports retain full data.
466
769
 
467
770
  **Is the extraction deterministic?**
468
- Yes. Same input produces byte-identical output. Critical for financial auditing.
771
+ Yes. Same input produces byte-identical output. Critical for
772
+ financial auditing.
469
773
 
470
774
  **Can it handle large files?**
471
- Yes. `parse_streaming()` is tested at 50,000 transactions (~25 MB) with bounded memory. Files over 50 MB use chunk-based streaming.
775
+ Yes. `parse_streaming()` is tested at 50,000 transactions (~25 MB)
776
+ with bounded memory. Files over 50 MB use chunk-based streaming.
777
+
778
+ See [FAQ.md](FAQ.md) for the complete FAQ covering data privacy,
779
+ technical specs, and treasury workflows.
780
+
781
+ ---
472
782
 
473
- See [FAQ.md](FAQ.md) for the complete FAQ covering data privacy, technical specs, and treasury workflows.
783
+ ## Documentation
784
+
785
+ | Document | Covers |
786
+ |---|---|
787
+ | [API reference](https://sebastienrousseau.github.io/bankstatementparser/) | Full API reference generated from docstrings (mkdocs + mkdocstrings). Build locally with `poetry install --with docs && poetry run mkdocs serve`. |
788
+ | [`docs/MAPPING.md`](docs/MAPPING.md) | ISO 20022 XML tag to DataFrame column mapping for all six formats. |
789
+ | [`FAQ.md`](FAQ.md) | Data privacy, determinism, technical specs, treasury workflows. |
790
+ | [`CONTRIBUTING.md`](CONTRIBUTING.md) | Signed-commit policy, the five local gates, PR guidelines. |
791
+ | [`CHANGELOG.md`](CHANGELOG.md) | Per-release notes following Keep a Changelog. |
792
+ | [`SECURITY.md`](.github/SECURITY.md) | Disclosure policy, supported versions, contact. |
793
+ | [`docs/compliance/`](docs/compliance/) | ISO 13485 validation, risk register, traceability matrix. |
794
+ | [`examples/hybrid/README.md`](examples/hybrid/README.md) | Hybrid-pipeline walkthrough: mock vs. live mode, verification matrix, Ollama smoke tests. |
474
795
 
475
796
  ---
476
797
 
477
- THE ARCHITECT ᛫ Sebastien Rousseau ᛫ https://sebastienrousseau.com
478
- THE ENGINE ᛞ EUXIS ᛫ Enterprise Unified Execution Intelligence System ᛫ https://euxis.co
798
+ ## License
799
+
800
+ Licensed under the
801
+ [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
802
+ See [LICENSE](LICENSE).
803
+
804
+ See [CHANGELOG.md](CHANGELOG.md) for release history.
805
+
806
+ <p align="right"><a href="#contents">Back to Top</a></p>
479
807