ing_kontoauszug_parser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b08a3098282dadebc68193749586e1ca664548d062dd022f4db358f21b0776a5
+  data.tar.gz: 9c107cf1bc1080515e0440401c0b6c7eceb5abe6db6d521f5424103c20579cf2
+SHA512:
+  metadata.gz: 5f36aa98eaf89d07fe86b6f8f5dc147092b9b495894ef4397c224191b9a1b508e97fbf9a97e3e1daae417eaadc0e73ed1068f00f12f9c0e22925548da68af9e1
+  data.tar.gz: 3b059deaba3f60f49d66c5b5db6da84fee6b568ed3cb7149cc109186497dad2d9ef773442ef0ef58188ead16f5342a6773f1debd72f11fd2555e1e85707d296b

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.1.0] - 2026-01-21
+
+### Added
+- Initial release of `IngKontoauszugParser` gem.
+- `StatementParser` class to parse ING Bank (Germany) statement PDFs into structured JSON.
+- PDF text extraction via poppler (fast) or pdf-reader (portable).
+- IBAN extraction and validation using ISO 13616 checksum.
+- Transaction parsing with dates, amounts, recipients, and narratives.
+- SEPA metadata extraction (`mandate_id`, `reference`) with ARN fallback.
+- Google Pay detection in transaction narratives.
+- CLI tool `bin/pdf_to_json` for direct PDF-to-JSON conversion.
+- Interactive console via `bin/console`.
+- Comprehensive test suite with Minitest.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Go
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,159 @@
+# ING Kontoauszug Parser
+
+A Ruby gem for parsing ING Bank (Germany) statement PDFs and text exports into structured JSON.
+
+## Why This Gem?
+
+ING Germany provides account statements as PDFs, but extracting transaction data programmatically is challenging:
+
+- **PDF text extraction is messy** – Column alignment, page breaks, and OCR artifacts make raw text unreliable
+- **No official export API** – ING doesn't provide a machine-readable format like CSV or OFX
+- **Manual categorization is tedious** – Hundreds of transactions need consistent tagging for budgeting
+
+This gem solves these problems by:
+
+1. **Extracting clean text** from statement PDFs using poppler (fast) or pdf-reader (portable)
+2. **Parsing transactions** into structured data with dates, amounts, recipients, and narratives
+3. **Extracting SEPA metadata** like mandate IDs and references for reconciliation
+4. **Validating IBANs** using the ISO 13616 checksum algorithm
+5. **Detecting payment methods** like Google Pay for automatic categorization
+
+Use it to build budgeting tools, import transactions into accounting software, or automate expense tracking.
+
+## Installation
+
+```ruby
+# Gemfile
+gem 'ing_kontoauszug_parser'
+```
+
+```bash
+bundle install
+```
+
+## Quick Start
+
+```ruby
+require 'ing_kontoauszug_parser'
+
+parser = IngKontoauszugParser::StatementParser.new
+result = parser.parse(file_path: 'statement.pdf')
+
+result[:header][:iban]              # => "DE89 3704 0044 0532 0130 00"
+result[:statements].first[:amount_eur_numeric]  # => "-31.49"
+```
+
+## API
+
+```ruby
+parser = IngKontoauszugParser::StatementParser.new
+
+# Parse PDF file
+result = parser.parse(file_path: 'statement.pdf')
+
+# Parse text export
+result = parser.parse_text(File.read('export.txt'))
+
+# Parse pre-split lines
+result = parser.parse_lines(lines)
+
+# Parse booking lines only (no header)
+statements = parser.parse_statement_lines(lines)
+```
+
+## Output Format
+
+```json
+{
+  "header": {
+    "iban": "DE89 3704 0044 0532 0130 00"
+  },
+  "statements": [
+    {
+      "booking_date": "01.08.2025",
+      "transfer_type": "Lastschrift",
+      "recipient": "Allianz Direct Vers.",
+      "amount_eur": "-31,49",
+      "amount_eur_numeric": "-31.49",
+      "amount_direction": "debit",
+      "value_date": "01.08.2025",
+      "narrative": "Versicherungsbeitrag August",
+      "mandate_id": "MA123456",
+      "reference": "RF123456789",
+      "google_pay": true
+    }
+  ]
+}
+```
+
+### Statement Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `booking_date` | string | Booking date (dd.mm.yyyy) |
+| `transfer_type` | string | Transaction type (e.g., Lastschrift, Gutschrift) |
+| `recipient` | string | Counterparty name |
+| `amount_eur` | string | Original amount with German formatting |
+| `amount_eur_numeric` | string | Normalized decimal (BigDecimal as string) |
+| `amount_direction` | string | `debit`, `credit`, or `neutral` |
+| `value_date` | string | Value date (dd.mm.yyyy) |
+| `narrative` | string | Transaction details |
+| `mandate_id` | string? | SEPA mandate ID (if present) |
+| `reference` | string? | SEPA reference or ARN (if present) |
+| `google_pay` | boolean? | `true` if Google Pay detected |
+
+## CLI Tools
+
+```bash
+# Parse PDF statement to JSON
+bin/pdf_to_json statement.pdf
+
+# Write to file instead of stdout
+bin/pdf_to_json -o output.json statement.pdf
+
+# Interactive console
+bin/console
+```
+
+## Error Handling
+
+PDF reader errors are wrapped as `IngKontoauszugParser::Error`:
+
+```ruby
+begin
+  parser.parse(file_path: 'encrypted.pdf')
+rescue IngKontoauszugParser::Error => e
+  puts e.message
+end
+```
+
+Non-fatal issues (e.g., statements missing value dates) are collected in an optional `warnings` array:
+
+```ruby
+result = parser.parse_text(text)
+result[:warnings]&.each { |w| puts "Warning: #{w}" }
+```
+
+## Development
+
+```bash
+bin/setup      # Install dependencies
+rake test      # Run tests
+rake rubocop   # Lint
+```
+
+## Requirements
+
+- Ruby 3.0+
+- pdf-reader gem
+- Input files must be UTF-8 encoded
+
+## Documentation
+
+- [Feature Summary](docs/FEATURE_SUMMARY.md)
+- [Technical Summary](docs/TECHNICAL_SUMMARY.md)
+- [Changelog](CHANGELOG.md)
+
+## Contributing
+
+Bug reports and pull requests welcome on GitHub.

data/bin/console

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'bundler/setup'
+require 'ing_kontoauszug_parser'
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/pdf_to_json

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'json'
+require 'optparse'
+require 'pdf/reader'
+
+$LOAD_PATH.unshift(File.expand_path('../lib', __dir__)) unless $LOAD_PATH.include?(File.expand_path('../lib', __dir__))
+require 'ing_kontoauszug_parser'
+
+options = {}
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: pdf_to_json [options] FILE'
+  opts.separator ''
+  opts.separator 'Parse an ING statement PDF directly into JSON.'
+  opts.separator ''
+
+  opts.on('-o', '--output FILE', 'Write JSON to FILE instead of STDOUT') do |file|
+    options[:output] = file
+  end
+
+  opts.on('-h', '--help', 'Show this help message') do
+    puts opts
+    exit
+  end
+
+  opts.on('-v', '--version', 'Show version') do
+    puts "ing_kontoauszug_parser #{IngKontoauszugParser::VERSION}"
+    exit
+  end
+end
+
+parser.parse!
+
+if ARGV.empty?
+  warn parser
+  exit 1
+end
+
+pdf_path = ARGV.first
+unless File.exist?(pdf_path)
+  warn "File not found: #{pdf_path}"
+  exit 1
+end
+
+begin
+  reader = PDF::Reader.new(pdf_path)
+rescue PDF::Reader::MalformedPDFError, PDF::Reader::EncryptedPDFError => e
+  warn "Could not open PDF: #{e.message}"
+  exit 1
+end
+
+full_text = reader.pages.map { |page| page.text.to_s }.join("\n")
+
+statement_parser = IngKontoauszugParser::StatementParser.new
+result = statement_parser.parse_text(full_text)
+
+json_output = JSON.pretty_generate(result)
+
+if options[:output]
+  File.write(options[:output], json_output)
+else
+  puts json_output
+end

data/bin/setup

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install

data/lib/ing_kontoauszug_parser/header.rb

@@ -0,0 +1,146 @@
+# frozen_string_literal: true
+
+module IngKontoauszugParser
+  # Extracts and validates IBANs from bank statement text.
+  #
+  # This module handles the extraction of International Bank Account Numbers
+  # from ING statement headers and validates them using the ISO 13616 mod-97
+  # checksum algorithm.
+  #
+  # == IBAN Format
+  #
+  # IBANs consist of:
+  # - 2-letter country code (e.g., "DE" for Germany)
+  # - 2 check digits
+  # - Up to 30 alphanumeric characters (Basic Bank Account Number - BBAN)
+  #
+  # Example: "DE89 3704 0044 0532 0130 00"
+  #
+  # == Validation Algorithm
+  #
+  # The ISO 13616 mod-97 algorithm:
+  # 1. Move the first 4 characters to the end
+  # 2. Convert letters to numbers (A=10, B=11, ..., Z=35)
+  # 3. Calculate modulo 97 of the resulting number
+  # 4. Valid if remainder equals 1
+  #
+  # @example Extract and validate IBAN from statement text
+  #   iban = Header.extract_iban(statement_text)
+  #   #=> "DE89 3704 0044 0532 0130 00"
+  #
+  # @example Validate an IBAN directly
+  #   Header.valid_iban?("DE89 3704 0044 0532 0130 00")  #=> true
+  #   Header.valid_iban?("DE00 0000 0000 0000 0000 00")  #=> false
+  #
+  # @api private
+  # @see https://en.wikipedia.org/wiki/International_Bank_Account_Number IBAN specification
+  module Header
+    # Pattern to match IBAN lines in ING statement format.
+    # Captures the IBAN portion after the "IBAN" label.
+    IBAN_REGEX = /IBAN\s+([A-Z0-9 ]{10,})/
+    private_constant :IBAN_REGEX
+
+    module_function
+
+    # Finds and extracts the IBAN from statement text.
+    #
+    # Scans the text for a line containing "IBAN" and extracts the account
+    # number that follows. By default, validates the extracted IBAN using
+    # the mod-97 checksum algorithm.
+    #
+    # @param text [String] statement text (typically the header section or full statement)
+    # @param validate [Boolean] whether to verify the IBAN checksum. Set to false
+    #   for faster extraction when validation is handled elsewhere.
+    # @return [String] the extracted IBAN with original spacing preserved
+    # @raise [IngKontoauszugParser::HeaderNotFound] if no IBAN line is found in the text
+    # @raise [IngKontoauszugParser::InvalidIBAN] if validation is enabled and the
+    #   checksum verification fails
+    #
+    # @example Extract with validation (default)
+    #   Header.extract_iban("IBAN DE89 3704 0044 0532 0130 00\n...")
+    #   #=> "DE89 3704 0044 0532 0130 00"
+    #
+    # @example Extract without validation
+    #   Header.extract_iban(text, validate: false)
+    def extract_iban(text, validate: true)
+      line = text.to_s.lines.find { |header_line| header_line.include?('IBAN') }
+      unless line
+        raise IngKontoauszugParser::HeaderNotFound,
+              'IBAN line not found in statement text'
+      end
+
+      match = line.match(IBAN_REGEX)
+      unless match
+        raise IngKontoauszugParser::HeaderNotFound,
+              'IBAN value not found in statement text'
+      end
+
+      iban = match[1].strip
+      validate_iban!(iban) if validate
+      iban
+    end
+
+    # Validates an IBAN using the ISO 13616 mod-97 checksum algorithm.
+    #
+    # The algorithm rearranges the IBAN (moving country code and check digits
+    # to the end), converts letters to numbers, and verifies that the resulting
+    # number modulo 97 equals 1.
+    #
+    # @param iban [String] the IBAN to validate. Spaces are allowed and will be
+    #   stripped before validation. Case-insensitive.
+    # @return [true] if the IBAN is valid
+    # @raise [IngKontoauszugParser::InvalidIBAN] if the checksum fails, with a
+    #   message containing the actual remainder for debugging
+    #
+    # @example Validate a correct IBAN
+    #   Header.validate_iban!("DE89 3704 0044 0532 0130 00")  #=> true
+    #
+    # @example Invalid IBAN raises exception
+    #   Header.validate_iban!("DE00 0000 0000 0000 0000 00")
+    #   #=> raises InvalidIBAN: "IBAN checksum validation failed..."
+    def validate_iban!(iban)
+      normalized = iban.gsub(/\s/, '').upcase
+      return true if normalized.length < 5 # Too short to validate meaningfully
+
+      # Move first 4 characters to end
+      rearranged = normalized[4..] + normalized[0, 4]
+
+      # Convert letters to numbers (A=10, B=11, ..., Z=35)
+      numeric_string = rearranged.chars.map do |char|
+        if char.match?(/[A-Z]/)
+          (char.ord - 'A'.ord + 10).to_s
+        else
+          char
+        end
+      end.join
+
+      # Perform mod-97 check
+      remainder = numeric_string.to_i % 97
+      return true if remainder == 1
+
+      raise IngKontoauszugParser::InvalidIBAN,
+            "IBAN checksum validation failed for #{iban} (remainder: #{remainder}, expected: 1)"
+    end
+
+    # Checks if an IBAN is valid without raising an exception.
+    #
+    # This is a convenience wrapper around {#validate_iban!} that returns
+    # a boolean instead of raising an exception on invalid IBANs.
+    #
+    # @param iban [String] the IBAN to validate
+    # @return [Boolean] true if the IBAN passes checksum validation, false otherwise
+    #
+    # @example Check validity
+    #   if Header.valid_iban?(user_input)
+    #     process_payment(user_input)
+    #   else
+    #     show_error("Invalid IBAN")
+    #   end
+    def valid_iban?(iban)
+      validate_iban!(iban)
+      true
+    rescue InvalidIBAN
+      false
+    end
+  end
+end

data/lib/ing_kontoauszug_parser/pdf_extractor.rb

@@ -0,0 +1,233 @@
+# frozen_string_literal: true
+
+require 'English'
+
+module IngKontoauszugParser
+  # Extracts text from PDF statement files for subsequent parsing.
+  #
+  # This class provides a unified interface to PDF text extraction with automatic
+  # backend selection. It supports two extraction methods:
+  #
+  # == Poppler Backend (Recommended)
+  #
+  # Uses the +pdftotext+ command-line utility from poppler-utils. This is
+  # 5-10x faster than the pure Ruby alternative and handles complex PDFs better.
+  # Install via your package manager:
+  #
+  #   # Debian/Ubuntu
+  #   apt-get install poppler-utils
+  #
+  #   # macOS
+  #   brew install poppler
+  #
+  # == PDF::Reader Backend (Fallback)
+  #
+  # Uses the pdf-reader gem for pure Ruby extraction. Automatically selected
+  # when poppler is not available. Supports parallel page processing for
+  # multi-page statements to improve performance.
+  #
+  # == Backend Selection
+  #
+  # By default, the fastest available backend is selected automatically:
+  # 1. Poppler if +pdftotext+ is found in PATH
+  # 2. PDF::Reader with parallel processing otherwise
+  #
+  # You can force a specific backend via the +:backend+ parameter.
+  #
+  # @example Automatic backend selection
+  #   lines = PdfExtractor.extract_lines('statement.pdf')
+  #
+  # @example Force specific backend
+  #   lines = PdfExtractor.extract_lines('statement.pdf', backend: :pdf_reader)
+  #
+  # @api private
+  # @see StatementParser#parse The public API that uses this class
+  class PdfExtractor
+    # Minimum page count before parallel processing is used with PDF::Reader.
+    # Below this threshold, thread overhead exceeds the parallelization benefit.
+    # @api private
+    PARALLEL_THRESHOLD = 4
+    private_constant :PARALLEL_THRESHOLD
+
+    # Checks whether the poppler +pdftotext+ utility is installed.
+    #
+    # The result is cached after the first call for performance. Use this
+    # to conditionally show backend recommendations to users.
+    #
+    # @return [Boolean] true if +pdftotext+ is found in PATH
+    def self.poppler_available?
+      return @poppler_available if defined?(@poppler_available)
+
+      @poppler_available = system('which pdftotext > /dev/null 2>&1')
+    end
+
+    # Extracts text lines from a PDF using the best available method.
+    #
+    # This is the primary entry point for PDF extraction. It automatically
+    # selects the fastest backend and returns an array of text lines ready
+    # for parsing by {TextParser}.
+    #
+    # @param file_path [String] absolute or relative path to the PDF file
+    # @param backend [Symbol, nil] force a specific backend:
+    #   - +:poppler+ - use pdftotext (fails if not installed)
+    #   - +:pdf_reader+ - use pdf-reader gem
+    #   - +nil+ - auto-select fastest available (default)
+    # @param parallel [Boolean] enable threaded page processing for pdf-reader.
+    #   Has no effect when using poppler backend. Default: true
+    # @return [Array<String>] text lines with trailing whitespace stripped
+    # @raise [IngKontoauszugParser::Error] if PDF extraction fails
+    # @raise [ArgumentError] if an unknown backend is specified
+    #
+    # @example Extract with automatic backend selection
+    #   lines = PdfExtractor.extract_lines('/path/to/statement.pdf')
+    #
+    # @example Force pdf-reader without parallelization
+    #   lines = PdfExtractor.extract_lines(path, backend: :pdf_reader, parallel: false)
+    def self.extract_lines(file_path, backend: nil, parallel: true)
+      backend ||= poppler_available? ? :poppler : :pdf_reader
+
+      case backend
+      when :poppler
+        extract_with_poppler(file_path)
+      when :pdf_reader
+        parallel ? extract_with_pdf_reader_parallel(file_path) : extract_with_pdf_reader(file_path)
+      else
+        raise ArgumentError, "Unknown PDF backend: #{backend}"
+      end
+    end
+
+    # Extracts text using the poppler +pdftotext+ command-line utility.
+    #
+    # Uses the +-layout+ flag to preserve text positioning, which is essential
+    # for correctly parsing the columnar ING statement format. The +-nopgbrk+
+    # flag removes page break characters for cleaner output.
+    #
+    # @param file_path [String] path to the PDF file
+    # @return [Array<String>] extracted text lines
+    # @raise [IngKontoauszugParser::Error] if pdftotext returns a non-zero exit code
+    def self.extract_with_poppler(file_path)
+      escaped_path = Shellwords.escape(file_path)
+      output = `pdftotext -layout -nopgbrk #{escaped_path} - 2>&1`
+
+      raise IngKontoauszugParser::Error, "pdftotext failed: #{output}" unless $CHILD_STATUS.success?
+
+      # Split into lines efficiently, stripping trailing whitespace
+      lines = []
+      output.each_line do |line|
+        lines << line.rstrip
+      end
+      lines
+    end
+
+    # Extracts text using the pdf-reader gem with optional parallelization.
+    #
+    # For multi-page statements (4+ pages), spawns threads to process pages
+    # concurrently. Smaller documents use sequential processing to avoid
+    # thread overhead. Results are sorted by page index to maintain order.
+    #
+    # @param file_path [String] path to the PDF file
+    # @return [Array<String>] extracted text lines in page order
+    # @raise [IngKontoauszugParser::Error] if pdf-reader encounters a PDF error
+    def self.extract_with_pdf_reader_parallel(file_path)
+      require 'pdf-reader'
+
+      reader = PDF::Reader.new(file_path)
+      pages = reader.pages.select { |p| p.respond_to?(:text) }
+
+      # Use sequential processing for small documents
+      return extract_pages_sequential(pages) if pages.length < PARALLEL_THRESHOLD
+
+      # Process pages in parallel using threads
+      extract_pages_parallel(pages)
+    rescue StandardError => e
+      raise IngKontoauszugParser::Error, e.message if pdf_reader_error?(e)
+
+      raise
+    end
+
+    # Extracts text using the pdf-reader gem with sequential processing.
+    #
+    # This is the simplest extraction method, processing pages one at a time.
+    # Use this for debugging or when thread safety is a concern. For production
+    # use, prefer {.extract_with_pdf_reader_parallel}.
+    #
+    # @param file_path [String] path to the PDF file
+    # @return [Array<String>] extracted text lines in page order
+    # @raise [IngKontoauszugParser::Error] if pdf-reader encounters a PDF error
+    def self.extract_with_pdf_reader(file_path)
+      require 'pdf-reader'
+
+      reader = PDF::Reader.new(file_path)
+      lines = []
+
+      reader.pages.each do |page|
+        next unless page.respond_to?(:text)
+
+        page.text.to_s.each_line do |line|
+          lines << line.rstrip
+        end
+      end
+
+      lines
+    rescue StandardError => e
+      raise IngKontoauszugParser::Error, e.message if pdf_reader_error?(e)
+
+      raise
+    end
+
+    # Determines if an error originated from the pdf-reader gem.
+    #
+    # Used to wrap pdf-reader exceptions in {IngKontoauszugParser::Error} for
+    # consistent error handling in the public API.
+    #
+    # @param error [StandardError] the caught exception
+    # @return [Boolean] true if error class is from PDF::Reader namespace
+    # @api private
+    def self.pdf_reader_error?(error)
+      error.class.name.start_with?('PDF::Reader::')
+    end
+
+    # Extracts text from pages one at a time.
+    #
+    # @param pages [Array<PDF::Reader::Page>] pages to process
+    # @return [Array<String>] concatenated text lines from all pages
+    # @api private
+    def self.extract_pages_sequential(pages)
+      lines = []
+      pages.each do |page|
+        page.text.to_s.each_line do |line|
+          lines << line.rstrip
+        end
+      end
+      lines
+    end
+
+    # Extracts text from pages concurrently using threads.
+    #
+    # Spawns one thread per page and collects results. Page order is preserved
+    # by sorting on page index before flattening.
+    #
+    # @param pages [Array<PDF::Reader::Page>] pages to process
+    # @return [Array<String>] concatenated text lines in page order
+    # @api private
+    def self.extract_pages_parallel(pages)
+      # Create threads to process each page
+      threads = pages.map.with_index do |page, idx|
+        Thread.new do
+          page_lines = []
+          page.text.to_s.each_line do |line|
+            page_lines << line.rstrip
+          end
+          [idx, page_lines]
+        end
+      end
+
+      # Collect results and sort by page index to maintain order
+      results = threads.map(&:value)
+      results.sort_by(&:first).flat_map(&:last)
+    end
+
+    # Pre-load shellwords for poppler extraction
+    require 'shellwords'
+  end
+end