fintl 0.1.0__tar.gz

fintl-0.1.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.3
+Name: fintl
+Version: 0.1.0
+Summary: Financial ETL CLI: parse, normalize, and explore your bank transaction data.
+Keywords: finance,etl,banking,transactions,cli
+Author: eschmidt42
+Author-email: eschmidt42 <11818904+eschmidt42@users.noreply.github.com>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial
+Requires-Dist: altair>=6.0.0
+Requires-Dist: beautifulsoup4>=4.14.3
+Requires-Dist: chardet>=7.3.0
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: instructor>=1.14.5
+Requires-Dist: polars>=1.39.3
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: pydantic-settings>=2.13.1
+Requires-Dist: python-dateutil>=2.9.0.post0
+Requires-Dist: rich>=14.3.3
+Requires-Dist: textual>=8.1.1
+Requires-Dist: xlsxwriter>=3.2.9
+Requires-Python: >=3.13
+Project-URL: Homepage, https://github.com/eschmidt42/fintl
+Project-URL: Repository, https://github.com/eschmidt42/fintl
+Project-URL: Issues, https://github.com/eschmidt42/fintl/issues
+Description-Content-Type: text/markdown
+
+# `fintl`
+
+[![CI](https://github.com/eschmidt42/fintl/actions/workflows/ci.yml/badge.svg)](https://github.com/eschmidt42/fintl/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/eschmidt42/fintl/branch/main/graph/badge.svg?token=FM1L1A7BQ8)](https://codecov.io/gh/eschmidt42/fintl)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
+
+> Financial ETL CLI: parse, normalize, and explore your bank transaction data.
+
+## TL;DR
+
+This tool helps you process, visualize and search your balance and transaction information that you have exported from your bank accounts.
+
+Currently supports DKB, Postbank, GLS and Scalable Capital Broker.
+
+Supported file formats: CSV, HTML, and PNG. PNG parsing uses a local [ollama](https://ollama.com) instance with a multimodal model — opt-in via `fintl.toml` (required only for Scalable broker PNG statements; gracefully skipped when not configured).
+
+**All your data stays on your machine. No need to trust another entity that is PSD2 certified.**
+
+## How to install
+
+```bash
+git clone https://github.com/eschmidt42/fintl.git
+cd fintl
+uv sync
+uv tool install .
+```
+
+After installation, `fintl` should be available on your `PATH`:
+
+```bash
+which fintl
+# e.g. /Users/YOURUSER/.local/bin/fintl
+```
+
+## How to use
+
+1. Configure your `~/.config/petprojects/fintl.toml`. For details see [here](./src/fintl/cli/README.md#fintltoml).
+2. Go to your bank account.
+3. Select your service, e.g. Giro.
+4. Export csv file or similar to `~/Downloads`, or directly your source dir for your bank / service.
+5. Optionally, if you've stored your file in `~/Downloads`, run `cd ~/Downloads` followed by `fintl store` (uses your `fintl.toml` from step 1).
+6. Optionally, if you want to process PNG screenshots via Ollama, start Ollama.
+7. Run the etl via `fintl etl` (also uses your `fintl.toml` from step 1).
+8. Upon success visualize / search your data via `fintl plot` or `fintl search`.
+
+[Please see here and below](./src/fintl/cli/README.md#top-level-usage) for more usage details.
+
+## Repo structure
+
+* `src/fintl/accounts_etl/` — core ETL logic: schemas, parsers, registry, runner
+* `src/fintl/cli/` — CLI entry point and subcommands (`etl`, `store`, `search`, `plot`)
+* `tests/` — tests for packages of this repo
+
+## Development
+
+Run tests:
+
+```bash
+uv run pytest -n auto tests
+```
+
+Type check:
+
+```bash
+uv run ty check src
+```
+
+Lint, format, type check, test and all the other good stuff:
+
+```bash
+pre-commit run --all-files
+```
+
+For release steps see [here](./docs/releases.md).

fintl-0.1.0/README.md

@@ -0,0 +1,78 @@
+# `fintl`
+
+[![CI](https://github.com/eschmidt42/fintl/actions/workflows/ci.yml/badge.svg)](https://github.com/eschmidt42/fintl/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/eschmidt42/fintl/branch/main/graph/badge.svg?token=FM1L1A7BQ8)](https://codecov.io/gh/eschmidt42/fintl)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.13+](https://img.shields.io/badge/python-3.13+-blue.svg)](https://www.python.org/)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
+
+> Financial ETL CLI: parse, normalize, and explore your bank transaction data.
+
+## TL;DR
+
+This tool helps you process, visualize and search your balance and transaction information that you have exported from your bank accounts.
+
+Currently supports DKB, Postbank, GLS and Scalable Capital Broker.
+
+Supported file formats: CSV, HTML, and PNG. PNG parsing uses a local [ollama](https://ollama.com) instance with a multimodal model — opt-in via `fintl.toml` (required only for Scalable broker PNG statements; gracefully skipped when not configured).
+
+**All your data stays on your machine. No need to trust another entity that is PSD2 certified.**
+
+## How to install
+
+```bash
+git clone https://github.com/eschmidt42/fintl.git
+cd fintl
+uv sync
+uv tool install .
+```
+
+After installation, `fintl` should be available on your `PATH`:
+
+```bash
+which fintl
+# e.g. /Users/YOURUSER/.local/bin/fintl
+```
+
+## How to use
+
+1. Configure your `~/.config/petprojects/fintl.toml`. For details see [here](./src/fintl/cli/README.md#fintltoml).
+2. Go to your bank account.
+3. Select your service, e.g. Giro.
+4. Export csv file or similar to `~/Downloads`, or directly your source dir for your bank / service.
+5. Optionally, if you've stored your file in `~/Downloads`, run `cd ~/Downloads` followed by `fintl store` (uses your `fintl.toml` from step 1).
+6. Optionally, if you want to process PNG screenshots via Ollama, start Ollama.
+7. Run the etl via `fintl etl` (also uses your `fintl.toml` from step 1).
+8. Upon success visualize / search your data via `fintl plot` or `fintl search`.
+
+[Please see here and below](./src/fintl/cli/README.md#top-level-usage) for more usage details.
+
+## Repo structure
+
+* `src/fintl/accounts_etl/` — core ETL logic: schemas, parsers, registry, runner
+* `src/fintl/cli/` — CLI entry point and subcommands (`etl`, `store`, `search`, `plot`)
+* `tests/` — tests for packages of this repo
+
+## Development
+
+Run tests:
+
+```bash
+uv run pytest -n auto tests
+```
+
+Type check:
+
+```bash
+uv run ty check src
+```
+
+Lint, format, type check, test and all the other good stuff:
+
+```bash
+pre-commit run --all-files
+```
+
+For release steps see [here](./docs/releases.md).

fintl-0.1.0/pyproject.toml

@@ -0,0 +1,63 @@
+[project]
+name = "fintl"
+version = "0.1.0"
+description = "Financial ETL CLI: parse, normalize, and explore your bank transaction data."
+readme = "README.md"
+license = { text = "MIT" }
+keywords = ["finance", "etl", "banking", "transactions", "cli"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: End Users/Desktop",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Office/Business :: Financial",
+]
+authors = [
+    { name = "eschmidt42", email = "11818904+eschmidt42@users.noreply.github.com" }
+]
+requires-python = ">=3.13"
+dependencies = [
+    "altair>=6.0.0",
+    "beautifulsoup4>=4.14.3",
+    "chardet>=7.3.0",
+    "httpx>=0.28.1",
+    "instructor>=1.14.5",
+    "polars>=1.39.3",
+    "pydantic>=2.12.5",
+    "pydantic-settings>=2.13.1",
+    "python-dateutil>=2.9.0.post0",
+    "rich>=14.3.3",
+    "textual>=8.1.1",
+    "xlsxwriter>=3.2.9",
+]
+
+[project.urls]
+Homepage = "https://github.com/eschmidt42/fintl"
+Repository = "https://github.com/eschmidt42/fintl"
+Issues = "https://github.com/eschmidt42/fintl/issues"
+
+[project.scripts]
+fintl = "fintl.cli.main:app"
+
+[build-system]
+requires = ["uv_build>=0.6.6,<0.11"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "inline-snapshot>=0.32.5",
+    "notebook>=7.5.5",
+    "pre-commit>=4.5.1",
+    "pytest>=9.0.2",
+    "pytest-cov>=7.1.0",
+    "pytest-xdist>=3.8.0",
+    "ruff>=0.15.7",
+    "ty>=0.0.25",
+    "vulture>=2.16",
+]
+
+[tool.inline-snapshot]
+format-command="ruff check --fix-only --stdin-filename {filename} | ruff format --stdin-filename {filename}"

fintl-0.1.0/src/fintl/accounts_etl/dkb/credit0.py

@@ -0,0 +1,242 @@
+import datetime
+import logging
+import re
+import typing as T
+from pathlib import Path
+
+import polars as pl
+
+from fintl.accounts_etl.exceptions import (
+    ExtractBalanceException,
+    ExtractTransactionsException,
+)
+from fintl.accounts_etl.file_helper import (
+    concatenate_new_information_to_history,
+    detect_new_raw_files,
+    detect_relevant_target_files,
+    get_parser_source_files,
+    store_balance,
+    store_transactions,
+)
+from fintl.accounts_etl.files import copy_new_files, load_lines, select_files_to_copy
+from fintl.accounts_etl.schemas import (
+    HASH_COLUMNS,
+    TRANSACTION_COLUMNS,
+    BalanceInfo,
+    Case,
+    Config,
+    DKBCreditParserEnum,
+    ProviderEnum,
+    ServiceEnum,
+)
+from fintl.accounts_etl.utils import (
+    detect_encoding,
+    find_line_with_pattern,
+    german_string_numbers_to_floats,
+    hash_transactions,
+    verify_transactions,
+)
+
+logger = logging.getLogger(__name__)
+
+CASE = Case(
+    provider=ProviderEnum.dkb.value,
+    service=ServiceEnum.credit.value,
+    parser=DKBCreditParserEnum.credit0.value,
+)
+
+
+def check_if_parser_applies(file_path: Path) -> bool:
+    is_file_name_match = (
+        re.search(
+            r"(\d{4}-\d{2}-\d{2}_to_\d{4}-\d{2}-\d{2}_\d{4}________\d{4}.csv)$",
+            str(file_path.name),
+        )
+        is not None
+    )
+    return is_file_name_match
+
+
+def extract_transactions(
+    case: Case, file_path: Path, lines: T.List[str], encoding: str
+) -> pl.DataFrame:
+    transaction_pattern: str = '^("?Umsatz)'  # start of transactions
+
+    date_format: str = "%d.%m.%Y"
+    date_cols: list = ["Belegdatum"]
+
+    ix_start_transactions, transactions_header = find_line_with_pattern(
+        lines, pattern=transaction_pattern
+    )
+    logger.debug(
+        f"{file_path=} has {ix_start_transactions=} and {transactions_header=}"
+    )
+
+    schema = {
+        "Umsatz abgerechnet und nicht im Saldo enthalten": pl.Utf8,
+        "Wertstellung": pl.Utf8,
+        "Belegdatum": pl.Utf8,
+        "Beschreibung": pl.Utf8,
+        "Betrag (EUR)": pl.Utf8,
+        "Ursprünglicher Betrag": pl.Utf8,
+    }
+    transactions = pl.read_csv(
+        file_path,
+        skip_rows=ix_start_transactions,
+        separator=";",
+        truncate_ragged_lines=True,
+        encoding=encoding,
+        schema=schema,
+    )
+    transactions = transactions.with_columns(
+        [pl.col(col).str.to_date(date_format) for col in date_cols],
+    )
+
+    transactions = transactions.with_columns(
+        pl.col("Betrag (EUR)")
+        .str.strip_chars_end()
+        .map_elements(german_string_numbers_to_floats, return_dtype=pl.Float64),
+    )
+    transactions = transactions.with_columns(
+        amount=pl.col("Betrag (EUR)"),
+        description=pl.col("Beschreibung"),
+        date=pl.col("Belegdatum"),
+        source=pl.when(pl.col("Betrag (EUR)") > 0)
+        .then(pl.col("Beschreibung"))
+        .otherwise(pl.lit("myself")),
+        recipient=pl.when(pl.col("Betrag (EUR)") < 0)
+        .then(pl.col("Beschreibung"))
+        .otherwise(pl.lit("myself")),
+        provider=pl.lit(case.provider),
+        service=pl.lit(case.service),
+        parser=pl.lit(case.parser),
+        file=pl.lit(str(file_path)),
+    )
+    transactions = hash_transactions(transactions, hash_columns=HASH_COLUMNS)
+
+    verify_transactions(TRANSACTION_COLUMNS, transactions, file_path)
+
+    transactions = transactions.select(TRANSACTION_COLUMNS)
+
+    return transactions
+
+
+def extract_balance(case: Case, file_path: Path, lines: T.List[str]) -> BalanceInfo:
+    balance_info_pattern: str = '^("?Saldo:)'  # start of balance info
+    ix_start_balance, balance_line = find_line_with_pattern(
+        lines, pattern=balance_info_pattern
+    )
+
+    logger.debug(f"{file_path=} has {ix_start_balance=} and {balance_line=}")
+
+    _lines = lines[ix_start_balance : ix_start_balance + 2]
+
+    total, date = _lines[0], _lines[1]
+
+    date = date.split(";")[1]
+    date = date.strip(";").strip('"')
+
+    date = [int(v) for v in date.split(".")]
+    date = datetime.date(date[2], date[1], date[0])
+
+    total = total.split(";")[1]
+    total = total.strip(";").strip(":").strip('"').split(" ")
+    amount, currency = total[0], total[1]
+    amount = float(amount)
+
+    return BalanceInfo(
+        date=date,
+        amount=amount,
+        currency=currency,
+        provider=case.provider,
+        service=case.service,
+        parser=case.parser,
+        file=str(file_path),
+    )
+
+
+def parse_csv_file(case: Case, file_path: Path) -> tuple[pl.DataFrame, BalanceInfo]:
+    encoding = detect_encoding(file_path)
+    logger.debug(f"{file_path=} has {encoding=}")
+
+    lines = load_lines(file_path, encoding)
+
+    try:
+        transactions = extract_transactions(case, file_path, lines, encoding)
+    except Exception as e:
+        msg = f"failed to parse {case=} transactions: {file_path=}"
+        logger.error(msg)
+        raise ExtractTransactionsException(msg) from e
+
+    try:
+        balance = extract_balance(case, file_path, lines)
+    except Exception as e:
+        msg = f"failed to parse {case=} balance: {file_path=}"
+        logger.error(msg)
+        raise ExtractBalanceException(msg) from e
+
+    return transactions, balance
+
+
+def parse_new_files(
+    case: Case,
+    new_files_to_parse: list[Path],
+    parsed_dir: Path,
+):
+    if len(new_files_to_parse) == 0:
+        logger.info("No new files to parse")
+        return
+
+    if not parsed_dir.exists():
+        logger.info(f"Creating {parsed_dir=}")
+        parsed_dir.mkdir(parents=True, exist_ok=True)
+
+    logger.info(f"Parsing {len(new_files_to_parse):_} new files to {parsed_dir=}")
+
+    for file_path in new_files_to_parse:
+        logger.debug(f"Parsing {file_path=} to {parsed_dir=}")
+        try:
+            transactions, balance = parse_csv_file(case, file_path)
+        except (ExtractBalanceException, ExtractTransactionsException):
+            continue  # already logged in parse_csv_file
+
+        store_transactions(parsed_dir, file_path, transactions)
+        store_balance(parsed_dir, file_path, balance)
+
+    logger.info(f"Finished parsing {len(new_files_to_parse):_d} new files")
+
+
+def main(config: Config):
+    logger.info(f"Processing {CASE=}")
+
+    # scan source files
+    relevant_source_files = get_parser_source_files(
+        CASE, config, check_if_parser_applies
+    )
+
+    # scan target files
+    raw_dir = config.get_raw_dir(CASE)
+    relevant_target_files = detect_relevant_target_files(raw_dir)
+
+    # select new source files to be processed
+    new_files_to_copy = select_files_to_copy(
+        relevant_source_files, relevant_target_files
+    )
+
+    # copy new source files
+    copy_new_files(raw_dir, new_files_to_copy)
+
+    # detect new raw files
+    parsed_dir = config.get_parsed_dir(CASE)
+    new_files_to_parse = detect_new_raw_files(
+        raw_dir, check_if_parser_applies, parsed_dir, CASE.provider, CASE.service
+    )
+
+    # parse new files to parquet -> transactions & balance
+    parse_new_files(CASE, new_files_to_parse, parsed_dir)
+
+    # extend pre-existing parquets for this parser
+    parser_dir = config.get_parser_dir(CASE)
+    concatenate_new_information_to_history(parser_dir, parsed_dir, new_files_to_parse)
+
+    logger.info(f"Done processing {CASE=}")