dpm-synth 1.0.0__py3-none-any.whl

dpm_synth-1.0.0.dist-info/METADATA

@@ -0,0 +1,156 @@
+Metadata-Version: 2.4
+Name: dpm-synth
+Version: 1.0.0
+Summary: Generate synthetic EBA xBRL-CSV reports that satisfy a module's DPM-XL validation rules
+Keywords: DPM,DPM-XL,EBA,XBRL,xBRL-CSV,test data,synthetic data,dpm-toolkit
+Author: Luuk Hopman
+Author-email: Luuk Hopman <github@luukhopman.nl>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Dist: dpm-xl>=1.0,<2
+Requires-Dist: dpm-xbrl>=1.0,<2
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/luukhopman/dpm-synth
+Project-URL: Documentation, https://luukhopman.github.io/dpm-synth/
+Project-URL: Source, https://github.com/luukhopman/dpm-synth
+Project-URL: Issues, https://github.com/luukhopman/dpm-synth/issues
+Project-URL: Changelog, https://github.com/luukhopman/dpm-synth/blob/main/CHANGELOG.md
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <a href="https://luukhopman.github.io/dpm-synth/"><img src="https://luukhopman.github.io/dpm-synth/assets/logo.svg" alt="dpm-synth" width="240"></a>
+</p>
+
+<p align="center"><em>Generate synthetic EBA reports that satisfy DPM-XL validation rules.</em></p>
+
+Generate synthetic EBA xBRL-CSV report packages that satisfy a module's
+DPM-XL validation rules. Give dpm-synth a template with a module's tables,
+cells and rules; it fills every cell with a plausible value, repairs the
+values until the rules pass, and writes a report package that reads and
+validates like a real submission. Use it to test loaders, data warehouses
+and validators for EBA reports, for any module, without real supervisory
+data.
+
+dpm-synth is an independent project, not affiliated with or endorsed by the
+European Banking Authority.
+
+## Installation
+
+```text
+pip install dpm-synth   # Python 3.11+; installs dpm-xl and dpm-xbrl
+```
+
+## Quickstart
+
+```python
+from dpmxbrl import write_xbrl_csv
+
+from dpmsynth import generate, load_template
+
+template = load_template("minimal.json")  # a module's tables, cells and rules
+result = generate(template, seed=7)
+print(result.passed, "of", result.rules, "rules pass; failing:", result.failed)
+write_xbrl_csv(result.report, "report.zip")
+```
+
+```text
+2 of 2 rules pass; failing: []
+```
+
+From the command line, with an exit status of 1 if some rules still fail:
+
+```console
+$ dpm-synth generate minimal.json -o report.zip --seed 7
+rules: 2, passed: 2, failed: 0, not evaluated: 0, not applicable: 0
+rounds: 2 (failing rules per round: 1, 0)
+wrote report.zip (3 facts)
+```
+
+The sample template `minimal.json` is in
+[`docs/samples`](https://github.com/luukhopman/dpm-synth/tree/main/docs/samples);
+templates for real EBA modules are exported from a DPM release by any tool
+that writes the
+[template format](https://luukhopman.github.io/dpm-synth/reference/template-format/).
+
+## Features
+
+- **Any module**: the `dpm-synth-template/1` format describes tables, open
+  keys, cells, data types, allowed items and DPM-XL rules.
+- **Rules satisfied**: an evaluate-and-repair loop over the rules, with
+  [dpm-xl](https://luukhopman.github.io/dpm-xl/), repairs sums,
+  differences, products, sign rules, `isnull`, `in` and `match`. On EBA
+  DPM 4.3 COREP_OF, 999 of the 1,051 rules that apply pass, in 15 rounds;
+  the result names the rest.
+- **Valid packages**: written by
+  [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/), with a fictional LEI,
+  and read back without broken EBA filing rules other than the file name.
+- **Reproducible**: the same template, seed and versions give the same
+  bytes.
+- **Large**: scaled packages stream open tables with millions of rows.
+
+## Toolkit
+
+**Part of the dpm toolkit**, three Python packages that work together:
+
+| Package | What it does |
+| :-- | :-- |
+| <a href="https://luukhopman.github.io/dpm-xl/"><img src="https://luukhopman.github.io/dpm-xl/assets/logo.svg" alt="dpm-xl" height="32"></a> | Parse and evaluate DPM-XL validation rules |
+| <a href="https://luukhopman.github.io/dpm-xbrl/"><img src="https://luukhopman.github.io/dpm-xbrl/assets/logo.svg" alt="dpm-xbrl" height="32"></a> | Read, write and check xBRL-CSV and XBRL-XML reports |
+| <a href="https://luukhopman.github.io/dpm-synth/"><img src="https://luukhopman.github.io/dpm-synth/assets/logo.svg" alt="dpm-synth" height="32"></a> | Generate test reports that satisfy DPM-XL rules **(this package)** |
+
+The [toolkit tour](https://luukhopman.github.io/dpm-xl/guides/toolkit-tour/)
+takes one report through all three: generated with dpm-synth, read with
+dpm-xbrl, then evaluated and explained with dpm-xl.
+
+## Documentation
+
+**Documentation: <https://luukhopman.github.io/dpm-synth/>**
+
+- [Getting started](https://luukhopman.github.io/dpm-synth/getting-started/)
+- [How-to guides](https://luukhopman.github.io/dpm-synth/guides/): write a
+  template, build one from a DPM release, load-test, reproducible
+  seeds, validate the output with dpm-xbrl and dpm-xl
+- [Concepts](https://luukhopman.github.io/dpm-synth/concepts/): the
+  evaluate-and-repair loop and its limits
+- [Reference](https://luukhopman.github.io/dpm-synth/reference/): the
+  [template format](https://luukhopman.github.io/dpm-synth/reference/template-format/),
+  the [CLI](https://luukhopman.github.io/dpm-synth/reference/cli/) and the
+  [Python API](https://luukhopman.github.io/dpm-synth/reference/api/)
+- For AI agents: [llms.txt](https://luukhopman.github.io/dpm-synth/llms.txt)
+  and [llms-full.txt](https://luukhopman.github.io/dpm-synth/llms-full.txt)
+- [Changelog](https://github.com/luukhopman/dpm-synth/blob/main/CHANGELOG.md),
+  [stability policy](https://luukhopman.github.io/dpm-synth/project/stability/)
+  and [security policy](https://luukhopman.github.io/dpm-synth/project/security/)
+
+## Development
+
+```text
+uv sync
+uv run pytest
+uv run ruff check
+uv run ruff format --check
+uv run ty check
+uv run python scripts/build_docs.py
+```
+
+See [contributing](https://luukhopman.github.io/dpm-synth/project/contributing/).
+
+## Licence
+
+MIT. See [LICENSE](https://github.com/luukhopman/dpm-synth/blob/main/LICENSE).

dpm_synth-1.0.0.dist-info/RECORD

@@ -0,0 +1,12 @@
+dpmsynth/__init__.py,sha256=hhwWYEgO_R-DgSWHjVV3ZoYZZfl5INmDX2HBpli9NgE,1244
+dpmsynth/_version.py,sha256=DO2ekebzHMRqi9gwDm8St8crsVeRButMhe_Ft5LEVbs,244
+dpmsynth/cli.py,sha256=uiN5AaNoR-LJAQUAawq0yM_Z0XxNxL3dB0kJfdEQhvY,9421
+dpmsynth/generator.py,sha256=Cus-EKkoltTy8Zfx1MloqGyOaL_4cUZqKr9Whq3e1DQ,43674
+dpmsynth/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dpmsynth/scale.py,sha256=6SwQ2qv7OqnQnBPfFXfkuClkOiJkh3rUQ-c7v6zA29U,6774
+dpmsynth/template.py,sha256=jttrLP2AqnavsUYK0MS7xfoE_iN4tPCfgd1NFVraqVA,17824
+dpm_synth-1.0.0.dist-info/licenses/LICENSE,sha256=I10cuw8RJQykDum4d6-HLudRBtwy3Rcup1-HpxIf2ps,1068
+dpm_synth-1.0.0.dist-info/WHEEL,sha256=si2p8vW03RnhyBnhWl4tj-ObJa78SpE-c9HsEtblGR0,80
+dpm_synth-1.0.0.dist-info/entry_points.txt,sha256=I0M5AkPFkWDTWbuqdx_ichZxWgKM1yXIUkjvXAqb0Sw,49
+dpm_synth-1.0.0.dist-info/METADATA,sha256=CH01AADNVabhYzSllvDAqFa1KPjPGUNp9dr8mkho0JU,6752
+dpm_synth-1.0.0.dist-info/RECORD,,

dpm_synth-1.0.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: uv 0.13.5
+Root-Is-Purelib: true
+Tag: py3-none-any

dpm_synth-1.0.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+dpm-synth = dpmsynth.cli:main
+

dpm_synth-1.0.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Luuk Hopman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dpmsynth/__init__.py

@@ -0,0 +1,58 @@
+"""Generate synthetic EBA xBRL-CSV reports that satisfy DPM-XL validation rules.
+
+Typical use::
+
+    from dpmxbrl import write_xbrl_csv
+
+    from dpmsynth import generate, load_template
+
+    result = generate(load_template("template.json"), seed=7)
+    write_xbrl_csv(result.report, "report.zip")
+    print(result.passed, result.failed)
+
+Everything in ``__all__`` is the public API; import it from ``dpmsynth``.
+"""
+
+from ._version import __version__
+from .generator import (
+    DEFAULT_ENTITY,
+    DEFAULT_REFERENCE_DATE,
+    LEI_PREFIX,
+    MAX_OPEN_ROWS,
+    GenerationResult,
+    SyntheticData,
+    build_report,
+    generate,
+)
+from .scale import write_scaled_package
+from .template import (
+    TEMPLATE_FORMAT,
+    ModuleTemplate,
+    OpenKey,
+    TemplateCell,
+    TemplateError,
+    TemplateRule,
+    TemplateTable,
+    load_template,
+)
+
+__all__ = [
+    "DEFAULT_ENTITY",
+    "DEFAULT_REFERENCE_DATE",
+    "LEI_PREFIX",
+    "MAX_OPEN_ROWS",
+    "TEMPLATE_FORMAT",
+    "GenerationResult",
+    "ModuleTemplate",
+    "OpenKey",
+    "SyntheticData",
+    "TemplateCell",
+    "TemplateError",
+    "TemplateRule",
+    "TemplateTable",
+    "__version__",
+    "build_report",
+    "generate",
+    "load_template",
+    "write_scaled_package",
+]

dpmsynth/_version.py

@@ -0,0 +1,8 @@
+"""The installed distribution version."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("dpm-synth")
+except PackageNotFoundError:  # pragma: no cover - not installed
+    __version__ = "0+unknown"

dpmsynth/cli.py

@@ -0,0 +1,283 @@
+"""Command-line interface: ``dpm-synth generate``.
+
+Exit statuses: 0 when every evaluable rule passes, 1 when some still fail
+(the package is written either way), 2 for a usage error, and 3 when the
+template cannot be read or is invalid, or the package cannot be written.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from collections.abc import Iterator, Sequence
+from datetime import date
+from pathlib import Path
+from typing import Any
+
+from dpmxbrl import write_xbrl_csv
+
+from ._version import __version__
+from .generator import DEFAULT_ENTITY, MAX_OPEN_ROWS, GenerationResult, generate
+from .scale import _MAX_ROWS, write_scaled_package
+from .template import TemplateError, load_template
+
+RULES_FAILED = 1
+"""Exit status when the package is written but some rules still fail."""
+
+INVALID_INPUT = 3
+"""Exit status when the template is unreadable or invalid, or the package
+cannot be written."""
+
+JSON_SCHEMA_VERSION = 1
+"""Version of the ``generate --json`` layout; it changes only with a new major."""
+
+_IDENTIFIER = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*")
+
+_EPILOG = """examples:
+  dpm-synth generate template.json -o report.zip
+  dpm-synth generate template.json -o report.zip --seed 7 --open-rows 3
+  dpm-synth generate template.json -o report.zip --json
+  dpm-synth generate template.json -o big.zip --scale-rows 1000000
+
+exit statuses: 0 every evaluable rule passes, 1 some rules still fail (the
+package is written), 2 usage error, 3 unreadable or invalid template, or the
+package cannot be written.
+
+The template format: https://luukhopman.github.io/dpm-synth/reference/template-format/"""
+
+
+class _Options(argparse.HelpFormatter):
+    """Lists ``-o, --output ZIP`` as Python 3.13 and later do, on every version."""
+
+    def _format_action_invocation(self, action: argparse.Action) -> str:
+        if not action.option_strings or action.nargs == 0:
+            return super()._format_action_invocation(action)
+        value = self._format_args(
+            action, self._get_default_metavar_for_optional(action)
+        )
+        return f"{', '.join(action.option_strings)} {value}"
+
+
+class _Main(_Options, argparse.RawDescriptionHelpFormatter):
+    """Keeps the line breaks of the description and the epilog."""
+
+
+def _whole(minimum: int, maximum: int | None = None) -> Any:
+    def parse(text: str) -> int:
+        try:
+            value = int(text)
+        except ValueError:
+            raise argparse.ArgumentTypeError(
+                f"expected a whole number, got {text!r}"
+            ) from None
+        if value < minimum:
+            raise argparse.ArgumentTypeError(
+                f"expected a whole number of {minimum} or more, got {text!r}"
+            )
+        if maximum is not None and value > maximum:
+            raise argparse.ArgumentTypeError(
+                f"expected a whole number of at most {maximum:_}, got {text!r}"
+            )
+        return value
+
+    parse.__name__ = "number"
+    return parse
+
+
+def _date(text: str) -> date:
+    try:
+        if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", text):
+            raise ValueError(text)  # fromisoformat also takes 20261231
+        return date.fromisoformat(text)
+    except ValueError:
+        raise argparse.ArgumentTypeError(
+            f"expected a date as YYYY-MM-DD, got {text!r}"
+        ) from None
+
+
+def _package(text: str) -> Path:
+    if not text.casefold().endswith(".zip"):
+        raise argparse.ArgumentTypeError(f"expected a .zip file, got {text!r}")
+    return Path(text)
+
+
+def _entity(text: str) -> str:
+    if not _IDENTIFIER.fullmatch(text):
+        raise argparse.ArgumentTypeError(
+            f"expected letters and digits, such as an LEI, got {text!r}"
+        )
+    return text
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="dpm-synth",
+        description="Generate synthetic EBA xBRL-CSV reports that satisfy a module's\n"
+        "DPM-XL validation rules.",
+        epilog=_EPILOG,
+        formatter_class=_Main,
+    )
+    parser.add_argument(
+        "--version", action="version", version=f"%(prog)s {__version__}"
+    )
+    commands = parser.add_subparsers(dest="command", required=True)
+    run = commands.add_parser(
+        "generate",
+        help="write an xBRL-CSV report package for a template",
+        formatter_class=_Options,
+        description="Generate values for every cell of a template, repair them "
+        "until the rules pass, and write an xBRL-CSV report package.",
+    )
+    run.add_argument("template", type=Path, help="dpm-synth-template/1 JSON file")
+    run.add_argument(
+        "-o",
+        "--output",
+        type=_package,
+        required=True,
+        metavar="ZIP",
+        help="report package to write (.zip)",
+    )
+    run.add_argument(
+        "--seed", type=int, default=0, metavar="N", help="random seed (default 0)"
+    )
+    run.add_argument(
+        "--open-rows",
+        type=_whole(1, MAX_OPEN_ROWS),
+        default=2,
+        metavar="N",
+        help=f"key rows per open table, at most {MAX_OPEN_ROWS} (default 2)",
+    )
+    run.add_argument(
+        "--max-rounds",
+        type=_whole(0),
+        default=40,
+        metavar="N",
+        help="most evaluate-and-repair rounds (default 40)",
+    )
+    run.add_argument(
+        "--scale-rows",
+        type=_whole(1, _MAX_ROWS),
+        metavar="N",
+        help="after solving, write every open table with N key rows "
+        "(load testing; rules over open-table totals will fail)",
+    )
+    run.add_argument(
+        "--reference-date",
+        type=_date,
+        metavar="YYYY-MM-DD",
+        help="reference date (default: the template's, else 2026-12-31)",
+    )
+    run.add_argument(
+        "--entity",
+        type=_entity,
+        default=DEFAULT_ENTITY,
+        metavar="LEI",
+        help=f"LEI of the reporting entity (default {DEFAULT_ENTITY})",
+    )
+    run.add_argument(
+        "--level",
+        choices=("CON", "IND"),
+        default="CON",
+        help="reporting level, the entity suffix (default CON)",
+    )
+    run.add_argument("--json", action="store_true", help="print a JSON summary")
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    """Run the command line.
+
+    Args:
+        argv: Arguments without the program name; ``sys.argv[1:]`` when
+            ``None``.
+
+    Returns:
+        int: The exit status: 0 when every evaluable rule passes, 1 when some
+        still fail, 3 when the template or the package cannot be read or
+        written. Usage errors exit with status 2 through ``SystemExit``.
+    """
+    arguments = _parser().parse_args(argv)
+    try:
+        template = load_template(arguments.template)
+    except TemplateError as error:
+        return _fail(error)
+    result = generate(
+        template,
+        seed=arguments.seed,
+        open_rows=arguments.open_rows,
+        max_rounds=arguments.max_rounds,
+        entity=arguments.entity,
+        reporting_level=arguments.level,
+        reference_date=arguments.reference_date,
+    )
+    try:
+        if arguments.scale_rows is None:
+            write_xbrl_csv(result.report, arguments.output)
+            facts = len(result.report.facts)
+        else:
+            written = write_scaled_package(
+                result.data, arguments.output, rows=arguments.scale_rows
+            )
+            facts = sum(written.values())
+    except (OSError, ValueError) as error:  # ReportError is a ValueError
+        return _fail(f"{arguments.output}: cannot write: {_reason(error)}")
+    if arguments.json:
+        print(json.dumps(_summary(result, arguments, facts), indent=2))
+    else:
+        for line in _lines(result):
+            print(line)
+        print(f"wrote {arguments.output} ({facts} facts)")
+    return RULES_FAILED if result.failed else 0
+
+
+def _fail(error: object) -> int:
+    print(f"error: {error}", file=sys.stderr)
+    return INVALID_INPUT
+
+
+def _reason(error: Exception) -> str:
+    if isinstance(error, OSError) and error.strerror:
+        return error.strerror
+    return str(error)
+
+
+def _summary(
+    result: GenerationResult, arguments: argparse.Namespace, facts: int
+) -> dict[str, Any]:
+    template = result.data.template
+    return {
+        "schema_version": JSON_SCHEMA_VERSION,
+        "output": str(arguments.output),
+        "release": template.release,
+        "module": template.module,
+        "seed": arguments.seed,
+        "rules": result.rules,
+        "passed": result.passed,
+        "failed": result.failed,
+        "not_evaluated": result.not_evaluated,
+        "not_applicable": result.not_applicable,
+        "rounds": result.rounds,
+        "history": result.history,
+        "repairs": result.repairs,
+        "facts": facts,
+    }
+
+
+def _lines(result: GenerationResult) -> Iterator[str]:
+    yield (
+        f"rules: {result.rules}, passed: {result.passed}, "
+        f"failed: {len(result.failed)}, not evaluated: {len(result.not_evaluated)}, "
+        f"not applicable: {len(result.not_applicable)}"
+    )
+    history = ", ".join(str(count) for count in result.history) or "-"
+    yield f"rounds: {result.rounds} (failing rules per round: {history})"
+    if result.failed:
+        shown = ", ".join(result.failed[:20])
+        more = len(result.failed) - 20
+        yield f"still failing: {shown}" + (f" and {more} more" if more > 0 else "")
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())