dpm-xbrl 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dpm_xbrl-0.1.0/LICENSE +21 -0
- dpm_xbrl-0.1.0/PKG-INFO +305 -0
- dpm_xbrl-0.1.0/README.md +276 -0
- dpm_xbrl-0.1.0/pyproject.toml +141 -0
- dpm_xbrl-0.1.0/pyproject.toml.orig +122 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/__init__.py +34 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/_package.py +282 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/_taxonomy.py +97 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/_util.py +86 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/_xmlsafe.py +112 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/cli.py +186 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/csv_reader.py +1058 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/filing.py +495 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/frames.py +388 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/model.py +153 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/py.typed +0 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/reader.py +86 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/writer.py +441 -0
- dpm_xbrl-0.1.0/src/dpmxbrl/xml_reader.py +1047 -0
dpm_xbrl-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Luuk Hopman
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
dpm_xbrl-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,305 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dpm-xbrl
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Read and write EBA-style xBRL-CSV and XBRL-XML regulatory reports (DPM taxonomies)
|
|
5
|
+
Keywords: EBA,XBRL,xBRL-CSV,regulatory reporting,report package
|
|
6
|
+
Author: Luuk Hopman
|
|
7
|
+
Author-email: Luuk Hopman <github@luukhopman.nl>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Topic :: Office/Business :: Financial
|
|
19
|
+
Classifier: Typing :: Typed
|
|
20
|
+
Requires-Dist: isal>=1.8,<2 ; extra == 'fast'
|
|
21
|
+
Requires-Dist: polars>=2.0,<3 ; extra == 'frames'
|
|
22
|
+
Requires-Python: >=3.11
|
|
23
|
+
Project-URL: Homepage, https://github.com/luukhopman/dpm-xbrl
|
|
24
|
+
Project-URL: Source, https://github.com/luukhopman/dpm-xbrl
|
|
25
|
+
Project-URL: Issues, https://github.com/luukhopman/dpm-xbrl/issues
|
|
26
|
+
Provides-Extra: fast
|
|
27
|
+
Provides-Extra: frames
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# dpm-xbrl
|
|
31
|
+
|
|
32
|
+
Read and write EBA-style regulatory XBRL reports in both serializations, xBRL-CSV
|
|
33
|
+
report packages and XBRL 2.1 XML instances, through one neutral fact model.
|
|
34
|
+
It targets reports built on the data point model (DPM) taxonomies of the
|
|
35
|
+
European Banking Authority and is tested on EBA reports. It is an independent
|
|
36
|
+
project, not affiliated with or endorsed by the EBA.
|
|
37
|
+
The package uses only the standard library and needs Python 3.11 or newer.
|
|
38
|
+
The optional `fast` extra adds `isal`, which inflates ZIP entries about twice
|
|
39
|
+
as fast as zlib; the bytes read are the same either way.
|
|
40
|
+
|
|
41
|
+
```python
|
|
42
|
+
from dpmxbrl import read_report
|
|
43
|
+
|
|
44
|
+
report = read_report("report.zip") # or a directory, report.json, .xbrl, or bytes
|
|
45
|
+
print(report.entity_id, report.reference_date, len(report.facts))
|
|
46
|
+
for fact in report.facts_for_template("C_01.00"):
|
|
47
|
+
print(fact.datapoint, fact.keys, fact.value)
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Model
|
|
51
|
+
|
|
52
|
+
`dpmxbrl.model` (re-exported from `dpmxbrl`) defines two frozen, slotted
|
|
53
|
+
dataclasses:
|
|
54
|
+
|
|
55
|
+
- `Fact`: `value` (lexical value as written, `None` when nil), `concept`
|
|
56
|
+
(prefixed qname), `datapoint` (xBRL-CSV datapoint code), `template`
|
|
57
|
+
(xBRL-CSV table template id, `None` for XML), `dimensions` (explicit dimension
|
|
58
|
+
qname to member qname), `typed_dimensions` (typed dimension qname to the typed
|
|
59
|
+
value text), `keys` (xBRL-CSV non-fact column name to cell value, in header
|
|
60
|
+
order), `unit` (`iso4217:EUR`, `xbrli:pure`, `num/denom` for divide units,
|
|
61
|
+
`a*b` for products), `decimals` (`None` means INF or unspecified) and
|
|
62
|
+
`location` (`reports/c_01.00.csv:3` with the 1-based physical line, or
|
|
63
|
+
`instance.xbrl#f12` with the fact id, `#factN` for the Nth fact without id).
|
|
64
|
+
- `Report`: `format`, `entry_point`, `entity_scheme`, `entity_id`,
|
|
65
|
+
`reference_date`, `base_currency`, `parameters`, `decimals_parameters`,
|
|
66
|
+
`filing_indicators`, `facts`, `namespaces`, `warnings` and `filing` (the
|
|
67
|
+
EBA filing rules the report breaks, see below).
|
|
68
|
+
|
|
69
|
+
`Report.facts_for_template(code)` compares template codes case-insensitively,
|
|
70
|
+
so filing-indicator codes such as `C_01.00` select facts from `c_01.00.csv`.
|
|
71
|
+
`local_name(qname)` returns the local part of a prefixed or Clark qname.
|
|
72
|
+
|
|
73
|
+
Qnames always use prefixes declared in `Report.namespaces`. XML keeps the
|
|
74
|
+
instance's own prefixes; a namespace declared again deeper in the document
|
|
75
|
+
under another prefix is rendered with the first prefix bound to it.
|
|
76
|
+
|
|
77
|
+
## Reading
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
read_report(source, *, name=None, metadata_resolver=None,
|
|
81
|
+
taxonomy_packages=(), limits=Limits()) -> Report
|
|
82
|
+
read_xbrl_csv(source, *, name=None, metadata_resolver=None,
|
|
83
|
+
taxonomy_packages=(), limits=Limits()) -> Report
|
|
84
|
+
read_xbrl_xml(source, *, name=None, limits=Limits()) -> Report
|
|
85
|
+
detect_format(source) -> "xbrl-csv" | "xbrl-xml"
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
`read_report` treats ZIP input (bytes or a path), a directory and a `.json`
|
|
89
|
+
path as xBRL-CSV and everything else as XBRL-XML. Every failure raises
|
|
90
|
+
`ReportError` (a `ValueError`) naming the file and, where known, the line.
|
|
91
|
+
|
|
92
|
+
### xBRL-CSV
|
|
93
|
+
|
|
94
|
+
- A ZIP or directory with exactly one `reports/*.json` report (normally
|
|
95
|
+
`reports/report.json`), optionally below one top-level folder, plus an
|
|
96
|
+
optional `META-INF/reportPackage.json` whose `documentType` must be
|
|
97
|
+
`https://xbrl.org/report-package/2023` or `.../2023/xbrl-csv`. A directory
|
|
98
|
+
may also hold `report.json` directly.
|
|
99
|
+
- `documentInfo.documentType` must be `https://xbrl.org/2021/xbrl-csv`; the
|
|
100
|
+
2019-2021 PWD/CR/PR draft URIs are accepted with a warning. Duplicate JSON
|
|
101
|
+
keys are rejected.
|
|
102
|
+
- `parameters.csv` (or the metadata's `parameterURL`) must have the header
|
|
103
|
+
`name,value`. `entityID` such as `lei:5299...` becomes `entity_id`; its
|
|
104
|
+
prefix is looked up in `namespaces` for `entity_scheme`, else kept as the
|
|
105
|
+
scheme. `refPeriod` must be a `YYYY-MM-DD` date. Parameters named
|
|
106
|
+
`decimals*` populate `decimals_parameters` (`#none` and `INF` map to `None`).
|
|
107
|
+
Inline JSON `parameters` are merged first; the CSV file overrides them.
|
|
108
|
+
- `FilingIndicators.csv` must have the header `templateID,reported`, with
|
|
109
|
+
`reported` one of `true`, `false`, `1`, `0`.
|
|
110
|
+
- Every other `*.csv` under the reports folder is a table. Its template id is
|
|
111
|
+
the file stem unless the metadata `tables` maps the file URL to a template.
|
|
112
|
+
|
|
113
|
+
**EBA convention (no metadata).** Tables need `datapoint` and `factValue`
|
|
114
|
+
columns; every other column is a key column kept in `Fact.keys`. Each row with
|
|
115
|
+
a non-empty `factValue` is one fact. `#nil` gives a nil fact, `#empty` and
|
|
116
|
+
`#none` are skipped, `##` escapes a leading `#`, and other `#` values are kept
|
|
117
|
+
as written; each case adds a warning. `Fact.decimals` stays `None`; consumers
|
|
118
|
+
pick the `decimals_parameters` entry for the fact's data type. Exact duplicate
|
|
119
|
+
rows are reported once; conflicting duplicates (same template, datapoint and
|
|
120
|
+
keys, different value) are all kept. Repeated findings are aggregated into one
|
|
121
|
+
warning with a count and the first location.
|
|
122
|
+
|
|
123
|
+
**Resolved metadata.** `report.json` and its `extends` chain are followed
|
|
124
|
+
recursively (cycle-safe, diamond-safe, depth-limited), merging `namespaces`,
|
|
125
|
+
`tableTemplates`, `tables`, `parameters`, `dimensions`, `decimals` and
|
|
126
|
+
`parameterURL`, with the extending document taking precedence. Relative URLs
|
|
127
|
+
resolve inside the package or against the extending document's URL. Absolute
|
|
128
|
+
URLs are resolved by `metadata_resolver(url)` first and then through the
|
|
129
|
+
`META-INF/catalog.xml` `rewriteURI`/`rewriteSystem` prefixes of the given
|
|
130
|
+
taxonomy package ZIPs. Nothing is fetched over the network. Table `url`s and
|
|
131
|
+
`parameterURL` resolve against the report's own JSON.
|
|
132
|
+
|
|
133
|
+
For a table whose template is resolved, the supported xBRL-CSV subset is:
|
|
134
|
+
template `columns`; fact columns (with `dimensions` or `propertiesFrom`);
|
|
135
|
+
property-group columns (`propertyGroups`, such as EBA's `datapoint` column keyed
|
|
136
|
+
by `dp...`); other columns as key columns; report, template, column and
|
|
137
|
+
property-group `dimensions` and `decimals`, in increasing precedence;
|
|
138
|
+
`$column` and `$parameter` references (row cell, then table parameters, then
|
|
139
|
+
report parameters), `$$` escapes and `#none`; core dimensions `concept`,
|
|
140
|
+
`unit`, `period`, `entity` and `language`; and taxonomy-defined dimensions. A
|
|
141
|
+
taxonomy-defined dimension value that is a qname with a declared prefix is an
|
|
142
|
+
explicit member; any other value is a typed value. `Fact.datapoint` is the
|
|
143
|
+
property-group key and `Fact.keys` keeps the key columns. A table whose
|
|
144
|
+
template cannot be resolved falls back to the EBA convention, and unresolved
|
|
145
|
+
`extends` produce one warning.
|
|
146
|
+
|
|
147
|
+
### XBRL-XML
|
|
148
|
+
|
|
149
|
+
- The prolog is scanned before parsing and any DOCTYPE (and with it any ENTITY
|
|
150
|
+
declaration) is rejected; external entities are never resolved. Parsing
|
|
151
|
+
streams with `iterparse` and clears each top-level element once processed.
|
|
152
|
+
- `link:schemaRef` gives `entry_point`. Contexts provide the entity identifier
|
|
153
|
+
and scheme, the reference date (instant, or the end date of a duration) and
|
|
154
|
+
`xbrldi:explicitMember`/`xbrldi:typedMember` from segment or scenario. A typed
|
|
155
|
+
member's value is the stripped text of its single child element; the child
|
|
156
|
+
element name is not kept.
|
|
157
|
+
- Units with one or more measures, or a divide, become strings. Facts are
|
|
158
|
+
top-level elements outside the `xbrli`, `link` and `find` namespaces with a
|
|
159
|
+
`contextRef`; `xsi:nil="true"` gives `value=None`; `decimals="INF"` gives
|
|
160
|
+
`None`. `precision` is ignored, and tuples and elements without `contextRef`
|
|
161
|
+
are skipped, each with a warning.
|
|
162
|
+
- `find:fIndicators/find:filingIndicator` provide filing indicators;
|
|
163
|
+
`find:filed` defaults to true.
|
|
164
|
+
- The report entity and reference date are the most common across facts, and
|
|
165
|
+
`base_currency` is the most common single-measure `iso4217` unit. More than
|
|
166
|
+
one entity or period adds a warning, because EBA requires one.
|
|
167
|
+
|
|
168
|
+
### Limits
|
|
169
|
+
|
|
170
|
+
`Limits(max_zip_entries=10_000, max_total_size=2 GiB, max_file_size=512 MiB,
|
|
171
|
+
max_facts=20_000_000, max_extends_depth=32)`. Report packages are also rejected
|
|
172
|
+
for absolute or `..` entry names, duplicate entries and encrypted entries, and
|
|
173
|
+
package directories for links that leave the directory. The file-size limit
|
|
174
|
+
also applies to XML instances and to files read from taxonomy packages. CSV
|
|
175
|
+
rows are streamed with the `csv` module (`utf-8-sig`), including from ZIPs.
|
|
176
|
+
An entry whose compressed data is damaged raises `ReportError` naming the entry.
|
|
177
|
+
|
|
178
|
+
### EBA filing rules
|
|
179
|
+
|
|
180
|
+
Both readers record the [EBA Filing Rules](https://www.eba.europa.eu/sites/default/files/2026-02/10436139-46ae-4a7d-880f-c399ba6de669/EBA%20Filing%20Rules%20v5.8_2026_02_25.pdf)
|
|
181
|
+
(version 5.8, February 2026) that a report breaks and that the file alone can show, in
|
|
182
|
+
`Report.filing`: one `FilingFinding` per rule and problem, with the rule's
|
|
183
|
+
number (`2.19`), its tag in the EBA document (`nilUsed`), a severity, a
|
|
184
|
+
message, how many times it occurs and up to five locations.
|
|
185
|
+
`dpmxbrl.filing.RULE_TITLES` names each rule. Severity is `error` where the
|
|
186
|
+
rule says MUST (EBA rejects the report) and `warning` where it says SHOULD;
|
|
187
|
+
file-naming findings are warnings because each supervisor confirms its own
|
|
188
|
+
naming.
|
|
189
|
+
|
|
190
|
+
- Both formats: one absolute taxonomy entry point (1.5, 2.2, 2.3); the
|
|
191
|
+
reporting subject's scheme, suffix, upper case and LEI check digits (2.8,
|
|
192
|
+
3.6); and the file name against `ReportSubject_Country_
|
|
193
|
+
FrameworkCodeModuleVersion_Module_ReferenceDate_CreationTimestamp`,
|
|
194
|
+
including the reporting subject, module and reference date.
|
|
195
|
+
- XBRL-XML: UTF-8 and no standalone declaration (1.4, 1.13); filing
|
|
196
|
+
indicators present, unique and in a context without segment or scenario
|
|
197
|
+
(1.6, 1.6.1); no `xsi:schemaLocation`, XInclude, `xml:base`,
|
|
198
|
+
`link:linkbaseRef`, footnotes, segments, forever or duration periods, or
|
|
199
|
+
non-dimensional scenario content (1.14, 1.15, 2.1, 2.4, 2.25, 2.11, 2.13, 2.14,
|
|
200
|
+
2.15); one entity and reference date, as plain dates (2.9, 2.10, 2.13); no
|
|
201
|
+
duplicate facts, and none differing only by unit (2.16, 2.16.1); `decimals`
|
|
202
|
+
rather than `precision`, and present on numeric facts (2.17, 2.18); no nil or
|
|
203
|
+
empty facts (2.19); unused or repeated contexts and units (2.7, 2.21, 2.22);
|
|
204
|
+
one reporting currency, and currency-of-denomination facts in the currency
|
|
205
|
+
their CUS or CUA member names (3.1); `xbrli:pure` for other numeric facts
|
|
206
|
+
(3.2); and namespace prefixes declared once, on the document element, and
|
|
207
|
+
used (3.4, 3.9, 3.10).
|
|
208
|
+
- xBRL-CSV: `META-INF/reportPackage.json`, `parameters.csv` with
|
|
209
|
+
`refPeriod`, and `FilingIndicators.csv`; the root folder named as the ZIP;
|
|
210
|
+
no `documentInfo.taxonomy` (1.5); a plain-date `refPeriod` (2.10); unique
|
|
211
|
+
filing indicators (1.6.1); no table file for a template indicated as not
|
|
212
|
+
reported (xBRL-CSV rule 6); no special values in fact or key columns
|
|
213
|
+
(`#nil`, `#empty` and any other value starting with `#`; 2.19 and rule 5);
|
|
214
|
+
no decimals suffix (rule 1); key values without leading or trailing spaces
|
|
215
|
+
(3.11).
|
|
216
|
+
|
|
217
|
+
Rules that need the taxonomy are left to the caller, which can build
|
|
218
|
+
`FilingFinding`s and merge them with `dpmxbrl.filing.FilingLog`: a filing
|
|
219
|
+
indicator for every template of the module and only valid codes (1.6, 1.6.3),
|
|
220
|
+
facts outside the reported templates (1.7.1), minimum decimals per data type
|
|
221
|
+
(2.18), fact values without leading or trailing spaces, which applies to
|
|
222
|
+
string data points only (3.11, `SPACES` in `dpmxbrl.filing`), an entry
|
|
223
|
+
point valid for the reference date (1.5), and `baseCurrency` when monetary
|
|
224
|
+
facts use it. Not checked: units against the XBRL unit registry (2.23),
|
|
225
|
+
decimal notation for rates (3.2), canonical prefixes (3.5),
|
|
226
|
+
generating-software information (2.26) and comments (2.5).
|
|
227
|
+
`read_xbrl_csv_frames` checks special fact values only; the decimals suffix,
|
|
228
|
+
spaces around key values and special key values are left to callers that
|
|
229
|
+
split values by data type, which can record them with the same messages
|
|
230
|
+
(`DECIMALS_SUFFIX`, `KEY_SPACES`, `SPECIAL_KEY_VALUE` in `dpmxbrl.filing`).
|
|
231
|
+
|
|
232
|
+
## Writing
|
|
233
|
+
|
|
234
|
+
```python
|
|
235
|
+
write_xbrl_csv(report, target, *, root=None) -> None
|
|
236
|
+
write_xbrl_xml(report, target, *, typed_domains=None) -> None
|
|
237
|
+
default_package_root(report) -> str
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
`write_xbrl_csv` writes a ZIP report package in EBA convention:
|
|
241
|
+
`<root>/META-INF/reportPackage.json`, `<root>/reports/report.json` (extending
|
|
242
|
+
`entry_point` and declaring `namespaces`), `parameters.csv` (`entityID`,
|
|
243
|
+
`refPeriod`, `baseCurrency`, `decimals*`, then any other parameter),
|
|
244
|
+
`FilingIndicators.csv` and one `<template>.csv` per template with header
|
|
245
|
+
`datapoint,factValue,<key columns in first-seen order>`. Nil values are
|
|
246
|
+
written as `#nil`, empty strings as `#empty` and a leading `#` as `##`. The root
|
|
247
|
+
defaults to `<entity id>_<reference date>`. Output is deterministic (sorted
|
|
248
|
+
templates, source fact order, fixed ZIP timestamps). Facts without a template
|
|
249
|
+
or datapoint raise `ReportError` before anything is written.
|
|
250
|
+
|
|
251
|
+
`write_xbrl_xml` writes `schemaRef`, filing indicators, contexts deduplicated
|
|
252
|
+
by dimensions (`c1`, `c2`, ..., entity plus an instant at `reference_date`,
|
|
253
|
+
dimensions in the scenario), units (`u1`, ...) and facts. A fact with a unit
|
|
254
|
+
is numeric and gets `decimals` (`INF` when unset) unless it is nil. Facts
|
|
255
|
+
without a concept, undeclared prefixes and a missing entity or reference date
|
|
256
|
+
raise `ReportError`. Because the reader does not keep typed-domain element
|
|
257
|
+
names, typed dimensions need `typed_domains={"eba_dim:qINC": "eba_typ:qCC"}`.
|
|
258
|
+
|
|
259
|
+
Reading a written report reproduces its facts in both formats; the tests
|
|
260
|
+
cover these round trips.
|
|
261
|
+
|
|
262
|
+
## CLI
|
|
263
|
+
|
|
264
|
+
```
|
|
265
|
+
dpm-xbrl inspect PATH [--json] [--taxonomy-package ZIP ...]
|
|
266
|
+
dpm-xbrl convert IN OUT [--taxonomy-package ZIP ...]
|
|
267
|
+
[--typed-domain DIMENSION=ELEMENT ...] [--root NAME]
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
`inspect` prints the format, entry point, entity, reference date, base
|
|
271
|
+
currency, filing indicators (reported of total), the fact count per template,
|
|
272
|
+
the warnings and the EBA filing rules the report breaks. `convert` chooses the output format from the suffix: `.zip`
|
|
273
|
+
writes xBRL-CSV, `.xbrl` or `.xml` writes XBRL-XML. Errors print `error: ...`
|
|
274
|
+
and exit with status 1.
|
|
275
|
+
|
|
276
|
+
## Development
|
|
277
|
+
|
|
278
|
+
```
|
|
279
|
+
uv sync
|
|
280
|
+
uv run pytest
|
|
281
|
+
uv run ruff check
|
|
282
|
+
uv run ruff format --check
|
|
283
|
+
uv run ty check
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
## Known gaps
|
|
287
|
+
|
|
288
|
+
- No taxonomy validation, schema lookups or calculation and formula checks.
|
|
289
|
+
Without a taxonomy, an explicit member and a typed value that looks like a
|
|
290
|
+
declared qname cannot be told apart in xBRL-CSV metadata.
|
|
291
|
+
- xBRL-CSV: no `rowIdColumn`, `links`, footnotes, `@start`/`@end` period
|
|
292
|
+
semantics, per-fact `period`/`entity`/`language` values (they are ignored),
|
|
293
|
+
or multiple reports per package. Taxonomy packages must be ZIPs.
|
|
294
|
+
- EBA-convention CSV facts carry no concept, dimensions, unit or decimals, so
|
|
295
|
+
they cannot be written as XML; XML facts carry no datapoint or template, so
|
|
296
|
+
they cannot be written as CSV. Converting between formats needs resolved
|
|
297
|
+
xBRL-CSV metadata.
|
|
298
|
+
- Empty-string facts do not survive an xBRL-CSV round trip (`#empty` is
|
|
299
|
+
skipped when read), and `Fact.decimals`, concepts and dimensions are not
|
|
300
|
+
written to EBA-convention CSV.
|
|
301
|
+
- XML: tuples, footnotes, `precision`, segment versus scenario placement,
|
|
302
|
+
forever periods and typed-domain element names are not kept. XML output puts
|
|
303
|
+
all dimensions in the scenario and uses one entity and instant. The XML
|
|
304
|
+
writer holds the instance text in memory before writing it.
|
|
305
|
+
- Fact locations in XML use the fact id rather than a line number.
|
dpm_xbrl-0.1.0/README.md
ADDED
|
@@ -0,0 +1,276 @@
|
|
|
1
|
+
# dpm-xbrl
|
|
2
|
+
|
|
3
|
+
Read and write EBA-style regulatory XBRL reports in both serializations, xBRL-CSV
|
|
4
|
+
report packages and XBRL 2.1 XML instances, through one neutral fact model.
|
|
5
|
+
It targets reports built on the data point model (DPM) taxonomies of the
|
|
6
|
+
European Banking Authority and is tested on EBA reports. It is an independent
|
|
7
|
+
project, not affiliated with or endorsed by the EBA.
|
|
8
|
+
The package uses only the standard library and needs Python 3.11 or newer.
|
|
9
|
+
The optional `fast` extra adds `isal`, which inflates ZIP entries about twice
|
|
10
|
+
as fast as zlib; the bytes read are the same either way.
|
|
11
|
+
|
|
12
|
+
```python
|
|
13
|
+
from dpmxbrl import read_report
|
|
14
|
+
|
|
15
|
+
report = read_report("report.zip") # or a directory, report.json, .xbrl, or bytes
|
|
16
|
+
print(report.entity_id, report.reference_date, len(report.facts))
|
|
17
|
+
for fact in report.facts_for_template("C_01.00"):
|
|
18
|
+
print(fact.datapoint, fact.keys, fact.value)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Model
|
|
22
|
+
|
|
23
|
+
`dpmxbrl.model` (re-exported from `dpmxbrl`) defines two frozen, slotted
|
|
24
|
+
dataclasses:
|
|
25
|
+
|
|
26
|
+
- `Fact`: `value` (lexical value as written, `None` when nil), `concept`
|
|
27
|
+
(prefixed qname), `datapoint` (xBRL-CSV datapoint code), `template`
|
|
28
|
+
(xBRL-CSV table template id, `None` for XML), `dimensions` (explicit dimension
|
|
29
|
+
qname to member qname), `typed_dimensions` (typed dimension qname to the typed
|
|
30
|
+
value text), `keys` (xBRL-CSV non-fact column name to cell value, in header
|
|
31
|
+
order), `unit` (`iso4217:EUR`, `xbrli:pure`, `num/denom` for divide units,
|
|
32
|
+
`a*b` for products), `decimals` (`None` means INF or unspecified) and
|
|
33
|
+
`location` (`reports/c_01.00.csv:3` with the 1-based physical line, or
|
|
34
|
+
`instance.xbrl#f12` with the fact id, `#factN` for the Nth fact without id).
|
|
35
|
+
- `Report`: `format`, `entry_point`, `entity_scheme`, `entity_id`,
|
|
36
|
+
`reference_date`, `base_currency`, `parameters`, `decimals_parameters`,
|
|
37
|
+
`filing_indicators`, `facts`, `namespaces`, `warnings` and `filing` (the
|
|
38
|
+
EBA filing rules the report breaks, see below).
|
|
39
|
+
|
|
40
|
+
`Report.facts_for_template(code)` compares template codes case-insensitively,
|
|
41
|
+
so filing-indicator codes such as `C_01.00` select facts from `c_01.00.csv`.
|
|
42
|
+
`local_name(qname)` returns the local part of a prefixed or Clark qname.
|
|
43
|
+
|
|
44
|
+
Qnames always use prefixes declared in `Report.namespaces`. XML keeps the
|
|
45
|
+
instance's own prefixes; a namespace declared again deeper in the document
|
|
46
|
+
under another prefix is rendered with the first prefix bound to it.
|
|
47
|
+
|
|
48
|
+
## Reading
|
|
49
|
+
|
|
50
|
+
```python
|
|
51
|
+
read_report(source, *, name=None, metadata_resolver=None,
|
|
52
|
+
taxonomy_packages=(), limits=Limits()) -> Report
|
|
53
|
+
read_xbrl_csv(source, *, name=None, metadata_resolver=None,
|
|
54
|
+
taxonomy_packages=(), limits=Limits()) -> Report
|
|
55
|
+
read_xbrl_xml(source, *, name=None, limits=Limits()) -> Report
|
|
56
|
+
detect_format(source) -> "xbrl-csv" | "xbrl-xml"
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
`read_report` treats ZIP input (bytes or a path), a directory and a `.json`
|
|
60
|
+
path as xBRL-CSV and everything else as XBRL-XML. Every failure raises
|
|
61
|
+
`ReportError` (a `ValueError`) naming the file and, where known, the line.
|
|
62
|
+
|
|
63
|
+
### xBRL-CSV
|
|
64
|
+
|
|
65
|
+
- A ZIP or directory with exactly one `reports/*.json` report (normally
|
|
66
|
+
`reports/report.json`), optionally below one top-level folder, plus an
|
|
67
|
+
optional `META-INF/reportPackage.json` whose `documentType` must be
|
|
68
|
+
`https://xbrl.org/report-package/2023` or `.../2023/xbrl-csv`. A directory
|
|
69
|
+
may also hold `report.json` directly.
|
|
70
|
+
- `documentInfo.documentType` must be `https://xbrl.org/2021/xbrl-csv`; the
|
|
71
|
+
2019-2021 PWD/CR/PR draft URIs are accepted with a warning. Duplicate JSON
|
|
72
|
+
keys are rejected.
|
|
73
|
+
- `parameters.csv` (or the metadata's `parameterURL`) must have the header
|
|
74
|
+
`name,value`. `entityID` such as `lei:5299...` becomes `entity_id`; its
|
|
75
|
+
prefix is looked up in `namespaces` for `entity_scheme`, else kept as the
|
|
76
|
+
scheme. `refPeriod` must be a `YYYY-MM-DD` date. Parameters named
|
|
77
|
+
`decimals*` populate `decimals_parameters` (`#none` and `INF` map to `None`).
|
|
78
|
+
Inline JSON `parameters` are merged first; the CSV file overrides them.
|
|
79
|
+
- `FilingIndicators.csv` must have the header `templateID,reported`, with
|
|
80
|
+
`reported` one of `true`, `false`, `1`, `0`.
|
|
81
|
+
- Every other `*.csv` under the reports folder is a table. Its template id is
|
|
82
|
+
the file stem unless the metadata `tables` maps the file URL to a template.
|
|
83
|
+
|
|
84
|
+
**EBA convention (no metadata).** Tables need `datapoint` and `factValue`
|
|
85
|
+
columns; every other column is a key column kept in `Fact.keys`. Each row with
|
|
86
|
+
a non-empty `factValue` is one fact. `#nil` gives a nil fact, `#empty` and
|
|
87
|
+
`#none` are skipped, `##` escapes a leading `#`, and other `#` values are kept
|
|
88
|
+
as written; each case adds a warning. `Fact.decimals` stays `None`; consumers
|
|
89
|
+
pick the `decimals_parameters` entry for the fact's data type. Exact duplicate
|
|
90
|
+
rows are reported once; conflicting duplicates (same template, datapoint and
|
|
91
|
+
keys, different value) are all kept. Repeated findings are aggregated into one
|
|
92
|
+
warning with a count and the first location.
|
|
93
|
+
|
|
94
|
+
**Resolved metadata.** `report.json` and its `extends` chain are followed
|
|
95
|
+
recursively (cycle-safe, diamond-safe, depth-limited), merging `namespaces`,
|
|
96
|
+
`tableTemplates`, `tables`, `parameters`, `dimensions`, `decimals` and
|
|
97
|
+
`parameterURL`, with the extending document taking precedence. Relative URLs
|
|
98
|
+
resolve inside the package or against the extending document's URL. Absolute
|
|
99
|
+
URLs are resolved by `metadata_resolver(url)` first and then through the
|
|
100
|
+
`META-INF/catalog.xml` `rewriteURI`/`rewriteSystem` prefixes of the given
|
|
101
|
+
taxonomy package ZIPs. Nothing is fetched over the network. Table `url`s and
|
|
102
|
+
`parameterURL` resolve against the report's own JSON.
|
|
103
|
+
|
|
104
|
+
For a table whose template is resolved, the supported xBRL-CSV subset is:
|
|
105
|
+
template `columns`; fact columns (with `dimensions` or `propertiesFrom`);
|
|
106
|
+
property-group columns (`propertyGroups`, such as EBA's `datapoint` column keyed
|
|
107
|
+
by `dp...`); other columns as key columns; report, template, column and
|
|
108
|
+
property-group `dimensions` and `decimals`, in increasing precedence;
|
|
109
|
+
`$column` and `$parameter` references (row cell, then table parameters, then
|
|
110
|
+
report parameters), `$$` escapes and `#none`; core dimensions `concept`,
|
|
111
|
+
`unit`, `period`, `entity` and `language`; and taxonomy-defined dimensions. A
|
|
112
|
+
taxonomy-defined dimension value that is a qname with a declared prefix is an
|
|
113
|
+
explicit member; any other value is a typed value. `Fact.datapoint` is the
|
|
114
|
+
property-group key and `Fact.keys` keeps the key columns. A table whose
|
|
115
|
+
template cannot be resolved falls back to the EBA convention, and unresolved
|
|
116
|
+
`extends` produce one warning.
|
|
117
|
+
|
|
118
|
+
### XBRL-XML
|
|
119
|
+
|
|
120
|
+
- The prolog is scanned before parsing and any DOCTYPE (and with it any ENTITY
|
|
121
|
+
declaration) is rejected; external entities are never resolved. Parsing
|
|
122
|
+
streams with `iterparse` and clears each top-level element once processed.
|
|
123
|
+
- `link:schemaRef` gives `entry_point`. Contexts provide the entity identifier
|
|
124
|
+
and scheme, the reference date (instant, or the end date of a duration) and
|
|
125
|
+
`xbrldi:explicitMember`/`xbrldi:typedMember` from segment or scenario. A typed
|
|
126
|
+
member's value is the stripped text of its single child element; the child
|
|
127
|
+
element name is not kept.
|
|
128
|
+
- Units with one or more measures, or a divide, become strings. Facts are
|
|
129
|
+
top-level elements outside the `xbrli`, `link` and `find` namespaces with a
|
|
130
|
+
`contextRef`; `xsi:nil="true"` gives `value=None`; `decimals="INF"` gives
|
|
131
|
+
`None`. `precision` is ignored, and tuples and elements without `contextRef`
|
|
132
|
+
are skipped, each with a warning.
|
|
133
|
+
- `find:fIndicators/find:filingIndicator` provide filing indicators;
|
|
134
|
+
`find:filed` defaults to true.
|
|
135
|
+
- The report entity and reference date are the most common across facts, and
|
|
136
|
+
`base_currency` is the most common single-measure `iso4217` unit. More than
|
|
137
|
+
one entity or period adds a warning, because EBA requires one.
|
|
138
|
+
|
|
139
|
+
### Limits
|
|
140
|
+
|
|
141
|
+
`Limits(max_zip_entries=10_000, max_total_size=2 GiB, max_file_size=512 MiB,
|
|
142
|
+
max_facts=20_000_000, max_extends_depth=32)`. Report packages are also rejected
|
|
143
|
+
for absolute or `..` entry names, duplicate entries and encrypted entries, and
|
|
144
|
+
package directories for links that leave the directory. The file-size limit
|
|
145
|
+
also applies to XML instances and to files read from taxonomy packages. CSV
|
|
146
|
+
rows are streamed with the `csv` module (`utf-8-sig`), including from ZIPs.
|
|
147
|
+
An entry whose compressed data is damaged raises `ReportError` naming the entry.
|
|
148
|
+
|
|
149
|
+
### EBA filing rules
|
|
150
|
+
|
|
151
|
+
Both readers record the [EBA Filing Rules](https://www.eba.europa.eu/sites/default/files/2026-02/10436139-46ae-4a7d-880f-c399ba6de669/EBA%20Filing%20Rules%20v5.8_2026_02_25.pdf)
|
|
152
|
+
(version 5.8, February 2026) that a report breaks and that the file alone can show, in
|
|
153
|
+
`Report.filing`: one `FilingFinding` per rule and problem, with the rule's
|
|
154
|
+
number (`2.19`), its tag in the EBA document (`nilUsed`), a severity, a
|
|
155
|
+
message, how many times it occurs and up to five locations.
|
|
156
|
+
`dpmxbrl.filing.RULE_TITLES` names each rule. Severity is `error` where the
|
|
157
|
+
rule says MUST (EBA rejects the report) and `warning` where it says SHOULD;
|
|
158
|
+
file-naming findings are warnings because each supervisor confirms its own
|
|
159
|
+
naming.
|
|
160
|
+
|
|
161
|
+
- Both formats: one absolute taxonomy entry point (1.5, 2.2, 2.3); the
|
|
162
|
+
reporting subject's scheme, suffix, upper case and LEI check digits (2.8,
|
|
163
|
+
3.6); and the file name against `ReportSubject_Country_
|
|
164
|
+
FrameworkCodeModuleVersion_Module_ReferenceDate_CreationTimestamp`,
|
|
165
|
+
including the reporting subject, module and reference date.
|
|
166
|
+
- XBRL-XML: UTF-8 and no standalone declaration (1.4, 1.13); filing
|
|
167
|
+
indicators present, unique and in a context without segment or scenario
|
|
168
|
+
(1.6, 1.6.1); no `xsi:schemaLocation`, XInclude, `xml:base`,
|
|
169
|
+
`link:linkbaseRef`, footnotes, segments, forever or duration periods, or
|
|
170
|
+
non-dimensional scenario content (1.14, 1.15, 2.1, 2.4, 2.25, 2.11, 2.13, 2.14,
|
|
171
|
+
2.15); one entity and reference date, as plain dates (2.9, 2.10, 2.13); no
|
|
172
|
+
duplicate facts, and none differing only by unit (2.16, 2.16.1); `decimals`
|
|
173
|
+
rather than `precision`, and present on numeric facts (2.17, 2.18); no nil or
|
|
174
|
+
empty facts (2.19); unused or repeated contexts and units (2.7, 2.21, 2.22);
|
|
175
|
+
one reporting currency, and currency-of-denomination facts in the currency
|
|
176
|
+
their CUS or CUA member names (3.1); `xbrli:pure` for other numeric facts
|
|
177
|
+
(3.2); and namespace prefixes declared once, on the document element, and
|
|
178
|
+
used (3.4, 3.9, 3.10).
|
|
179
|
+
- xBRL-CSV: `META-INF/reportPackage.json`, `parameters.csv` with
|
|
180
|
+
`refPeriod`, and `FilingIndicators.csv`; the root folder named as the ZIP;
|
|
181
|
+
no `documentInfo.taxonomy` (1.5); a plain-date `refPeriod` (2.10); unique
|
|
182
|
+
filing indicators (1.6.1); no table file for a template indicated as not
|
|
183
|
+
reported (xBRL-CSV rule 6); no special values in fact or key columns
|
|
184
|
+
(`#nil`, `#empty` and any other value starting with `#`; 2.19 and rule 5);
|
|
185
|
+
no decimals suffix (rule 1); key values without leading or trailing spaces
|
|
186
|
+
(3.11).
|
|
187
|
+
|
|
188
|
+
Rules that need the taxonomy are left to the caller, which can build
|
|
189
|
+
`FilingFinding`s and merge them with `dpmxbrl.filing.FilingLog`: a filing
|
|
190
|
+
indicator for every template of the module and only valid codes (1.6, 1.6.3),
|
|
191
|
+
facts outside the reported templates (1.7.1), minimum decimals per data type
|
|
192
|
+
(2.18), fact values without leading or trailing spaces, which applies to
|
|
193
|
+
string data points only (3.11, `SPACES` in `dpmxbrl.filing`), an entry
|
|
194
|
+
point valid for the reference date (1.5), and `baseCurrency` when monetary
|
|
195
|
+
facts use it. Not checked: units against the XBRL unit registry (2.23),
|
|
196
|
+
decimal notation for rates (3.2), canonical prefixes (3.5),
|
|
197
|
+
generating-software information (2.26) and comments (2.5).
|
|
198
|
+
`read_xbrl_csv_frames` checks special fact values only; the decimals suffix,
|
|
199
|
+
spaces around key values and special key values are left to callers that
|
|
200
|
+
split values by data type, which can record them with the same messages
|
|
201
|
+
(`DECIMALS_SUFFIX`, `KEY_SPACES`, `SPECIAL_KEY_VALUE` in `dpmxbrl.filing`).
|
|
202
|
+
|
|
203
|
+
## Writing
|
|
204
|
+
|
|
205
|
+
```python
|
|
206
|
+
write_xbrl_csv(report, target, *, root=None) -> None
|
|
207
|
+
write_xbrl_xml(report, target, *, typed_domains=None) -> None
|
|
208
|
+
default_package_root(report) -> str
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
`write_xbrl_csv` writes a ZIP report package in EBA convention:
|
|
212
|
+
`<root>/META-INF/reportPackage.json`, `<root>/reports/report.json` (extending
|
|
213
|
+
`entry_point` and declaring `namespaces`), `parameters.csv` (`entityID`,
|
|
214
|
+
`refPeriod`, `baseCurrency`, `decimals*`, then any other parameter),
|
|
215
|
+
`FilingIndicators.csv` and one `<template>.csv` per template with header
|
|
216
|
+
`datapoint,factValue,<key columns in first-seen order>`. Nil values are
|
|
217
|
+
written as `#nil`, empty strings as `#empty` and a leading `#` as `##`. The root
|
|
218
|
+
defaults to `<entity id>_<reference date>`. Output is deterministic (sorted
|
|
219
|
+
templates, source fact order, fixed ZIP timestamps). Facts without a template
|
|
220
|
+
or datapoint raise `ReportError` before anything is written.
|
|
221
|
+
|
|
222
|
+
`write_xbrl_xml` writes `schemaRef`, filing indicators, contexts deduplicated
|
|
223
|
+
by dimensions (`c1`, `c2`, ..., entity plus an instant at `reference_date`,
|
|
224
|
+
dimensions in the scenario), units (`u1`, ...) and facts. A fact with a unit
|
|
225
|
+
is numeric and gets `decimals` (`INF` when unset) unless it is nil. Facts
|
|
226
|
+
without a concept, undeclared prefixes and a missing entity or reference date
|
|
227
|
+
raise `ReportError`. Because the reader does not keep typed-domain element
|
|
228
|
+
names, typed dimensions need `typed_domains={"eba_dim:qINC": "eba_typ:qCC"}`.
|
|
229
|
+
|
|
230
|
+
Reading a written report reproduces its facts in both formats; the tests
|
|
231
|
+
cover these round trips.
|
|
232
|
+
|
|
233
|
+
## CLI
|
|
234
|
+
|
|
235
|
+
```
|
|
236
|
+
dpm-xbrl inspect PATH [--json] [--taxonomy-package ZIP ...]
|
|
237
|
+
dpm-xbrl convert IN OUT [--taxonomy-package ZIP ...]
|
|
238
|
+
[--typed-domain DIMENSION=ELEMENT ...] [--root NAME]
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
`inspect` prints the format, entry point, entity, reference date, base
|
|
242
|
+
currency, filing indicators (reported of total), the fact count per template,
|
|
243
|
+
the warnings and the EBA filing rules the report breaks. `convert` chooses the output format from the suffix: `.zip`
|
|
244
|
+
writes xBRL-CSV, `.xbrl` or `.xml` writes XBRL-XML. Errors print `error: ...`
|
|
245
|
+
and exit with status 1.
|
|
246
|
+
|
|
247
|
+
## Development
|
|
248
|
+
|
|
249
|
+
```
|
|
250
|
+
uv sync
|
|
251
|
+
uv run pytest
|
|
252
|
+
uv run ruff check
|
|
253
|
+
uv run ruff format --check
|
|
254
|
+
uv run ty check
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
## Known gaps
|
|
258
|
+
|
|
259
|
+
- No taxonomy validation, schema lookups or calculation and formula checks.
|
|
260
|
+
Without a taxonomy, an explicit member and a typed value that looks like a
|
|
261
|
+
declared qname cannot be told apart in xBRL-CSV metadata.
|
|
262
|
+
- xBRL-CSV: no `rowIdColumn`, `links`, footnotes, `@start`/`@end` period
|
|
263
|
+
semantics, per-fact `period`/`entity`/`language` values (they are ignored),
|
|
264
|
+
or multiple reports per package. Taxonomy packages must be ZIPs.
|
|
265
|
+
- EBA-convention CSV facts carry no concept, dimensions, unit or decimals, so
|
|
266
|
+
they cannot be written as XML; XML facts carry no datapoint or template, so
|
|
267
|
+
they cannot be written as CSV. Converting between formats needs resolved
|
|
268
|
+
xBRL-CSV metadata.
|
|
269
|
+
- Empty-string facts do not survive an xBRL-CSV round trip (`#empty` is
|
|
270
|
+
skipped when read), and `Fact.decimals`, concepts and dimensions are not
|
|
271
|
+
written to EBA-convention CSV.
|
|
272
|
+
- XML: tuples, footnotes, `precision`, segment versus scenario placement,
|
|
273
|
+
forever periods and typed-domain element names are not kept. XML output puts
|
|
274
|
+
all dimensions in the scenario and uses one entity and instant. The XML
|
|
275
|
+
writer holds the instance text in memory before writing it.
|
|
276
|
+
- Fact locations in XML use the fact id rather than a line number.
|