@fin.cx/opendata 4.7.0 → 4.9.0

package/readme.md

@@ -1,8 +1,8 @@
 # @fin.cx/opendata
 
-`@fin.cx/opendata` is a TypeScript toolbox for open business and market data: stock and crypto prices, SEC fundamentals, lookups in the German registers through the register portal of the federal states, the statutory public holidays of the German federal states, the base rate of § 247 BGB, and the German consumer price index for index clauses. It is built for programmers who want scriptable APIs and real workflows instead of a thin wrapper around one remote endpoint.
+`@fin.cx/opendata` is a TypeScript toolbox for open business and market data: stock and crypto prices, SEC fundamentals, lookups in the German registers through the register portal of the federal states, the statutory public holidays of the German federal states, the base rate of § 247 BGB, the German consumer price index for index clauses, and the figures of published financial statements. It is built for programmers who want scriptable APIs and real workflows instead of a thin wrapper around one remote endpoint.
 
-It gives you seven practical lanes in one package:
+It gives you nine practical lanes in one package:
 
 - `StockPriceService` for stock and crypto prices with caching, retries, failover, historical ranges, and intraday support
 - `FundamentalsService` and `StockDataService` for SEC fundamentals and combined price + fundamentals payloads
@@ -11,6 +11,8 @@ It gives you seven practical lanes in one package:
 - the base rate, from `@fin.cx/opendata/baserate`, for the Basiszinssatz of § 247 BGB per half-year as the Deutsche Bundesbank publishes it, for default interest under § 288 BGB
 - VAT ID confirmation, from `@fin.cx/opendata/vatid`, over the BZSt's eVatR REST API (the confirmation of § 18e UStG, simple or qualified) and the European Commission's VIES REST API, without any storage
 - the consumer price index, from `@fin.cx/opendata/cpi`, for the Verbraucherpreisindex für Deutschland of the Statistisches Bundesamt as the Deutsche Bundesbank republishes it, with the arithmetic of index clauses (Wertsicherungsklauseln, § 557b BGB), without any storage
+- published financial statements, from `@fin.cx/opendata/financialstatements`, for the balance sheet and the income statement under the HGB from the HTML or the PDF the Unternehmensregister shows, and the facts of an ESEF report, read from a document the caller obtained, without any request or storage
+- GLEIF Level 1 data, from `@fin.cx/opendata/gleif`, for streamed Golden Copy full/delta imports, single-LEI lookup, and German registration-authority codes mapped to XJustiz register courts
 
 ## Issue Reporting and Security
 
@@ -179,6 +181,69 @@ The figures are the Statistisches Bundesamt's, under the Datenlizenz Deutschland
 
 A change in percent, a threshold or a scaled amount is not a published figure. Mark it as such, for example "eigene Berechnung", next to the source note.
 
+### Published Financial Statements
+
+Everything here comes from `@fin.cx/opendata/financialstatements`. That entry point imports nothing else of the package, makes no request and keeps nothing. It reads a statement from the bytes the caller obtained, such as a statement saved from the Unternehmensregister, and returns what the caller stores: the filing, the document byte for byte, and what could be read from it. Its types hold no storage of their own, so any store can keep them.
+
+- `readFinancialStatement({ filing, bytes, mediaType, now? })`
+  A statement as a caller keeps it (`IFinancialStatement`): the filing as the caller states it (`filing`, an `IFiling`), the document with its media type and SHA-256 (`raw`), the statement read from it (`parsed`), why none was (`parseFailure`), when it was obtained (`obtainedAt`) and the publisher's origin note (`notice`). An ESEF report, on its own or in its package, goes to the ESEF reader, and so does HTML that holds inline XBRL; any other HTML or PDF goes to `parseStatementDocument`.
+- `parseStatementDocument(bytes, mediaType)`
+  A statement under the HGB from the HTML or the PDF a register shows (`IHgbStatement`): the balance sheet (§ 266 HGB) and, where one is published, the income statement (§ 275 HGB, by the total cost or the cost of sales method), each printed line with its outline code, label, kind and the amounts of both years, the unit (EUR, TEUR or Mio. EUR), the periods, and the key figures: balance sheet total, equity, net income and revenue. A PDF is read from its text layer.
+- `parseEsefPackage(zipBytes)`, `parseEsefReport(xhtmlBytes, reportPath?)`
+  A listed issuer's annual financial report in the European Single Electronic Format (`IEsefStatement`): the LEI, every numeric fact its inline XBRL tags (concept, period, dimensions, unit, and the value with scale and sign applied), and the key figures of its consolidated statements from the IFRS concepts `Assets`, `Equity`, `ProfitLoss` and `Revenue`. A package must hold exactly one report under `reports/`.
+- `unternehmensregisterNotice(publishedOn, deposited?)`, `unternehmensregisterOrigin`
+  The Unternehmensregister's origin note, "Auszug aus dem Unternehmensregister", with the day a document was posted or deposited.
+- `maxStatementBytes` (100 MiB), `maxPdfPages` (500), `maxEsefFacts` (100,000), `maxEsefPackageEntries` (10,000)
+- `FinancialStatementError` with the codes `invalid_input`, `unsupported_media_type`, `too_large`, `invalid_package`, `unreadable`, `no_text` and `unparseable`
+
+Amounts are decimal strings in the currency itself, the unit's factor applied (`1.234` printed in TEUR is `'1234000'`), never floats.
+
+```typescript
+import { FinancialStatementError, readFinancialStatement } from '@fin.cx/opendata/financialstatements';
+
+const statement = await readFinancialStatement({
+  filing: {
+    source: 'unternehmensregister',
+    kind: 'annual_accounts',
+    companyName: 'Muster Werkzeugbau GmbH',
+    fiscalYearStart: '2024-01-01',
+    fiscalYearEnd: '2024-12-31',
+    publishedOn: '2025-11-03',
+  },
+  bytes, // the PDF as saved from the register, a Uint8Array
+  mediaType: 'application/pdf',
+});
+// statement.raw: { bytes, mediaType: 'application/pdf', sha256: '...' }
+// statement.notice: 'Auszug aus dem Unternehmensregister, eingestellt am 03.11.2025'
+
+if (statement.parsed?.standard === 'hgb') {
+  statement.parsed.keyFigures.current;
+  // { totalAssets: '500000.00', equity: '300000.00', netIncome: '40000.00', revenue: '1000000.00' }
+} else if (statement.parseFailure) {
+  statement.parseFailure; // { code: 'unparseable', message: 'No balance sheet was found.' }
+}
+
+try {
+  await readFinancialStatement({ filing: statement.filing, bytes: scan, mediaType: 'image/png' });
+} catch (error) {
+  if (error instanceof FinancialStatementError) {
+    console.log(error.code); // 'unsupported_media_type': nothing is kept
+  }
+}
+```
+
+#### Read, never guessed
+
+`parsed` is null rather than guessed. Everything `parseStatementDocument` returns was printed and adds up: each position with parts equals their sum, each side of the balance sheet its total, both totals each other, and the lines of the income statement its net income, whether it prints its expenses with a minus or as amounts to subtract. A balance sheet whose current year does not add up, whose columns do not state their year or unit, that holds text where an amount belongs, or that is in a currency other than the euro ends in `unparseable`; `readFinancialStatement` then keeps the document with `parsed` null and the reason in `parseFailure`. A prior year that does not add up, and an income statement that does not, are left out with a note in `notes`. A PDF without a text layer, such as a scan, ends in `no_text`: nothing is recognised from images.
+
+The ESEF reader takes values as the report tags them. It reads the formats `num-dot-decimal`, `num-comma-decimal`, `fixed-zero` and `zerodash` (and the older `numdotdecimal` and `numcommadecimal`); a fact in another format is left out and counted in `notes`. A key figure tagged twice with values that differ at the coarser of their `decimals` is left out with a note.
+
+#### Where the documents come from
+
+The entry point makes no request; the caller obtains each document and passes its bytes in. The statements of financial years beginning after 31 December 2021 are published in the Unternehmensregister, earlier ones in the Bundesanzeiger. Both show a statement only after a security check (a captcha); the Unternehmensregister shows it as HTML and usually also as PDF. A balance sheet a micro company deposited (§ 326 Abs. 2 HGB) is retrieved only for a fee and after registration. Listed issuers publish their ESEF reports on their own sites as well.
+
+The Unternehmensregister's terms of use allow copying the statements it makes accessible by print or as a file (§ 3 Abs. 3) and mark them with the origin note "Auszug aus dem Unternehmensregister" and the day the document was posted, or for a deposited balance sheet the day it was deposited (§ 3 Abs. 5). A filing from the Unternehmensregister therefore needs `publishedOn`, and `notice` gives the note, so that a kept copy keeps it.
+
 ## Before You Start
 
 - No stock or fundamentals providers are pre-registered. You must register them before calling the services.
@@ -323,6 +388,78 @@ console.log({
 });
 ```
 
+### GLEIF Level 1 data (`@fin.cx/opendata/gleif`)
+
+`GleifClient` discovers the current GLEIF Golden Copy publication, streams its LEI-CDF 3.1 ZIP/CSV records, or looks up one LEI through GLEIF's public JSON:API. It keeps no files or data. `fetch` is injectable; every request has a signal and a time limit, with no retries. Breaking a `for await` loop cancels the download. Only publication metadata and single-record JSON responses are collected (at most 2 MiB); the ZIP and CSV are read incrementally with backpressure and a default 1,048,576-character limit per CSV row. A completed stream checks the compressed byte and unfiltered record counts against the publication descriptor.
+
+```ts
+import {
+  GleifClient,
+  courtFromRegistrationAuthority,
+  registrationAuthorityFromCourt,
+  parseRegistrationAuthorityEntityId,
+  registerReferenceFromRegistrationAuthority,
+} from '@fin.cx/opendata/gleif';
+
+const gleif = new GleifClient({ userAgent: 'Example Books (+https://example.com/contact)' });
+const publish = await gleif.getLatestPublish();
+// Save this descriptor in your own store to identify the import's publication.
+for await (const record of gleif.readGoldenCopy(publish.fullFile, { country: 'DE' })) {
+  console.log(record.lei, record.legalName.name, record.registrationAuthority);
+}
+// Or discover and stream in one call. Other kinds: GoldenCopy, LastDay, LastWeek, LastMonth.
+for await (const record of gleif.streamRecords({ kind: 'IntraDay', jurisdiction: 'DE' })) {
+  console.log(record.lei, record.lastUpdateDate);
+}
+const record = await gleif.lookup('529900W18LQJJN6SJ336');
+const court = courtFromRegistrationAuthority('RA000222'); // { code: 'H1101', name: 'Bremen' }
+const raCode = registrationAuthorityFromCourt('H1101'); // RA000222
+const number = parseRegistrationAuthorityEntityId('HRB 35230HB'); // { type: 'HRB', number: '35230 HB' }
+const reference = registerReferenceFromRegistrationAuthority('RA000222', 'HRB 35230HB');
+// IRegisterReference: { court, type: 'HRB', number: '35230 HB' }
+```
+
+The exact yielded/lookup contract is `IGleifRecord`:
+
+```ts
+interface IGleifRecord {
+  lei: string;
+  legalName: IGleifName;
+  otherNames: IGleifName[];
+  transliteratedOtherNames: IGleifName[];
+  legalForm?: { elfCode?: string; other?: string };
+  legalAddress: IGleifAddress;
+  headquartersAddress: IGleifAddress;
+  jurisdiction?: string;
+  entityStatus: string;
+  entityCategory?: string;
+  registrationAuthority?: { id?: string; otherId?: string; entityId?: string };
+  registrationStatus: string;
+  initialRegistrationDate: string;
+  lastUpdateDate: string;
+}
+```
+
+`IGleifName` carries `name`, optional `language`, and optional published other-name `type`. `IGleifAddress` carries `addressLines`, `city`, `country`, and optional `language`, `region`, `postalCode`, `addressNumber`, `addressNumberWithinBuilding` and `mailRouting`. Empty optional source fields are absent, other-name lists are arrays, and dates remain the published ISO 8601 strings, including fractional seconds. ELF codes remain codes; no legal-form designation is invented. Status and category strings remain as published. `country` compares the legal address's ISO country, `jurisdiction` compares the exact legal jurisdiction; when both are supplied, both must match. Filtering happens after parsing each record, before yielding it.
+
+The full file and delta descriptors are `IGleifGoldenCopyFile` (`url`, `format: 'csv'`, `cdfVersion: 'LEI_3.1'`, `kind`, `recordCount`, `size`). `getLatestPublish()` returns `IGleifPublish` (`publishDate`, `fullFile`, `deltaFiles`). Options include `fetch`, `goldenCopyBaseUrl`, `restBaseUrl`, `userAgent` and `timeoutMs` (default 30 minutes per request, including consumer time). Read calls accept `country`, `jurisdiction`, `signal` and `maxRowChars`. `lookup()` and discovery accept `signal`. `parseGleifRestRecord(json)` maps an already obtained JSON:API response without fetching. `GleifError` exposes `code` and, for HTTP errors, `httpStatus`.
+
+GLEIF publishes full files three times daily and deltas for the preceding 8 hours (`IntraDay`), 24 hours (`LastDay`), 7 days (`LastWeek`) and 31 days (`LastMonth`). Level 1 deltas contain the current version of new or changed LEI records. They contain no per-record change kind: this module does not infer insert/update/delete events. The deletion flag described in the manual concerns Level 2 relationships, which this module does not read. A delta overlaps earlier windows; callers own their import checkpoint and idempotent persistence. Discovery selects the CSV format from GLEIF's descriptors rather than constructing file download URLs. Unsupported CDF versions or missing columns fail explicitly.
+
+The generated German RA ↔ XJustiz crosswalk uses the official **GLEIF Registration Authorities List v1.9, 2026-09-30**. Its 177 German authorities contain 147 mapped courts and 30 non-court authorities with unmapped reasons. Of opendata's 150 court codes, 147 are mapped; `Y1303` (Arnstadt Zweigstelle Ilmenau), `Y1208` (Pößneck Zweigstelle Bad Lobenstein) and `R1105` (Langenfeld) have no distinct RA in that version. No parent/successor mapping is inferred. `germanRegistrationAuthorities`, `gleifUnmappedCourts` and `gleifRegistrationAuthoritiesSource` expose the list and its provenance. Unknown/unmapped codes and unparseable register identifiers return `undefined`. Register identifiers must state their type (HRA, HRB, GnR, PR, VR or GsR); leading zeros and location suffixes are preserved.
+
+Reproduce the code list offline with `tsx scripts/generate.gleif-crosswalk.ts`, or verify it with `--check`. `--refresh` fetches only the small official RA CSV and updates the committed German subset and generated list. For a new RA-list version, update the version/date constants in the generator, review new names and mapping ambiguity, then refresh and run the exhaustive tests. The generator never contacts a register portal.
+
+Official sources, read on 2026-10-13:
+
+- [Golden Copy and Delta Files overview](https://www.gleif.org/en/lei-data/gleif-golden-copy) and [specification/user manual v2.2](https://www.gleif.org/lei-data/gleif-golden-copy/2022-02-23_gleif-golden-copy-and-delta-files_v2.2-final.pdf), especially §§1.5–1.6 and 2.2: ZIP compression, formats, delta windows and API discovery at `https://goldencopy.gleif.org/api/v2/golden-copies/publishes/latest`. `leidata-preview.gleif.org` is the documented legacy domain for the same API.
+- [LEI-CDF 3.1 format](https://www.gleif.org/en/lei-data/access-and-use-lei-data/level-1-data-lei-cdf-3-1-format) and [official XSD](https://www.gleif.org/lei-data/access-and-use-lei-data/level-1-data-lei-cdf-3-1-format/2021-03-04_lei-cdf-v3-1.xsd). CSV uses the published dotted columns, including `Entity.LegalName`, numbered other/transliterated names, `Entity.RegistrationAuthority.RegistrationAuthorityID` / `RegistrationAuthorityEntityID`, `Entity.LegalForm.EntityLegalFormCode`, addresses and registration dates. The parser reads the header rather than assuming column positions; fixtures preserve all 338 published columns.
+- [GLEIF REST API documentation](https://api.gleif.org/docs): `GET https://api.gleif.org/api/v1/lei-records/{lei}`, JSON:API `data.attributes.entity` and `data.attributes.registration`, including `registeredAt.id` and `registeredAs`.
+- [RA-list publication page](https://www.gleif.org/en/about-lei/code-lists/gleif-registration-authorities-list) and [v1.9 CSV](https://www.gleif.org/lei-data/code-lists/gleif-registration-authorities-list/2026-09-30_ra-list-v1.9.csv).
+- [GLEIF LEI Data Terms of Use](https://www.gleif.org/en/meta/lei-data-terms-of-use): data available through its access service are provided under [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).
+
+Fixtures are 100 records each from the 2026-10-13 00:00 full and intra-day CSVs, publication descriptors, and the REST documentation's single-record response. Tests make no GLEIF requests; ZIPs are created from these small CSV samples in memory. Provenance is in `test/fixtures/gleif/readme.md`. Full-population imports and GLEIF request-rate limits have not been verified; only bounded live samples, documented shapes and offline fixtures were checked.
+
 ### Register Lookups (Handelsregister)
 
 `HandelsregisterClient` speaks to the register portal of the federal states over plain HTTP, the way a browser would: it opens the search form, sends the search, reads the result list and, for one entry, downloads the structured register content (SI, an XJustiz XML file) and parses it.
@@ -613,11 +750,20 @@ Four holidays apply only in part of a state:
 - `bundesbankCpiBaseUrl`, `bundesbankVpiMonthlyKey`, `bundesbankVpiAnnualKey`, `bundesbankVpiFirstMonth`, `bundesbankVpiFirstYear`, `destatisLicence`, `defaultCpiTimeoutMs`
 - `CpiError` with the codes `invalid_input`, `before_first_period`, `not_published`, `base_mismatch`, `series_withdrawn`, `busy`, `unavailable`, `unreachable`, `timeout`, `aborted` and `unexpected_answer`
 
+### Financial statements (`@fin.cx/opendata/financialstatements`)
+
+- `readFinancialStatement({ filing, bytes, mediaType, now? })`
+- `parseStatementDocument(bytes, mediaType)`
+- `parseEsefPackage(zipBytes)`, `parseEsefReport(xhtmlBytes, reportPath?)`
+- `unternehmensregisterNotice(publishedOn, deposited?)`, `unternehmensregisterOrigin`, `maxStatementBytes`, `maxPdfPages`, `maxEsefFacts`, `maxEsefPackageEntries`
+- the types `IFiling`, `IFinancialStatement`, `IHgbStatement`, `IEsefStatement`, `IKeyFigures`, `IStatementPosition` and their parts
+- `FinancialStatementError` with the codes `invalid_input`, `unsupported_media_type`, `too_large`, `invalid_package`, `unreadable`, `no_text` and `unparseable`
+
 ## Data Sources
 
-The package ships no market or register data. Prices and fundamentals come at run time from the providers you register (Marketstack, CoinGecko, SEC EDGAR), and register entries come at run time from the register portal of the federal states (handelsregister.de), one lookup at a time.
+The package ships no market or register data. Prices and fundamentals come at run time from the providers you register (Marketstack, CoinGecko, SEC EDGAR), and register entries come at run time from the register portal of the federal states (handelsregister.de), one lookup at a time. Financial statements come from the caller: `@fin.cx/opendata/financialstatements` reads the documents it is given and fetches none.
 
-The package carries two kinds of data: the holiday rules of `@fin.cx/opendata/holidays` (`ts/holidays/data.holidays.ts`), described at the end of this section, and the code tables that `@fin.cx/opendata/handelsregister` uses to decode the structured register content (`ts/handelsregister/data.codelists.ts`). The code tables are official code lists, published on XRepository (https://www.xrepository.de), and were taken on 2026-09-23:
+The GLEIF module also carries the generated RA ↔ court crosswalk described above. The package carries two other kinds of data: the holiday rules of `@fin.cx/opendata/holidays` (`ts/holidays/data.holidays.ts`), described at the end of this section, and the code tables that `@fin.cx/opendata/handelsregister` uses to decode the structured register content (`ts/handelsregister/data.codelists.ts`). The code tables are official code lists, published on XRepository (https://www.xrepository.de), and were taken on 2026-09-23:
 
 | Table | Code list | Issued by |
 | --- | --- | --- |
@@ -674,6 +820,10 @@ The consumer price index of `@fin.cx/opendata/cpi` is read at run time, one requ
 
 - Not verified: limits on the number of requests to the Bundesbank's API (none is documented) and terms of use specific to it. The Bundesbank's terms for reusing statistics exclude data of third parties, so the figures are used under the Statistisches Bundesamt's licence. Whether the Bundesbank keeps the series of an old base for a while after a change of base year is not known; today the key of 2015=100 answers 404.
 
+The reader of `@fin.cx/opendata/financialstatements` follows the outlines of the balance sheet (§ 266 HGB) and the income statement (§ 275 HGB), the terms of use of the Unternehmensregister (`https://www.unternehmensregister.de/i18n-doc/D061_UReg_nutz_0118_de.pdf`, § 3) and its help page on searching, which names the security check, both read on 2026-10-10. The ESEF reader was also run on two reports issuers published, which are not part of the repository.
+
+- Not verified: the markup the Unternehmensregister and the Bundesanzeiger show a statement in today, and the PDF they make of it. No live statement was opened, to stay clear of the security check; the HTML and PDF readers are tested against documents written in the shape of published statements, not against saved ones. Whether readers can obtain a statement as filed in XML or XBRL is not known either; the help pages name HTML and PDF only.
+
 ## Behavior And Caveats
 
 - Historical market data is cached aggressively because it is treated as immutable.
@@ -681,6 +831,7 @@ The consumer price index of `@fin.cx/opendata/cpi` is read at run time, one requ
 - `StockDataService` is convenience-first. `StockPriceService` is the better low-level choice if you care about request-shape-specific cache behavior.
 - The register lookup reads the portal's German pages and asks for German explicitly. It depends on the portal's current page structure: when that changes, calls end in `unexpected_response` rather than in guessed data.
 - The structured register content is a non-binding service of the portal; the current printout is binding. Results carry `binding: false` and the portal's note.
+- The statement reader depends on how a statement is laid out: when a layout is not one it can read with certainty, the document is kept with `parsed` null rather than read in part or guessed. It reads the balance sheet and the income statement under the HGB and the numeric facts of ESEF reports; the notes and the management report are not read.
 - There is no hosted backend in this package. Everything is meant to run from your own TypeScript/Node process.
 
 ## Testing
@@ -689,7 +840,7 @@ The consumer price index of `@fin.cx/opendata/cpi` is read at run time, one requ
 pnpm test
 ```
 
-Some market data tests hit live remote services, so they may depend on network access or optional API credentials. The register lookup is tested against minimal hand-written pages and structured content with synthetic values only; the tests never contact the register portal. The VAT ID clients are tested against the answers the two services gave on 2026-09-27 to their documented test values (`test/fixtures/vatid/`); the tests never contact them. The consumer price index client is tested against the answers the Bundesbank's API gave on 2026-10-05 and two answers written by hand from them (`test/fixtures/cpi/`); the tests never contact it.
+GLEIF tests use only small real published fixtures (`test/fixtures/gleif/`) and injected fetches, including cancellation, malformed input, optional names and exhaustive crosswalk coverage. Some market data tests hit live remote services, so they may depend on network access or optional API credentials. The register lookup is tested against minimal hand-written pages and structured content with synthetic values only; the tests never contact the register portal. The VAT ID clients are tested against the answers the two services gave on 2026-09-27 to their documented test values (`test/fixtures/vatid/`); the tests never contact them. The consumer price index client is tested against the answers the Bundesbank's API gave on 2026-10-05 and two answers written by hand from them (`test/fixtures/cpi/`); the tests never contact it. The financial statement reader is tested against HTML statements written by hand in the shape of the register's (`test/fixtures/financialstatements/`), PDFs the tests write themselves, and ESEF reports and packages the tests write, all with synthetic values; the tests contact no register.
 
 ## License and Legal Information
 

package/ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@fin.cx/opendata',
-  version: '4.7.0',
-  description: 'Open business and financial data for TypeScript: register lookups on the German register portal (handelsregister.de) without storage, the statutory public holidays of the German federal states, the base rate of § 247 BGB, the German consumer price index with the arithmetic of index clauses, and stock and crypto prices and fundamentals from several providers.'
+  version: '4.9.0',
+  description: 'Open business and financial data for TypeScript: register lookups on the German register portal (handelsregister.de) without storage, the statutory public holidays of the German federal states, the base rate of § 247 BGB, the German consumer price index with the arithmetic of index clauses, published financial statements read from their HTML, PDF or ESEF documents, and stock and crypto prices and fundamentals from several providers.'
 }

package/ts/financialstatements/amounts.ts

@@ -0,0 +1,181 @@
+import type { IStatementUnit } from './interfaces.js';
+
+/** millionths of a euro per euro: amounts are kept as whole millionths, so no sum loses a digit */
+const microsPerEuro = 1_000_000n;
+
+/** a step at or below one cent: the amount was printed to the cent and is exact */
+const exactStep = 10_000n;
+
+/**
+ * An amount as this entry point computes with it: millionths of a euro, and the rounding it may
+ * carry. `slack` is twice the largest rounding error, also in millionths: zero for an amount printed
+ * to the cent, the step of the last printed digit for one printed rounded, and the sum of its terms'
+ * slack for a sum.
+ */
+export interface IAmount {
+  micros: bigint;
+  slack: bigint;
+  /** the digits after the decimal point the amount is written with */
+  fractionDigits: number;
+}
+
+/** a number as printed, before its unit is applied */
+export interface IPrintedAmount {
+  /** the printed digits as one integer: `1.234,5` is 12345 */
+  digits: bigint;
+  /** the digits after the decimal comma */
+  decimals: number;
+  /** the unit printed with the number, where one is */
+  unit: IStatementUnit | null;
+}
+
+export const unitEur: IStatementUnit = { label: 'EUR', factor: '1' };
+export const unitTeur: IStatementUnit = { label: 'TEUR', factor: '1000' };
+export const unitMioEur: IStatementUnit = { label: 'Mio. EUR', factor: '1000000' };
+
+/** the euro after a prefix such as `Mio.` or `T` */
+const euroSuffix = String.raw`(?:€|EUR\b|Euro\b|EURO\b)`;
+
+/** the units the statements print, millions and thousands before plain euros */
+const unitPatterns: Array<{ unit: IStatementUnit; pattern: string }> = [
+  { unit: unitMioEur, pattern: String.raw`\b(?:Mio\.?|Millionen)\s*${euroSuffix}` },
+  { unit: unitTeur, pattern: String.raw`\b(?:T|Tsd\.?\s*|Tausend\s+)${euroSuffix}` },
+  { unit: unitEur, pattern: String.raw`(?:€|\b${euroSuffix})` },
+];
+
+const unitAlternation = unitPatterns.map((entryArg) => `(${entryArg.pattern})`).join('|');
+
+/** the units a text names, in the order it names them */
+export const unitsIn = (textArg: string): IStatementUnit[] => {
+  const result: IStatementUnit[] = [];
+  for (const match of textArg.matchAll(new RegExp(unitAlternation, 'g'))) {
+    const index = match.slice(1).findIndex((groupArg) => groupArg !== undefined);
+    result.push(unitPatterns[index].unit);
+  }
+  return result;
+};
+
+/** a text with the units it names blanked out */
+export const withoutUnits = (textArg: string): string => textArg.replace(new RegExp(unitAlternation, 'g'), ' ');
+
+/** the currencies other than the euro a statement may name */
+const otherCurrencySource = String.raw`\b(?:USD|US-Dollar|CHF|GBP|JPY|CNY|DKK|SEK|NOK|PLN|CZK|HUF)\b|\$|£|¥`;
+
+/** whether a text names a currency other than the euro */
+export const namesOtherCurrency = (textArg: string): boolean => new RegExp(otherCurrencySource).test(textArg);
+
+/** a text with the currencies other than the euro it names blanked out */
+export const withoutOtherCurrencies = (textArg: string): string => textArg.replace(new RegExp(otherCurrencySource, 'g'), ' ');
+
+/** the unit alone, `TEUR` or `(in TEUR)`, as a line of its own */
+const unitOnlyPattern = new RegExp(String.raw`^\(?\s*(?:in\s+)?(?:${unitAlternation})\s*\)?$`);
+
+/** the explicit phrases a statement states its unit with: `Angaben in TEUR`, `(Beträge in Euro)` */
+const unitPhrasePattern = new RegExp(
+  String.raw`(?:\(\s*in\s+|\b(?:Angaben|Beträge|Betrag|Werte|Zahlen|Alle Beträge|Zahlenangaben)\s+in\s+)(?:${unitAlternation})`
+);
+
+/** the unit a line of context states, `Angaben in TEUR` or a line that is only a unit; null if none */
+export const unitStatedBy = (textArg: string): IStatementUnit | null => {
+  const text = normalizeSpaces(textArg);
+  if (text.length > 200) {
+    return null;
+  }
+  const phrase = unitPhrasePattern.exec(text);
+  if (phrase) {
+    return unitsIn(phrase[0])[0] ?? null;
+  }
+  return unitOnlyPattern.test(text) ? unitsIn(text)[0] ?? null : null;
+};
+
+/** collapses the spaces statements print, no-break and narrow ones among them */
+export const normalizeSpaces = (textArg: string): string => textArg.replace(/\s+/g, ' ').trim();
+
+const dashOnlyPattern = /^[-–—−]{1,3}$/;
+const numberPattern = /^([-–—−])?\s*(\d{1,3}(?:[. ]\d{3})+|\d+)(?:,(\d{1,6}))?$/;
+
+/**
+ * Reads an amount as German statements print it: `1.234.567,89`, `-1 234`, `– 12,5`, a dash alone
+ * for zero, with or without a unit before or after it (`EUR 1.234,56`, `1.234 T€`). Returns null for
+ * anything else, a year or a note number among them for the caller to tell apart.
+ */
+export const parsePrintedAmount = (textArg: string): IPrintedAmount | null => {
+  let text = normalizeSpaces(textArg);
+  if (text === '') {
+    return null;
+  }
+  let unit: IStatementUnit | null = null;
+  const leading = new RegExp(String.raw`^(?:${unitAlternation})\s*`).exec(text);
+  if (leading) {
+    unit = unitsIn(leading[0])[0];
+    text = text.slice(leading[0].length);
+  } else {
+    const trailing = new RegExp(String.raw`\s*(?:${unitAlternation})$`).exec(text);
+    if (trailing && trailing.index > 0) {
+      unit = unitsIn(trailing[0])[0];
+      text = text.slice(0, trailing.index);
+    }
+  }
+  if (dashOnlyPattern.test(text)) {
+    return { digits: 0n, decimals: 0, unit };
+  }
+  const match = numberPattern.exec(text);
+  if (!match) {
+    return null;
+  }
+  const decimals = match[3]?.length ?? 0;
+  const magnitude = BigInt(match[2].replace(/[. ]/g, '') + (match[3] ?? ''));
+  return { digits: match[1] ? -magnitude : magnitude, decimals, unit };
+};
+
+/** whether a text reads as an amount rather than as a year such as `2024` or `2024 EUR` */
+export const readsAsAmount = (textArg: string): boolean =>
+  parsePrintedAmount(textArg) !== null && !/^(?:19|20)\d{2}$/.test(normalizeSpaces(withoutUnits(textArg)));
+
+/** an amount in a unit: `1.234` in TEUR is 1,234,000 euros, rounded to the thousand */
+export const toAmount = (printedArg: IPrintedAmount, unitArg: IStatementUnit): IAmount => {
+  const step = BigInt(unitArg.factor) * 10n ** BigInt(6 - printedArg.decimals);
+  return {
+    micros: printedArg.digits * step,
+    slack: step <= exactStep ? 0n : step,
+    fractionDigits: fractionDigitsOf(step),
+  };
+};
+
+const fractionDigitsOf = (stepArg: bigint): number => {
+  let digits = 6;
+  for (let step = stepArg; digits > 0 && step % 10n === 0n; step /= 10n) {
+    digits--;
+  }
+  return digits;
+};
+
+export const zeroAmount: IAmount = { micros: 0n, slack: 0n, fractionDigits: 0 };
+
+export const addAmounts = (leftArg: IAmount, rightArg: IAmount): IAmount => ({
+  micros: leftArg.micros + rightArg.micros,
+  slack: leftArg.slack + rightArg.slack,
+  fractionDigits: Math.max(leftArg.fractionDigits, rightArg.fractionDigits),
+});
+
+export const negateAmount = (amountArg: IAmount): IAmount => ({ ...amountArg, micros: -amountArg.micros });
+
+export const sumAmounts = (amountsArg: IAmount[]): IAmount => amountsArg.reduce(addAmounts, zeroAmount);
+
+/**
+ * Whether two amounts can be the same figure: equal, or for figures printed rounded, no further
+ * apart than both their roundings together allow.
+ */
+export const amountsAgree = (leftArg: IAmount, rightArg: IAmount): boolean => {
+  const difference = leftArg.micros - rightArg.micros;
+  return (difference < 0n ? -difference : difference) * 2n <= leftArg.slack + rightArg.slack;
+};
+
+/** an amount as a decimal string in euros, written with its own digits: `1234000`, `-12.50` */
+export const formatAmount = (amountArg: IAmount): string => {
+  const negative = amountArg.micros < 0n;
+  const magnitude = negative ? -amountArg.micros : amountArg.micros;
+  const whole = (magnitude / microsPerEuro).toString();
+  const fraction = (magnitude % microsPerEuro).toString().padStart(6, '0').slice(0, amountArg.fractionDigits);
+  return `${negative ? '-' : ''}${whole}${fraction === '' ? '' : `.${fraction}`}`;
+};

package/ts/financialstatements/classes.financialstatementerror.ts

@@ -0,0 +1,35 @@
+/**
+ * Why a document was not read:
+ * - `invalid_input`: arguments this entry point does not take: no bytes, a date that is not one, a
+ *   statement from the Unternehmensregister without the day it was posted;
+ * - `unsupported_media_type`: a media type other than HTML, PDF, XHTML or a ZIP package;
+ * - `too_large`: a document, or the report in a package, larger than `maxStatementBytes`, a package
+ *   with more than `maxEsefPackageEntries` entries, a PDF with more than `maxPdfPages` pages, or a
+ *   report with more than `maxEsefFacts` facts;
+ * - `invalid_package`: a package that is not a readable ZIP, is encrypted, or does not hold exactly
+ *   one report under `reports/`;
+ * - `unreadable`: a document that does not open as what its media type says: a damaged or
+ *   password-protected PDF, HTML in an unknown encoding, XHTML that is not well-formed;
+ * - `no_text`: a PDF without a text layer, such as a scan;
+ * - `unparseable`: a readable document in which no statement was found with certainty: no balance
+ *   sheet, columns that do not say their year or unit, or current-year figures that do not add up.
+ */
+export type TFinancialStatementErrorCode =
+  | 'invalid_input'
+  | 'unsupported_media_type'
+  | 'too_large'
+  | 'invalid_package'
+  | 'unreadable'
+  | 'no_text'
+  | 'unparseable';
+
+/** The typed error of `@fin.cx/opendata/financialstatements`. */
+export class FinancialStatementError extends Error {
+  public readonly code: TFinancialStatementErrorCode;
+
+  constructor(code: TFinancialStatementErrorCode, message: string, options?: { cause?: unknown }) {
+    super(message, options);
+    this.name = 'FinancialStatementError';
+    this.code = code;
+  }
+}

package/ts/financialstatements/constants.ts

@@ -0,0 +1,24 @@
+/** The largest document read, and the largest report a package may hold: 100 MiB. */
+export const maxStatementBytes = 100 * 1024 * 1024;
+
+/** The most pages a PDF may have. */
+export const maxPdfPages = 500;
+
+/** The most numeric facts an ESEF report may tag. */
+export const maxEsefFacts = 100_000;
+
+/** The most entries an ESEF package may hold. */
+export const maxEsefPackageEntries = 10_000;
+
+/** The origin note the Unternehmensregister marks what it gives out with (Nutzungsbedingungen § 3 Abs. 5). */
+export const unternehmensregisterOrigin = 'Auszug aus dem Unternehmensregister';
+
+/**
+ * The full origin note of a document from the Unternehmensregister: "Auszug aus dem
+ * Unternehmensregister" and the day the document was posted, or for a deposited balance sheet the day
+ * it was deposited (§ 326 Abs. 2 HGB).
+ */
+export const unternehmensregisterNotice = (publishedOnArg: string, depositedArg = false): string => {
+  const [year, month, day] = publishedOnArg.split('-');
+  return `${unternehmensregisterOrigin}, ${depositedArg ? 'hinterlegt' : 'eingestellt'} am ${day}.${month}.${year}`;
+};

package/ts/financialstatements/dates.ts

@@ -0,0 +1,89 @@
+import type { TIsoDate } from './interfaces.js';
+
+const monthNumbers: Record<string, number> = {
+  januar: 1, jan: 1, jänner: 1, februar: 2, feb: 2, märz: 3, mär: 3, maerz: 3, april: 4, apr: 4, mai: 5,
+  juni: 6, jun: 6, juli: 7, jul: 7, august: 8, aug: 8, september: 9, sep: 9, sept: 9, oktober: 10, okt: 10,
+  november: 11, nov: 11, dezember: 12, dez: 12,
+};
+
+/** `31.12.2024`, `31. Dezember 2024`, `1.1.`, `01. Jan.`: a day, a month and maybe a year */
+const datePattern = /(\d{1,2})\.\s*(?:(\d{1,2})\.|([A-Za-zÄäÖöÜü]{3,9})\.?(?=[\s\d]|$))\s*((?:19|20)\d{2})?/g;
+
+interface IDateMatch {
+  day: number;
+  month: number;
+  year: number | null;
+  index: number;
+  end: number;
+}
+
+const isoOf = (yearArg: number, monthArg: number, dayArg: number): TIsoDate | null => {
+  const date = new Date(Date.UTC(yearArg, monthArg - 1, dayArg));
+  if (date.getUTCFullYear() !== yearArg || date.getUTCMonth() !== monthArg - 1 || date.getUTCDate() !== dayArg) {
+    return null;
+  }
+  return date.toISOString().slice(0, 10);
+};
+
+const dateMatches = (textArg: string): IDateMatch[] => {
+  const result: IDateMatch[] = [];
+  for (const match of textArg.matchAll(datePattern)) {
+    const month = match[2] !== undefined ? Number(match[2]) : monthNumbers[match[3].toLowerCase()];
+    const day = Number(match[1]);
+    if (month === undefined || month < 1 || month > 12 || day < 1 || day > 31) {
+      continue;
+    }
+    result.push({
+      day,
+      month,
+      year: match[4] !== undefined ? Number(match[4]) : null,
+      index: match.index,
+      end: match.index + match[0].length,
+    });
+  }
+  return result;
+};
+
+/** the last full date a text names, `31.12.2024` or `31. Dezember 2024`; null if none */
+export const lastDateIn = (textArg: string): TIsoDate | null => {
+  const full = dateMatches(textArg).filter((matchArg) => matchArg.year !== null);
+  const last = full[full.length - 1];
+  return last ? isoOf(last.year as number, last.month, last.day) : null;
+};
+
+/** a text with the dates it names blanked out, `31.12.2024`, `1. Januar` and the like */
+export const withoutDates = (textArg: string): string => {
+  let text = textArg;
+  for (const match of dateMatches(textArg).reverse()) {
+    text = text.slice(0, match.index) + ' '.repeat(match.end - match.index) + text.slice(match.end);
+  }
+  return text;
+};
+
+/** the last year a text names on its own, outside a date; null if none */
+export const lastYearIn = (textArg: string): number | null => {
+  const years = [...withoutDates(textArg).matchAll(/\b((?:19|20)\d{2})\b/g)];
+  return years.length > 0 ? Number(years[years.length - 1][1]) : null;
+};
+
+/**
+ * The period a line of a statement names: `vom 1. Januar bis 31. Dezember 2024` and
+ * `01.01.-31.12.2024` give both days, `zum 31. Dezember 2024` the last. A first day printed without
+ * its year is in the last day's year, or the year before where it would fall after the last day.
+ */
+export const periodIn = (textArg: string): { start: TIsoDate | null; end: TIsoDate | null } => {
+  const matches = dateMatches(textArg);
+  for (let index = 0; index + 1 < matches.length; index++) {
+    const first = matches[index];
+    const second = matches[index + 1];
+    const between = textArg.slice(first.end, second.index);
+    if (second.year === null || !/^\s*(?:bis(?:\s+(?:zum|einschließlich))?|[-–—])\s*$/i.test(between)) {
+      continue;
+    }
+    const end = isoOf(second.year, second.month, second.day);
+    const startYear =
+      first.year ?? (first.month * 100 + first.day <= second.month * 100 + second.day ? second.year : second.year - 1);
+    return { start: end ? isoOf(startYear, first.month, first.day) : null, end };
+  }
+  return { start: null, end: lastDateIn(textArg) };
+};