yggio 0.0.16 → 0.0.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/NOTICE ADDED
@@ -0,0 +1,334 @@
1
+ yggio
2
+ Copyright (c) yggio contributors
3
+
4
+ FIX Orchestra (FIX Trading Community)
5
+
6
+ This product includes a FIX dictionary derived from OrchestraFIXLatest.xml
7
+ (FIX.Latest EP309) of the FIX Trading Community orchestrations repository,
8
+ https://github.com/FIXTradingCommunity/orchestrations, commit
9
+ 099914dd0edd49a699326f0441776d6e21cfaf93, licensed under the Apache License,
10
+ Version 2.0. The derived files are config/seeds/fix.parquet,
11
+ rust/market/src/fix/seed.parquet, rust/market/src/fix/constants.rs and
12
+ rust/market/src/fix/fields.rs. The metadata of OrchestraFIXLatest.xml states
13
+ "Copyright (c) FIX Protocol Ltd. All Rights Reserved".
14
+
15
+ Underlyings seed (config/seeds/underlyings.parquet, rust/market/src/underlying/seed.parquet)
16
+
17
+ The underlyings seed is maintained by hand (config/underlyings/underlyings.json)
18
+ and folds in the records the repository's generators write from these sources:
19
+
20
+ - ESMA FIRDS and FITRS (registers.esma.europa.eu): reproduction authorised
21
+ provided the source is acknowledged. Source: ESMA. Transformed by
22
+ scripts/seed_venues.py, US and Canadian ISINs included (read 2026-11-12;
23
+ each identifier as ESMA publishes it under its notice, no national number
24
+ an ISIN embeds - a CUSIP - stored): ISIN, CFI, FISN, full
25
+ name, issuer LEI, notional currency and each listing's venue and first
26
+ trading day kept; the home listing is the one on FITRS's most relevant
27
+ market (RlvntMkt).
28
+ - ESMA Benchmarks Register (registers.esma.europa.eu): source acknowledged.
29
+ Transformed by scripts/seed_identifiers.py: each benchmark's ISIN, its
30
+ administrator's LEI and an index CFI derived from its type; no names.
31
+ - OpenFIGI (www.openfigi.com): FIGIs dedicated to the public domain (terms
32
+ of service of 2018-11-27, read 2026-11-12: "FIGI Identifiers may be freely
33
+ reproduced, distributed, transmitted, used, modified, built upon, or
34
+ otherwise exploited by anyone for any purpose"; a ticker or exchange code
35
+ the API returns is licensed by no clause, so those stay as shipped in
36
+ 0.0.16 pending the owner's ruling and nothing beyond FIGIs is added from
37
+ this source). Transformed by scripts/seed_venues.py: venue FIGI, ticker, Bloomberg code,
38
+ share-class FIGI and per-country composite FIGI.
39
+ - European Central Bank (www.ecb.europa.eu): Source: ECB, modified. The
40
+ eligible marketable assets list (issuer residence, maturity, denomination,
41
+ debt CFI; at most 1,200 new ISINs), the EXR series keys (pairs only, no
42
+ rates) and the MFI list (RIAD codes onto LEI rows). The data is free of
43
+ charge; a product that includes it must say so to its buyers.
44
+ - ISO 10383 market identifier codes (www.iso20022.org/market-identifier-codes,
45
+ SWIFT as registration authority): rust/src/mic/tables.rs and the seed's
46
+ MIC rows, written by scripts/generate_mic_table.py with
47
+ config/mics/aliases.json. Its terms of use (version October 2016, read
48
+ 2026-11-12): "the material on this site is intended to be used and
49
+ reproduced freely by all interested users ... You are granted a
50
+ non-exclusive right to access and use the information on this website
51
+ excluding the right to reproduce all or substantial part of this website
52
+ for the purpose of making such reproductions available to third parties."
53
+ The list ships under the first sentence (the owner's ruling, 2026-11-12).
54
+ - GLEIF (www.gleif.org): LEI data CC0 1.0 - the Golden Copy and its
55
+ relationships, the ISO 20275 ELF code list.
56
+ - SWIFT BIC-to-LEI mapping, published by GLEIF:
57
+ "SWIFT © and database rights October 2026. All rights reserved. This
58
+ Mapping Table has been developed by SWIFT. Any use of the Mapping Table, in
59
+ whole or in part, is subject to the BIC/LEI Mapping Table License Agreement
60
+ as published with the Mapping Table available on GLEIF's website. The
61
+ Mapping Table is updated monthly. For the latest BIC information and
62
+ updates, always refer to www.swift.com/bic ." Transformed: each LEI's BICs,
63
+ upper-cased, sorted, kept once.
64
+ - Wikidata (www.wikidata.org): CC0 1.0. ISINs' issuers and tickers, and
65
+ organisation PermIDs as Wikidata states them; Open PermID (LSEG,
66
+ CC BY 4.0) is their likely origin. No instrument or quote PermID and no RIC
67
+ is shipped.
68
+ - EDINET (Financial Services Agency, Japan): Public Data License (Version
69
+ 1.0), https://www.digital.go.jp/resources/open_data/public_data_license_v1.0.
70
+ Edited: each listed filer's EDINET code and corporate number joined by
71
+ securities code to an instrument the seed holds; nothing else kept.
72
+ - Taiwan Stock Exchange Corporation, 2026, 上市公司基本資料 (t187ap03_L);
73
+ Taipei Exchange, 2026, 上櫃公司基本資料 (t187ap03_O); Securities and
74
+ Futures Bureau, Financial Supervisory Commission and Securities Investment
75
+ Trust & Consulting Association, 2026, 投信投顧公會境內基金短線交易限制
76
+ (data.gov.tw dataset 21404); each version of 2026-11-12, licensed under the
77
+ Open Government Data License, version 1.0 (https://data.gov.tw/license).
78
+ Edited: stock code, unified business number, name, listing date and an ISIN
79
+ derived from the code (checked against Wikidata); fund and class ISINs.
80
+ - FINOS Common Domain Model 7.9.0, the CDM Working Group, Community
81
+ Specification License 1.0
82
+ (https://github.com/finos/common-domain-model/tree/7.9.0): the
83
+ FloatingRateIndexEnum display names, keyed INDEX:{name}.
84
+ - Trust Wallet assets (github.com/trustwallet/assets), Copyright (c)
85
+ 2019-2023 Trust Wallet, and ethereum-lists/chains
86
+ (github.com/ethereum-lists/chains), Copyright (c) 2026, each under the MIT
87
+ License: token addresses and chain ids only, no logos.
88
+ - cosmos/chain-registry (github.com/cosmos/chain-registry) at commit
89
+ d6d22e51f4b520ed993f3a939ab64d7979e2fcd3, licensed under a Creative
90
+ Commons Attribution 4.0 International License
91
+ (https://creativecommons.org/licenses/by/4.0/): chain ids and coin types,
92
+ written as CAIP-2 and CAIP-19 slip44 ids (CC0), and decimals only; no
93
+ denoms, CoinGecko ids, logos, descriptions or socials.
94
+ - ChainAgnostic/namespaces (github.com/ChainAgnostic/namespaces), each
95
+ namespace's CAIP-2 page, copyright and related rights waived via CC0 1.0
96
+ (terms read 2026-11-12): the CAIP-2 chain id of each digital asset native
97
+ to one chain (scripts/seed_fx_pairs.py CHAINS), on its spot pairs; and,
98
+ from the CAIP-19 specification (github.com/ChainAgnostic/CAIPs, CC0 1.0)
99
+ and the stellar and xrpl namespaces' CAIP-19 pages, the CAIP-19 types of
100
+ BTC, ETH, LTC, XLM and XRP (ASSET_TYPES).
101
+ - Not embedded, recorded as read on 2026-11-12: SLIP-0044
102
+ (github.com/satoshilabs/slips, CC BY-SA 4.0), the coin types every other
103
+ CAIP-19 type needs, and the DTI registry of the Digital Token Identifier
104
+ Foundation (dtif.org; its terms grant use, download and redistribution
105
+ of the Registry on condition that it is not modified so as to mislead and
106
+ no one is charged for it, but its download needs a DTIF account): both are
107
+ folded by a person into their own store with scripts/seed_local.py's
108
+ asset mode.
109
+ - Source: BIS Triennial Central Bank Survey 2025, annex Table 25.2, OTC
110
+ foreign exchange turnover by currency (www.bis.org, read 2026-11-12; the
111
+ BIS permits reproduction with the source stated): the ranking of the 30
112
+ most traded currencies (scripts/seed_fx_pairs.py MOST_TRADED), which
113
+ orders the FX pairs' tiers; no figure is copied.
114
+ - Contains public sector information licensed under the Open Government
115
+ Licence v3.0 (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/);
116
+ source: UK Debt Management Office, gilts in issue (report D1A,
117
+ https://www.dmo.gov.uk/terms-of-use/).
118
+ - Source: Dutch State Treasury Agency (DSTA), DSL issuance results; CC0 1.0
119
+ (https://creativecommons.org/publicdomain/zero/1.0/,
120
+ https://english.dsta.nl/copyright).
121
+ - Quelle: Deutsche Bundesbank; fields copied unaltered
122
+ (https://www.bundesbank.de/de/startseite/benutzerhinweise/nutzungsbedingungen-fuer-den-allgemeinen-gebrauch-der-website-763554):
123
+ the listed Federal securities.
124
+ - Source: Hong Kong Monetary Authority, via DATA.GOV.HK
125
+ (https://data.gov.hk/en/terms-and-conditions); the data is the Government
126
+ of the HKSAR's and the Hong Kong Monetary Authority's: Exchange Fund Bills
127
+ and Notes and HKSAR Government bonds outstanding, with their XHKG stock codes.
128
+ - Fuente: D. G. del Tesoro y Política Financiera (https://www.tesoro.es/aviso-legal),
129
+ reproduced faithfully; this information may be obtained free of charge at
130
+ https://www.tesoro.es/deuda-publica/valores-del-tesoro/valores-en-circulacion.
131
+ - Source: U.S. Department of the Treasury, Bureau of the Fiscal Service, Fiscal
132
+ Data (https://fiscaldata.treasury.gov/about-us/): the Monthly Statement of the
133
+ Public Debt, table III (detail of marketable Treasury securities outstanding);
134
+ "The data on this site is available to copy, adapt, redistribute, or
135
+ otherwise use for non-commercial and commercial purposes." (terms read
136
+ 2026-11-12). The rows' identifiers are the issuer's own as published in
137
+ that file: each ISIN is the Treasury's published CUSIP under the US prefix,
138
+ closed by its ISO 6166 check digit.
139
+ - Source: ECB, modified (https://www.ecb.europa.eu/services/using-our-site/disclaimer/html/index.en.html):
140
+ the Eurosystem's list of eligible marketable assets, its central-government
141
+ lines; this information may be obtained free of charge at
142
+ https://www.ecb.europa.eu/mopo/coll/assets/html/list-MID.en.html. Terms read
143
+ 2026-11-12: "users of this website may make free use of the information
144
+ obtained directly from it subject to the following conditions: When such
145
+ information is distributed or reproduced, it must appear accurately and the
146
+ ECB must be cited as the source." The modification: each line's issuer is
147
+ stated as its government's LEI and its CFI from the list's asset type and
148
+ coupon definition.
149
+ - 出典:財務省ウェブサイト(https://www.mof.go.jp/jgbs/reference/appendix/index.htm);
150
+ 「国債の入札結果ヒストリカルデータ」(財務省)を加工して作成;
151
+ 公共データ利用規約(第1.0版)(https://www.digital.go.jp/resources/open_data/public_data_license_v1.0). Terms
152
+ read 2026-11-12 (https://www.mof.go.jp/about_mof/notice/index.html): the
153
+ site's content is under PDL 1.0, compatible with CC BY 4.0, commercial use
154
+ allowed. The edit: each series' ISIN is derived by the Securities
155
+ Identification Code Committee's published rule (the rule only, not its
156
+ document); the MOF states none.
157
+ - Source: Bank of Canada (https://www.bankofcanada.ca/terms/), modified: its
158
+ Valet auction records of Government of Canada bonds and treasury bills; this
159
+ content is available on that website free of charge. Terms read 2026-11-12:
160
+ "the Bank permits you to freely use, copy, distribute and transmit its
161
+ website content" with attribution, changes indicated and, where sold, notice
162
+ that it is free on the Bank's website. The rows' identifiers are the
163
+ issuer's own as published in that file. The change: each security is read
164
+ once, at its latest maturity, with its government's LEI and a CFI.
165
+ - Source: Finansinspektionen (fi.se), the PDMR transactions register, on
166
+ the date the seed's provenance (yggio.seed.provenance) records; FI's data
167
+ may be freely processed and re-used. Only the
168
+ instrument's ISIN, issuer LEI, name, currency and venue are kept; no person
169
+ is read.
170
+ - 出典:国税庁法人番号公表サイト(国税庁); 国税庁法人番号公表サイト(国税庁)
171
+ (https://www.houjin-bangou.nta.go.jp/download/zenken/index.html)を加工して作成.
172
+ Public Data License 1.0; edited: corporate numbers of entities the seed
173
+ holds, as LEI rows' aliases.
174
+ - ASIC - Company Dataset, Australian Securities and Investments Commission,
175
+ https://data.gov.au/data/dataset/asic-companies, licensed under Creative
176
+ Commons Attribution 3.0 Australia; changed: only the ACN of companies the
177
+ seed holds an LEI of is kept.
178
+ - Contains information from Entities Registered with ACRA accessed on
179
+ the date the seed's provenance (yggio.seed.provenance) records, from
180
+ data.gov.sg which is made available under the terms of the
181
+ Singapore Open Data Licence version 1.0 https://data.gov.sg/open-data-licence;
182
+ only UENs of entities the seed holds, no address.
183
+ - Unicode CLDR 49 (release-49), Copyright © 1991-2026 Unicode, Inc., under
184
+ the Unicode License v3 (https://www.unicode.org/license.txt; its text is
185
+ below): currency
186
+ codes, numeric codes, minor units, regional validity and English names, and
187
+ the country names (rust/src/country/tables.rs and the CCY: and COUNTRY:
188
+ rows), the offshore renminbi CNH among the codes (read 2026-11-12: CLDR
189
+ keeps it current for China beside CNY; ISO 4217 assigns it no code), and
190
+ every code CLDR has ended in every region with the day it ended.
191
+ - Listed futures and option series (the hand document's FF: and OC: rows
192
+ and the INDEX: rows they are written on), stated by hand on 2026-11-12
193
+ from each venue's public contract specification: the contract's root,
194
+ month cycle, expiry day, multiplier and tick, which are facts of the
195
+ contract. No venue's product list or file is copied. The bulk product
196
+ lists stay local (scripts/seed_local.py). Their terms were read on
197
+ 2026-11-12:
198
+ - Cboe (cboe.com/terms) and Eurex (eurex.com disclaimer): no
199
+ distribution without written consent.
200
+ - ICE (ice.com/terms-of-use): personal, non-commercial use, no extraction.
201
+ - HKEX (Terms of Use): no distribution without written permission.
202
+ - TMX (tmx.com/en/terms-of-use): republishing needs prior written
203
+ permission.
204
+ - CME Group, OCC, Euronext, Nasdaq Nordic, JPX, KRX, NSE, ASX, SGX,
205
+ TAIFEX, BME/MEFF and Borsa Italiana: terms unreadable from the
206
+ research host, so treated as not confirmed and kept local.
207
+ - The bases those futures are written on, stated by hand on 2026-11-14:
208
+ the commodities INDEX:COPPER (lbs), INDEX:CORN, INDEX:SOYBEANS and
209
+ INDEX:WHEAT (Bu), each in USD - physical facts of the deliverable,
210
+ named after the contract specification above, no venue page copied - and
211
+ the identity alone (name, symbol, currency, an ISIN its administrator
212
+ states) of four indices, no level, constituent or weight: INDEX:VIX
213
+ (Cboe, cboe.com/tradable_products/vix; terms cboe.com/terms: "You may not
214
+ otherwise copy, reproduce, alter, store ... create a derivative work"),
215
+ DE000A0C3QF1 (VSTOXX, STOXX Ltd, stoxx.com/index/v2tx: EUR, V2TX, DE000A0C3QF1;
216
+ terms pages refused the research host), INDEX:XIN9 (FTSE Russell, the
217
+ XIN9 factsheet: base currency CNH; its disclaimer: "No part of this
218
+ information may be reproduced ... without prior written permission") and
219
+ INDEX:HSCEI (Hang Seng Indexes, the hsceie factsheet: HKD; its disclaimer
220
+ page unreadable from the research host). Every page read 2026-11-14. The
221
+ futures on bond deliverable baskets (FGBL, FGBM, FGBS, FGBX, ZB, ZF, ZN,
222
+ ZT) are written on no single base and state none.
223
+
224
+ Underlyings share-alike partition (config/seeds/underlyings-sharealike.parquet,
225
+ rust/market/src/underlying/seed-sharealike.parquet)
226
+
227
+ This file is kept apart from the permissive seed: it is the derived database
228
+ of the sources below, offered under each one's share-alike licence. Its rows
229
+ hold codes the seed holds none of.
230
+
231
+ - Contains information from Comissão de Valores Mobiliários (CVM), Cias
232
+ Abertas: Formulário Cadastral (FCA) (https://dados.cvm.gov.br/dataset/cia_aberta-doc-fca),
233
+ made available under the Open Database License (ODbL) v1.0
234
+ (https://opendatacommons.org/licenses/odbl/1-0/); this partition is a
235
+ derived database of it, offered under the ODbL v1.0. ISINs from Wikidata
236
+ (CC0 1.0).
237
+ - Superintendencia Financiera de Colombia - SUPERFINANCIERA, Formato 351
238
+ (https://www.datos.gov.co/d/ur2p-h4yf), licensed under Creative Commons
239
+ Attribution-ShareAlike 4.0 International
240
+ (https://creativecommons.org/licenses/by-sa/4.0/); adapted: instrument
241
+ ISINs and nemotécnicos only, no holdings or prices; this partition is the
242
+ adapted database, offered under CC BY-SA 4.0.
243
+
244
+ FX rates seeds (config/seeds/fxrates.parquet, rust/market/src/fxrates/seed.parquet,
245
+ config/seeds/fxrates/<frequency>/<year>.parquet)
246
+
247
+ The FX rates seeds are written by rust/market/src/fxrates/seed.rs from the
248
+ publishers' own files, each read by the crate's adapter for it; each file's
249
+ footer names its sources and when they were fetched. The embedded seed holds
250
+ the ECB's and the World Bank's rows; the repository's files every source's:
251
+
252
+ - European Central Bank (www.ecb.europa.eu): Source: ECB, modified
253
+ (https://www.ecb.europa.eu/home/disclaimer/html/index.en.html): cite the
254
+ ECB; state modifications; tell buyers it is free if the data goes into a
255
+ sold product. Modified: the euro foreign exchange reference rates, each as
256
+ the pair EUR/{CCY} at 14:15 Europe/Berlin of its date, the currencies the
257
+ ECB no longer fixes included.
258
+ - Source: Board of Governors of the Federal Reserve System (H.10), public
259
+ domain (https://www.federalreserve.gov/disclaimer.htm). Modified: the noon
260
+ buying rates in New York, each at 12:00 America/New_York of its date, a
261
+ series quoted in US dollars per unit as the pair {CCY}/USD, every other as
262
+ USD/{CCY}.
263
+ - Source: BIS (WS_XRU): the use of the statistics is unrestricted provided
264
+ the BIS is cited as the source, the use is not potentially misleading, and
265
+ inclusion in a commercial product results in no additional charge
266
+ (https://data.bis.org/help/legal). Modified: the daily bilateral US dollar
267
+ exchange rates alone, each as the pair USD/{CCY} at 00:00 UTC of its day.
268
+ - Source: World Bank Commodity Price Data (The Pink Sheet), CC BY 4.0
269
+ (https://datacatalog.worldbank.org/public-licenses). Modified: the monthly
270
+ average prices of Brent, WTI and Henry Hub natural gas and of gold, silver
271
+ and platinum alone, each dated the first instant UTC of its month.
272
+
273
+ The MIT License: Permission is hereby granted, free of charge, to any person
274
+ obtaining a copy of this software and associated documentation files (the
275
+ "Software"), to deal in the Software without restriction, including without
276
+ limitation the rights to use, copy, modify, merge, publish, distribute,
277
+ sublicense, and/or sell copies of the Software, and to permit persons to whom
278
+ the Software is furnished to do so, subject to the following conditions: The
279
+ above copyright notice and this permission notice shall be included in all
280
+ copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS
281
+ IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
282
+ LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
283
+ PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
284
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
285
+ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
286
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
287
+
288
+ Unicode License v3 (Unicode CLDR 49)
289
+
290
+ UNICODE LICENSE V3
291
+
292
+ COPYRIGHT AND PERMISSION NOTICE
293
+
294
+ Copyright © 1991-2026 Unicode, Inc.
295
+
296
+ NOTICE TO USER: Carefully read the following legal agreement. BY
297
+ DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
298
+ SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
299
+ TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
300
+ DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
301
+
302
+ Permission is hereby granted, free of charge, to any person obtaining a
303
+ copy of data files and any associated documentation (the "Data Files") or
304
+ software and any associated documentation (the "Software") to deal in the
305
+ Data Files or Software without restriction, including without limitation
306
+ the rights to use, copy, modify, merge, publish, distribute, and/or sell
307
+ copies of the Data Files or Software, and to permit persons to whom the
308
+ Data Files or Software are furnished to do so, provided that either (a)
309
+ this copyright and permission notice appear with all copies of the Data
310
+ Files or Software, or (b) this copyright and permission notice appear in
311
+ associated Documentation.
312
+
313
+ THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
314
+ KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
315
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
316
+ THIRD PARTY RIGHTS.
317
+
318
+ IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
319
+ BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
320
+ OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
321
+ WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
322
+ ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
323
+ FILES OR SOFTWARE.
324
+
325
+ Except as contained in this notice, the name of a copyright holder shall
326
+ not be used in advertising or otherwise to promote the sale, use or other
327
+ dealings in these Data Files or Software without prior written
328
+ authorization of the copyright holder.
329
+
330
+ Clear Street FIX documentation (Apache License 2.0)
331
+
332
+ The vendor FIX dialect in config/fix/dialects/clear-street/ states fields
333
+ from the Clear Street FIX documentation, https://clear-street.github.io/FIX-docs,
334
+ licensed under the Apache License, Version 2.0.
package/README.md CHANGED
@@ -300,33 +300,37 @@ const path = require('node:path')
300
300
  const arrow = require('apache-arrow')
301
301
  const { Field, IOBase, fields, iceberg } = require('yggio')
302
302
 
303
- // A location is absolute: a handle is a URL, and a relative name names no root.
304
- const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggio-'))
305
-
306
- const schema = iceberg.assignFieldIds(
307
- fields.struct('row', [Field.from('id: int64'), Field.from('venue: utf8')], {
308
- nullable: false,
309
- }),
310
- )
311
- const rows = new arrow.Table({
312
- id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
313
- venue: arrow.vectorFromArray(['XNAS', 'XNYS'], new arrow.Utf8()),
314
- })
303
+ async function main() {
304
+ // A location is absolute: a handle is a URL, and a relative name names no root.
305
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggio-'))
306
+
307
+ const schema = iceberg.assignFieldIds(
308
+ fields.struct('row', [Field.from('id: int64'), Field.from('venue: utf8')], {
309
+ nullable: false,
310
+ }),
311
+ )
312
+ const rows = new arrow.Table({
313
+ id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
314
+ venue: arrow.vectorFromArray(['XNAS', 'XNYS'], new arrow.Utf8()),
315
+ })
316
+
317
+ // A handle reads and writes records through the encoding its name declares.
318
+ const file = new IOBase(path.join(root, 'trades.parquet'))
319
+ const written = await file.overwriteArrowTable(rows)
320
+ console.assert(written.toString() === 'read 2 rows, wrote 2, skipped 0')
321
+ console.assert(file.readArrowReader().intoTable().numRows === 2)
322
+ console.assert(file.isIo())
323
+ console.assert(file.rowSize === 2)
324
+ console.assert(file.columnSize === 2)
325
+
326
+ // An Iceberg table is a folder, and a folder is all it ever touches.
327
+ const table = iceberg.IcebergTable.create(path.join(root, 'trades'), schema, ['venue'])
328
+ await table.append(rows)
329
+ console.assert(table.currentSnapshot.operation === 'append')
330
+ console.assert(table.dataFiles().length === 2)
331
+ }
315
332
 
316
- // A handle reads and writes records through the encoding its name declares.
317
- const file = new IOBase(path.join(root, 'trades.parquet'))
318
- const written = file.overwriteArrowTable(rows)
319
- console.assert(written.toString() === 'read 2 rows, wrote 2, skipped 0')
320
- console.assert(file.readArrowReader().intoTable().numRows === 2)
321
- console.assert(file.isIo())
322
- console.assert(file.rowSize === 2)
323
- console.assert(file.columnSize === 2)
324
-
325
- // An Iceberg table is a folder, and a folder is all it ever touches.
326
- const table = iceberg.IcebergTable.create(path.join(root, 'trades'), schema, ['venue'])
327
- table.append(rows)
328
- console.assert(table.currentSnapshot.operation === 'append')
329
- console.assert(table.dataFiles().length === 2)
333
+ main()
330
334
  ```
331
335
 
332
336
  `PartitionSpec`, `PartitionField`, `Snapshot`, `SnapshotRef`, `ManifestFile`,
@@ -358,8 +362,10 @@ Configured intent uses `writeArrowReader`, `writeArrowTable`,
358
362
  The required mode is `'overwrite'`, `'append'`, or `'merge'` and is checked
359
363
  before a one-shot reader, exporter, or iterable is inspected.
360
364
 
361
- Every record write answers an `IOResult` - an async record write a promise of
362
- one - stating the rows it read from its source (`readRows`), wrote
365
+ Every record write answers a promise of an `IOResult`, its core write run on
366
+ the libuv pool, and every Iceberg commit (`append`, `overwrite`,
367
+ `overwriteWhere`, `merge`, `mergeWhere`, a schema update's `commit`) a promise
368
+ of its own answer; a refusal is a rejected promise. The `IOResult` states the rows it read from its source (`readRows`), wrote
363
369
  (`writtenRows`) and skipped (`skippedRows`: kept out by the options' `where`,
364
370
  or cut off a last batch by a bound), as the core counted them. A write cut
365
371
  into several commits answers their sum; `add` sums two results the same way.
@@ -373,20 +379,22 @@ computed fresh after `close()`; JavaScript keeps no parallel count or schema.
373
379
  Counts saturate at `Number.MAX_SAFE_INTEGER` rather than becoming imprecise.
374
380
 
375
381
  Record iterables cross in bounded IPC chunks (`options.batchRowSize`, or
376
- 65,536 rows), each chunk one batch. Synchronous iterables are pulled lazily through one native reader. With
377
- no `options.commitBatchNum`, async iterables spool those bounded chunks to a
378
- private temporary file before the one native write, preserving the
379
- destination's own cadence while bounding memory; the spool is removed on
380
- success or failure.
382
+ 65,536 rows), each chunk one batch. A native write runs on a worker thread that cannot call
383
+ back into JavaScript, so a record iterable - synchronous or asynchronous - is
384
+ pulled on the JavaScript thread, one bounded chunk at a time. With no
385
+ `options.commitBatchNum` the chunks are spooled to one private temporary file,
386
+ which a native reader replays for the one write and one publication; the spool
387
+ is removed on success or failure. A handle is refused by name (`a write is in
388
+ flight on this handle`) if it is used before the write on it is awaited.
381
389
 
382
390
  A positive `options.commitBatchNum` publishes every that many whole batches,
383
391
  then the remainder; a cadence never cuts a batch, so a chunk is the unit it
384
392
  counts.
385
- The async path alternates one awaited chunk with one opaque Rust-session push,
393
+ With a cadence the path alternates one chunk (awaited, for an async source) with one push,
386
394
  so every complete cadence is visible before the next source pull and a later
387
395
  failure drops only the incomplete cadence. Global row and byte limits remain
388
396
  one operation-wide budget. A zero limit does not inspect the source: append is
389
- a synchronous no-op answering the empty `IOResult`, overwrite publishes an
397
+ a no-op answering the empty `IOResult`, overwrite publishes an
390
398
  explicitly typed empty value, and a limited merge is rejected.
391
399
 
392
400
  Every page of the [documentation](https://platob.github.io/yggio/) shows its
@@ -0,0 +1,44 @@
1
+ 'use strict'
2
+
3
+ // Boundary cost of the expression layer over a serie: a Selector, a Filter
4
+ // and a Plan applied to one record column, and a Plan run end to end. The row
5
+ // count is YGGIO_BENCH_ROWS (default 1000000).
6
+
7
+ const { performance } = require('node:perf_hooks')
8
+
9
+ const arrow = require('apache-arrow')
10
+
11
+ const { Filter, Plan, Selector, Serie } = require('yggio')
12
+
13
+ const rows = Number.parseInt(process.env.YGGIO_BENCH_ROWS ?? '1000000', 10)
14
+ if (!Number.isSafeInteger(rows) || rows <= 0) {
15
+ throw new RangeError('YGGIO_BENCH_ROWS must be a positive safe integer')
16
+ }
17
+
18
+ const table = arrow.tableFromArrays({
19
+ id: Float64Array.from({ length: rows }, (_, index) => index),
20
+ price: Float64Array.from({ length: rows }, (_, index) => (index % 997) + 0.5),
21
+ })
22
+ const serie = Serie.fromArrowBatch(table.batches[0])
23
+
24
+ function benchmark(name, operation) {
25
+ operation()
26
+ const runs = []
27
+ for (let round = 0; round < 5; round += 1) {
28
+ const started = performance.now()
29
+ operation()
30
+ runs.push(performance.now() - started)
31
+ }
32
+ runs.sort((a, b) => a - b)
33
+ console.log(`${name.padEnd(44)} ${((runs[2] * 1e6) / rows).toFixed(1).padStart(10)} ns/row`)
34
+ }
35
+
36
+ const selector = new Selector('id, price * 2 as doubled')
37
+ const filter = new Filter('price > 500')
38
+ const plan = new Plan('select id, price * 2 as doubled from rows where price > 500')
39
+ const where = new Plan('where price > 500')
40
+
41
+ benchmark('Selector.applySerie', () => selector.applySerie(serie).length)
42
+ benchmark('Filter.applySerie', () => filter.applySerie(serie).length)
43
+ benchmark('Plan(where).applySerie', () => where.applySerie(serie).length)
44
+ benchmark('Plan(select where).applySerie', () => plan.applySerie(serie).length)
package/benchmarks/fix.js CHANGED
@@ -42,6 +42,16 @@ function benchmarkStreams(name, rounds, operation) {
42
42
  console.log(`${name}: ${rate.toLocaleString('en-US')} operations/second`)
43
43
  }
44
44
 
45
+ // A stream answered as a Promise: the same rounds, each awaited.
46
+ async function benchmarkPromises(name, rounds, operation) {
47
+ for (let index = 0; index < Math.min(rounds, 5); index += 1) await operation()
48
+ const started = performance.now()
49
+ for (let index = 0; index < rounds; index += 1) await operation()
50
+ const elapsed = performance.now() - started
51
+ const rate = Math.round((rounds * 1_000) / elapsed)
52
+ console.log(`${name}: ${rate.toLocaleString('en-US')} operations/second`)
53
+ }
54
+
45
55
  // A load costs milliseconds, so it runs a hundredth of the hit count.
46
56
  function benchmarkLoad(name, operation) {
47
57
  const rounds = Math.max(1, Math.round(iterations / 1_000))
@@ -209,6 +219,7 @@ if (drain(new fix.MarketMessages(msgpluginidOptions).fromTextLines(MSGPLUGINID_L
209
219
  if (drain(new fix.MarketMessages(seedOptions).fromLines(LINES)) !== LINES.length) throw new Error('line cardinality mismatch')
210
220
  const sink = { write() {} }
211
221
 
222
+ ;(async () => {
212
223
  try {
213
224
  benchmark('fix/tag_hit', () => registry.getFieldByTag(55))
214
225
  benchmark('fix/alternate_tag_hit', () => registry.getFieldByTag(20))
@@ -331,7 +342,7 @@ try {
331
342
  benchmarkStreams(`fix/arrow_reader_over_parse_lines/${LINES.length}`, streams, () =>
332
343
  new fix.MarketMessages(seedOptions).fromLines(LINES).intoArrowReader(fixedSchema).intoTable().numRows,
333
344
  )
334
- benchmarkStreams(`fix/write_arrow_reader/${LINES.length}`, streams, () =>
345
+ await benchmarkPromises(`fix/write_arrow_reader/${LINES.length}`, streams, () =>
335
346
  new fix.MarketMessages(seedOptions).fromArrowReader(BatchReader.fromIpc(parsedIpc)).intoWriter(sink),
336
347
  )
337
348
  benchmarkLoad('fix/from_handle_seed', () => fix.FixRegistry.fromHandle(SEED))
@@ -341,3 +352,4 @@ try {
341
352
  } finally {
342
353
  fs.rmSync(workspace, { recursive: true, force: true })
343
354
  }
355
+ })()
@@ -0,0 +1,51 @@
1
+ 'use strict'
2
+
3
+ // Boundary cost of bulk FIX parse: one stream crossing against one call per
4
+ // message. `MarketMessages.fromLines` pulls an array of lines through one
5
+ // native stream and `intoSerie` reads it on a worker thread and answers a held Serie; `MarketMessage.fromFixMessage`
6
+ // crosses once per line. ClOrdID and ExecID vary per line. The count is
7
+ // YGGIO_BENCH_MESSAGES (default 10000).
8
+
9
+ const path = require('node:path')
10
+ const { performance } = require('node:perf_hooks')
11
+
12
+ const { fix } = require('yggio')
13
+
14
+ const count = Number.parseInt(process.env.YGGIO_BENCH_MESSAGES ?? '10000', 10)
15
+ if (!Number.isSafeInteger(count) || count <= 0) {
16
+ throw new RangeError('YGGIO_BENCH_MESSAGES must be a positive safe integer')
17
+ }
18
+
19
+ const SEED = path.join(__dirname, '..', '..', 'config', 'seeds', 'fix.parquet')
20
+ const options = new fix.MarketMessagesOptions(fix.FixRegistry.fromHandle(SEED))
21
+ const texts = Array.from(
22
+ { length: count },
23
+ (_, index) =>
24
+ `8=FIX.4.4|35=8|37=O1|11=C${index + 1}|17=E${index + 1}|150=F|39=2|40=2|59=1|54=1|55=AAPL|15=USD|` +
25
+ '32=10|31=101|30=XNAS|60=20261030-10:00:00|10=0|',
26
+ )
27
+ const lines = texts.map((text) => Buffer.from(text))
28
+
29
+ async function benchmark(name, operation) {
30
+ await operation()
31
+ const runs = []
32
+ for (let round = 0; round < 5; round += 1) {
33
+ const started = performance.now()
34
+ await operation()
35
+ runs.push(performance.now() - started)
36
+ }
37
+ runs.sort((a, b) => a - b)
38
+ const nanoseconds = (runs[2] * 1e6) / count
39
+ console.log(`${name.padEnd(44)} ${nanoseconds.toFixed(1).padStart(12)} ns/message`)
40
+ }
41
+
42
+ const handler = () => new fix.MarketMessages(options)
43
+ async function main() {
44
+ await benchmark('fromLines(Buffer).intoSerie().length', async () => (await handler().fromLines(lines).intoSerie()).length)
45
+ await benchmark('fromLines(string).intoSerie().length', async () => (await handler().fromLines(texts).intoSerie()).length)
46
+ await benchmark('fromFixMessage per line', () => {
47
+ for (const line of lines) fix.MarketMessage.fromFixMessage(line, options)
48
+ })
49
+ }
50
+
51
+ main()
@@ -0,0 +1,45 @@
1
+ 'use strict'
2
+
3
+ // Boundary cost of the FIX view on one message: a keyed read against the
4
+ // typed getters and the entries tree. Mirrors python/benchmarks/fix_view.py;
5
+ // Node has no `getEntryByTag`, so `getByTag(32)` and `get(32)` stand for it.
6
+ // The count is YGGIO_BENCH_ITERATIONS (default 20000).
7
+
8
+ const path = require('node:path')
9
+ const { performance } = require('node:perf_hooks')
10
+
11
+ const { fix } = require('yggio')
12
+
13
+ const iterations = Number.parseInt(process.env.YGGIO_BENCH_ITERATIONS ?? '20000', 10)
14
+ if (!Number.isSafeInteger(iterations) || iterations <= 0) {
15
+ throw new RangeError('YGGIO_BENCH_ITERATIONS must be a positive safe integer')
16
+ }
17
+
18
+ const SEED = path.join(__dirname, '..', '..', 'config', 'seeds', 'fix.parquet')
19
+ const FILL = Buffer.from(
20
+ '8=FIX.4.4|35=8|37=O1|11=C1|17=E1|150=F|39=2|40=2|59=1|54=1|55=AAPL|15=USD|' +
21
+ '32=10|31=101|30=XNAS|60=20261030-10:00:00|10=0|',
22
+ )
23
+ const options = new fix.MarketMessagesOptions(fix.FixRegistry.fromHandle(SEED))
24
+ const message = fix.MarketMessage.fromFixMessage(FILL, options)
25
+
26
+ function measure(name, operation) {
27
+ for (let index = 0; index < Math.min(iterations, 1_000); index += 1) operation()
28
+ const runs = []
29
+ for (let round = 0; round < 5; round += 1) {
30
+ const started = performance.now()
31
+ for (let index = 0; index < iterations; index += 1) operation()
32
+ runs.push(performance.now() - started)
33
+ }
34
+ runs.sort((a, b) => a - b)
35
+ console.log(`${name.padEnd(44)} ${((runs[2] * 1e6) / iterations).toFixed(1).padStart(12)} ns/op`)
36
+ }
37
+
38
+ measure('getByTag(32)', () => message.getByTag(32))
39
+ measure('get(32)', () => message.get(32))
40
+ measure('get("LastQty")', () => message.get('LastQty'))
41
+ measure('getByTag(32).asJs()', () => message.getByTag(32).asJs())
42
+ measure('lastqty (typed getter)', () => message.lastqty)
43
+ measure('side (typed getter)', () => message.side)
44
+ measure('uuid (typed getter)', () => message.uuid)
45
+ measure('entries()', () => message.entries())