orderbook-dbengine 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,470 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: orderbook-dbengine
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Low-latency Level 2 orderbook database engine for HFT
|
|
5
|
+
Keywords: orderbook,hft,database,trading,low-latency
|
|
6
|
+
Author: kmacewicz
|
|
7
|
+
License: Apache-2.0
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
11
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
12
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
13
|
+
Classifier: Operating System :: MacOS
|
|
14
|
+
Classifier: Programming Language :: C++
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Topic :: Database :: Database Engines/Servers
|
|
17
|
+
Classifier: Topic :: Office/Business :: Financial :: Investment
|
|
18
|
+
Project-URL: Homepage, https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine
|
|
19
|
+
Project-URL: Repository, https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine
|
|
20
|
+
Project-URL: Documentation, https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine/tree/master/docs
|
|
21
|
+
Requires-Python: >=3.9
|
|
22
|
+
Provides-Extra: compression
|
|
23
|
+
Requires-Dist: lz4>=4.0; extra == "compression"
|
|
24
|
+
Provides-Extra: test
|
|
25
|
+
Requires-Dist: pytest>=7.0; extra == "test"
|
|
26
|
+
Requires-Dist: pytest-timeout>=2.1; extra == "test"
|
|
27
|
+
Requires-Dist: requests>=2.28; extra == "test"
|
|
28
|
+
Requires-Dist: lz4>=4.0; extra == "test"
|
|
29
|
+
Description-Content-Type: text/markdown
|
|
30
|
+
|
|
31
|
+
# orderbook-dbengine
|
|
32
|
+
|
|
33
|
+
[](https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine/actions/workflows/ci.yml)
|
|
34
|
+
[](https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine/actions/workflows/codeql.yml)
|
|
35
|
+
[](LICENSE)
|
|
36
|
+
[](https://en.cppreference.com/w/cpp/20)
|
|
37
|
+
|
|
38
|
+
A purpose-built C++20 database engine for Level 2 orderbook data in high-frequency trading environments. Designed for sub-microsecond update latency and millions of updates per second on a single core — and the measured numbers, including the workloads where it **loses**, are in [How it compares](#how-it-compares) rather than in a sentence.
|
|
39
|
+
|
|
40
|
+
Built by [Smart Data Engines](https://smartdataengines.com), who build custom database engines for specific hardware and workloads.
|
|
41
|
+
|
|
42
|
+
> **Deployment notes:** the engine runs natively on the host; there is no containerised deployment path by design. The wire protocol authenticates client sessions, replication links and multi-master peers by challenge-response (`--auth-secret-file`, `--cluster-secret-file`). **All three surfaces can be encrypted** with `--tls-client`, `--tls-replication` and `--tls-multi-master` (TLS 1.3, no configurable floor). Both shipped clients verify the certificate chain *and* the name by default; on the node links TLS is **mutual** and cannot be configured otherwise, which is what binds the exchange to the connection and closes the relay that authentication alone cannot. Every surface is off by default. See [SECURITY.md](SECURITY.md).
|
|
43
|
+
|
|
44
|
+
## How it compares
|
|
45
|
+
|
|
46
|
+
Measured against natively installed competitors on one machine, by
|
|
47
|
+
`python -m benchmarks.comparative.run --rows 200000 --rounds 12`. **Control floor 1.92%** over
|
|
48
|
+
twelve interleaved rounds: this machine does not separate differences smaller than that, so anything
|
|
49
|
+
below it is reported as indistinguishable rather than as a win. That floor was **21.2%** on the
|
|
50
|
+
machine an earlier table was measured on, which is most of why the harness measures its own.
|
|
51
|
+
|
|
52
|
+
Amazon EC2 m8a.xlarge, AMD EPYC 9R45, 4 cores, 15.3 GiB, Amazon Elastic Block Store on nvme0n1p1 → nvme0n1, ext4, kernel 7.0.0, Ubuntu 26.04, GCC 15.2, Release, at `4b92718`. Clock: what `/proc/cpuinfo` read at the start, a moment's figure rather than a rated one.
|
|
53
|
+
200,000 rows, 50 symbols, 20 levels, seed 7.
|
|
54
|
+
|
|
55
|
+
| System | Version | Ingest (rows/s) | Time-range query (4000 rows) | Why it is not a like-for-like number |
|
|
56
|
+
|---|---|---|---|---|
|
|
57
|
+
| **orderbook-dbengine** | 0.1.0 | **480,898** | **0.678 ms** (0.664–0.744) | 64 updates per round trip: there is still no bulk-load path over the wire, so this sends rows/64 requests. The answer is text, read into rows by the Python client's `query_rows()` |
|
|
58
|
+
| ClickHouse | 26.9.12.8 | 1,307,364 | 1.342 ms (1.267–1.528) | the whole CSV in one request; the query through clickhouse-driver, with the uncompressed cache on |
|
|
59
|
+
| TimescaleDB | 2.30.2 / PG 16.15 | 511,001 | 0.676 ms (0.658–0.881) | `\copy` of the whole CSV; `timescaledb-tune` applied; the query through psycopg, its results in binary |
|
|
60
|
+
| kdb+ | — | NOT MEASURED | NOT MEASURED | needs a vendor registration, and whether its free edition's numbers may be published here is a licence question rather than a technical one |
|
|
61
|
+
|
|
62
|
+
**One of the four comparable pairs is a win, one a tie and two are losses**, classified by the
|
|
63
|
+
harness against its own measured floor rather than by inspection:
|
|
64
|
+
|
|
65
|
+
- **the time-range query** against ClickHouse — **49.5% apart** against a 1.92% floor. A win.
|
|
66
|
+
- **the time-range query** against TimescaleDB — **0.4% apart, inside the floor**. Indistinguishable,
|
|
67
|
+
which is the harness's word for it and not a win.
|
|
68
|
+
- **ingest** against ClickHouse — **63.2% apart** against a 1.92% floor. A loss.
|
|
69
|
+
- **ingest** against TimescaleDB — **5.9% apart** against a 1.92% floor. A loss.
|
|
70
|
+
|
|
71
|
+
**What the query column measures.** Every figure runs from the request to the answer as rows of
|
|
72
|
+
Python ints, and each system is asked through the fastest Python client measured for it: psycopg
|
|
73
|
+
with binary results for TimescaleDB, clickhouse-driver for ClickHouse, the client's own
|
|
74
|
+
`query_rows()` for this engine (#226, #229). Of the engine's 0.678 ms its reply takes 0.190 ms to
|
|
75
|
+
arrive, read and not parsed; the rest is Python turning its text into ints, where psycopg and
|
|
76
|
+
clickhouse-driver build their rows from binary in compiled code.
|
|
77
|
+
|
|
78
|
+
**What changed since the previous table, and what it is safe to say about it.** The previous table,
|
|
79
|
+
of 19 September, had this engine losing the time-range query to both, 35.6% and 21.9% apart. That
|
|
80
|
+
comparison was unequal four ways, found together (#226):
|
|
81
|
+
- the competitors' clocks stopped before they parsed their answers, and the engine's after it;
|
|
82
|
+
- the competitors were asked through their slowest Python clients, `psql` and HTTP;
|
|
83
|
+
- the engine's adapter parsed its answer the slowest of three ways measured;
|
|
84
|
+
- Python's garbage collector fell inside timed calls, every system's.
|
|
85
|
+
|
|
86
|
+
Each is corrected here. The machine changed too - the m9g.xlarge the previous table ran on has since
|
|
87
|
+
been replaced, and this is the host the competitors are installed on - and so did the engine, with segment
|
|
88
|
+
format 3 (#219), decoded columns held between queries (#220) and conditions that narrow the read
|
|
89
|
+
(#47). So no figure here is compared with that table's.
|
|
90
|
+
|
|
91
|
+
One row of this dataset is one book level, so rows and levels are the same count in this table. They
|
|
92
|
+
are **not** the same count in the paragraph below about the wire, and that difference used to be
|
|
93
|
+
published as a ratio.
|
|
94
|
+
|
|
95
|
+
The ingest column is a limit of the *protocol* rather than of the storage engine: **there is no
|
|
96
|
+
bulk-load path over the wire**, so this harness sends a round trip per 64 book updates (#141) while
|
|
97
|
+
the SQL systems receive the whole CSV in one request.
|
|
98
|
+
|
|
99
|
+
It used to say the round trip was the whole of the difference — "446,219 updates/s in process
|
|
100
|
+
against 4,012 updates/s through the wire, a factor of 111" — and that was wrong twice. Both numbers
|
|
101
|
+
said "updates" while one meant a single level and the other twenty, so a factor of twenty of the
|
|
102
|
+
111 was the word. And the remainder is not the round trip. Holding the volume and the client fixed
|
|
103
|
+
and changing only the path, at 20,000,000 levels on the m9g.xlarge an earlier table was measured on:
|
|
104
|
+
|
|
105
|
+
| | wall ns per level | server CPU ns per level |
|
|
106
|
+
|---|---|---|
|
|
107
|
+
| in process (`bench_engine BM_IngestionThroughputBatched`) | **1014** | **67** |
|
|
108
|
+
| the same update over a socket, C++ client | **1059** | 525 |
|
|
109
|
+
|
|
110
|
+
**Wall-clock throughput is the same to within about 4%.** The protocol costs about eight times the
|
|
111
|
+
storage path's CPU per level and almost nothing in throughput, because throughput is bounded by
|
|
112
|
+
neither: server CPU per level is flat at 525–530 ns across a tenfold change in volume while wall per
|
|
113
|
+
level climbs and then stops. The full series, the caveats and what is still unexplained are in
|
|
114
|
+
[`benchmarks/on-a-bigger-machine.md`](benchmarks/on-a-bigger-machine.md).
|
|
115
|
+
|
|
116
|
+
What that does not excuse is the ingest column itself, and against ClickHouse the gap **still
|
|
117
|
+
widens with volume**: at five times the rows (1,000,000, six rounds, floor 0.66%,
|
|
118
|
+
[`…-672503ce.md`](benchmarks/comparative/results/2026-10-10-672503ce.md)) ClickHouse loads **3.04×**
|
|
119
|
+
what it does at 200,000 and this engine **1.30×**, so 63.2% apart becomes **84.3% apart**.
|
|
120
|
+
|
|
121
|
+
**At five times the rows this engine is faster than TimescaleDB on ingest** — 624,447 rows/s against
|
|
122
|
+
480,363, **23.1% apart against a 0.66% floor** — where at 200,000 rows it is 5.9% slower. This engine
|
|
123
|
+
loads **1.30×** what it did and TimescaleDB **0.94×**, so the crossing is mostly this engine gaining
|
|
124
|
+
from volume. It is a cross-run comparison, which the in-run floor does not govern.
|
|
125
|
+
|
|
126
|
+
**And with 20,000 rows an answer the time-range query is ClickHouse's.** In the same run this engine
|
|
127
|
+
answered in 4.049 ms against ClickHouse's 1.908, **52.9% apart**, a loss, and TimescaleDB's 4.584,
|
|
128
|
+
**11.7% apart**, a win. Its reply took 0.908 ms of the 4.049. The rest is the Python client parsing
|
|
129
|
+
20,000 rows of text, where clickhouse-driver decodes ClickHouse's native column blocks in compiled
|
|
130
|
+
code. In Python, then, reading the engine's answer costs more than the engine takes to give it, and
|
|
131
|
+
the more rows, the more.
|
|
132
|
+
|
|
133
|
+
### What it costs, which is a different question from who finishes first
|
|
134
|
+
|
|
135
|
+
Wall-clock on a four-core box conflates "faster per core" with "uses more cores", and this engine's
|
|
136
|
+
name contains a claim about cost. Measured over 2,000,000 levels by
|
|
137
|
+
[`scripts/measure_cpu_cost.py`](scripts/measure_cpu_cost.py), counting each server's
|
|
138
|
+
`utime+stime+cutime+cstime` and each client's own CPU including its live children, in five runs on an
|
|
139
|
+
**Amazon EC2 m8a.xlarge** (4 vCPU, AMD EPYC 9R45, Ubuntu 26.04, GCC 15.2, ext4) on **7 October 2026**,
|
|
140
|
+
master `96d1dd9`, against ClickHouse 26.9.12.8 and TimescaleDB 2.30.2 on PostgreSQL 16.15, installed
|
|
141
|
+
natively and tuned as [`benchmarks/install_competitors.md`](benchmarks/install_competitors.md) says.
|
|
142
|
+
Each cell is the range of the five:
|
|
143
|
+
|
|
144
|
+
| system | wall | levels/s | server CPU | client CPU | **levels per server CPU-second** | server cores |
|
|
145
|
+
|---|---|---|---|---|---|---|
|
|
146
|
+
| orderbook | 3.149 - 3.223 s | 620,633 - 635,186 | 0.400 - 0.410 s | 3.158 - 3.229 s | **4,878,049 - 5,000,000** | **0.13** |
|
|
147
|
+
| clickhouse | 0.357 - 0.410 s | 4,875,006 - 5,601,321 | 0.890 - 1.220 s | 0.026 - 0.028 s | **1,639,344 - 2,247,191** | **2.44 - 2.97** |
|
|
148
|
+
| timescaledb | 4.064 - 4.523 s | 442,143 - 492,164 | 3.160 - 3.200 s | 0.043 - 0.063 s | 625,000 - 632,911 | 0.71 - 0.79 |
|
|
149
|
+
|
|
150
|
+
The engine's server CPU was 0.400 or 0.410 s in every run, which the kernel counts in 10 ms ticks: the
|
|
151
|
+
figures stand for 0.395 - 0.415 s, a 2.5% resolution, and its levels per CPU-second for 4.82 - 5.06
|
|
152
|
+
million. Its lowest against each competitor's highest, per server CPU-second the engine ingests
|
|
153
|
+
**2.17 times** what ClickHouse does and **7.71 times** what TimescaleDB does. This is a different
|
|
154
|
+
script from the comparative run above and carries no floor of its own, so it is given as the figures
|
|
155
|
+
and their ratio rather than borrowed into the vocabulary that run owns.
|
|
156
|
+
|
|
157
|
+
**Where it was before.** On 6 October, at `3157ce7`, the same script on the same machine put the
|
|
158
|
+
engine's server CPU at 0.290 s in all five runs - 6,896,552 levels per CPU-second, 3.07 times
|
|
159
|
+
ClickHouse's best - in segment format 2. Format 3 (#219) spends the difference at a seal, which
|
|
160
|
+
searches for each column's smallest encoding: a seal with no encodings to reuse searches all of them,
|
|
161
|
+
and in this three-second run each of the 50 symbols seals only a few times, so that cost is nearly
|
|
162
|
+
all of it - 37 - 46% more server CPU than format 2, run for run. Over long runs, which reuse what a
|
|
163
|
+
symbol's last seal chose, the server spent 3.5 - 7.5% more CPU than format 2 for the same levels and
|
|
164
|
+
ingested them faster, with a batch's p999 a sixth of format 2's (#219). On 19 September, on an
|
|
165
|
+
m9g.xlarge, the script had put the engine at 2,272,727 against ClickHouse's 1,754,386; it read the
|
|
166
|
+
engine's CPU as zero from #151 until #218.
|
|
167
|
+
|
|
168
|
+
ClickHouse still wins the clock, by spending **2.44 - 2.97 cores** where the engine spends **0.13**:
|
|
169
|
+
about twenty times the parallelism for eight to nine times the throughput. TimescaleDB costs **7.7 -
|
|
170
|
+
8.0 times** the CPU per level of the engine.
|
|
171
|
+
|
|
172
|
+
Three things belong beside that rather than after it. ClickHouse is doing **more** work per level -
|
|
173
|
+
parsing text and building compressed parts where the engine receives binary frames and appends - so
|
|
174
|
+
what this compares is the cost of loading the data each way it is loaded, not of the same work.
|
|
175
|
+
**Our client burns eight times what our server burns**: this ingest column measures the harness's
|
|
176
|
+
Python as much as the protocol, and a C++ client sends the same updates with a fraction of it - see
|
|
177
|
+
`benchmarks/wire_load.cpp` and [`benchmarks/on-a-bigger-machine.md`](benchmarks/on-a-bigger-machine.md).
|
|
178
|
+
And the bytes it keeps: by [`scripts/measure_storage.py`](scripts/measure_storage.py) on the same
|
|
179
|
+
machine, versions and day, each dataset loaded, flushed and left 90 s for the merges, bytes on disk a
|
|
180
|
+
row - the engine's segments, ClickHouse's table as the harness creates it and with five codec sets
|
|
181
|
+
an expert might choose, TimescaleDB's hypertable compressed, in one chunk spanning the data:
|
|
182
|
+
|
|
183
|
+
| dataset | rows | **orderbook** | ClickHouse as installed | ClickHouse, the best of five codec sets | TimescaleDB, compressed |
|
|
184
|
+
|---|---|---|---|---|---|
|
|
185
|
+
| Binance, diff stream: 10 pairs, 10 minutes | 266,717 | **3.78** | 7.04 | 4.31 - `GCD` before `Delta` or `T64`, `DoubleDelta` on the time, `ZSTD(3)` | 17.29 |
|
|
186
|
+
| Binance, top 20 snapshots: the same pairs and minutes | 1,347,400 | **0.53** | 1.17 | 0.87 - `ZSTD(1)`, `DoubleDelta` on the time | 12.45 |
|
|
187
|
+
| the comparative dataset: 50 synthetic symbols | 200,000 | **1.89** | 4.23 | 1.87 - `Delta` or `T64`, `DoubleDelta` on the time, `ZSTD(9)` | 7.86 |
|
|
188
|
+
|
|
189
|
+
On the two recordings of real books the engine keeps **45 - 54%** of what ClickHouse keeps as
|
|
190
|
+
installed, and **61 - 88%** of what it keeps under the best of the five codec sets measured for each;
|
|
191
|
+
on the comparative dataset's random walks it is level with that best (1.89 against 1.87) and keeps
|
|
192
|
+
45% of ClickHouse as installed. These are a seal's bytes: each symbol's data is one segment here, so
|
|
193
|
+
nothing merges, and a merge's ZSTD tier takes the recordings to about 2.94 and 0.33 bytes a row (#219).
|
|
194
|
+
As the file system allocates them, the comparative dataset's fifty segments of 4,000 rows pay a 4 KiB
|
|
195
|
+
block twice each - 3.07 bytes a row against ClickHouse's best 2.09 - where the recordings' larger
|
|
196
|
+
segments take 3.99 and 0.61 against 4.47 and 0.95. The figures come with their reading cost: a format-3
|
|
197
|
+
read decodes every column format 2 read raw. A segment's decoded columns are then held between
|
|
198
|
+
queries (`--decoded-cache-mb`, #220). A query asked again on the recordings takes 0.20 - 0.85 of format
|
|
199
|
+
2's time from the page cache on the m9g.xlarge and the m8a.xlarge (medians of three rounds' p50).
|
|
200
|
+
A segment's first read takes 1.25 - 1.55 times format 2's (#223).
|
|
201
|
+
|
|
202
|
+
The engine's share of the query column, its reply against its parse, is **measured in each run** and
|
|
203
|
+
stated in its results file rather than subtracted. An earlier version of this page carried a parsing
|
|
204
|
+
constant larger than the smallest query median in the same table, because the constant had been
|
|
205
|
+
measured on a different machine and written into the report as prose. The one after it said the
|
|
206
|
+
parse was in every system's figure, when the clocks put it in the engine's alone (#226).
|
|
207
|
+
|
|
208
|
+
Full run, with every tuning declaration and every refusal: [`benchmarks/comparative/results/2026-10-10-287a4e15.md`](benchmarks/comparative/results/2026-10-10-287a4e15.md). The write-up of
|
|
209
|
+
what the m9g.xlarge the earlier tables ran on found, including the nine defects it exposed and the
|
|
210
|
+
prediction registered before it booted: [`benchmarks/on-a-bigger-machine.md`](benchmarks/on-a-bigger-machine.md).
|
|
211
|
+
To reproduce it, install the competitors natively first, and the two Python clients the harness
|
|
212
|
+
times them through — [`benchmarks/install_competitors.md`](benchmarks/install_competitors.md); nothing
|
|
213
|
+
in the harness installs anything, and a containerised competitor would measure the container.
|
|
214
|
+
## Features
|
|
215
|
+
|
|
216
|
+
- **SoA (Struct-of-Arrays) buffer** with seqlock for lock-free concurrent reads
|
|
217
|
+
- **Write-Ahead Log (WAL)** with CRC32C checksums and crash recovery
|
|
218
|
+
- **Columnar segments on disk** — time-partitioned, every column of a segment in one checksummed
|
|
219
|
+
file, each in whichever of its candidate encodings is smallest for it (segment format 3, #219);
|
|
220
|
+
written whole when a symbol has enough rows or its oldest are ten seconds old, and merged in the
|
|
221
|
+
background into segments of up to 262 144 rows, so their number follows the data rather than
|
|
222
|
+
uptime
|
|
223
|
+
- **Aggregation engine** (VWAP, spread, mid-price, imbalance, etc.) with optional AVX2/AVX-512 SIMD,
|
|
224
|
+
reachable over the wire protocol: every result carries its scale factor and distinguishes an empty
|
|
225
|
+
aggregate from a zero
|
|
226
|
+
- **SQL-like query language** with time-range filters and aggregations. A `SELECT` answers the
|
|
227
|
+
columns it names, in the order it names them, and the server decodes only the columns needed
|
|
228
|
+
to answer it — including a column a predicate reads and the answer does not carry
|
|
229
|
+
- **The live book, on the wire** — `BOOK <symbol> <exchange> [depth]` returns the current levels of
|
|
230
|
+
both sides from the structure the engine updates in place, under one seqlock read, bids first and
|
|
231
|
+
each side in its own order. Distinct from `SELECT`, which reads history and answers with every
|
|
232
|
+
version of a level it has stored: this answers with one row per level. Measured on an m9g.xlarge
|
|
233
|
+
over loopback, ten levels per side is **9.7 µs p50** of which 8.0 µs is the round trip, and the
|
|
234
|
+
marginal cost is 45–53 ns per level. Two of the seven columns are properties of the read rather
|
|
235
|
+
than of a level, so every row carries the same `timestamp_ns` and `sequence_number` — which is
|
|
236
|
+
the number to resume a `SUBSCRIBE` from. Also in the Python client (`book()`) and in `ob_cli`
|
|
237
|
+
- **Streaming subscriptions, pushed** — `SUBSCRIBE 'SYM'.'EXCH'` over the wire and the server sends
|
|
238
|
+
rows as they are written, prefixed `PUSH <id>` with all seven columns, whatever select list the
|
|
239
|
+
subscription named: a `PUSH` line has no header, so a narrowed push would change what a field
|
|
240
|
+
means with nothing for a client to check against. A
|
|
241
|
+
bounded queue per subscriber, and a consumer that stops reading is disconnected rather than
|
|
242
|
+
allowed to grow the server's memory. Also available embedded (`Engine::subscribe()`,
|
|
243
|
+
`ob_subscribe()`) and from the Python client (`subscribe()` / `poll()`)
|
|
244
|
+
- **TCP server** — connect remotely via telnet/nc, like PostgreSQL or ClickHouse
|
|
245
|
+
- **Backup and restore** — `BACKUP` takes a cut at one moment into the server's `--backup-dir`
|
|
246
|
+
(hard links on the data directory's filesystem, a checksummed copy on another), `ob_backup` asks
|
|
247
|
+
for one from cron, and `ob_restore` checks a backup whole before it writes anything and restores
|
|
248
|
+
it into an empty directory. No restore to a moment between two backups yet: see
|
|
249
|
+
[docs/operations.md](docs/operations.md), "Backing up and restoring a node"
|
|
250
|
+
- **Multi-master replication** — write to any node, automatic conflict resolution via HLC + LWW
|
|
251
|
+
- **Fuzzed parsers** — libFuzzer harnesses over wire command parsing, multi-master framing and WAL
|
|
252
|
+
deserialization, with an in-repo corpus and a bounded run on every pull request. Each asserts a
|
|
253
|
+
property rather than merely surviving, and the harnesses themselves are checked by planting
|
|
254
|
+
defects in the parsers: see [fuzz/README.md](fuzz/README.md)
|
|
255
|
+
- **C API** for FFI integration (Python, Rust, Go, etc.)
|
|
256
|
+
- **Python bindings** — local (ctypes) or remote (TCP), same API
|
|
257
|
+
|
|
258
|
+
## Quick Start
|
|
259
|
+
|
|
260
|
+
### Build
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
|
264
|
+
cmake --build build
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
It needs GCC 12 or newer, or clang 15, and the development packages of liblz4, libzstd, libcurl and
|
|
268
|
+
OpenSSL 3; CMake refuses an older compiler, saying so (#213), and CI builds the packages with GCC 12
|
|
269
|
+
(#221). On Ubuntu 24.04:
|
|
270
|
+
`sudo apt install build-essential cmake pkg-config liblz4-dev libzstd-dev libcurl4-openssl-dev libssl-dev` - what CI
|
|
271
|
+
installs, with OpenSSL named rather than taken as libcurl's dependency. On Amazon Linux
|
|
272
|
+
2023, whose default compiler is GCC 11:
|
|
273
|
+
`sudo dnf install gcc14-c++ cmake lz4-devel libzstd-devel libcurl-devel openssl-devel`, then configure with
|
|
274
|
+
`CC=gcc14-gcc CXX=gcc14-g++` in front of the first command.
|
|
275
|
+
|
|
276
|
+
### Run the interactive CLI
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
./build/ob_cli /tmp/my_orderbook
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
```
|
|
283
|
+
ob> insert BTC-USD BINANCE bid 6500000 150
|
|
284
|
+
ob> insert BTC-USD BINANCE ask 6510000 80
|
|
285
|
+
ob> flush
|
|
286
|
+
ob> query SELECT * FROM 'BTC-USD'.'BINANCE' WHERE timestamp BETWEEN 0 AND 9999999999999999999
|
|
287
|
+
ob> quit
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
### Run the TCP server
|
|
291
|
+
|
|
292
|
+
```bash
|
|
293
|
+
./build/ob_tcp_server --port 5555 --data-dir /tmp/ob_data
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
Connect with any TCP client:
|
|
297
|
+
|
|
298
|
+
```bash
|
|
299
|
+
$ nc localhost 5555
|
|
300
|
+
OK ob_tcp_server v0.1.0
|
|
301
|
+
PING
|
|
302
|
+
PONG
|
|
303
|
+
INSERT BTC-USD BINANCE bid 6500000 150 3
|
|
304
|
+
OK
|
|
305
|
+
FLUSH
|
|
306
|
+
OK
|
|
307
|
+
SELECT * FROM 'BTC-USD'.'BINANCE' WHERE timestamp BETWEEN 0 AND 9999999999999999999
|
|
308
|
+
OK
|
|
309
|
+
timestamp_ns price quantity order_count side level sequence_number
|
|
310
|
+
1700000000000 6500000 150 3 0 0 1
|
|
311
|
+
|
|
312
|
+
QUIT
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
### Use from Python
|
|
316
|
+
|
|
317
|
+
```bash
|
|
318
|
+
pip install . # the client, pure Python
|
|
319
|
+
# local mode needs the engine's C API library: a release package installs it, or build it -
|
|
320
|
+
cmake --build build --target orderbook_shared
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
```python
|
|
324
|
+
from orderbook_engine import OrderbookEngine
|
|
325
|
+
|
|
326
|
+
# Local mode (in-process, ctypes)
|
|
327
|
+
engine = OrderbookEngine("/tmp/ob_data")
|
|
328
|
+
|
|
329
|
+
# Or TCP mode (connect to running ob_tcp_server)
|
|
330
|
+
engine = OrderbookEngine(host="192.168.1.10", port=5555)
|
|
331
|
+
|
|
332
|
+
engine.insert("BTC-USD", "BINANCE", "bid",
|
|
333
|
+
prices=[6_500_000, 6_499_000],
|
|
334
|
+
qtys=[150, 200])
|
|
335
|
+
|
|
336
|
+
# Or several updates in one round trip, with an outcome for each of them. Measured on an
|
|
337
|
+
# m9g.xlarge: 1,542,355 levels/s at 64 per call against 969,204 one at a time.
|
|
338
|
+
from orderbook_engine import BookUpdate
|
|
339
|
+
engine.insert_batch([BookUpdate("BTC-USD", "BINANCE", "bid", [6_500_000], [150]),
|
|
340
|
+
BookUpdate("ETH-USD", "BINANCE", "ask", [3_100_000], [40])])
|
|
341
|
+
engine.flush()
|
|
342
|
+
|
|
343
|
+
rows = engine.query_all("BTC-USD", "BINANCE")
|
|
344
|
+
for row in rows:
|
|
345
|
+
print(row.price, row.quantity, row.side)
|
|
346
|
+
|
|
347
|
+
engine.close()
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
### Run a Multi-Master Cluster
|
|
351
|
+
|
|
352
|
+
Start three nodes that all accept writes with automatic replication:
|
|
353
|
+
|
|
354
|
+
```bash
|
|
355
|
+
# Requires etcd running on localhost:2379
|
|
356
|
+
# Node 1
|
|
357
|
+
./build/ob_tcp_server --port 5555 --data-dir /tmp/mm1 \
|
|
358
|
+
--coordinator-endpoints http://127.0.0.1:2379 --node-id mm1 \
|
|
359
|
+
--multi-master --mm-node-id 1 --mm-replication-port 6001 --replication-port 6001
|
|
360
|
+
|
|
361
|
+
# Node 2
|
|
362
|
+
./build/ob_tcp_server --port 5556 --data-dir /tmp/mm2 \
|
|
363
|
+
--coordinator-endpoints http://127.0.0.1:2379 --node-id mm2 \
|
|
364
|
+
--multi-master --mm-node-id 2 --mm-replication-port 6002 --replication-port 6002
|
|
365
|
+
|
|
366
|
+
# Node 3
|
|
367
|
+
./build/ob_tcp_server --port 5557 --data-dir /tmp/mm3 \
|
|
368
|
+
--coordinator-endpoints http://127.0.0.1:2379 --node-id mm3 \
|
|
369
|
+
--multi-master --mm-node-id 3 --mm-replication-port 6003 --replication-port 6003
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
Write to any node — data replicates automatically:
|
|
373
|
+
|
|
374
|
+
```python
|
|
375
|
+
from orderbook_engine import OrderbookEngine
|
|
376
|
+
|
|
377
|
+
engine = OrderbookEngine(hosts=["localhost:5555", "localhost:5556", "localhost:5557"])
|
|
378
|
+
engine.insert("BTC-USD", "BINANCE", "bid", prices=[6_500_000], qtys=[150])
|
|
379
|
+
engine.close()
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
### Run tests
|
|
383
|
+
|
|
384
|
+
```bash
|
|
385
|
+
# -j1 is required: network tests bind fixed ports and fail under parallel execution.
|
|
386
|
+
ctest --test-dir build --output-on-failure -j1
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
GTest and RapidCheck. The suite's size and its measured runtime live in
|
|
390
|
+
[docs/roadmap.md](docs/roadmap.md), recorded against the commit that measured them rather than
|
|
391
|
+
repeated here — this page said "510 tests" for long enough that the number had halved against
|
|
392
|
+
reality. Integration tests additionally need a native `etcd` binary — see
|
|
393
|
+
[tests/integration/README.md](tests/integration/README.md).
|
|
394
|
+
|
|
395
|
+
### Coverage
|
|
396
|
+
|
|
397
|
+
The required `coverage` job builds the tree with instrumentation, runs the whole suite under it and
|
|
398
|
+
**fails below a 58% line floor** — so the green CI badge at the top of this page already asserts
|
|
399
|
+
that floor. It also gates three things that are not percentages: that the tree builds with
|
|
400
|
+
coverage, that the suite passes under it, and that the instrumentation still **reaches the
|
|
401
|
+
libraries**, which is the check that failed silently for as long as the option existed.
|
|
402
|
+
|
|
403
|
+
Last measured: **66.2% of 14,562 lines (9,640 covered)**, functions 78.5%, branches 36.3%, on
|
|
404
|
+
commit `d2929e4` —
|
|
405
|
+
[run 34953853236](https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine/actions/runs/34953853236).
|
|
406
|
+
The per-file breakdown is in that job's summary and attached to it as an artifact.
|
|
407
|
+
|
|
408
|
+
**There is deliberately no coverage badge**, and the reasoning is in
|
|
409
|
+
[docs/roadmap.md](docs/roadmap.md) under item 37: a percentage badge needs either a third-party
|
|
410
|
+
account with this repository's reports flowing to it or write access for CI to push the number
|
|
411
|
+
somewhere, and neither buys anything the floor above does not already assert. The figure is quoted
|
|
412
|
+
with its denominator for the same reason: before the instrumentation was fixed this repository could
|
|
413
|
+
have published "59.0%" truthfully while measuring 6 of 34 source files.
|
|
414
|
+
|
|
415
|
+
### Run benchmarks
|
|
416
|
+
|
|
417
|
+
```bash
|
|
418
|
+
# C++ (native)
|
|
419
|
+
./build/benchmarks/bench_engine
|
|
420
|
+
|
|
421
|
+
# Python
|
|
422
|
+
python python/benchmark.py
|
|
423
|
+
python python/benchmark.py --mode tcp --host 127.0.0.1 --port 5555
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
## Build Options
|
|
427
|
+
|
|
428
|
+
| Option | Default | Description |
|
|
429
|
+
|--------|---------|-------------|
|
|
430
|
+
| `OB_BUILD_TESTS` | ON | Build tests and fetch gtest/rapidcheck |
|
|
431
|
+
| `OB_BUILD_FUZZERS` | OFF | Build libFuzzer harnesses over the parsers. Requires Clang and `OB_ENABLE_ASAN`; see [fuzz/README.md](fuzz/README.md) |
|
|
432
|
+
| `OB_ENABLE_AVX2` | OFF | Enable AVX2 SIMD for aggregation |
|
|
433
|
+
| `OB_ENABLE_AVX512` | OFF | Enable AVX-512 SIMD for aggregation |
|
|
434
|
+
| `OB_ENABLE_COVERAGE` | OFF | Enable gcov/llvm-cov instrumentation |
|
|
435
|
+
| `OB_ENABLE_ASAN` | OFF | AddressSanitizer + UndefinedBehaviorSanitizer. Build in a separate tree |
|
|
436
|
+
| `OB_ENABLE_TSAN` | OFF | ThreadSanitizer. Cannot be combined with `OB_ENABLE_ASAN` |
|
|
437
|
+
|
|
438
|
+
## Project Structure
|
|
439
|
+
|
|
440
|
+
```
|
|
441
|
+
include/orderbook/ C++ headers (public API)
|
|
442
|
+
src/ Implementation files
|
|
443
|
+
tests/ Unit + property-based tests
|
|
444
|
+
benchmarks/ Google Benchmark suite
|
|
445
|
+
fuzz/ libFuzzer harnesses over the parsers, with their seed corpus
|
|
446
|
+
tools/ CLI tool (ob_cli) and TCP server (ob_tcp_server)
|
|
447
|
+
python/ Python bindings and benchmark script
|
|
448
|
+
docs/ Documentation
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
## Documentation
|
|
452
|
+
|
|
453
|
+
See the [docs/](docs/) directory:
|
|
454
|
+
|
|
455
|
+
- [Architecture Overview](docs/architecture.md)
|
|
456
|
+
- [CLI Reference](docs/cli.md)
|
|
457
|
+
- [Query Language](docs/query-language.md)
|
|
458
|
+
- [Python Bindings](docs/python.md)
|
|
459
|
+
- [C API Reference](docs/c-api.md)
|
|
460
|
+
- [Storage Format](docs/storage.md)
|
|
461
|
+
- [Operations](docs/operations.md) - backups, failover, the dashboard and alert rules
|
|
462
|
+
- [Upgrading without stopping](docs/upgrading.md)
|
|
463
|
+
- [Releasing](docs/releasing.md) - what a release contains, how one is cut and verified; [Changelog](CHANGELOG.md)
|
|
464
|
+
- [Benchmarks](benchmarks/README.md)
|
|
465
|
+
- [Roadmap](docs/roadmap.md)
|
|
466
|
+
- [Repository security](docs/github-security.md)
|
|
467
|
+
|
|
468
|
+
## License
|
|
469
|
+
|
|
470
|
+
Apache License 2.0 — see [LICENSE](LICENSE).
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
orderbook_engine/__init__.py,sha256=W1wJhxYgQiXJ7B-tRu-oFHfxRc3naxOSL65MAkJZ01s,116668
|
|
2
|
+
orderbook_engine/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
|
|
3
|
+
orderbook_dbengine-0.1.0.dist-info/METADATA,sha256=r3xo6yN_A-TN8-VeeiXuSq99s-784sOAAHgcgtPWGqM,28202
|
|
4
|
+
orderbook_dbengine-0.1.0.dist-info/WHEEL,sha256=WyBUD_oHJV7P5xhqzT5Yyx5kVVFvC--N311fBxRrW9o,95
|
|
5
|
+
orderbook_dbengine-0.1.0.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
|
|
6
|
+
orderbook_dbengine-0.1.0.dist-info/RECORD,,
|