@usherlabs/cex-broker 0.2.52 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (147) hide show
  1. package/README.md +116 -8
  2. package/dist/commands/cli.js +1515 -467
  3. package/dist/index.js +1516 -468
  4. package/dist/index.js.map +31 -26
  5. package/dist/proto/node.descriptor.ts +3 -1
  6. package/dist/proto/node.proto +2 -0
  7. package/package.json +5 -28
  8. package/dist/commands/cex-canonical-orderbook-export.d.ts +0 -23
  9. package/dist/commands/cex-canonical-orderbook-export.js +0 -34275
  10. package/dist/commands/cex-canonical-orderbook-export.js.map +0 -266
  11. package/dist/commands/market-data-vendor-backfill.d.ts +0 -24
  12. package/dist/commands/market-data-vendor-backfill.js +0 -34930
  13. package/dist/commands/market-data-vendor-backfill.js.map +0 -268
  14. package/dist/handlers/execute-action/context.d.ts +0 -50
  15. package/dist/handlers/execute-action/deposit.d.ts +0 -2
  16. package/dist/handlers/execute-action/handler.d.ts +0 -19
  17. package/dist/handlers/execute-action/index.d.ts +0 -10
  18. package/dist/handlers/execute-action/internal-transfer.d.ts +0 -2
  19. package/dist/handlers/execute-action/order-book-call.d.ts +0 -3
  20. package/dist/handlers/execute-action/orders.d.ts +0 -2
  21. package/dist/handlers/execute-action/pass-through.d.ts +0 -2
  22. package/dist/handlers/execute-action/perp-config.d.ts +0 -2
  23. package/dist/handlers/execute-action/registry.d.ts +0 -5
  24. package/dist/handlers/execute-action/treasury-call.d.ts +0 -2
  25. package/dist/handlers/execute-action/withdraw.d.ts +0 -2
  26. package/dist/handlers/subscribe/broker-lifecycle.d.ts +0 -13
  27. package/dist/handlers/subscribe/handler.d.ts +0 -20
  28. package/dist/handlers/subscribe/index.d.ts +0 -2
  29. package/dist/handlers/types.d.ts +0 -23
  30. package/dist/helpers/account-balance-archive-poller.d.ts +0 -19
  31. package/dist/helpers/auth.d.ts +0 -5
  32. package/dist/helpers/balance-update-archive-consumer.d.ts +0 -16
  33. package/dist/helpers/binance-user-data-normalization.d.ts +0 -3
  34. package/dist/helpers/binance-user-data-stream.d.ts +0 -58
  35. package/dist/helpers/broker-execution-archive/capture.d.ts +0 -41
  36. package/dist/helpers/broker-execution-archive/index.d.ts +0 -7
  37. package/dist/helpers/broker-execution-archive/journal-export.d.ts +0 -20
  38. package/dist/helpers/broker-execution-archive/loss-journal.d.ts +0 -36
  39. package/dist/helpers/broker-execution-archive/redact.d.ts +0 -7
  40. package/dist/helpers/broker-execution-archive/rows.d.ts +0 -134
  41. package/dist/helpers/broker-execution-archive/types.d.ts +0 -26
  42. package/dist/helpers/broker-execution-archive/withdrawal-observation-tracker.d.ts +0 -23
  43. package/dist/helpers/broker-execution-archive/writer.d.ts +0 -105
  44. package/dist/helpers/broker.d.ts +0 -38
  45. package/dist/helpers/canonical-orderbook-export/exact-selection.d.ts +0 -21
  46. package/dist/helpers/canonical-orderbook-export/exporter.d.ts +0 -26
  47. package/dist/helpers/canonical-orderbook-export/parquet-projection.d.ts +0 -19
  48. package/dist/helpers/constants.d.ts +0 -36
  49. package/dist/helpers/deposit-archive-poller.d.ts +0 -66
  50. package/dist/helpers/deposit.d.ts +0 -5
  51. package/dist/helpers/exchange-credentials.d.ts +0 -10
  52. package/dist/helpers/fill-archive-poller.d.ts +0 -42
  53. package/dist/helpers/grpc/broker.d.ts +0 -5
  54. package/dist/helpers/grpc/callbacks.d.ts +0 -9
  55. package/dist/helpers/grpc/payload.d.ts +0 -9
  56. package/dist/helpers/grpc/status.d.ts +0 -8
  57. package/dist/helpers/index.d.ts +0 -44
  58. package/dist/helpers/logger.d.ts +0 -13
  59. package/dist/helpers/market-data-archive/canonical-orderbook.d.ts +0 -20
  60. package/dist/helpers/market-data-archive/capture-context.d.ts +0 -53
  61. package/dist/helpers/market-data-archive/capture-contract.d.ts +0 -30
  62. package/dist/helpers/market-data-archive/capture.d.ts +0 -22
  63. package/dist/helpers/market-data-archive/index.d.ts +0 -13
  64. package/dist/helpers/market-data-archive/legacy-migration.d.ts +0 -37
  65. package/dist/helpers/market-data-archive/ohlcv-bar-tracker.d.ts +0 -11
  66. package/dist/helpers/market-data-archive/ohlcv-bootstrap.d.ts +0 -1
  67. package/dist/helpers/market-data-archive/ohlcv-history.d.ts +0 -8
  68. package/dist/helpers/market-data-archive/orderbook-depth.d.ts +0 -5
  69. package/dist/helpers/market-data-archive/orderbook-sampler.d.ts +0 -12
  70. package/dist/helpers/market-data-archive/parse-stream.d.ts +0 -26
  71. package/dist/helpers/market-data-archive/rows.d.ts +0 -14
  72. package/dist/helpers/market-data-archive/types.d.ts +0 -83
  73. package/dist/helpers/market-data-preparation/conformance-fixtures.d.ts +0 -128
  74. package/dist/helpers/market-data-preparation/contracts.d.ts +0 -2970
  75. package/dist/helpers/market-data-preparation/file-job.d.ts +0 -38
  76. package/dist/helpers/market-data-preparation/required-clock-qualification.d.ts +0 -43
  77. package/dist/helpers/market-data-preparation/source-tape-dependencies.d.ts +0 -19
  78. package/dist/helpers/market-data-preparation/source-tape-operation.d.ts +0 -109
  79. package/dist/helpers/market-data-source-forensics.d.ts +0 -400
  80. package/dist/helpers/market-data-vendor-backfill/archive-reader.d.ts +0 -26
  81. package/dist/helpers/market-data-vendor-backfill/batching.d.ts +0 -10
  82. package/dist/helpers/market-data-vendor-backfill/conformance-fixtures.d.ts +0 -78
  83. package/dist/helpers/market-data-vendor-backfill/contracts.d.ts +0 -358
  84. package/dist/helpers/market-data-vendor-backfill/core.d.ts +0 -101
  85. package/dist/helpers/market-data-vendor-backfill/cryptohftdata.d.ts +0 -118
  86. package/dist/helpers/market-data-vendor-backfill/forwarder-client.d.ts +0 -25
  87. package/dist/helpers/market-data-vendor-backfill/identity.d.ts +0 -10
  88. package/dist/helpers/market-data-vendor-backfill/manifests.d.ts +0 -61
  89. package/dist/helpers/market-data-vendor-backfill/promotion.d.ts +0 -10
  90. package/dist/helpers/market-data-vendor-backfill/qualification.d.ts +0 -20
  91. package/dist/helpers/market-data-vendor-backfill/selection.d.ts +0 -30
  92. package/dist/helpers/market-data-vendor-backfill/semantic-verification.d.ts +0 -25
  93. package/dist/helpers/market-type.d.ts +0 -16
  94. package/dist/helpers/order-activity-tracker.d.ts +0 -22
  95. package/dist/helpers/order-book.d.ts +0 -79
  96. package/dist/helpers/order-telemetry.d.ts +0 -50
  97. package/dist/helpers/otel.d.ts +0 -83
  98. package/dist/helpers/passive-order.d.ts +0 -8
  99. package/dist/helpers/public-market-data-feed/identity.d.ts +0 -12
  100. package/dist/helpers/public-market-data-feed/index.d.ts +0 -5
  101. package/dist/helpers/public-market-data-feed/orderbook-coalescing-evidence.d.ts +0 -79
  102. package/dist/helpers/public-market-data-feed/orderbook-profile.d.ts +0 -54
  103. package/dist/helpers/public-market-data-feed/subscriber-buffer.d.ts +0 -31
  104. package/dist/helpers/public-market-data-feed/supervisor.d.ts +0 -82
  105. package/dist/helpers/shared/errors.d.ts +0 -14
  106. package/dist/helpers/shared/guards.d.ts +0 -2
  107. package/dist/helpers/source-tape-sandbox.d.ts +0 -82
  108. package/dist/helpers/source-tape.d.ts +0 -165
  109. package/dist/helpers/stream-health-publisher.d.ts +0 -41
  110. package/dist/helpers/trace-context.d.ts +0 -8
  111. package/dist/helpers/transfer-network.d.ts +0 -12
  112. package/dist/helpers/travel-rule-deposit-reconciler.d.ts +0 -159
  113. package/dist/helpers/travel-rule.d.ts +0 -75
  114. package/dist/helpers/treasury-discovery.d.ts +0 -17
  115. package/dist/helpers/user-asset-archive-poller.d.ts +0 -19
  116. package/dist/helpers/user-data-stream-supervisor.d.ts +0 -27
  117. package/dist/helpers/verity.d.ts +0 -5
  118. package/dist/index.d.ts +0 -56
  119. package/dist/market-data-preparation/fixtures/conformance-v3.json +0 -621
  120. package/dist/market-data-preparation/policies/capability-policy.json +0 -39
  121. package/dist/market-data-preparation/policies/resource-policy.json +0 -16
  122. package/dist/market-data-preparation/policies/source-tape-capability-v1.json +0 -35
  123. package/dist/market-data-preparation/schema-manifest.json +0 -66
  124. package/dist/market-data-preparation/schemas/archive-selection-v1.schema.json +0 -224
  125. package/dist/market-data-preparation/schemas/backfill-request-v1.schema.json +0 -166
  126. package/dist/market-data-preparation/schemas/backfill-result-v2.schema.json +0 -212
  127. package/dist/market-data-preparation/schemas/canonical-orderbook-export-request-v1.schema.json +0 -61
  128. package/dist/market-data-preparation/schemas/canonical-orderbook-export-result-v2.schema.json +0 -252
  129. package/dist/market-data-preparation/schemas/order-book-depth-summary-parquet-projection-v1.schema.json +0 -256
  130. package/dist/market-data-preparation/schemas/order-book-levels-parquet-projection-v1.schema.json +0 -220
  131. package/dist/market-data-preparation/schemas/preparation-product-pin-v2.schema.json +0 -176
  132. package/dist/market-data-preparation/schemas/promotion-receipt-v1.schema.json +0 -175
  133. package/dist/market-data-preparation/schemas/required-clock-v1.schema.json +0 -47
  134. package/dist/market-data-preparation/schemas/source-forensics-ledger-v1.schema.json +0 -463
  135. package/dist/market-data-preparation/schemas/source-qualification-record-v1.schema.json +0 -394
  136. package/dist/market-data-preparation.d.ts +0 -11
  137. package/dist/market-data-preparation.js +0 -37488
  138. package/dist/market-data-preparation.js.map +0 -277
  139. package/dist/market-data-vendor-backfill.d.ts +0 -21
  140. package/dist/market-data-vendor-backfill.js +0 -22463
  141. package/dist/market-data-vendor-backfill.js.map +0 -199
  142. package/dist/proto/node.descriptor.d.ts +0 -127
  143. package/dist/proto-loader-options.d.ts +0 -7
  144. package/dist/proto-package-definition.d.ts +0 -2
  145. package/dist/schemas/action-payloads.d.ts +0 -88
  146. package/dist/server.d.ts +0 -10
  147. package/dist/types.d.ts +0 -100
package/README.md CHANGED
@@ -31,17 +31,20 @@ See [SERVICES_ARCHITECTURE.md](SERVICES_ARCHITECTURE.md) for the authoritative b
31
31
  ## 🛠️ Installation
32
32
 
33
33
  1. **Clone the repository:**
34
+
34
35
  ```bash
35
36
  git clone <repository-url>
36
37
  cd cex-broker
37
38
  ```
38
39
 
39
40
  2. **Install dependencies:**
41
+
40
42
  ```bash
41
43
  bun install
42
44
  ```
43
45
 
44
46
  3. **Generate protobuf types:**
47
+
45
48
  ```bash
46
49
  bun run proto-gen
47
50
  ```
@@ -137,7 +140,7 @@ The broker **defaults to spot everywhere** unless a request explicitly opts into
137
140
  Pass `marketType` in action payloads (string map) or subscribe `options`:
138
141
 
139
142
  | `marketType` | Meaning |
140
- |--------------|---------|
143
+ | -------------- | --------- |
141
144
  | omitted / `spot` | Spot markets and spot balances |
142
145
  | `swap` or `perp` | Perpetuals (resolves symbols like `ETH/USDC:USDC` on Hyperliquid) |
143
146
  | `future` | Dated futures where supported |
@@ -175,9 +178,11 @@ For split futures exchanges (`binanceusdm`, `krakenfutures`, `kucoinfutures`), r
175
178
  Two capability-gated actions complement `Action.Call`:
176
179
 
177
180
  | Action | Value | Requires CCXT | Purpose |
178
- |--------|-------|---------------|---------|
181
+ | -------- | ------- | --------------- | --------- |
179
182
  | `GetPerpConfigState` | `14` | `fetchPositions` | Read positions and per-symbol leverage/margin mode |
180
183
  | `SetPerpConfigState` | `15` | `setLeverage` | Set leverage (and margin mode) for a symbol |
184
+ | `FetchMarketRules` | `16` | `loadMarkets` | Read typed spot execution constraints for one pair |
185
+ | `Batch` | `17` | Action-specific | Execute an allowlisted sequence of read-only actions |
181
186
 
182
187
  `GetPerpConfigState` payload: optional `symbol`, optional `params` (JSON).
183
188
 
@@ -302,6 +307,7 @@ The service exposes a gRPC interface with two main methods:
302
307
  Execute trading operations on supported exchanges.
303
308
 
304
309
  **Request:**
310
+
305
311
  ```protobuf
306
312
  message ActionRequest {
307
313
  Action action = 1; // The action to perform
@@ -312,13 +318,16 @@ message ActionRequest {
312
318
  ```
313
319
 
314
320
  **Response:**
321
+
315
322
  ```protobuf
316
323
  message ActionResponse {
317
- string result = 2; // JSON string of the result data or ZK proof
324
+ string result = 1; // JSON string of the result data
325
+ string proof = 2; // Optional action-specific proof
318
326
  }
319
327
  ```
320
328
 
321
329
  **Available Actions:**
330
+
322
331
  - `NoAction` (0): No operation
323
332
  - `Deposit` (1): Confirm deposit transaction
324
333
  - `Withdraw` (2): Withdraw funds
@@ -328,8 +337,15 @@ message ActionResponse {
328
337
  - `FetchBalances` (6): Get account balances. Supports `balanceType`: "free", "used", "total" (defaults to "total").
329
338
  - `FetchDepositAddresses` (7): Get deposit addresses for a token/network
330
339
  - `FetchTicker` (8): Get ticker information
331
- - `FetchCurrency` (9): Get currency metadata (networks, fees, etc.) for a symbol
332
- - `Call` (10): Generic method invocation on the underlying broker instance. Provide `functionName`, optional `args` array, and optional `params` object.
340
+ - `FetchCurrency` (9): Get `cex-transfer-network-evidence/v1` for one asset and required network.
341
+ - `Call` (10): Generic method invocation on the underlying broker instance. Provide `functionName`, optional `args` array, and optional `params` object. This action is not batchable.
342
+ - `FetchAccountId` (11): Get the selected account identifier.
343
+ - `FetchFees` (12): Get authenticated `cex-trading-fee-evidence/v1` for one slash-delimited spot pair.
344
+ - `InternalTransfer` (13): Transfer between configured exchange accounts.
345
+ - `GetPerpConfigState` (14): Read positions and leverage/margin configuration.
346
+ - `SetPerpConfigState` (15): Change leverage/margin configuration.
347
+ - `FetchMarketRules` (16): Get `cex-market-rule-evidence/v1` for one active spot pair.
348
+ - `Batch` (17): Execute up to 32 registry-approved read-only child actions sequentially, with a 256 KiB encoded request limit.
333
349
 
334
350
  #### Order Book Call Methods
335
351
 
@@ -451,20 +467,106 @@ const depositAddressRequest = {
451
467
  symbol: "USDT"
452
468
  };
453
469
 
454
- // Fetch currency metadata
470
+ // Fetch one transfer-network fact (breaking hard-cut contract)
455
471
  const fetchCurrencyRequest = {
456
472
  action: 9, // FetchCurrency
473
+ payload: { network: "BEP20" },
474
+ cex: "mexc",
475
+ symbol: "USDC"
476
+ };
477
+
478
+ // Fetch authenticated account commission for one pair
479
+ const fetchFeesRequest = {
480
+ action: 12, // FetchFees
457
481
  payload: {},
458
- cex: "binance",
459
- symbol: "USDT"
482
+ cex: "mexc",
483
+ symbol: "ARB/USDC"
484
+ };
485
+
486
+ // Fetch typed market rules for one pair
487
+ const fetchMarketRulesRequest = {
488
+ action: 16, // FetchMarketRules
489
+ payload: {},
490
+ cex: "mexc",
491
+ symbol: "ARB/USDC"
492
+ };
493
+
494
+ // Batch independent read-only actions. `requests` is a JSON string because
495
+ // ActionRequest.payload remains map<string, string>.
496
+ const batchRequest = {
497
+ action: 17, // Batch
498
+ cex: "mexc",
499
+ symbol: "",
500
+ payload: {
501
+ requests: JSON.stringify([
502
+ { id: "fees", action: 12, symbol: "ARB/USDC", payload: {} },
503
+ { id: "rules", action: 16, symbol: "ARB/USDC", payload: {} },
504
+ {
505
+ id: "network",
506
+ action: 9,
507
+ symbol: "USDC",
508
+ payload: { network: "BEP20" }
509
+ }
510
+ ])
511
+ }
460
512
  };
461
513
  ```
462
514
 
515
+ #### Read-only batch and venue evidence contracts
516
+
517
+ `Batch` inherits the outer exchange and credential/account selection. Children
518
+ cannot override routing. The v1 batchable set is `FetchBalances`, `FetchTicker`,
519
+ `FetchCurrency`, `FetchAccountId`, `FetchFees`, `GetPerpConfigState`, and
520
+ `FetchMarketRules`. `Call`, order/transfer actions, deposit discovery,
521
+ configuration writes, nested batches, batches over 32 children, and encoded
522
+ `payload.requests` values over 256 KiB are rejected before any child runs.
523
+ Structurally invalid batches return gRPC `INVALID_ARGUMENT`; runtime child
524
+ failures remain in an outer-`OK` `cex-broker-action-batch/v1` envelope with the
525
+ same gRPC status as the corresponding unary call.
526
+
527
+ `FetchFees` no longer accepts token-only symbols, `includeAllFees`, or
528
+ `includeFundingFees`. It uses authenticated `fetchTradingFee` and never treats
529
+ `loadMarkets`, market maker/taker defaults, or `broker.fees` as account
530
+ commission. `FetchCurrency` requires one network and no longer returns an
531
+ unscoped raw currency object.
532
+
533
+ These facts have distinct authority:
534
+
535
+ - account commission is the selected account's current maker/taker schedule;
536
+ - realized fill commission belongs to executed trade/fill records;
537
+ - transfer-network fees are inventory-movement costs;
538
+ - Maker DEX venue-fee revenue is protocol liquidity-provider revenue.
539
+
540
+ Each successful evidence result has its own observation timestamp and canonical
541
+ source digest. The broker does not fabricate historical effective intervals or
542
+ coerce observed pair values to an expected fee profile.
543
+
544
+ A batch response preserves each action-specific JSON result string:
545
+
546
+ ```json
547
+ {
548
+ "schemaVersion": "cex-broker-action-batch/v1",
549
+ "responses": [
550
+ {
551
+ "id": "fees",
552
+ "action": 12,
553
+ "symbol": "ARB/USDC",
554
+ "response": {
555
+ "result": "{\"schemaVersion\":\"cex-trading-fee-evidence/v1\"}",
556
+ "proof": ""
557
+ },
558
+ "error": null
559
+ }
560
+ ]
561
+ }
562
+ ```
563
+
463
564
  ### Subscribe (Streaming)
464
565
 
465
566
  Real-time streaming of market data and account updates.
466
567
 
467
568
  **Request:**
569
+
468
570
  ```protobuf
469
571
  message SubscribeRequest {
470
572
  string cex = 1; // CEX identifier
@@ -475,6 +577,7 @@ message SubscribeRequest {
475
577
  ```
476
578
 
477
579
  **Response Stream:**
580
+
478
581
  ```protobuf
479
582
  message SubscribeResponse {
480
583
  string data = 1; // JSON string of the streaming data
@@ -485,6 +588,7 @@ message SubscribeResponse {
485
588
  ```
486
589
 
487
590
  **Available Subscription Types:**
591
+
488
592
  - `NO_ACTION` (0): Compatibility default; resolved to `ORDERBOOK`
489
593
  - `ORDERBOOK` (1): Real-time order book updates
490
594
  - `TRADES` (2): Live trade feed
@@ -832,6 +936,7 @@ cex-broker/
832
936
  The broker automatically supports all exchanges available in CCXT. To add a new exchange:
833
937
 
834
938
  1. Add your API credentials to environment variables:
939
+
835
940
  ```env
836
941
  CEX_BROKER_<EXCHANGE>_API_KEY=your_api_key
837
942
  CEX_BROKER_<EXCHANGE>_API_SECRET=your_api_secret
@@ -846,12 +951,14 @@ The broker automatically supports all exchanges available in CCXT. To add a new
846
951
  Secondary brokers provide redundancy and load balancing:
847
952
 
848
953
  1. Configure secondary API keys:
954
+
849
955
  ```env
850
956
  CEX_BROKER_BINANCE_API_KEY_1=secondary_key_1
851
957
  CEX_BROKER_BINANCE_API_SECRET_1=secondary_secret_1
852
958
  ```
853
959
 
854
960
  2. Use secondary brokers in your gRPC calls:
961
+
855
962
  ```typescript
856
963
  const metadata = new grpc.Metadata();
857
964
  metadata.set('use-secondary-key', '1'); // Use secondary broker
@@ -894,6 +1001,7 @@ for (const [currency, info] of Object.entries(currencies)) {
894
1001
  ```
895
1002
 
896
1003
  **Common Network Identifiers:**
1004
+
897
1005
  - `BEP20` / `BSC`: Binance Smart Chain
898
1006
  - `ETH` / `ERC20`: Ethereum
899
1007
  - `TRC20`: Tron