dccd 2.3.3__tar.gz → 3.0.0__tar.gz

{dccd-2.3.3 → dccd-3.0.0}/CHANGELOG.md

@@ -6,10 +6,136 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+### Changed
+
+### Fixed
+
+### Deprecated
+
+### Removed
+
+## [3.0.0] - 2026-06-07
+
+### Added
+
+- Reworked web UI split by concern: a read-only enriched **Inventory** (data
+  freshness, OHLC gap detection, on-disk size, per-exchange totals) and two
+  collection pages — **Historical** and **Live** — each with data-type tabs and
+  per-exchange accordions. Jobs are created, edited (first date) and deleted
+  inline on the page; the Live page shows a real-time liveness indicator (last
+  trade/quote + age) fed by a throttled stream heartbeat over SSE. (#76)
+- Job CRUD over the API: `POST /api/jobs/create|delete|update`, backed by
+  `AppConfig.add_job`/`remove_job`/`update_job_start` (persisted to `config.yml`).
+- `ParquetStore.inventory()` now reports on-disk `bytes` and, for OHLC,
+  `expected_rows`/`missing_rows` (gap detection) at no extra read cost.
+- `EventBus` fan-out to multiple SSE consumers and a `StreamSampleEvent`
+  liveness sample emitted (throttled) by `operations.stream`.
+- UI polish: nav reorganised into `Collect ▾`/`System ▾` dropdowns; **Inventory**
+  renamed **Data** (`/inventory`→`/data`) with data-type tabs; reworked Live
+  liveness — seeded from the last on-disk data point so a page refresh shows
+  freshness immediately (no "waiting…"), span-aware dot, a freshness label that
+  is a live relative "N min ago" counter under 24h and an absolute date beyond,
+  and no noise age for fresh trades, with client-side number formatting;
+  order-book cadence (`snapshot_interval`) shown and settable;
+  Storage shows on-disk sizes; Dashboard adds a KPI bar and clearer sections;
+  Logs reoriented around recent runs with human run labels. The Config page no
+  longer duplicates job management (jobs live on Historical/Live; raw edit via
+  its JSON tab). `GET /api/jobs` now returns `start`/`every`/`snapshot_interval`/
+  `depth`. (#76)
+- Cursor-based trades pagination: the engine now follows each adapter's opaque
+  cursor until a window is drained, instead of advancing by a fixed time window.
+  Fixes silent loss of >95% of trades on every liquid pair (all exchanges).
+- UI: single-line top bar (brand + nav on one row); per-job **Schedule** on
+  Historical (a recurring backfill cron — Off/hourly/daily/custom, independent of
+  the span but `≥` it), reconciled live via `Scheduler.sync_intervals`; **Run
+  all** (global) and per-exchange run; timezone-aware date display driven by
+  `settings.timezone` (`local`/`UTC`/zoneinfo). OHLC removed from Live (collected
+  via Historical schedule); order books removed from Historical (no REST
+  history). `POST /api/jobs/update` now also sets `every` (schedule); new
+  `manual` trigger kind for never-auto-run jobs.
+- Bearer auth on `/api/*` when `settings.ui_auth_token` is set, with a `?token=`
+  fallback for Server-Sent Events; `settings.ui_allow_origins` for opt-in CORS.
+- Public async `Client.read()` and `Client.stream()`; `Client` wires adapters
+  via `service_factory` (single source of truth).
+- Network-marked end-to-end tests (`pytest -m network`) validating pagination
+  against live exchange APIs.
+
+### Fixed
+
+- Data loss on merge: writing into an existing legacy v2 Parquet file no longer
+  silently overwrites it; existing rows are canonicalised and preserved.
+- Provenance is now actually written into the Parquet footer (was computed but
+  dropped).
+- Custom ISO start date for backfill no longer raises (`JobParams.start`).
+- Historical *first date* edit no longer reverts on reload: `GET /api/jobs` was
+  not returning `start`, so the UI reset the field after every refresh. (#76)
+- Live order-book streams reported a crossed/incorrect best bid-ask: the WS
+  adapters emitted unmerged diff levels. binance/okx/bitmex now use full
+  snapshot channels (`@depth<N>`, `books5`, `orderBook10`) and bybit
+  reconstructs full state from snapshot+deltas (like kraken); best bid/ask is
+  computed defensively (`max` bid / `min` ask). (#76)
+- Order-book Live liveness was incoherent with its cadence: it sampled the WS
+  every second while only one snapshot per ``snapshot_interval`` is captured. The
+  liveness sample is now emitted when a snapshot is actually saved, so its age
+  counts up to the interval and resets (matching the "Δ Ns" cadence). (#76)
+- `dccd inventory` no longer crashes on OHLC datasets.
+- Streams with no real implementation (Coinbase OHLC/order book, Bitfinex order
+  book) are rejected with `NoCapability` instead of "running" with zero output.
+- `history="recent"` exchanges (Kraken OHLC) are clamped + warned instead of
+  silently returning wrong deep history.
+- Kraken live OHLC timestamps were epoch 0 (1970): the WS adapter read a
+  non-existent `timestamp_open`; it now parses `interval_begin` (ISO-8601).
+- `mypy dccd/` runs and passes again (it had been aborting on the dev Sphinx).
+
+### Changed / Removed
+
+- Docs/examples swept to v3: README drops the removed `dccd migrate` command and
+  the "Migrating from v2" section; `examples/` rewritten to the v3 `Client` and
+  `dccd.application` daemon wiring with a v3 `jobs:` config, and the stale v2
+  `historical_downloader.ipynb` removed. (#82)
+- Honest OHLC fidelity: Coinbase `quote_volume` is null (was a fabricated
+  `close×volume`); Kraken now fills its native trade count.
+- Removed the dead `parallel` backfill flag, the unused `Page` model and the
+  unused bundled `htmx.min.js`.
+- Removed the v2→v3 Parquet migration tool entirely: `dccd migrate`,
+  `POST /api/migrate`, the Storage-page migrate card, `dccd/storage/migrate.py`,
+  and the `migrate` operation in the registry.
+
+> v3 is a full hexagonal rewrite. It **removes** the v2 daemon web UI shipped in
+> 2.4.0 (`dccd/daemon/*`) and replaces it with `dccd/interfaces/` (api/cli/ui).
+
+## [2.4.0] - 2026-06-04
+
+### Added
+
+- `dccd/daemon/api.py` — web UI and JSON API (FastAPI + Jinja2 + htmx): a thin
+  HTTP layer over the existing daemon modules exposing dashboard (live health
+  metrics), inventory (stored data coverage), jobs (histo/stream list + add/remove
+  + live backfill progress), logs (tail), config (view/validate/save the YAML),
+  and storage (rclone status + manual sync). JSON-only API (`/api/*`) with
+  dumb-shell templates, so the front-end can be swapped without touching the API.
+  Optional Bearer-token auth via `settings.ui_auth_token`
+- `dccd/daemon/cli.py` — `dccd ui`: serve the web UI standalone; the UI is also
+  started automatically (background thread) by `dccd start` when the `[ui]` extra
+  is installed
+- `dccd/daemon/config.py` — `SettingsConfig.ui_host`, `ui_port`, `ui_auth_token`:
+  web UI bind address, port, and optional auth token
+- `dccd/daemon/backfill.py` — `progress_callback` and `stop_event` on
+  `_BackfillBase.run()` / `run_backfill()`: let the UI report live progress and
+  cancel a running backfill (defaults keep CLI behaviour unchanged)
+- `dccd/daemon/stream_manager.py` — `SyncService` writes
+  `{local_path}/.dccd/last_sync.json` after each successful remote push, so the UI
+  can display the last sync time
+- `pyproject.toml` — new optional extra `[ui]` (`fastapi`, `uvicorn[standard]`,
+  `jinja2`); install with `pip install dccd[daemon,ui]`
+
 ## [2.3.3] - 2026-05-31
 
 ### Added
 
+
 - `doc/source/` — complete Sphinx documentation overhaul: redesigned homepage
   with sphinx-design cards, captioned toctrees (Getting Started / Data Collection /
   Reference), new pages (`installation`, `quickstart`, `changelog`, `cli`,

dccd-3.0.0/CLAUDE.md

@@ -0,0 +1,276 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+> **Claude-oriented developer brief**: [`doc/dev/`](doc/dev/) contains an
+> orientation pack written specifically for Claude Code — overview, architecture,
+> design decisions & rationale, the per-exchange capability matrix, testing
+> methodology + findings, current status, and the roadmap. Start at
+> [`doc/dev/README.md`](doc/dev/README.md) for a fuller picture than this file
+> gives. `CLAUDE.md` remains authoritative for commands and invariants.
+
+## Commands
+
+```bash
+# Dev install (Python 3.11+)
+pip install -e ".[dev]"
+
+# Run full unit suite (network E2E excluded by default via -m 'not network')
+pytest
+
+# Run a single test file
+pytest dccd/tests/v3/test_domain.py -v
+
+# Real-exchange end-to-end tests (hit live APIs; opt-in)
+pytest -m network
+
+# Lint
+ruff check dccd/
+
+# Type check (strict on domain/; mypy assumes python 3.12 — see note below)
+mypy dccd/
+
+# Build Sphinx docs (must produce 0 warnings)
+cd doc && make html
+
+# UI smoke test (headless browser; start an isolated `dccd ui` first)
+pip install playwright && playwright install chromium
+python doc/dev/ui_smoke.py http://127.0.0.1:8137
+```
+
+> **mypy** assumes `python_version = 3.12` (in `pyproject.toml`): the dev/docs
+> env ships Sphinx whose source uses 3.12 `type` statements, which made mypy
+> abort under 3.11. dccd supports 3.11–3.13, so 3.12 semantics are safe.
+
+## Git Flow
+
+**Branch model:**
+```
+master          ← stable releases only (tagged vX.Y.Z)
+  └── develop   ← integration branch
+        ├── feat/<topic>   new feature or modernization axis
+        ├── fix/<topic>    bug fix
+        ├── chore/<topic>  tooling, CI, deps
+        └── docs/<topic>   documentation only
+```
+
+**Rules — always follow these before committing or pushing:**
+1. **Never commit directly to `master`.**
+2. **Never commit directly to `develop`** — always use a feature branch + PR.
+3. Branch off `develop`: `git checkout develop && git checkout -b feat/my-topic`
+4. Open a PR into `develop` when done. `develop` → `master` only at release time.
+
+**Commit style (Conventional Commits):**
+```
+feat: add Bybit futures OHLC capability
+fix: correct paginator window for Coinbase
+chore: upgrade httpx to 0.28
+docs: update README for v3 install
+```
+
+Do not add `Co-Authored-By` trailers to commits — this is a personal repo.
+
+**Before every commit:** run `pytest`. It must pass.
+
+**One PR = one concern, small and disposable.** Even a large plan ships as
+*several* small atomic PRs — never one fourre-tout branch. A PR you couldn't throw
+away without losing unrelated good work is too big: split it. This is what makes
+`/abandon-task` (kill a bad PR, keep the lesson) viable.
+
+### Dev loop & docs of record
+
+The iterative loop is tooled by skills, with three tracked docs as the sources of
+truth:
+
+| Doc | Holds | Updated by |
+|-----|-------|-----------|
+| `doc/dev/07-roadmap.md` | open work (single source) | `/pick-task` reads · `/finish-task`, `/abandon-task` update |
+| `doc/dev/03-decisions.md` | the *why* — ADR journal (+ settled rationale) | `/finish-task` (accepted), `/abandon-task` (rejected/tombstone) |
+| `doc/dev/06-status.md` | where things stand | `/finish-task`, `/groom-docs` |
+
+`CHANGELOG.md` + git log stay authoritative for *what* shipped. The loop:
+`/pick-task` (smallest slice → branch) → plan (split big plans into small PRs) →
+`/finish-task` (tests, ADR entry, status, PR) **or** `/abandon-task` (salvage the
+lesson + close the PR); `/groom-docs` periodically keeps `doc/dev/` lean and true.
+
+**Model per task** (advisory — you set it via `/model`, or a skill spawns a
+subagent with an explicit `model`; subagents otherwise *inherit* the parent):
+
+| Model | For |
+|-------|-----|
+| `opus` | judgement, design, decisions, planning, review |
+| `sonnet` | implementation — code, tests, docstrings |
+| `haiku` | mechanical fan-out (doc scans, checklists) — spawn it explicitly as a subagent |
+
+## Architecture (v3 — hexagonal)
+
+### Three usage modes
+
+1. **Python API** — `async with Client() as c: await c.backfill(...)`.
+2. **CLI** — `dccd` command (backfill, stream, start, ui, …).
+3. **HTTP API / UI** — FastAPI server + Jinja2 templates (`dccd ui` or `dccd start`).
+
+### Package structure
+
+```
+dccd/
+  domain/          # Pure, sync, zero I/O — models, capabilities, transforms
+  transport/       # Async HTTP (httpx), WebSocket base, RateLimiter, Paginator
+  sources/         # Exchange adapters (Source protocols + registry)
+  storage/         # ParquetStore, RunsStore (SQLite), RemoteStorage
+  application/     # Operations (backfill, stream), Scheduler, EventBus, Config
+  interfaces/
+    api/           # FastAPI app (1:1 with OperationRegistry)
+    cli/           # Typer CLI (asyncio.run)
+    ui/            # Jinja2 templates (pure HTTP client of api/)
+  tests/v3/        # All tests
+```
+
+### Domain layer (`domain/`)
+
+Pure, synchronous, no I/O. Never import from transport/sources/storage.
+
+| Module | Contents |
+|--------|----------|
+| `symbol.py` | `Symbol(base, quote)` — normalises XBT→BTC |
+| `types.py` | `DataType` enum: `ohlc`, `trades`, `orderbook` |
+| `records.py` | `OHLCBar`, `Trade`, `OrderBookSnapshot` (ns timestamps) |
+| `dataset.py` | `DatasetId`, `Provenance` |
+| `capability.py` | `Capability` — declared per adapter per (data_type × transport × mode) |
+| `timeutils.py` | Helpers: `s_to_ns`, `align_ns`, `span_label`, `binance_interval`, … |
+| `transforms.py` | `aggregate_ohlc(trades, span)` — pure computation |
+| `errors.py` | `NoCapability`, `CoverageError` |
+
+**All internal timestamps are nanoseconds UTC (int64).**
+
+### Transport layer (`transport/`)
+
+Async only. Drives I/O; domain stays pure.
+
+| Module | Contents |
+|--------|----------|
+| `http.py` | `AsyncHTTPClient` — httpx wrapper with retry/backoff |
+| `ws.py` | `WebSocketBase` — `stream_raw()` async generator with exponential reconnect |
+| `ratelimit.py` | `RateLimiter` — token-bucket per exchange |
+| `paginate.py` | `paginate_ohlc`, `paginate_trades` — generic forward paginator |
+
+**Paginator contract**: callers must pass a closure with `symbol` (and `span` for OHLC) already bound:
+
+```python
+async def _fetch(start_ns, end_ns, limit):
+    return await adapter.fetch_ohlc_page(symbol, span, start_ns, end_ns, limit)
+async for bar in paginate_ohlc(_fetch, cap, start_ns, end_ns, span):
+    ...
+```
+
+### Source adapters (`sources/`)
+
+One class per exchange implementing Source protocol mixins:
+
+- `OHLCHistory`, `TradesHistory`, `OrderBookSnapshotREST` — REST historical
+- `OHLCLive`, `TradesLive`, `OrderBookLive` — WebSocket live
+
+Adapters declare their capabilities via `capabilities() -> list[Capability]`.
+
+| Exchange | Notes |
+|----------|-------|
+| `binance.py` | Full history OHLC+trades, depth 5000 |
+| `coinbase.py` | 300 candles/req (Paginator handles automatically) |
+| `kraken.py` | OHLC: 720 recent only (`history="recent"`); trades: full via `since` cursor |
+| `bybit.py` | No spot trades history (capability not declared → `NoCapability` early) |
+| `okx.py` | `history-candles` + `history-trades` for deep history |
+| `bitfinex.py` | Up to 10 000 items per request |
+| `bitmex.py` | Bucketed OHLC (1m/5m/1h/1d only), full trades |
+
+**WS adapters** extend `WebSocketBase` and use `self.stream_raw()` (NOT a custom `_stream_raw` — the base handles reconnect).
+
+### Storage (`storage/`)
+
+| Module | Contents |
+|--------|----------|
+| `parquet.py` | `ParquetStore` — read/write Parquet (ns, provenance, dedup); `inventory()` enriched with on-disk `bytes` and (OHLC only) `expected_rows`/`missing_rows` gap detection at zero extra I/O |
+| `runs_sqlite.py` | `RunsStore` — SQLite WAL, append-only job run history |
+| `remote.py` | `RemoteStorage` — rclone sync |
+
+**Layout**: `{data_path}/{exchange}/ohlc/{pair}/{span}/YYYY.parquet` (annual) and `.../trades/{pair}/YYYY-MM-DD.parquet` (daily).
+
+### Application (`application/`)
+
+| Module | Contents |
+|--------|----------|
+| `config.py` | `AppConfig` + `JobConfig` — Pydantic v2, validates exchange names + span-for-OHLC; runtime CRUD (`add_job`, `remove_job`, `update_job_start`) normalises mutations to single-pair entries (multi-pair configs are read but split on edit) |
+| `events.py` | `EventBus` — pub/sub with **multi-queue fan-out** (`add_queue`/`remove_queue`, `enable_queue` alias) so Live + Logs + Dashboard consume concurrently; events: `ProgressEvent`, `LogEvent`, `StatusEvent`, `StreamSampleEvent` |
+| `jobs.py` | `JobSpec`, `JobRun`, `Trigger`, `JobParams` |
+| `operations.py` | `backfill()`, `stream()` (emits throttled `StreamSampleEvent` ≤1/s for Live liveness), `read()`, `inventory()` |
+| `scheduler.py` | `Scheduler` — async interval/supervised/once job orchestration; `sync_streams()` reconciles stream workers and `sync_intervals()` reconciles recurring backfill loops (start/cancel/restart on cadence change, keyed by spec id) — both stop+drop deleted ones |
+| `registry.py` | `REGISTRY` — maps operation names to schemas (parity enforcement) |
+| `monitor.py` | `HealthMonitor` — EventBus subscriber, webhook alerts |
+| `service_factory.py` | `build_registry()`, `build_store()`, `build_runs_store()` — **single source of truth for wiring** |
+
+**Adding a new exchange**: add the adapter to `sources/`, register it in `service_factory.build_registry()`.
+
+### Interfaces (`interfaces/`)
+
+- `api/app.py` — FastAPI `create_app()`, lifespan context manager, module-level Pydantic request models. Job CRUD lives here: `POST /api/jobs/{create,delete,update}` (body-based to allow `/`/`:` in ids), all routed through the async `_persist_and_refresh` helper (writes YAML, updates `app.state`, calls `scheduler.sync_streams` **and** `scheduler.sync_intervals` to reconcile recurring backfills live). `POST /api/jobs/update` edits `start` and/or the recurring `every` (schedule). `GET /api/jobs` exposes `start`/`every`/`trigger`/`snapshot_interval`/`depth` so the UI can render and preserve them. `POST /api/jobs/run` + `/api/jobs/run-all` trigger configured backfills on demand. SSE at `GET /api/events` uses `add_queue`/`remove_queue` for multi-consumer fan-out.
+- `cli/main.py` — Typer commands, all import from `service_factory`
+- `ui/` — Jinja2 templates + static files. Nav: `Dashboard` · `Data` flat, plus `Collect ▾` (Historical/Live) and `System ▾` (Logs/Config/Storage) dropdowns. Pages are **split by concern**:
+  - **Data** (`data.html`, route `/data`; `/inventory` 307-redirects here) — read-only view of what's on disk: DataType tabs → per-exchange accordions with totals, freshness dot, OHLC gap %, on-disk size, file count. No action buttons.
+  - **Historical** (`historical.html`) — backfill jobs (**OHLC + Trades only**; order books have no REST history): DataType tabs → exchange accordions → one row per dataset with editable `first_date` (defaults to the dataset's earliest stored bar), a **Schedule** select (Off/hourly/daily/custom → `every`; `manual` trigger when off), real coverage bar, inline Run/Delete. **Run all** (header) + per-exchange **Run all**. New jobs default to `manual`.
+  - **Live** (`live.html`) — stream jobs (**Trades + Order Book only**; OHLC is collected via the Historical schedule, not streamed): same tab/accordion shape, with a liveness indicator fed by `StreamSampleEvent` over SSE (numeric `value`/`bid`/`ask`, formatted client-side via `fmtNum`). Liveness is **seeded from the last on-disk point** (inventory `max_ts`) so a refresh shows freshness without waiting for a live sample. The dot's "fresh" window is span-aware (order-book `snapshot_interval` / short for trades); the freshness label is a relative "N ago" under 24h (`fmtFreshness`) and an absolute date beyond, or the last-run date-time when stopped. Cadence column + `snapshot_interval` field for order book. Inline Start/Stop/Delete.
+  - Single top bar carries the brand (logo · `dccd` · version) left and the nav right. Dates render in `settings.timezone` (`local`/`UTC`/zoneinfo) via `DCCD_TZ` in `fmtNs`/`fmtDate`; relative ages are tz-independent.
+  - `dashboard.html` (KPIs + Active now / Recent runs / Data), `logs.html` (recent runs first, live console secondary, human run labels), `config.html` (Settings incl. `timezone`/Alerts/Storage + Raw JSON — **no jobs form**; jobs are managed on Historical/Live), `storage.html` (sizes via `fmtBytes`; no migrate tool).
+
+**UI↔API contract**: UI is a pure HTTP client of the API — no direct calls to application layer. Inline job create/edit/delete on Historical/Live go through `/api/jobs/*`; the Config page no longer manages jobs (edit the `jobs` array via its Raw JSON tab if needed).
+
+## Testing conventions
+
+Tests live in `dccd/tests/v3/`. No doctests (removed `--doctest-modules` from `addopts`).
+
+Coverage is measured on every run (`--cov=dccd`). CI matrix: Python 3.11–3.13.
+
+Key test files:
+- `test_domain.py` + `test_domain_extended.py` — domain models, transforms, config validation
+- `test_sources.py` — capability declarations, protocol compliance, symbol mapping
+- `test_storage.py` + `test_storage_extended.py` — ParquetStore, dedup keys, gap detection
+- `test_application.py` — EventBus (multi-queue fan-out, `sample`), JobSpec, OperationRegistry parity, `AppConfig` job CRUD (incl. multi-pair split)
+- `test_api.py` — FastAPI endpoints (incl. auth, backfill cancel, `/api/jobs/{create,delete,update}`, stream-delete unregisters worker) via TestClient
+- `test_transport.py` — AsyncHTTPClient concurrency safety
+- `test_backfill_lookback.py` — bounded default lookback per data type
+- `test_network.py` — **real-exchange** E2E (`@pytest.mark.network`, opt-in)
+
+**Test the chain on real data, not just the pieces.** A green unit suite missed
+a backfill writing 0 rows, a store losing 58 % of trades, and a "Stop" button
+that did nothing. For any data path: run the real operation, read what landed on
+Parquet, and compare it to what was requested. Back up before any in-place
+mutation. Full methodology + the catalogue of bugs this surfaced:
+[`doc/dev/05-testing.md`](doc/dev/05-testing.md);
+UI smoke test: `doc/dev/ui_smoke.py`.
+
+### Invariants — do not regress
+
+- **Trades pagination is cursor-based** (per-adapter opaque cursor); never
+  advance trades by a fixed time window. OHLC snaps the start to the *bar* (span),
+  not the window.
+- **Dedup key is per data type** (`ParquetStore._dedup_subset`): OHLC=`TS`,
+  trades=`tid`(else composite), order book=`(TS,side,price)`. `TS` alone is
+  unique only for OHLC.
+- **Declared capabilities must be honest**: don't declare a WS channel or
+  `history` depth that isn't implemented — the engine rejects undeclared ones.
+- **All timestamps ns UTC int64**; legacy frames pass through
+  `ParquetStore.canonicalize()` before any `concat`.
+- **First `start=last` backfill is bounded** per type (`_DEFAULT_LOOKBACK_NS`);
+  backfills are cancellable (`stop_event` → `DELETE /api/backfill/{id}`).
+- **Adapters share one reference-counted HTTP client** (concurrency-safe).
+- **`ui_auth_token` enforces Bearer on `/api/*`**; CORS is not wildcard.
+- **Stream worker set is reconciled, not append-only**: deleting a stream job must
+  `Scheduler.sync_streams()` so its worker is stopped and dropped (never left
+  running/controllable after its config is gone).
+- **`EventBus` fans out to all registered queues**; SSE consumers register via
+  `add_queue` and must `remove_queue` on disconnect (done in the `/api/events`
+  `finally`).
+
+## Dependencies
+
+Core (Python 3.11+): `httpx`, `websockets`, `pydantic>=2`, `polars`, `pyarrow`, `numpy`, `scipy`  
+Daemon extra: `pyyaml`, `typer`, `tqdm`, `uvicorn`, `fastapi`, `jinja2`, `apscheduler>=3.10,<4`  
+Dev extra: `pytest`, `pytest-asyncio`, `pytest-cov`, `ruff`, `mypy`, `interrogate`

dccd-3.0.0/PKG-INFO

@@ -0,0 +1,250 @@
+Metadata-Version: 2.4
+Name: dccd
+Version: 3.0.0
+Summary: Download Crypto Currency Data — hexagonal architecture, async-first.
+Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
+Project-URL: Documentation, https://download-crypto-currencies-data.readthedocs.io/
+Project-URL: Changelog, https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/blob/master/CHANGELOG.md
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Office/Business :: Financial
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: httpx>=0.27
+Requires-Dist: websockets>=12.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: polars>=0.20
+Requires-Dist: pyarrow>=13
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.10
+Requires-Dist: SQLAlchemy>=2.0
+Provides-Extra: daemon
+Requires-Dist: pyyaml>=6.0; extra == "daemon"
+Requires-Dist: typer>=0.12; extra == "daemon"
+Requires-Dist: tqdm>=4.64; extra == "daemon"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "daemon"
+Requires-Dist: fastapi>=0.110; extra == "daemon"
+Requires-Dist: jinja2>=3.1; extra == "daemon"
+Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: interrogate>=1.5; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Requires-Dist: typer>=0.12; extra == "dev"
+Requires-Dist: tqdm>=4.64; extra == "dev"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "dev"
+Requires-Dist: fastapi>=0.110; extra == "dev"
+Requires-Dist: jinja2>=3.1; extra == "dev"
+Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
+Requires-Dist: httpx>=0.27; extra == "dev"
+Provides-Extra: doc
+Requires-Dist: sphinx>=7.0; extra == "doc"
+Requires-Dist: furo; extra == "doc"
+Requires-Dist: numpydoc; extra == "doc"
+Requires-Dist: sphinx-design; extra == "doc"
+Requires-Dist: sphinx-copybutton; extra == "doc"
+Requires-Dist: sphinx-click; extra == "doc"
+Requires-Dist: autodoc-pydantic>=2.0; extra == "doc"
+Requires-Dist: pyyaml>=6.0; extra == "doc"
+Requires-Dist: fastapi>=0.110; extra == "doc"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "doc"
+Requires-Dist: jinja2>=3.1; extra == "doc"
+Dynamic: license-file
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/develop/doc/source/_static/logo-dark-transparent.svg">
+  <img alt="dccd logo" src="https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/develop/doc/source/_static/logo-light-transparent.svg" height="180px" align="left">
+</picture>
+
+# **Download Crypto-Currency Data** — v3
+
+[![Python versions](https://img.shields.io/pypi/pyversions/dccd)](https://pypi.org/project/dccd/)
+[![PyPI](https://img.shields.io/pypi/v/dccd.svg)](https://pypi.org/project/dccd/)
+[![CI](https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/actions/workflows/ci.yml/badge.svg)](https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/actions/workflows/ci.yml)
+[![License](https://img.shields.io/github/license/ArthurBernard/Download_Crypto_Currencies_Data.svg)](https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/blob/master/LICENSE.txt)<br>
+[![Documentation](https://readthedocs.org/projects/download-crypto-currencies-data/badge/?version=latest)](https://download-crypto-currencies-data.readthedocs.io/en/latest/)
+[![Coverage](https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data/branch/master/graph/badge.svg)](https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data)
+
+---
+
+**dccd** downloads crypto-currency market data (OHLCV, trades, order book)
+from 7 exchanges via REST and WebSocket. Data is stored as Parquet files with
+nanosecond-precision timestamps.
+
+## Architecture (v3)
+
+Hexagonal architecture — business logic is fully separated from interfaces:
+
+```
+Interfaces: CLI · HTTP API · Web UI · Python Client
+                        ↓
+          Application: backfill, stream, read, inventory
+                        ↓
+  Domain ← Sources (7 exchange adapters) ← Transport (httpx · WS · Paginator)
+                        ↓
+             Storage: ParquetStore + RunsStore (SQLite)
+```
+
+- **Async-first** — httpx + websockets, one event loop; CLI via `asyncio.run`
+- **Nanosecond timestamps** — uniform int64 UTC throughout the store
+- **Generic Paginator** — no per-exchange chunking; Coinbase 300-limit is a capability declaration
+- **NoCapability early** — Bybit no spot trades history, Kraken OHLC recent-only → clear error
+- **Four iso-functional interfaces** — same operations everywhere (parity test enforces this)
+
+## Supported exchanges
+
+You pick a **data type** (OHLC · trades · order book) and an **operation** —
+**backfill** (history) or **stream** (live):
+
+| Exchange | Backfill (history) | Stream (live) |
+|----------|--------------------|---------------|
+| Binance  | OHLC · trades · book | OHLC · trades · book |
+| Coinbase | OHLC · book · trades *(recent)* | trades |
+| Kraken   | OHLC *(720 recent)* · trades · book | OHLC · trades · book |
+| Bybit    | OHLC · book | OHLC · trades · book |
+| OKX      | OHLC · trades · book | OHLC · trades · book |
+| Bitfinex | OHLC · trades · book | OHLC · trades |
+| BitMEX   | OHLC *(1m/5m/1h/1d)* · trades · book | OHLC · trades · book |
+
+Trades backfill is **cursor-paginated** (drains the full window, not just the
+first page). *recent* = no deep history via the public API (a deeper request is
+rejected/clamped early, never silently truncated); Bybit spot has no trade
+history. **Order-book backfill** is a single snapshot — use a stream to record
+the book over time. Stream channels are only listed where really implemented
+(undeclared ones raise `NoCapability`).
+
+### OHLC field fidelity
+
+Not every exchange returns every OHLC field natively. Missing fields are stored
+as `null` (never fabricated):
+
+| Exchange | `quote_volume` | `trades` (count) |
+|----------|----------------|------------------|
+| Binance  | ✅ native      | ✅ native |
+| Bybit / OKX | ✅ native   | — null |
+| Kraken   | ✅ (vwap × volume, exact) | ✅ native |
+| Coinbase / Bitfinex / BitMEX | — null | — null |
+
+## Installation
+
+```bash
+# Core — Python 3.11+
+pip install dccd
+
+# With scheduler, CLI, and web UI
+pip install "dccd[daemon]"
+
+# Development
+pip install "dccd[dev]"
+```
+
+## Quick start
+
+### Python API
+
+```python
+import asyncio
+from dccd import Client
+
+async def main():
+    async with Client() as c:
+        result = await c.backfill("binance", "BTC/USDT", data_type="ohlc", span=3600)
+        print(f"Wrote {result['rows_written']} rows")
+        for ds in c.inventory():
+            print(ds)
+
+asyncio.run(main())
+```
+
+### CLI
+
+```bash
+dccd validate --config config.yml      # validate config
+dccd backfill --config config.yml      # run all backfill jobs
+dccd backfill -e binance -s BTC/USDT --type ohlc --span 3600  # ad-hoc
+dccd stream   --config config.yml      # run WebSocket stream jobs
+dccd start    --config config.yml      # full daemon + UI
+dccd ui       --config config.yml      # UI only (no scheduler)
+dccd inventory --config config.yml     # list stored datasets
+dccd status   --config config.yml      # show recent runs
+```
+
+### Configuration (`config.yml`)
+
+```yaml
+settings:
+  data_path: ./data/crypto
+  timezone: UTC
+  ui_port: 8080
+
+jobs:
+  - exchange: binance
+    pairs: [BTC/USDT, ETH/USDT]
+    data_type: ohlc
+    span: 3600
+    trigger_kind: interval
+    every: 3600
+
+  - exchange: kraken
+    pairs: [BTC/USD]
+    data_type: trades
+    operation: stream
+    trigger_kind: supervised
+
+storage:
+  remotes:
+    - provider: rclone
+      remote: "mynas:crypto/"
+  sync_interval: 3600
+```
+
+### HTTP API (when `dccd ui` or `dccd start` is running)
+
+```
+GET  /api/operations          list registered operations
+POST /api/backfill            start a backfill job
+GET  /api/backfill/{run_id}   poll run status
+GET  /api/streams             list stream jobs + state
+POST /api/streams/start       start a stream job
+POST /api/streams/stop        stop a stream job
+POST /api/read                read stored data (≤1 000 rows)
+GET  /api/events              SSE stream of progress/log/status events
+GET  /api/inventory           list all datasets
+GET  /health                  liveness check
+```
+
+## Data layout
+
+```
+{data_path}/
+  {exchange}/
+    ohlc/{pair}/{span}/YYYY.parquet       # annual, ns timestamps
+    trades/{pair}/YYYY-MM-DD.parquet      # daily
+    orderbook/{pair}/YYYY-MM-DD.parquet   # daily
+    .dccd/runs.db                         # SQLite job run history
+```
+
+All timestamps are **nanoseconds UTC** (int64).
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest             # 141 tests
+ruff check dccd/   # lint
+mypy dccd/         # type check (strict on domain/)
+```