dccd 2.1.0__tar.gz → 2.3.0__tar.gz

{dccd-2.1.0 → dccd-2.3.0}/CHANGELOG.md

@@ -6,6 +6,71 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-05-22
+
+### Added
+
+- `dccd/storage.py` — `DataStore.is_period_complete(year)`: checks whether an annual parquet file contains all expected candles; `DataStore.missing_intervals(start, end)`: gap-detection — complete past years are skipped, incomplete years resume from the last saved row (#41)
+- `dccd/daemon/backfill.py` — `_BackfillBase.run()` now iterates over `DataStore.missing_intervals()` instead of a single sliding window from `last_saved`; complete years are never re-downloaded (#41)
+- `dccd/storage.py` — new `DataStore` class: unified read/write interface for OHLC, trades, and orderbook; `save(df)` (merge-on-TS, annual OHLC / daily trades+orderbook), `load(start, end)`, `existing_periods()`, `last_timestamp()` (#39)
+- `dccd/tools/date_time.py` — `span_label(span)` converts seconds to short directory labels (``'1m'``, ``'1h'``, ``'1d'``…); `_SPAN_LABEL` mapping exported (#39)
+- `doc/source/storage.rst` — Sphinx page for `DataStore` with directory layout examples (#39)
+
+### Changed
+
+- `dccd collect` (formerly `dccd run`) — renamed to clarify the distinction: `collect` = one incremental batch, `backfill` = full historical download with gap detection, `start` = continuous daemon (#41)
+- New storage arborescence: ``{data_path}/{exchange}/ohlc/{pair}/{span}/YYYY.parquet``, ``…/trades/{pair}/YYYY-MM-DD.parquet``, ``…/orderbook/{pair}/YYYY-MM-DD.parquet`` — replaces the old ``{Exchange}/Data/Clean_Data/{per}/{pair}/`` layout (#39)
+- `dccd/histo_dl/exchange.py` — `save()`, `_get_last_date()`, `save_trades()`, `save_orderbook()` now delegate to `DataStore`; removed `last_df`, `_set_by_period`, `_name_file`, `_excel_format`; removed unused `set_hierarchy()` (#39, #41)
+- `dccd/histo_dl/{binance,bybit,coinbase,okx}.py` — removed `full_path` overrides (base class sets the correct path via `DataStore`) (#39)
+- `dccd/daemon/backfill.py`, `scheduler.py` — removed `by_period` parameter; `save()` call simplified (#39)
+- `dccd/daemon/stream_manager.py` — WebSocket save path now built from `DataStore.directory` (#39)
+- `dccd/daemon/config.py` — `HistoJob.by_period` field removed; granularity is automatic (#39)
+
+- `dccd/histo_dl/exchange.py` — `save()` now supports `form='parquet'`; previously only `'xlsx'` and `'csv'` were handled (#35)
+- `config.yml` — ready-to-use daemon config for minutely OHLC + real-time orderbook/trades on Binance, Kraken, and Bybit (#35)
+- `dccd/daemon/backfill.py` — `OHLCBackfill` and `KrakenBackfill` strategy classes with shared retry/progress/save loop; `make_job()` factory; `run_backfill()` orchestrator; tqdm progress bars and optional `--parallel` execution (#38)
+- `dccd/daemon/cli.py` — `dccd backfill` command: reads all job definitions from config, supports `--exchange` / `--pairs` filters, `--start`, `--parallel`, and `--dry-run` flags (#38)
+- `dccd/daemon/config.py` — `SettingsConfig` with `data_path` and `timezone` fields; `CollectorConfig.settings` propagates `data_path` to `StorageConfig.local_path` when not set explicitly (#38)
+
+### Removed
+
+- `scripts/backfill.py` — replaced by `dccd backfill` CLI command and `dccd.daemon.backfill` module (#38)
+
+### Fixed
+
+- `dccd/histo_dl/exchange.py` — `save(form='parquet')` was silently ignored (logged a warning instead of writing the file) (#35)
+- `dccd/histo_dl/exchange.py` — `_sort_data()` crashed with a ValueError when the API returned fewer candles than the expected window size; index is now derived from actual data (#36)
+- `dccd/histo_dl/exchange.py` — `by_period='M'` produced minute-level file names (strftime `%M`) instead of year-month; added `_PERIOD_FMT` mapping so `'M'` → `'%Y-%m'` (#36)
+- `dccd/histo_dl/exchange.py` — `self.end` now reflects the last candle timestamp so window-loop callers advance correctly (was stuck at `now` for Kraken) (#36)
+- `dccd/histo_dl/binance.py` — missing `limit=1000` parameter caused Binance to return only 500 candles per request (#36)
+- `dccd/histo_dl/bybit.py` — `limit` was 200; raised to 1 000 to match the API maximum (#36)
+- `dccd/histo_dl/exchange.py` — `_sort_data()` dropped the minute just before a window boundary when the last trade arrived ≥2 spans early; grid now uses `self.end` directly as the exclusive stop (#36)
+
+## [2.2.0] - 2026-05-17
+
+### Added
+
+- `dccd/histo_dl/exchange.py` — `import_trades(start, end)` and `import_orderbook(depth)` public methods on `ImportDataCryptoCurrencies`; `_sort_trades` / `_sort_orderbook` helpers validate via Pydantic, sort and deduplicate; `trades_df` / `orderbook_df` attributes; `save_trades` / `save_orderbook` save helpers (#31)
+- `dccd/histo_dl/{binance,kraken,bybit,okx,coinbase}.py` — `_import_trades(start, end)` and `_import_orderbook(depth)` implemented for all five exchanges; Binance and Kraken support full history via paginated endpoints; Bybit (≤ 1 000) and Coinbase (≤ 100) return recent-only snapshots (#31)
+- `dccd/models.py` — `Trade.tid` made optional (`int | None`); `OrderBookEntry` gains required `side` field (`'bid'` or `'ask'`) and `count` made optional (`int | None`) (#31)
+- `dccd/daemon/health.py` — `HealthMonitor`: rotating log handler (10 MB × 5 files), per-job metrics JSON, and optional Slack/Discord webhook alerts on consecutive failures; `JobMetrics` dataclass (#30)
+- `dccd/daemon/cli.py` — `dccd` CLI (`validate`, `run`, `start`, `status`, `add` commands) via typer; `[project.scripts]` entrypoint; `typer>=0.12` added to the `daemon` extra (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager` (one thread per `(exchange, pair)`, auto-restart on crash) and `SyncService` (periodic rclone push to all remotes, decoupled from collection) (#26)
+- `dccd/daemon/config.py` — declarative YAML config with Pydantic v2: `CollectorConfig`, `HistoJob`, `StreamJob`, `StorageConfig`, `AlertConfig`, `RemoteConfig`, `load_config()` (#25)
+- `dccd/daemon/storage.py` — `RemoteStorage.push()` via rclone; supports multiple remotes and root-path sync (#25, #26)
+- `dccd/daemon/scheduler.py` — `build_histo_scheduler()` (APScheduler 3.x), `run_histo_job()`, `run_once()` (#25)
+- `examples/config.example.yml` — annotated reference config for the daemon (#25)
+- `examples/daemon_example.py` — programmatic daemon example in 6 steps (#30)
+- `pyproject.toml` — `[daemon]` optional extra (`pyyaml`, `apscheduler`, `typer`) (#25, #30)
+
+### Changed
+
+- `dccd/daemon/scheduler.py` — `run_histo_job`, `build_histo_scheduler`, `run_once` accept an optional `health: HealthMonitor` parameter (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager.__init__` accepts optional `health: HealthMonitor`; `_run_forever` records success/failure on each iteration (#30)
+- `dccd/daemon/config.py` — `StorageConfig.remote` replaced by `remotes: list[RemoteConfig]` and `sync_interval: int` (#26)
+- `dccd/histo_dl/{binance,coinbase,bybit,okx,kraken}.py` — `format_pair(crypto, fiat)` extracted as a static method, independently testable (#29)
+- `dccd/continuous_dl/exchange.py` — unified `__call__`, `_push_trades`, `_push_book_updates`, `_get_book_state`, `_restore_book_state` in base class; separate `set_trades_saver` / `set_book_saver`; crash-recovery checkpoint; `snapshot_ts` injected into every snapshot payload (#28, #29)
+
 ## [2.1.0] - 2026-05-15
 
 ### Added

{dccd-2.1.0 → dccd-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.1.0
+Version: 2.3.0
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -33,6 +33,11 @@ Requires-Dist: pydantic>=2.0
 Provides-Extra: io
 Requires-Dist: pyarrow>=13; extra == "io"
 Requires-Dist: polars>=0.20; extra == "io"
+Provides-Extra: daemon
+Requires-Dist: pyyaml>=6.0; extra == "daemon"
+Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
+Requires-Dist: typer>=0.12; extra == "daemon"
+Requires-Dist: tqdm>=4.64; extra == "daemon"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.4; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
@@ -41,12 +46,18 @@ Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: interrogate>=1.5; extra == "dev"
 Requires-Dist: mypy>=1.0; extra == "dev"
 Requires-Dist: pandas-stubs>=2.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
+Requires-Dist: typer>=0.12; extra == "dev"
+Requires-Dist: tqdm>=4.64; extra == "dev"
 Provides-Extra: doc
 Requires-Dist: sphinx>=7.0; extra == "doc"
 Requires-Dist: furo; extra == "doc"
 Requires-Dist: numpydoc; extra == "doc"
 Requires-Dist: sphinx-design; extra == "doc"
 Requires-Dist: sphinx-copybutton; extra == "doc"
+Requires-Dist: pyyaml>=6.0; extra == "doc"
+Requires-Dist: apscheduler<4,>=3.10; extra == "doc"
 Dynamic: license-file
 
 =============================
@@ -103,6 +114,10 @@ With optional Parquet / Polars support::
 
     $ pip install "dccd[io]"
 
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
 From source::
 
     $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
@@ -115,15 +130,15 @@ Supported exchanges
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
 +==================+============+=============+=================+==========+===========+================+
-| Binance          | ✓          |             |                 |          | ✓         | ✓              |
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Coinbase         | ✓          |             |                 |          |           |                |
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
@@ -132,6 +147,8 @@ Supported exchanges
 
 \* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
+
 Presentation
 ============
 
@@ -144,6 +161,14 @@ Presentation
     Stream real-time data (order book, trades) via WebSocket with automatic
     reconnection and configurable processing/saving callbacks.
 
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
 Output formats
 --------------
 
@@ -155,7 +180,9 @@ Parquet files can be read back as either a ``pandas.DataFrame`` or a
 Quick start
 ===========
 
-Historical data (pandas)::
+Historical data (pandas):
+
+.. code-block:: python
 
     from dccd.histo_dl import FromBinance
 
@@ -164,15 +191,21 @@ Historical data (pandas)::
     obj.save(form='parquet')
     df = obj.get_data()            # pandas DataFrame
 
-Polars output::
+Polars output:
+
+.. code-block:: python
 
     df_pl = obj.get_data(format='polars')
 
-Incremental update (resume from last saved point)::
+Incremental update (resume from last saved point):
+
+.. code-block:: python
 
     obj.import_data(start='last', end='now').save(form='parquet')
 
-Other exchanges::
+Other exchanges:
+
+.. code-block:: python
 
     from dccd.histo_dl import FromKraken, FromBybit, FromOKX
 
@@ -180,6 +213,99 @@ Other exchanges::
     FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
     FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
 
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    settings:
+      data_path: /data/crypto/
+      timezone: UTC
+
+    storage:
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+CLI quick start:
+
+.. code-block:: bash
+
+    # Validate the config
+    dccd validate --config config.yml
+
+    # Backfill all OHLC history defined in config (resumable)
+    dccd backfill --config config.yml --start "2020-01-01 00:00:00"
+
+    # Dry run — estimate windows and time without downloading
+    dccd backfill --config config.yml --dry-run
+
+    # Backfill only one exchange
+    dccd backfill --config config.yml --exchange kraken
+
+    # One incremental batch per job, then exit (for cron)
+    dccd collect --config config.yml
+
+    # Continuous daemon (Ctrl-C to stop)
+    dccd start --config config.yml
+
+Python API:
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
 Links
 =====
 

{dccd-2.1.0 → dccd-2.3.0}/README.rst

@@ -52,6 +52,10 @@ With optional Parquet / Polars support::
 
     $ pip install "dccd[io]"
 
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
 From source::
 
     $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
@@ -64,15 +68,15 @@ Supported exchanges
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
 +==================+============+=============+=================+==========+===========+================+
-| Binance          | ✓          |             |                 |          | ✓         | ✓              |
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Coinbase         | ✓          |             |                 |          |           |                |
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
@@ -81,6 +85,8 @@ Supported exchanges
 
 \* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
+
 Presentation
 ============
 
@@ -93,6 +99,14 @@ Presentation
     Stream real-time data (order book, trades) via WebSocket with automatic
     reconnection and configurable processing/saving callbacks.
 
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
 Output formats
 --------------
 
@@ -104,7 +118,9 @@ Parquet files can be read back as either a ``pandas.DataFrame`` or a
 Quick start
 ===========
 
-Historical data (pandas)::
+Historical data (pandas):
+
+.. code-block:: python
 
     from dccd.histo_dl import FromBinance
 
@@ -113,15 +129,21 @@ Historical data (pandas)::
     obj.save(form='parquet')
     df = obj.get_data()            # pandas DataFrame
 
-Polars output::
+Polars output:
+
+.. code-block:: python
 
     df_pl = obj.get_data(format='polars')
 
-Incremental update (resume from last saved point)::
+Incremental update (resume from last saved point):
+
+.. code-block:: python
 
     obj.import_data(start='last', end='now').save(form='parquet')
 
-Other exchanges::
+Other exchanges:
+
+.. code-block:: python
 
     from dccd.histo_dl import FromKraken, FromBybit, FromOKX
 
@@ -129,6 +151,99 @@ Other exchanges::
     FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
     FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
 
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    settings:
+      data_path: /data/crypto/
+      timezone: UTC
+
+    storage:
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+CLI quick start:
+
+.. code-block:: bash
+
+    # Validate the config
+    dccd validate --config config.yml
+
+    # Backfill all OHLC history defined in config (resumable)
+    dccd backfill --config config.yml --start "2020-01-01 00:00:00"
+
+    # Dry run — estimate windows and time without downloading
+    dccd backfill --config config.yml --dry-run
+
+    # Backfill only one exchange
+    dccd backfill --config config.yml --exchange kraken
+
+    # One incremental batch per job, then exit (for cron)
+    dccd collect --config config.yml
+
+    # Continuous daemon (Ctrl-C to stop)
+    dccd start --config config.yml
+
+Python API:
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
 Links
 =====
 

{dccd-2.1.0 → dccd-2.3.0}/dccd/continuous_dl/binance.py

@@ -29,7 +29,6 @@ import time
 # Third party packages
 # Local packages
 from dccd.continuous_dl.exchange import ContinuousDownloader
-from dccd.process_data import set_marketdepth, set_orders, set_trades
 from dccd.tools.io import IODataBase
 
 __all__ = [
@@ -119,7 +118,7 @@ class DownloadBinanceData(ContinuousDownloader):
     """
 
     def __init__(self, pair: str = 'BTCUSDT', time_step: int = 60,
-                 until: int | None = 3600) -> None:
+                 until: int | None = 3600, checkpoint_dir: str | None = None) -> None:
         """ Initialize object. """
         if until is None:
             until = 0
@@ -128,13 +127,14 @@ class DownloadBinanceData(ContinuousDownloader):
 
         self.pair = pair
         url = _BINANCE_WS_URL.format(sym=pair.lower())
-        ContinuousDownloader.__init__(self, url, time_step=time_step, STOP=until)
+        ContinuousDownloader.__init__(self, url, time_step=time_step, STOP=until,
+                                      checkpoint_dir=checkpoint_dir)
         self._parser_data = {
             'trades': self.parser_trades,
             'book': self.parser_book,
         }
         self.logger = logging.getLogger(__name__)
-        self.d: dict[str, float] = {}
+        self._load_checkpoint()
 
     async def _subscribe(self, **kwargs: object) -> None:
         """ Wait for connection; Binance streams are declared in the URL. """
@@ -165,8 +165,7 @@ class DownloadBinanceData(ContinuousDownloader):
             The ``data`` field from the combined-stream trade envelope.
 
         """
-        for trade in _parser_trades(data):
-            self._raw_parser(trade)
+        self._push_trades(_parser_trades(data))
 
     def parser_book(self, data: dict) -> None:
         """ Parse and update the order book from a depth message.
@@ -177,18 +176,7 @@ class DownloadBinanceData(ContinuousDownloader):
             The ``data`` field from the combined-stream depth envelope.
 
         """
-        updates = _parser_book(data)
-        for price, qty in updates.items():
-            if qty == 0:
-                self.d.pop(price, None)
-            else:
-                self.d[price] = qty
-        self._data[self.t] = dict(self.d)
-
-    def _raw_parser(self, data: object) -> None:
-        if self.t not in self._data:
-            self._data[self.t] = []
-        self._data[self.t].append(data)  # type: ignore[union-attr]
+        self._push_book_updates(_parser_book(data))
 
 
 def get_trades_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
@@ -210,8 +198,7 @@ def get_trades_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_trades)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_trades_saver(IODataBase(path, method=form))
     downloader(pair=pair)
 
 
@@ -234,8 +221,7 @@ def get_orderbook_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_marketdepth)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_book_saver(IODataBase(path, method=form))
     downloader(pair=pair)
 
 
@@ -246,7 +232,8 @@ def get_data_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
     Parameters
     ----------
     path : str
-        Path to save data.
+        Root path; trades saved under ``<path>/trades/``, book under
+        ``<path>/book/``.
     pair : str, optional
         Trading pair in Binance format (e.g. 'BTCUSDT'), default is 'BTCUSDT'.
     time_step : int, optional
@@ -258,6 +245,6 @@ def get_data_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_orders)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_trades_saver(IODataBase(f'{path}/trades', method=form))
+    downloader.set_book_saver(IODataBase(f'{path}/book', method=form))
     downloader(pair=pair)

{dccd-2.1.0 → dccd-2.3.0}/dccd/continuous_dl/bitfinex.py

@@ -31,7 +31,6 @@ Low level API
 """
 
 # Built-in packages
-import asyncio
 import logging
 import time
 from typing import Any
@@ -117,7 +116,8 @@ class DownloadBitfinexData(ContinuousDownloader):
 
     """
 
-    def __init__(self, time_step: int = 60, until: int | None = 3600) -> None:
+    def __init__(self, time_step: int = 60, until: int | None = 3600,
+                 checkpoint_dir: str | None = None) -> None:
         """ Initialize object.
 
         Parameters
@@ -127,6 +127,9 @@ class DownloadBitfinexData(ContinuousDownloader):
         until : int or None, optional
             Seconds to run, or a future Unix timestamp to stop at.
             Default is ``3600``.
+        checkpoint_dir : str or None, optional
+            Directory to write the order-book crash-recovery checkpoint.
+            Disabled when ``None`` (default).
 
         """
         if until is None:
@@ -135,7 +138,7 @@ class DownloadBitfinexData(ContinuousDownloader):
             until -= int(time.time())
 
         ContinuousDownloader.__init__(self, 'bitfinex', time_step=time_step,
-                                      STOP=until)
+                                      STOP=until, checkpoint_dir=checkpoint_dir)
 
         self._parser_data: dict[str, Any] = {
             'book': self.parser_book,
@@ -144,7 +147,7 @@ class DownloadBitfinexData(ContinuousDownloader):
             'trades_raw': self.parser_raw_trades,
         }
         self.logger = logging.getLogger(__name__)
-        self.d: dict[str, Any] = {}
+        self._load_checkpoint()
 
     def parser_raw_book(self, data: list[Any]) -> None:
         """ Parse raw order book, each timestep set in a list all orders.
@@ -177,7 +180,9 @@ class DownloadBitfinexData(ContinuousDownloader):
         else:
             self.d.pop(parsed['price'])
 
-        self._data[self.t] = {v['price']: v['amount'] for v in self.d.values()}  # type: ignore[assignment]
+        self._data.setdefault(self.t, {'trades': [], 'book': {}})['book'] = {
+            v['price']: v['amount'] for v in self.d.values()
+        }
 
     def parser_raw_trades(self, data: list[Any]) -> None:
         """ Parse raw trade data tick-by-tick.
@@ -248,18 +253,9 @@ class DownloadBitfinexData(ContinuousDownloader):
 
         """
         self.parser = self.get_parser(channel)
-
         channel = channel[:-4] if channel[-4:] == '_raw' else channel
-
         self.logger.info('Try connect WS and set {} stream.'.format(channel))
-
-        self.loop = asyncio.get_event_loop()
-        self.loop.run_until_complete(asyncio.gather(
-            self._connect(channel=channel, **kwargs),
-            self._loop()
-        ))
-
-        return self
+        return super().__call__(channel=channel, **kwargs)  # type: ignore[return-value]
 
 
 # =========================================================================== #