dccd 2.1.0__tar.gz → 2.2.0__tar.gz

{dccd-2.1.0 → dccd-2.2.0}/CHANGELOG.md

@@ -6,6 +6,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.2.0] - 2026-05-17
+
+### Added
+
+- `dccd/histo_dl/exchange.py` — `import_trades(start, end)` and `import_orderbook(depth)` public methods on `ImportDataCryptoCurrencies`; `_sort_trades` / `_sort_orderbook` helpers validate via Pydantic, sort and deduplicate; `trades_df` / `orderbook_df` attributes; `save_trades` / `save_orderbook` save helpers (#31)
+- `dccd/histo_dl/{binance,kraken,bybit,okx,coinbase}.py` — `_import_trades(start, end)` and `_import_orderbook(depth)` implemented for all five exchanges; Binance and Kraken support full history via paginated endpoints; Bybit (≤ 1 000) and Coinbase (≤ 100) return recent-only snapshots (#31)
+- `dccd/models.py` — `Trade.tid` made optional (`int | None`); `OrderBookEntry` gains required `side` field (`'bid'` or `'ask'`) and `count` made optional (`int | None`) (#31)
+- `dccd/daemon/health.py` — `HealthMonitor`: rotating log handler (10 MB × 5 files), per-job metrics JSON, and optional Slack/Discord webhook alerts on consecutive failures; `JobMetrics` dataclass (#30)
+- `dccd/daemon/cli.py` — `dccd` CLI (`validate`, `run`, `start`, `status`, `add` commands) via typer; `[project.scripts]` entrypoint; `typer>=0.12` added to the `daemon` extra (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager` (one thread per `(exchange, pair)`, auto-restart on crash) and `SyncService` (periodic rclone push to all remotes, decoupled from collection) (#26)
+- `dccd/daemon/config.py` — declarative YAML config with Pydantic v2: `CollectorConfig`, `HistoJob`, `StreamJob`, `StorageConfig`, `AlertConfig`, `RemoteConfig`, `load_config()` (#25)
+- `dccd/daemon/storage.py` — `RemoteStorage.push()` via rclone; supports multiple remotes and root-path sync (#25, #26)
+- `dccd/daemon/scheduler.py` — `build_histo_scheduler()` (APScheduler 3.x), `run_histo_job()`, `run_once()` (#25)
+- `examples/config.example.yml` — annotated reference config for the daemon (#25)
+- `examples/daemon_example.py` — programmatic daemon example in 6 steps (#30)
+- `pyproject.toml` — `[daemon]` optional extra (`pyyaml`, `apscheduler`, `typer`) (#25, #30)
+
+### Changed
+
+- `dccd/daemon/scheduler.py` — `run_histo_job`, `build_histo_scheduler`, `run_once` accept an optional `health: HealthMonitor` parameter (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager.__init__` accepts optional `health: HealthMonitor`; `_run_forever` records success/failure on each iteration (#30)
+- `dccd/daemon/config.py` — `StorageConfig.remote` replaced by `remotes: list[RemoteConfig]` and `sync_interval: int` (#26)
+- `dccd/histo_dl/{binance,coinbase,bybit,okx,kraken}.py` — `format_pair(crypto, fiat)` extracted as a static method, independently testable (#29)
+- `dccd/continuous_dl/exchange.py` — unified `__call__`, `_push_trades`, `_push_book_updates`, `_get_book_state`, `_restore_book_state` in base class; separate `set_trades_saver` / `set_book_saver`; crash-recovery checkpoint; `snapshot_ts` injected into every snapshot payload (#28, #29)
+
 ## [2.1.0] - 2026-05-15
 
 ### Added

{dccd-2.1.0 → dccd-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.1.0
+Version: 2.2.0
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -33,6 +33,10 @@ Requires-Dist: pydantic>=2.0
 Provides-Extra: io
 Requires-Dist: pyarrow>=13; extra == "io"
 Requires-Dist: polars>=0.20; extra == "io"
+Provides-Extra: daemon
+Requires-Dist: pyyaml>=6.0; extra == "daemon"
+Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
+Requires-Dist: typer>=0.12; extra == "daemon"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.4; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
@@ -41,12 +45,17 @@ Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: interrogate>=1.5; extra == "dev"
 Requires-Dist: mypy>=1.0; extra == "dev"
 Requires-Dist: pandas-stubs>=2.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
+Requires-Dist: typer>=0.12; extra == "dev"
 Provides-Extra: doc
 Requires-Dist: sphinx>=7.0; extra == "doc"
 Requires-Dist: furo; extra == "doc"
 Requires-Dist: numpydoc; extra == "doc"
 Requires-Dist: sphinx-design; extra == "doc"
 Requires-Dist: sphinx-copybutton; extra == "doc"
+Requires-Dist: pyyaml>=6.0; extra == "doc"
+Requires-Dist: apscheduler<4,>=3.10; extra == "doc"
 Dynamic: license-file
 
 =============================
@@ -103,6 +112,10 @@ With optional Parquet / Polars support::
 
     $ pip install "dccd[io]"
 
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
 From source::
 
     $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
@@ -115,15 +128,15 @@ Supported exchanges
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
 +==================+============+=============+=================+==========+===========+================+
-| Binance          | ✓          |             |                 |          | ✓         | ✓              |
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Coinbase         | ✓          |             |                 |          |           |                |
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
@@ -132,6 +145,8 @@ Supported exchanges
 
 \* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
+
 Presentation
 ============
 
@@ -144,6 +159,14 @@ Presentation
     Stream real-time data (order book, trades) via WebSocket with automatic
     reconnection and configurable processing/saving callbacks.
 
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
 Output formats
 --------------
 
@@ -155,7 +178,9 @@ Parquet files can be read back as either a ``pandas.DataFrame`` or a
 Quick start
 ===========
 
-Historical data (pandas)::
+Historical data (pandas):
+
+.. code-block:: python
 
     from dccd.histo_dl import FromBinance
 
@@ -164,15 +189,21 @@ Historical data (pandas)::
     obj.save(form='parquet')
     df = obj.get_data()            # pandas DataFrame
 
-Polars output::
+Polars output:
+
+.. code-block:: python
 
     df_pl = obj.get_data(format='polars')
 
-Incremental update (resume from last saved point)::
+Incremental update (resume from last saved point):
+
+.. code-block:: python
 
     obj.import_data(start='last', end='now').save(form='parquet')
 
-Other exchanges::
+Other exchanges:
+
+.. code-block:: python
 
     from dccd.histo_dl import FromKraken, FromBybit, FromOKX
 
@@ -180,6 +211,73 @@ Other exchanges::
     FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
     FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
 
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    storage:
+      local_path: /data/crypto/
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+        by_period: Y
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
 Links
 =====
 

{dccd-2.1.0 → dccd-2.2.0}/README.rst

@@ -52,6 +52,10 @@ With optional Parquet / Polars support::
 
     $ pip install "dccd[io]"
 
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
 From source::
 
     $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
@@ -64,15 +68,15 @@ Supported exchanges
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
 +==================+============+=============+=================+==========+===========+================+
-| Binance          | ✓          |             |                 |          | ✓         | ✓              |
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Coinbase         | ✓          |             |                 |          |           |                |
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
-| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
 | Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
 +------------------+------------+-------------+-----------------+----------+-----------+----------------+
@@ -81,6 +85,8 @@ Supported exchanges
 
 \* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
+
 Presentation
 ============
 
@@ -93,6 +99,14 @@ Presentation
     Stream real-time data (order book, trades) via WebSocket with automatic
     reconnection and configurable processing/saving callbacks.
 
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
 Output formats
 --------------
 
@@ -104,7 +118,9 @@ Parquet files can be read back as either a ``pandas.DataFrame`` or a
 Quick start
 ===========
 
-Historical data (pandas)::
+Historical data (pandas):
+
+.. code-block:: python
 
     from dccd.histo_dl import FromBinance
 
@@ -113,15 +129,21 @@ Historical data (pandas)::
     obj.save(form='parquet')
     df = obj.get_data()            # pandas DataFrame
 
-Polars output::
+Polars output:
+
+.. code-block:: python
 
     df_pl = obj.get_data(format='polars')
 
-Incremental update (resume from last saved point)::
+Incremental update (resume from last saved point):
+
+.. code-block:: python
 
     obj.import_data(start='last', end='now').save(form='parquet')
 
-Other exchanges::
+Other exchanges:
+
+.. code-block:: python
 
     from dccd.histo_dl import FromKraken, FromBybit, FromOKX
 
@@ -129,6 +151,73 @@ Other exchanges::
     FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
     FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
 
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    storage:
+      local_path: /data/crypto/
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+        by_period: Y
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
 Links
 =====
 

{dccd-2.1.0 → dccd-2.2.0}/dccd/continuous_dl/binance.py

@@ -29,7 +29,6 @@ import time
 # Third party packages
 # Local packages
 from dccd.continuous_dl.exchange import ContinuousDownloader
-from dccd.process_data import set_marketdepth, set_orders, set_trades
 from dccd.tools.io import IODataBase
 
 __all__ = [
@@ -119,7 +118,7 @@ class DownloadBinanceData(ContinuousDownloader):
     """
 
     def __init__(self, pair: str = 'BTCUSDT', time_step: int = 60,
-                 until: int | None = 3600) -> None:
+                 until: int | None = 3600, checkpoint_dir: str | None = None) -> None:
         """ Initialize object. """
         if until is None:
             until = 0
@@ -128,13 +127,14 @@ class DownloadBinanceData(ContinuousDownloader):
 
         self.pair = pair
         url = _BINANCE_WS_URL.format(sym=pair.lower())
-        ContinuousDownloader.__init__(self, url, time_step=time_step, STOP=until)
+        ContinuousDownloader.__init__(self, url, time_step=time_step, STOP=until,
+                                      checkpoint_dir=checkpoint_dir)
         self._parser_data = {
             'trades': self.parser_trades,
             'book': self.parser_book,
         }
         self.logger = logging.getLogger(__name__)
-        self.d: dict[str, float] = {}
+        self._load_checkpoint()
 
     async def _subscribe(self, **kwargs: object) -> None:
         """ Wait for connection; Binance streams are declared in the URL. """
@@ -165,8 +165,7 @@ class DownloadBinanceData(ContinuousDownloader):
             The ``data`` field from the combined-stream trade envelope.
 
         """
-        for trade in _parser_trades(data):
-            self._raw_parser(trade)
+        self._push_trades(_parser_trades(data))
 
     def parser_book(self, data: dict) -> None:
         """ Parse and update the order book from a depth message.
@@ -177,18 +176,7 @@ class DownloadBinanceData(ContinuousDownloader):
             The ``data`` field from the combined-stream depth envelope.
 
         """
-        updates = _parser_book(data)
-        for price, qty in updates.items():
-            if qty == 0:
-                self.d.pop(price, None)
-            else:
-                self.d[price] = qty
-        self._data[self.t] = dict(self.d)
-
-    def _raw_parser(self, data: object) -> None:
-        if self.t not in self._data:
-            self._data[self.t] = []
-        self._data[self.t].append(data)  # type: ignore[union-attr]
+        self._push_book_updates(_parser_book(data))
 
 
 def get_trades_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
@@ -210,8 +198,7 @@ def get_trades_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_trades)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_trades_saver(IODataBase(path, method=form))
     downloader(pair=pair)
 
 
@@ -234,8 +221,7 @@ def get_orderbook_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_marketdepth)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_book_saver(IODataBase(path, method=form))
     downloader(pair=pair)
 
 
@@ -246,7 +232,8 @@ def get_data_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
     Parameters
     ----------
     path : str
-        Path to save data.
+        Root path; trades saved under ``<path>/trades/``, book under
+        ``<path>/book/``.
     pair : str, optional
         Trading pair in Binance format (e.g. 'BTCUSDT'), default is 'BTCUSDT'.
     time_step : int, optional
@@ -258,6 +245,6 @@ def get_data_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
 
     """
     downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
-    downloader.set_process_data(set_orders)
-    downloader.set_saver(IODataBase(path, method=form))
+    downloader.set_trades_saver(IODataBase(f'{path}/trades', method=form))
+    downloader.set_book_saver(IODataBase(f'{path}/book', method=form))
     downloader(pair=pair)

{dccd-2.1.0 → dccd-2.2.0}/dccd/continuous_dl/bitfinex.py

@@ -31,7 +31,6 @@ Low level API
 """
 
 # Built-in packages
-import asyncio
 import logging
 import time
 from typing import Any
@@ -117,7 +116,8 @@ class DownloadBitfinexData(ContinuousDownloader):
 
     """
 
-    def __init__(self, time_step: int = 60, until: int | None = 3600) -> None:
+    def __init__(self, time_step: int = 60, until: int | None = 3600,
+                 checkpoint_dir: str | None = None) -> None:
         """ Initialize object.
 
         Parameters
@@ -127,6 +127,9 @@ class DownloadBitfinexData(ContinuousDownloader):
         until : int or None, optional
             Seconds to run, or a future Unix timestamp to stop at.
             Default is ``3600``.
+        checkpoint_dir : str or None, optional
+            Directory to write the order-book crash-recovery checkpoint.
+            Disabled when ``None`` (default).
 
         """
         if until is None:
@@ -135,7 +138,7 @@ class DownloadBitfinexData(ContinuousDownloader):
             until -= int(time.time())
 
         ContinuousDownloader.__init__(self, 'bitfinex', time_step=time_step,
-                                      STOP=until)
+                                      STOP=until, checkpoint_dir=checkpoint_dir)
 
         self._parser_data: dict[str, Any] = {
             'book': self.parser_book,
@@ -144,7 +147,7 @@ class DownloadBitfinexData(ContinuousDownloader):
             'trades_raw': self.parser_raw_trades,
         }
         self.logger = logging.getLogger(__name__)
-        self.d: dict[str, Any] = {}
+        self._load_checkpoint()
 
     def parser_raw_book(self, data: list[Any]) -> None:
         """ Parse raw order book, each timestep set in a list all orders.
@@ -177,7 +180,9 @@ class DownloadBitfinexData(ContinuousDownloader):
         else:
             self.d.pop(parsed['price'])
 
-        self._data[self.t] = {v['price']: v['amount'] for v in self.d.values()}  # type: ignore[assignment]
+        self._data.setdefault(self.t, {'trades': [], 'book': {}})['book'] = {
+            v['price']: v['amount'] for v in self.d.values()
+        }
 
     def parser_raw_trades(self, data: list[Any]) -> None:
         """ Parse raw trade data tick-by-tick.
@@ -248,18 +253,9 @@ class DownloadBitfinexData(ContinuousDownloader):
 
         """
         self.parser = self.get_parser(channel)
-
         channel = channel[:-4] if channel[-4:] == '_raw' else channel
-
         self.logger.info('Try connect WS and set {} stream.'.format(channel))
-
-        self.loop = asyncio.get_event_loop()
-        self.loop.run_until_complete(asyncio.gather(
-            self._connect(channel=channel, **kwargs),
-            self._loop()
-        ))
-
-        return self
+        return super().__call__(channel=channel, **kwargs)  # type: ignore[return-value]
 
 
 # =========================================================================== #

{dccd-2.1.0 → dccd-2.2.0}/dccd/continuous_dl/bitmex.py

@@ -31,7 +31,6 @@ Low level API
 """
 
 # Built-in packages
-import asyncio
 import time
 from datetime import datetime as dt
 from typing import Any
@@ -180,8 +179,8 @@ class DownloadBitmexData(ContinuousDownloader):
             'orderBookL2_25': self.parser_book,
             'trade': self.parser_trades,
         }
-        self.d: dict[int, Any] = {}
         self.start = False
+        self._load_checkpoint()
 
     def parser_book(self, data: dict[str, Any]) -> None:
         """ Parse and maintain a local copy of the order book.
@@ -214,7 +213,9 @@ class DownloadBitmexData(ContinuousDownloader):
             else:
                 self.logger.error('Unknown action {}: {}'.format(action, data))
 
-        self._data[self.t] = {v['price']: v['amount'] for v in self.d.values()}  # type: ignore[assignment]
+        self._data.setdefault(self.t, {'trades': [], 'book': {}})['book'] = {
+            v['price']: v['amount'] for v in self.d.values()
+        }
 
     def parser_trades(self, data: dict[str, Any]) -> None:
         """ Parse trade data and accumulate records for the current timestep.
@@ -226,15 +227,12 @@ class DownloadBitmexData(ContinuousDownloader):
             key with a list of trade records.
 
         """
-        i, _data = 0, []
-        for d in data['data']:
-            _data += [_parser_trades(d, i)]
-            i += 1
+        slot = self._data.setdefault(self.t, {'trades': [], 'book': {}})
+        for i, d in enumerate(data['data']):
+            slot['trades'].append(_parser_trades(d, i))
 
-        if self.t in self._data.keys():
-            self._data[self.t] += _data
-        else:
-            self._data[self.t] = _data
+    def _restore_book_state(self, state: dict[int, Any]) -> None:  # type: ignore[override]
+        self.d = {int(k): v for k, v in state.items()}
 
     async def on_message(self, data: dict[str, Any] | list[Any]) -> None:
         """ Route an incoming websocket message to the appropriate parser. """
@@ -270,16 +268,8 @@ class DownloadBitmexData(ContinuousDownloader):
 
         """
         self.parser = self.get_parser(args[0])
-
         self.logger.info('Try connect WS and set {} stream.'.format(args[0]))
-
-        self.loop = asyncio.get_event_loop()
-        self.loop.run_until_complete(asyncio.gather(
-            self._connect(args=':'.join(args)),
-            self._loop()
-        ))
-
-        return self
+        return super().__call__(args=':'.join(args))  # type: ignore[return-value]
 
 
 # =========================================================================== #