dccd 2.0.2__tar.gz → 2.1.0__tar.gz

{dccd-2.0.2 → dccd-2.1.0}/CHANGELOG.md

@@ -6,6 +6,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.1.0] - 2026-05-15
+
+### Added
+
+- `dccd/tests/test_binance.py`, `test_kraken.py`, `test_bybit.py`, `test_okx.py`, `test_coinbase.py` — REST error-scenario tests: HTTP 500 and malformed response for every exchange (#22)
+- `dccd/continuous_dl/binance.py` — `DownloadBinanceData` streaming trades and order book via Binance combined WebSocket streams (#20)
+- `dccd/continuous_dl/kraken.py` — `DownloadKrakenData` streaming trades, order book, and OHLCV via Kraken WebSocket v2 (#20)
+- `dccd/continuous_dl/okx.py` — `DownloadOKXData` streaming trades, order book, and candles via OKX WebSocket v5 (#20)
+- `get_trades_*`, `get_orderbook_*`, `get_data_*` high-level helpers for Binance, Kraken, and OKX (#20)
+- `dccd/tests/test_binance_ws.py`, `test_kraken_ws.py`, `test_okx_ws.py` — 34 new tests for the new WS modules (#20)
+- `README.rst` and `doc/source/index.rst` — exchange support matrix table (REST/WS × data type) (#20)
+- `dccd/tests/test_websocket.py`, `test_bitfinex.py`, `test_bitmex.py`, `test_bybit_ws.py` — tests for `continuous_dl` and `BasisWebSocket`; coverage lifted from excluded to 82% overall (#12)
+- `dccd/tests/test_histo_dl.py` — tests for `_get_last_date` (xlsx, csv, parquet, empty directory) (#12)
+- `pyproject.toml` — `pytest-asyncio>=0.23` added to dev dependencies (#12)
+
+### Fixed
+
+- `dccd/tools/date_time.py` — `span_to_str` and `str_to_span` now cover all spans supported by the exchanges: 180 s (3 m), 900 s (15 m), 14400 s (4 h), 21600 s (6 h), 28800 s (8 h), 43200 s (12 h), 259200 s (3 d), 1296000 s (15 d), 2592000 s (1 M); previously any span outside the original 7 values returned `None` and silently broke the save path (#21)
+- `dccd/histo_dl/kraken.py` — `import_data` now raises `UserWarning` when `end` is passed, as the Kraken OHLC API does not support a custom end date and silently ignored the parameter (#21)
+
+### Changed
+
+- `dccd/continuous_dl/exchange.py` — `get_parser()` now raises `KeyError` on unknown key instead of falling back to the removed debug parser; `_loop()` awaits `is_connect` instead of `_data` (#22)
+- `dccd/continuous_dl/bitmex.py`, `dccd/continuous_dl/bybit.py` — added numpydoc docstrings on `_parser_trades()` and `_parser_book()` (#22)
+- `dccd/histo_dl/exchange.py` — `ImportDataCryptoCurrencies` docstring updated: `See Also` lists all five exchanges; `platform` parameter documents all supported values; fixed typos (#22)
+- `dccd/histo_dl/exchange.py` — `ImportDataCryptoCurrencies` now inherits from `ABC` and `_import_data` is decorated with `@abstractmethod`, preventing accidental instantiation of the base class (#21)
+- `dccd/histo_dl/binance.py`, `coinbase.py`, `bybit.py`, `okx.py`, `kraken.py` — added `from __future__ import annotations`, `from typing import Any`, and full type hints on `_import_data` and `import_data` signatures (#21)
+- `dccd/histo_dl/exchange.py` — `_get_last_date` now reads `.csv` and `.parquet` files in addition to `.xlsx` instead of falling back to 2012-01-01 (#12)
+- `dccd/histo_dl/exchange.py` — completed numpydoc docstrings for `_get_last_date`, `_set_by_period`, `_name_file`, `_excel_format`, `_sort_data`, `set_hierarchy` (#12)
+- `dccd/tools/io.py` — documented `driver`, `username`, `password`, `host`, `port` parameters of `save_as_sql` (#12)
+- `dccd/continuous_dl/exchange.py` — documented `time_step=None` tick-by-tick behaviour in `ContinuousDownloader` (#12)
+- `dccd/continuous_dl/bitfinex.py` — resolved all inline TODOs, added full type annotations, removed dead `__main__` block (#12)
+- `dccd/continuous_dl/bitmex.py` — resolved all inline TODOs, added full type annotations, fixed undefined `pair` variable in `get_data_bitmex`, removed dead `__main__` block (#12)
+- `pyproject.toml` — removed `bitfinex` and `bitmex` from mypy `ignore_errors` override; lifted `continuous_dl/*` and `tools/websocket.py` from coverage omit (#12)
+
+### Removed
+
+- `ContinuousDownloader._parser_debug()` — dead method, never called; `dccd/tools/__init__.py` commented-out imports removed (#22)
+
 ## [2.0.2] - 2026-05-15
 
 ### Changed

{dccd-2.0.2 → dccd-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.0.2
+Version: 2.1.0
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -35,6 +35,7 @@ Requires-Dist: pyarrow>=13; extra == "io"
 Requires-Dist: polars>=0.20; extra == "io"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
 Requires-Dist: pytest-cov>=4.1; extra == "dev"
 Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: interrogate>=1.5; extra == "dev"
@@ -75,7 +76,11 @@ Download Crypto-Currency Data
     :target: https://download-crypto-currencies-data.readthedocs.io/en/latest/
     :alt: Documentation Status
 
-.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/master/interrogate_badge_output.svg
+.. image:: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data/branch/master/graph/badge.svg
+    :target: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data
+    :alt: Coverage
+
+.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/badges/interrogate_badge.svg
     :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
     :alt: Docstring Coverage
 
@@ -107,23 +112,25 @@ From source::
 Supported exchanges
 ===================
 
-+------------------+------------+-----------------+
-| Exchange         | Historical | Real-time       |
-+==================+============+=================+
-| Binance          | ✓          |                 |
-+------------------+------------+-----------------+
-| Coinbase         | ✓          |                 |
-+------------------+------------+-----------------+
-| Kraken           | ✓          |                 |
-+------------------+------------+-----------------+
-| Bybit            | ✓          | ✓               |
-+------------------+------------+-----------------+
-| OKX              | ✓          |                 |
-+------------------+------------+-----------------+
-| Bitfinex         |            | ✓               |
-+------------------+------------+-----------------+
-| Bitmex           |            | ✓               |
-+------------------+------------+-----------------+
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
++==================+============+=============+=================+==========+===========+================+
+| Binance          | ✓          |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Coinbase         | ✓          |             |                 |          |           |                |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitmex           |            |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+
+\* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
 Presentation
 ============

{dccd-2.0.2 → dccd-2.1.0}/README.rst

@@ -25,7 +25,11 @@ Download Crypto-Currency Data
     :target: https://download-crypto-currencies-data.readthedocs.io/en/latest/
     :alt: Documentation Status
 
-.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/master/interrogate_badge_output.svg
+.. image:: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data/branch/master/graph/badge.svg
+    :target: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data
+    :alt: Coverage
+
+.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/badges/interrogate_badge.svg
     :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
     :alt: Docstring Coverage
 
@@ -57,23 +61,25 @@ From source::
 Supported exchanges
 ===================
 
-+------------------+------------+-----------------+
-| Exchange         | Historical | Real-time       |
-+==================+============+=================+
-| Binance          | ✓          |                 |
-+------------------+------------+-----------------+
-| Coinbase         | ✓          |                 |
-+------------------+------------+-----------------+
-| Kraken           | ✓          |                 |
-+------------------+------------+-----------------+
-| Bybit            | ✓          | ✓               |
-+------------------+------------+-----------------+
-| OKX              | ✓          |                 |
-+------------------+------------+-----------------+
-| Bitfinex         |            | ✓               |
-+------------------+------------+-----------------+
-| Bitmex           |            | ✓               |
-+------------------+------------+-----------------+
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
++==================+============+=============+=================+==========+===========+================+
+| Binance          | ✓          |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Coinbase         | ✓          |             |                 |          |           |                |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Kraken           | ✓          |             |                 | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bybit            | ✓          |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| OKX              | ✓          |             |                 | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitmex           |            |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+
+\* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
 
 Presentation
 ============

{dccd-2.0.2 → dccd-2.1.0}/dccd/continuous_dl/__init__.py

@@ -9,8 +9,8 @@
 """ Module to download continuously data.
 
 Module to download continuously data (orderbook, trades, etc.) and update
-automatically the database. *Currently only supports Bitfinex and Bitmex
-exchanges.*
+automatically the database. Supports Binance, Bitfinex, Bitmex, Bybit,
+Kraken, and OKX exchanges via WebSocket.
 
 .. currentmodule:: dccd.continuous_dl
 
@@ -18,8 +18,12 @@ exchanges.*
    :maxdepth: 1
    :caption: Contents:
 
+   continuous_dl.binance
    continuous_dl.bitfinex
    continuous_dl.bitmex
+   continuous_dl.bybit
+   continuous_dl.kraken
+   continuous_dl.okx
 
 """
 
@@ -28,12 +32,18 @@ exchanges.*
 # Third party packages
 
 # Local packages
-from . import bitfinex, bitmex, bybit, exchange
+from . import binance, bitfinex, bitmex, bybit, exchange, kraken, okx
+from .binance import *
 from .bitfinex import *
 from .bitmex import *
 from .bybit import *
+from .kraken import *
+from .okx import *
 
 __all__ = ['exchange']
+__all__ += binance.__all__
 __all__ += bitfinex.__all__
 __all__ += bitmex.__all__
 __all__ += bybit.__all__
+__all__ += kraken.__all__
+__all__ += okx.__all__

dccd-2.1.0/dccd/continuous_dl/binance.py

@@ -0,0 +1,263 @@
+#!/usr/bin/env python3
+# coding: utf-8
+
+""" Objects and functions to download data from Binance exchange (WebSocket).
+
+.. currentmodule:: dccd.continuous_dl.binance
+
+High level API
+--------------
+
+.. autofunction:: get_data_binance
+.. autofunction:: get_orderbook_binance
+.. autofunction:: get_trades_binance
+
+Low level API
+-------------
+
+.. autoclass:: dccd.continuous_dl.binance.DownloadBinanceData
+   :members: set_process_data, set_saver
+   :special-members: __call__
+   :show-inheritance:
+
+"""
+
+# Built-in packages
+import logging
+import time
+
+# Third party packages
+# Local packages
+from dccd.continuous_dl.exchange import ContinuousDownloader
+from dccd.process_data import set_marketdepth, set_orders, set_trades
+from dccd.tools.io import IODataBase
+
+__all__ = [
+    'DownloadBinanceData', 'get_data_binance', 'get_orderbook_binance',
+    'get_trades_binance',
+]
+
+_BINANCE_WS_URL = 'wss://stream.binance.com:9443/stream?streams={sym}@trade/{sym}@depth50@100ms'
+
+
+def _parser_trades(data: dict) -> list[dict]:
+    """ Parse a trade message from Binance combined stream.
+
+    Parameters
+    ----------
+    data : dict
+        The ``data`` field of a combined-stream trade message.
+
+    Returns
+    -------
+    list of dict
+        Each dict has keys: ``tid``, ``timestamp``, ``price``, ``amount``,
+        ``type`` ('buy' or 'sell').
+
+    """
+    return [{
+        'tid': data['t'],
+        'timestamp': int(data['T']) / 1000,
+        'price': float(data['p']),
+        'amount': float(data['q']),
+        'type': 'sell' if data['m'] else 'buy',
+    }]
+
+
+def _parser_book(data: dict) -> dict:
+    """ Parse a depth message from Binance combined stream.
+
+    Parameters
+    ----------
+    data : dict
+        The ``data`` field of a combined-stream depth message.
+
+    Returns
+    -------
+    dict
+        Price levels as strings; bids positive, asks prefixed with ``'-'``
+        and negative. Zero quantity means the level was removed.
+
+    """
+    book: dict[str, float] = {}
+    for bid in data.get('b', []):
+        book[bid[0]] = float(bid[1])
+    for ask in data.get('a', []):
+        book['-' + ask[0]] = -float(ask[1])
+    return book
+
+
+class DownloadBinanceData(ContinuousDownloader):
+    """ Download data continuously from Binance via combined WebSocket streams.
+
+    Parameters
+    ----------
+    pair : str
+        Trading pair symbol in Binance format (e.g. 'BTCUSDT').
+    time_step : int, optional
+        Seconds between data snapshots, default is 60.
+    until : int, optional
+        Seconds to run or stop timestamp, default is 3600.
+
+    Attributes
+    ----------
+    host : str
+        WebSocket URL (combined stream endpoint).
+    is_connect : bool
+        True if connected.
+    ts : int
+        Snapshot interval in seconds.
+    until : int
+        Stop timestamp.
+
+    Methods
+    -------
+    set_process_data
+    set_saver
+    __call__
+
+    """
+
+    def __init__(self, pair: str = 'BTCUSDT', time_step: int = 60,
+                 until: int | None = 3600) -> None:
+        """ Initialize object. """
+        if until is None:
+            until = 0
+        elif until > time.time():
+            until -= int(time.time())
+
+        self.pair = pair
+        url = _BINANCE_WS_URL.format(sym=pair.lower())
+        ContinuousDownloader.__init__(self, url, time_step=time_step, STOP=until)
+        self._parser_data = {
+            'trades': self.parser_trades,
+            'book': self.parser_book,
+        }
+        self.logger = logging.getLogger(__name__)
+        self.d: dict[str, float] = {}
+
+    async def _subscribe(self, **kwargs: object) -> None:
+        """ Wait for connection; Binance streams are declared in the URL. """
+        await self.wait_that('ws')
+        self.is_connect = True
+
+    async def on_message(self, msg: dict) -> None:
+        """ Dispatch incoming combined-stream WebSocket messages.
+
+        Parameters
+        ----------
+        msg : dict
+            Combined-stream envelope with ``stream`` and ``data`` keys.
+
+        """
+        stream = msg.get('stream', '')
+        if '@trade' in stream:
+            self.parser_trades(msg['data'])
+        elif '@depth' in stream:
+            self.parser_book(msg['data'])
+
+    def parser_trades(self, data: dict) -> None:
+        """ Parse and store a trade message.
+
+        Parameters
+        ----------
+        data : dict
+            The ``data`` field from the combined-stream trade envelope.
+
+        """
+        for trade in _parser_trades(data):
+            self._raw_parser(trade)
+
+    def parser_book(self, data: dict) -> None:
+        """ Parse and update the order book from a depth message.
+
+        Parameters
+        ----------
+        data : dict
+            The ``data`` field from the combined-stream depth envelope.
+
+        """
+        updates = _parser_book(data)
+        for price, qty in updates.items():
+            if qty == 0:
+                self.d.pop(price, None)
+            else:
+                self.d[price] = qty
+        self._data[self.t] = dict(self.d)
+
+    def _raw_parser(self, data: object) -> None:
+        if self.t not in self._data:
+            self._data[self.t] = []
+        self._data[self.t].append(data)  # type: ignore[union-attr]
+
+
+def get_trades_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
+                       until: int = 3600, form: str = 'csv') -> None:
+    """ Download trades data from Binance via WebSocket.
+
+    Parameters
+    ----------
+    path : str
+        Path to save data.
+    pair : str, optional
+        Trading pair in Binance format (e.g. 'BTCUSDT'), default is 'BTCUSDT'.
+    time_step : int, optional
+        Seconds between snapshots, default is 60.
+    until : int, optional
+        Duration in seconds or stop timestamp, default is 3600.
+    form : str, optional
+        Save format ('csv', 'parquet', etc.), default is 'csv'.
+
+    """
+    downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
+    downloader.set_process_data(set_trades)
+    downloader.set_saver(IODataBase(path, method=form))
+    downloader(pair=pair)
+
+
+def get_orderbook_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
+                          until: int = 3600, form: str = 'csv') -> None:
+    """ Download order book data from Binance via WebSocket.
+
+    Parameters
+    ----------
+    path : str
+        Path to save data.
+    pair : str, optional
+        Trading pair in Binance format (e.g. 'BTCUSDT'), default is 'BTCUSDT'.
+    time_step : int, optional
+        Seconds between snapshots, default is 60.
+    until : int, optional
+        Duration in seconds or stop timestamp, default is 3600.
+    form : str, optional
+        Save format ('csv', 'parquet', etc.), default is 'csv'.
+
+    """
+    downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
+    downloader.set_process_data(set_marketdepth)
+    downloader.set_saver(IODataBase(path, method=form))
+    downloader(pair=pair)
+
+
+def get_data_binance(path: str, pair: str = 'BTCUSDT', time_step: int = 60,
+                     until: int = 3600, form: str = 'csv') -> None:
+    """ Download order book and trades data from Binance via WebSocket.
+
+    Parameters
+    ----------
+    path : str
+        Path to save data.
+    pair : str, optional
+        Trading pair in Binance format (e.g. 'BTCUSDT'), default is 'BTCUSDT'.
+    time_step : int, optional
+        Seconds between snapshots, default is 60.
+    until : int, optional
+        Duration in seconds or stop timestamp, default is 3600.
+    form : str, optional
+        Save format ('csv', 'parquet', etc.), default is 'csv'.
+
+    """
+    downloader = DownloadBinanceData(pair=pair, time_step=time_step, until=until)
+    downloader.set_process_data(set_orders)
+    downloader.set_saver(IODataBase(path, method=form))
+    downloader(pair=pair)