dccd 2.0.2__tar.gz → 2.2.0__tar.gz

dccd-2.2.0/CHANGELOG.md

@@ -0,0 +1,140 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [2.2.0] - 2026-05-17
+
+### Added
+
+- `dccd/histo_dl/exchange.py` — `import_trades(start, end)` and `import_orderbook(depth)` public methods on `ImportDataCryptoCurrencies`; `_sort_trades` / `_sort_orderbook` helpers validate via Pydantic, sort and deduplicate; `trades_df` / `orderbook_df` attributes; `save_trades` / `save_orderbook` save helpers (#31)
+- `dccd/histo_dl/{binance,kraken,bybit,okx,coinbase}.py` — `_import_trades(start, end)` and `_import_orderbook(depth)` implemented for all five exchanges; Binance and Kraken support full history via paginated endpoints; Bybit (≤ 1 000) and Coinbase (≤ 100) return recent-only snapshots (#31)
+- `dccd/models.py` — `Trade.tid` made optional (`int | None`); `OrderBookEntry` gains required `side` field (`'bid'` or `'ask'`) and `count` made optional (`int | None`) (#31)
+- `dccd/daemon/health.py` — `HealthMonitor`: rotating log handler (10 MB × 5 files), per-job metrics JSON, and optional Slack/Discord webhook alerts on consecutive failures; `JobMetrics` dataclass (#30)
+- `dccd/daemon/cli.py` — `dccd` CLI (`validate`, `run`, `start`, `status`, `add` commands) via typer; `[project.scripts]` entrypoint; `typer>=0.12` added to the `daemon` extra (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager` (one thread per `(exchange, pair)`, auto-restart on crash) and `SyncService` (periodic rclone push to all remotes, decoupled from collection) (#26)
+- `dccd/daemon/config.py` — declarative YAML config with Pydantic v2: `CollectorConfig`, `HistoJob`, `StreamJob`, `StorageConfig`, `AlertConfig`, `RemoteConfig`, `load_config()` (#25)
+- `dccd/daemon/storage.py` — `RemoteStorage.push()` via rclone; supports multiple remotes and root-path sync (#25, #26)
+- `dccd/daemon/scheduler.py` — `build_histo_scheduler()` (APScheduler 3.x), `run_histo_job()`, `run_once()` (#25)
+- `examples/config.example.yml` — annotated reference config for the daemon (#25)
+- `examples/daemon_example.py` — programmatic daemon example in 6 steps (#30)
+- `pyproject.toml` — `[daemon]` optional extra (`pyyaml`, `apscheduler`, `typer`) (#25, #30)
+
+### Changed
+
+- `dccd/daemon/scheduler.py` — `run_histo_job`, `build_histo_scheduler`, `run_once` accept an optional `health: HealthMonitor` parameter (#30)
+- `dccd/daemon/stream_manager.py` — `StreamManager.__init__` accepts optional `health: HealthMonitor`; `_run_forever` records success/failure on each iteration (#30)
+- `dccd/daemon/config.py` — `StorageConfig.remote` replaced by `remotes: list[RemoteConfig]` and `sync_interval: int` (#26)
+- `dccd/histo_dl/{binance,coinbase,bybit,okx,kraken}.py` — `format_pair(crypto, fiat)` extracted as a static method, independently testable (#29)
+- `dccd/continuous_dl/exchange.py` — unified `__call__`, `_push_trades`, `_push_book_updates`, `_get_book_state`, `_restore_book_state` in base class; separate `set_trades_saver` / `set_book_saver`; crash-recovery checkpoint; `snapshot_ts` injected into every snapshot payload (#28, #29)
+
+## [2.1.0] - 2026-05-15
+
+### Added
+
+- `dccd/tests/test_binance.py`, `test_kraken.py`, `test_bybit.py`, `test_okx.py`, `test_coinbase.py` — REST error-scenario tests: HTTP 500 and malformed response for every exchange (#22)
+- `dccd/continuous_dl/binance.py` — `DownloadBinanceData` streaming trades and order book via Binance combined WebSocket streams (#20)
+- `dccd/continuous_dl/kraken.py` — `DownloadKrakenData` streaming trades, order book, and OHLCV via Kraken WebSocket v2 (#20)
+- `dccd/continuous_dl/okx.py` — `DownloadOKXData` streaming trades, order book, and candles via OKX WebSocket v5 (#20)
+- `get_trades_*`, `get_orderbook_*`, `get_data_*` high-level helpers for Binance, Kraken, and OKX (#20)
+- `dccd/tests/test_binance_ws.py`, `test_kraken_ws.py`, `test_okx_ws.py` — 34 new tests for the new WS modules (#20)
+- `README.rst` and `doc/source/index.rst` — exchange support matrix table (REST/WS × data type) (#20)
+- `dccd/tests/test_websocket.py`, `test_bitfinex.py`, `test_bitmex.py`, `test_bybit_ws.py` — tests for `continuous_dl` and `BasisWebSocket`; coverage lifted from excluded to 82% overall (#12)
+- `dccd/tests/test_histo_dl.py` — tests for `_get_last_date` (xlsx, csv, parquet, empty directory) (#12)
+- `pyproject.toml` — `pytest-asyncio>=0.23` added to dev dependencies (#12)
+
+### Fixed
+
+- `dccd/tools/date_time.py` — `span_to_str` and `str_to_span` now cover all spans supported by the exchanges: 180 s (3 m), 900 s (15 m), 14400 s (4 h), 21600 s (6 h), 28800 s (8 h), 43200 s (12 h), 259200 s (3 d), 1296000 s (15 d), 2592000 s (1 M); previously any span outside the original 7 values returned `None` and silently broke the save path (#21)
+- `dccd/histo_dl/kraken.py` — `import_data` now raises `UserWarning` when `end` is passed, as the Kraken OHLC API does not support a custom end date and silently ignored the parameter (#21)
+
+### Changed
+
+- `dccd/continuous_dl/exchange.py` — `get_parser()` now raises `KeyError` on unknown key instead of falling back to the removed debug parser; `_loop()` awaits `is_connect` instead of `_data` (#22)
+- `dccd/continuous_dl/bitmex.py`, `dccd/continuous_dl/bybit.py` — added numpydoc docstrings on `_parser_trades()` and `_parser_book()` (#22)
+- `dccd/histo_dl/exchange.py` — `ImportDataCryptoCurrencies` docstring updated: `See Also` lists all five exchanges; `platform` parameter documents all supported values; fixed typos (#22)
+- `dccd/histo_dl/exchange.py` — `ImportDataCryptoCurrencies` now inherits from `ABC` and `_import_data` is decorated with `@abstractmethod`, preventing accidental instantiation of the base class (#21)
+- `dccd/histo_dl/binance.py`, `coinbase.py`, `bybit.py`, `okx.py`, `kraken.py` — added `from __future__ import annotations`, `from typing import Any`, and full type hints on `_import_data` and `import_data` signatures (#21)
+- `dccd/histo_dl/exchange.py` — `_get_last_date` now reads `.csv` and `.parquet` files in addition to `.xlsx` instead of falling back to 2012-01-01 (#12)
+- `dccd/histo_dl/exchange.py` — completed numpydoc docstrings for `_get_last_date`, `_set_by_period`, `_name_file`, `_excel_format`, `_sort_data`, `set_hierarchy` (#12)
+- `dccd/tools/io.py` — documented `driver`, `username`, `password`, `host`, `port` parameters of `save_as_sql` (#12)
+- `dccd/continuous_dl/exchange.py` — documented `time_step=None` tick-by-tick behaviour in `ContinuousDownloader` (#12)
+- `dccd/continuous_dl/bitfinex.py` — resolved all inline TODOs, added full type annotations, removed dead `__main__` block (#12)
+- `dccd/continuous_dl/bitmex.py` — resolved all inline TODOs, added full type annotations, fixed undefined `pair` variable in `get_data_bitmex`, removed dead `__main__` block (#12)
+- `pyproject.toml` — removed `bitfinex` and `bitmex` from mypy `ignore_errors` override; lifted `continuous_dl/*` and `tools/websocket.py` from coverage omit (#12)
+
+### Removed
+
+- `ContinuousDownloader._parser_debug()` — dead method, never called; `dccd/tools/__init__.py` commented-out imports removed (#22)
+
+## [2.0.2] - 2026-05-15
+
+### Changed
+
+- `README.rst` — added PyPI status, docstring coverage, and downloads badges
+
+## [2.0.1] - 2026-05-14
+
+### Changed
+
+- Docstrings `See Also` updated in `FromBinance`, `FromKraken`, `FromCoinbase` — replaced defunct `FromGDax`/`FromPoloniex` with `FromBybit`/`FromOKX`
+- `doc/source/index.rst` — exchange lists updated (Bybit, OKX added); all exchange RST pages added to toctree (previously orphaned)
+- `dccd/__init__.py` module docstring — exchange list updated
+- `pyproject.toml` — added `Documentation` and `Changelog` project URLs
+- `README.rst` — added exchange support table, output format section, multi-exchange Quick start examples
+- `examples/historical_downloader.py` — rewritten with modern API (Binance + Parquet)
+
+## [2.0.0] - 2026-05-14
+
+### Added
+
+- `pyproject.toml` (PEP 517/518) — replaces `setup.py` (#5)
+- GitHub Actions CI (`.github/workflows/ci.yml`) — matrix Python 3.10/3.11/3.12/3.13, jobs `test` and `lint` (#5)
+- `dccd/histo_dl/coinbase.py` — `FromCoinbase` class replacing the defunct GDAX module (#5)
+- `.githooks/pre-push` — Git Flow enforcement (#5)
+- `CONTRIBUTING.md` — development setup, Git Flow, commit conventions (#5)
+- `CHANGELOG.md` (#5)
+- `.pre-commit-config.yaml` — hooks `ruff` (lint + fix) et `ruff-format` (#7)
+- `dccd/tests/test_date_time.py`, `test_io.py`, `test_process_data.py` — couverture ≥ 80 % (#8)
+- `.github/workflows/badges.yml` — badge couverture docstrings via `interrogate` (#8)
+- `dccd/histo_dl/bybit.py` — `FromBybit` : téléchargement historique Bybit v5 REST (#9)
+- `dccd/continuous_dl/bybit.py` — `DownloadBybitData` : stream WebSocket Bybit v5 (#9)
+- `dccd/histo_dl/okx.py` — `FromOKX` : téléchargement historique OKX v5 REST (#9)
+- `dccd/models.py` — `OHLCBar`, `Trade`, `OrderBookEntry` : validation pydantic des réponses API (#9)
+- `IODataBase.save_as_parquet` — format Parquet via pyarrow (optionnel `dccd[io]`) (#9)
+- `IODataBase.save_as_polars` — format Polars, Parquet sous le capot (optionnel `dccd[io]`) (#9)
+- `ImportDataCryptoCurrencies.get_data(format='polars')` — retourne un `pl.DataFrame` (#9)
+- `dccd/tools/date_time.py`, `tools/io.py`, `histo_dl/exchange.py`, `continuous_dl/exchange.py`, `tools/websocket.py` — type hints complets (#10)
+- `.github/workflows/release.yml` — publication automatique PyPI + GitHub Release sur tag `v*` via OIDC (#10)
+
+### Changed
+
+- **Breaking:** minimum Python version is now 3.10 (dropped 3.5–3.9) (#5)
+- **Breaking:** minimum dependency versions bumped — `pandas>=2.0`, `SQLAlchemy>=2.0`, `numpy>=1.26`, `requests>=2.28`, `websockets>=12.0`, `scipy>=1.10` (#5)
+- Replaced `xlrd` + `xlsxwriter` with `openpyxl` for Excel I/O (#5)
+- `dccd/histo_dl/exchange.py`: `df.append()` → `pd.concat()`, `ffill()`, `openpyxl` engine (#5)
+- `dccd/tools/io.py`: `SQLAlchemy URL()` → `URL.create()`, `df.append()` → `pd.concat()` (#5)
+- Version now managed via `importlib.metadata` (#5)
+- `dccd/tools/websocket.py`: `asyncio.get_event_loop().run_until_complete()` → `asyncio.run()` (#7)
+- `dccd/tests/conftest.py`: fixtures `tmp_data_path` + mocks HTTP — les tests ne font plus d'appels réseau (#7)
+- `doc/source/conf.py` : thème scipy → furo, extensions modernisées (#8)
+- `dccd/histo_dl/binance.py` : API v1 → v3 (#9)
+- `dccd/histo_dl/exchange.py` : `_fetch()` avec retry tenacity sur HTTP 429 (#9)
+- `dccd/tools/websocket.py` : reconnexion automatique avec `max_retries` et `retry_delay` (#9)
+- `print()` remplacés par `logging` dans `exchange.py`, `binance.py`, `date_time.py` (#9)
+- `pyproject.toml` : `mypy>=1.0` + `pandas-stubs>=2.0` dans `dev`, section `[tool.mypy]` (#10)
+- `.github/workflows/ci.yml` : étape `mypy dccd/` ajoutée dans le job `lint` (#10)
+- `dccd/tools/websocket.py` : arguments mutables `conn={}` / `subs={}` corrigés en `None` (#10)
+
+### Fixed
+
+- `dccd/tools/io.py` : logique CSV inversée dans `save_as_csv` — le fichier existant était écrasé au lieu d'être appendé (#8)
+
+### Removed
+
+- **Breaking:** `FromPoloniex` and `dccd/histo_dl/poloniex.py` — Poloniex exchange shut down in 2024 (#5)
+- **Breaking:** `FromGDax` and `dccd/histo_dl/gdax.py` — GDAX API endpoint defunct; replaced by `FromCoinbase` (#5)
+- `setup.py`, `tox.ini`, `requirements.txt`, `doc-requirements.txt`, `.travis.yml` (#5)

{dccd-2.0.2 → dccd-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.0.2
+Version: 2.2.0
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -33,19 +33,29 @@ Requires-Dist: pydantic>=2.0
 Provides-Extra: io
 Requires-Dist: pyarrow>=13; extra == "io"
 Requires-Dist: polars>=0.20; extra == "io"
+Provides-Extra: daemon
+Requires-Dist: pyyaml>=6.0; extra == "daemon"
+Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
+Requires-Dist: typer>=0.12; extra == "daemon"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
 Requires-Dist: pytest-cov>=4.1; extra == "dev"
 Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: interrogate>=1.5; extra == "dev"
 Requires-Dist: mypy>=1.0; extra == "dev"
 Requires-Dist: pandas-stubs>=2.0; extra == "dev"
+Requires-Dist: pyyaml>=6.0; extra == "dev"
+Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
+Requires-Dist: typer>=0.12; extra == "dev"
 Provides-Extra: doc
 Requires-Dist: sphinx>=7.0; extra == "doc"
 Requires-Dist: furo; extra == "doc"
 Requires-Dist: numpydoc; extra == "doc"
 Requires-Dist: sphinx-design; extra == "doc"
 Requires-Dist: sphinx-copybutton; extra == "doc"
+Requires-Dist: pyyaml>=6.0; extra == "doc"
+Requires-Dist: apscheduler<4,>=3.10; extra == "doc"
 Dynamic: license-file
 
 =============================
@@ -75,7 +85,11 @@ Download Crypto-Currency Data
     :target: https://download-crypto-currencies-data.readthedocs.io/en/latest/
     :alt: Documentation Status
 
-.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/master/interrogate_badge_output.svg
+.. image:: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data/branch/master/graph/badge.svg
+    :target: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data
+    :alt: Coverage
+
+.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/badges/interrogate_badge.svg
     :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
     :alt: Docstring Coverage
 
@@ -98,6 +112,10 @@ With optional Parquet / Polars support::
 
     $ pip install "dccd[io]"
 
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
 From source::
 
     $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
@@ -107,23 +125,27 @@ From source::
 Supported exchanges
 ===================
 
-+------------------+------------+-----------------+
-| Exchange         | Historical | Real-time       |
-+==================+============+=================+
-| Binance          | ✓          |                 |
-+------------------+------------+-----------------+
-| Coinbase         | ✓          |                 |
-+------------------+------------+-----------------+
-| Kraken           | ✓          |                 |
-+------------------+------------+-----------------+
-| Bybit            | ✓          | ✓               |
-+------------------+------------+-----------------+
-| OKX              | ✓          |                 |
-+------------------+------------+-----------------+
-| Bitfinex         |            | ✓               |
-+------------------+------------+-----------------+
-| Bitmex           |            | ✓               |
-+------------------+------------+-----------------+
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
++==================+============+=============+=================+==========+===========+================+
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitmex           |            |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+
+\* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
+
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
 
 Presentation
 ============
@@ -137,6 +159,14 @@ Presentation
     Stream real-time data (order book, trades) via WebSocket with automatic
     reconnection and configurable processing/saving callbacks.
 
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
 Output formats
 --------------
 
@@ -148,7 +178,9 @@ Parquet files can be read back as either a ``pandas.DataFrame`` or a
 Quick start
 ===========
 
-Historical data (pandas)::
+Historical data (pandas):
+
+.. code-block:: python
 
     from dccd.histo_dl import FromBinance
 
@@ -157,15 +189,21 @@ Historical data (pandas)::
     obj.save(form='parquet')
     df = obj.get_data()            # pandas DataFrame
 
-Polars output::
+Polars output:
+
+.. code-block:: python
 
     df_pl = obj.get_data(format='polars')
 
-Incremental update (resume from last saved point)::
+Incremental update (resume from last saved point):
+
+.. code-block:: python
 
     obj.import_data(start='last', end='now').save(form='parquet')
 
-Other exchanges::
+Other exchanges:
+
+.. code-block:: python
 
     from dccd.histo_dl import FromKraken, FromBybit, FromOKX
 
@@ -173,6 +211,73 @@ Other exchanges::
     FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
     FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
 
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    storage:
+      local_path: /data/crypto/
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+        by_period: Y
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
 Links
 =====
 

dccd-2.2.0/README.rst

@@ -0,0 +1,227 @@
+=============================
+Download Crypto-Currency Data
+=============================
+
+.. image:: https://img.shields.io/pypi/pyversions/dccd
+    :alt: PyPI - Python Version
+
+.. image:: https://img.shields.io/pypi/v/dccd.svg
+    :target: https://pypi.org/project/dccd/
+    :alt: PyPI
+
+.. image:: https://img.shields.io/pypi/status/dccd.svg?colorB=blue
+    :target: https://pypi.org/project/dccd/
+    :alt: PyPI - Status
+
+.. image:: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/actions/workflows/ci.yml/badge.svg
+    :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/actions/workflows/ci.yml
+    :alt: CI
+
+.. image:: https://img.shields.io/github/license/ArthurBernard/Download_Crypto_Currencies_Data.svg
+    :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/blob/master/LICENSE.txt
+    :alt: License
+
+.. image:: https://readthedocs.org/projects/download-crypto-currencies-data/badge/?version=latest
+    :target: https://download-crypto-currencies-data.readthedocs.io/en/latest/
+    :alt: Documentation Status
+
+.. image:: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data/branch/master/graph/badge.svg
+    :target: https://codecov.io/gh/ArthurBernard/Download_Crypto_Currencies_Data
+    :alt: Coverage
+
+.. image:: https://raw.githubusercontent.com/ArthurBernard/Download_Crypto_Currencies_Data/badges/interrogate_badge.svg
+    :target: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
+    :alt: Docstring Coverage
+
+.. image:: https://pepy.tech/badge/dccd
+    :target: https://pepy.tech/project/dccd
+    :alt: Downloads
+
+Python package to download crypto-currency data (OHLCV, trades, order book) from multiple
+exchanges via REST and WebSocket APIs. Data can be saved to CSV, Excel, SQLite, PostgreSQL,
+or Parquet.
+
+Installation
+============
+
+From pip::
+
+    $ pip install dccd
+
+With optional Parquet / Polars support::
+
+    $ pip install "dccd[io]"
+
+With autonomous daemon support (APScheduler + PyYAML)::
+
+    $ pip install "dccd[daemon]"
+
+From source::
+
+    $ git clone https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
+    $ cd Download_Crypto_Currencies_Data
+    $ pip install -e .
+
+Supported exchanges
+===================
+
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Exchange         | REST OHLCV | REST Trades | REST Order Book | WS OHLCV | WS Trades | WS Order Book  |
++==================+============+=============+=================+==========+===========+================+
+| Binance          | ✓          | ✓           | ✓               |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Coinbase         | ✓          | ✓†          | ✓               |          |           |                |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Kraken           | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bybit            | ✓          | ✓†          | ✓               |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| OKX              | ✓          | ✓           | ✓               | ✓        | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitfinex         |            |             |                 | ✓\*      | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+| Bitmex           |            |             |                 |          | ✓         | ✓              |
++------------------+------------+-------------+-----------------+----------+-----------+----------------+
+
+\* Bitfinex WS OHLCV is aggregated from the trades stream via ``get_ohlc_bitfinex``.
+
+† Recent trades only (Bybit ≤ 1 000, Coinbase ≤ 100) — no deep historical pagination via the public REST API.
+
+Presentation
+============
+
+**Historical Downloader** ``dccd.histo_dl``
+    Download OHLCV data via REST APIs and save to disk. Supports chunked
+    requests, automatic retry on rate-limit (HTTP 429), and incremental
+    updates from the last saved timestamp.
+
+**Continuous Downloader** ``dccd.continuous_dl``
+    Stream real-time data (order book, trades) via WebSocket with automatic
+    reconnection and configurable processing/saving callbacks.
+
+**Daemon** ``dccd.daemon``
+    Autonomous, server-side collector driven by a YAML config.  Runs REST
+    jobs on a schedule (APScheduler), opens WebSocket streams for real-time
+    collection, and periodically syncs all local data to one or more remote
+    destinations (NAS, S3, SFTP, …) via rclone.  Multiple remotes and a
+    configurable sync interval are supported; collection is never blocked by
+    remote availability.
+
+Output formats
+--------------
+
+Historical data can be saved as **CSV**, **Excel** (``.xlsx``), **SQLite**,
+**PostgreSQL** (via SQLAlchemy), or **Parquet** (requires ``dccd[io]``).
+Parquet files can be read back as either a ``pandas.DataFrame`` or a
+``polars.DataFrame``.
+
+Quick start
+===========
+
+Historical data (pandas):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance
+
+    obj = FromBinance('/path/to/data/', 'BTC', 3600, fiat='USDT')
+    obj.import_data(start='2024-01-01 00:00:00', end='2024-12-31 00:00:00')
+    obj.save(form='parquet')
+    df = obj.get_data()            # pandas DataFrame
+
+Polars output:
+
+.. code-block:: python
+
+    df_pl = obj.get_data(format='polars')
+
+Incremental update (resume from last saved point):
+
+.. code-block:: python
+
+    obj.import_data(start='last', end='now').save(form='parquet')
+
+Other exchanges:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromKraken, FromBybit, FromOKX
+
+    FromKraken('/path/', 'ETH', 3600).import_data(start='2024-01-01', end='now').save()
+    FromBybit('/path/', 'BTC', 86400).import_data(start='2024-01-01', end='now').save()
+    FromOKX('/path/', 'BTC', 3600).import_data(start='2024-01-01', end='now').save()
+
+Trades (historical or recent):
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromBinance, FromKraken
+
+    obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
+    obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
+    obj.save_trades(form='csv')
+    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+
+    # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
+    FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
+
+Order book snapshot:
+
+.. code-block:: python
+
+    from dccd.histo_dl import FromOKX
+
+    obj = FromOKX('/path/', 'BTC', 3600)
+    obj.import_orderbook(depth=50)
+    obj.save_orderbook(form='csv')
+    df = obj.orderbook_df    # columns: side, price, amount, count
+
+Daemon (autonomous collector) — ``config.yml``:
+
+.. code-block:: yaml
+
+    storage:
+      local_path: /data/crypto/
+      remotes:
+        - provider: rclone
+          remote: "mynas:crypto/"
+      sync_interval: 3600
+
+    histo_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT, ETH/USDT]
+        span: 3600
+        format: parquet
+        by_period: Y
+
+    stream_jobs:
+      - exchange: binance
+        pairs: [BTC/USDT]
+        channels: [trades, book]
+        time_step: 60
+
+.. code-block:: python
+
+    from dccd.daemon.config import load_config
+    from dccd.daemon.scheduler import run_once, build_histo_scheduler
+    from dccd.daemon.stream_manager import StreamManager
+
+    config = load_config('config.yml')
+
+    # One-shot: download all histo jobs once, then exit
+    run_once(config)
+
+    # Daemon mode: periodic REST + live WebSocket streams
+    scheduler = build_histo_scheduler(config)
+    scheduler.start()
+
+    mgr = StreamManager(config)
+    mgr.start()      # runs until mgr.stop() is called
+
+Links
+=====
+
+- PyPI: https://pypi.org/project/dccd/
+- Documentation: https://download-crypto-currencies-data.readthedocs.io/
+- Source: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data
+- Changelog: https://github.com/ArthurBernard/Download_Crypto_Currencies_Data/blob/master/CHANGELOG.md

{dccd-2.0.2 → dccd-2.2.0}/dccd/continuous_dl/__init__.py

@@ -9,8 +9,8 @@
 """ Module to download continuously data.
 
 Module to download continuously data (orderbook, trades, etc.) and update
-automatically the database. *Currently only supports Bitfinex and Bitmex
-exchanges.*
+automatically the database. Supports Binance, Bitfinex, Bitmex, Bybit,
+Kraken, and OKX exchanges via WebSocket.
 
 .. currentmodule:: dccd.continuous_dl
 
@@ -18,8 +18,12 @@ exchanges.*
    :maxdepth: 1
    :caption: Contents:
 
+   continuous_dl.binance
    continuous_dl.bitfinex
    continuous_dl.bitmex
+   continuous_dl.bybit
+   continuous_dl.kraken
+   continuous_dl.okx
 
 """
 
@@ -28,12 +32,18 @@ exchanges.*
 # Third party packages
 
 # Local packages
-from . import bitfinex, bitmex, bybit, exchange
+from . import binance, bitfinex, bitmex, bybit, exchange, kraken, okx
+from .binance import *
 from .bitfinex import *
 from .bitmex import *
 from .bybit import *
+from .kraken import *
+from .okx import *
 
 __all__ = ['exchange']
+__all__ += binance.__all__
 __all__ += bitfinex.__all__
 __all__ += bitmex.__all__
 __all__ += bybit.__all__
+__all__ += kraken.__all__
+__all__ += okx.__all__