dccd 2.2.0__tar.gz → 2.3.1__tar.gz

{dccd-2.2.0 → dccd-2.3.1}/CHANGELOG.md

@@ -6,6 +6,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.3.1] - 2026-05-24
+
+### Fixed
+
+- `dccd/storage.py` — `DataStore.missing_intervals` now detects the gap **before** the first saved row when the requested `start` predates `file_min`; previously only the trailing gap (after `file_max`) was returned, causing `dccd backfill --start <early-date>` to silently skip all historical data before the first existing candle (#46)
+- `dccd/histo_dl/coinbase.py` — raise `RuntimeError` when Coinbase returns HTTP 200 with a JSON dict (e.g. `{"message": "..."}` for near-future windows) instead of silently iterating dict keys and crashing with `ValueError` (#45)
+- `dccd/histo_dl/coinbase.py` — additional guard: raise `RuntimeError` when Coinbase returns a JSON list whose first element is not itself a list/tuple (e.g. `["message"]`); previously caused `float("m")` `ValueError` (#45)
+- `dccd/histo_dl/exchange.py` — `_sort_data` no longer raises `KeyError: 'TS'` when the API returns empty data; returns early with an empty `self.df` so the backfill skips the window cleanly (#45)
+- `dccd/histo_dl/exchange.py` — `_sort_data` strips any candle at or beyond `self.end` before merging; exchanges with inclusive endpoint semantics (Coinbase) no longer cause `_advance` to overshoot by one span per window, preventing drift that accumulated into near-future requests (#45)
+- `dccd/histo_dl/okx.py` — raise `RuntimeError` when OKX response code is not `"0"`, letting the backfill retry/skip logic handle API-level errors (#45)
+- `dccd/histo_dl/okx.py` — switch `_import_data` from `/market/candles` to `/market/history-candles`; the former only serves the last ~24 h of 1-minute bars and silently returns empty data for older windows (#45)
+
+## [2.3.0] - 2026-05-22
+
+### Added
+
+- `dccd/storage.py` — `DataStore.is_period_complete(year)`: checks whether an annual parquet file contains all expected candles; `DataStore.missing_intervals(start, end)`: gap-detection — complete past years are skipped, incomplete years resume from the last saved row (#41)
+- `dccd/daemon/backfill.py` — `_BackfillBase.run()` now iterates over `DataStore.missing_intervals()` instead of a single sliding window from `last_saved`; complete years are never re-downloaded (#41)
+- `dccd/storage.py` — new `DataStore` class: unified read/write interface for OHLC, trades, and orderbook; `save(df)` (merge-on-TS, annual OHLC / daily trades+orderbook), `load(start, end)`, `existing_periods()`, `last_timestamp()` (#39)
+- `dccd/tools/date_time.py` — `span_label(span)` converts seconds to short directory labels (``'1m'``, ``'1h'``, ``'1d'``…); `_SPAN_LABEL` mapping exported (#39)
+- `doc/source/storage.rst` — Sphinx page for `DataStore` with directory layout examples (#39)
+
+### Changed
+
+- `dccd collect` (formerly `dccd run`) — renamed to clarify the distinction: `collect` = one incremental batch, `backfill` = full historical download with gap detection, `start` = continuous daemon (#41)
+- New storage arborescence: ``{data_path}/{exchange}/ohlc/{pair}/{span}/YYYY.parquet``, ``…/trades/{pair}/YYYY-MM-DD.parquet``, ``…/orderbook/{pair}/YYYY-MM-DD.parquet`` — replaces the old ``{Exchange}/Data/Clean_Data/{per}/{pair}/`` layout (#39)
+- `dccd/histo_dl/exchange.py` — `save()`, `_get_last_date()`, `save_trades()`, `save_orderbook()` now delegate to `DataStore`; removed `last_df`, `_set_by_period`, `_name_file`, `_excel_format`; removed unused `set_hierarchy()` (#39, #41)
+- `dccd/histo_dl/{binance,bybit,coinbase,okx}.py` — removed `full_path` overrides (base class sets the correct path via `DataStore`) (#39)
+- `dccd/daemon/backfill.py`, `scheduler.py` — removed `by_period` parameter; `save()` call simplified (#39)
+- `dccd/daemon/stream_manager.py` — WebSocket save path now built from `DataStore.directory` (#39)
+- `dccd/daemon/config.py` — `HistoJob.by_period` field removed; granularity is automatic (#39)
+
+- `dccd/histo_dl/exchange.py` — `save()` now supports `form='parquet'`; previously only `'xlsx'` and `'csv'` were handled (#35)
+- `config.yml` — ready-to-use daemon config for minutely OHLC + real-time orderbook/trades on Binance, Kraken, and Bybit (#35)
+- `dccd/daemon/backfill.py` — `OHLCBackfill` and `KrakenBackfill` strategy classes with shared retry/progress/save loop; `make_job()` factory; `run_backfill()` orchestrator; tqdm progress bars and optional `--parallel` execution (#38)
+- `dccd/daemon/cli.py` — `dccd backfill` command: reads all job definitions from config, supports `--exchange` / `--pairs` filters, `--start`, `--parallel`, and `--dry-run` flags (#38)
+- `dccd/daemon/config.py` — `SettingsConfig` with `data_path` and `timezone` fields; `CollectorConfig.settings` propagates `data_path` to `StorageConfig.local_path` when not set explicitly (#38)
+
+### Removed
+
+- `scripts/backfill.py` — replaced by `dccd backfill` CLI command and `dccd.daemon.backfill` module (#38)
+
+### Fixed
+
+- `dccd/histo_dl/exchange.py` — `save(form='parquet')` was silently ignored (logged a warning instead of writing the file) (#35)
+- `dccd/histo_dl/exchange.py` — `_sort_data()` crashed with a ValueError when the API returned fewer candles than the expected window size; index is now derived from actual data (#36)
+- `dccd/histo_dl/exchange.py` — `by_period='M'` produced minute-level file names (strftime `%M`) instead of year-month; added `_PERIOD_FMT` mapping so `'M'` → `'%Y-%m'` (#36)
+- `dccd/histo_dl/exchange.py` — `self.end` now reflects the last candle timestamp so window-loop callers advance correctly (was stuck at `now` for Kraken) (#36)
+- `dccd/histo_dl/binance.py` — missing `limit=1000` parameter caused Binance to return only 500 candles per request (#36)
+- `dccd/histo_dl/bybit.py` — `limit` was 200; raised to 1 000 to match the API maximum (#36)
+- `dccd/histo_dl/exchange.py` — `_sort_data()` dropped the minute just before a window boundary when the last trade arrived ≥2 spans early; grid now uses `self.end` directly as the exclusive stop (#36)
+
 ## [2.2.0] - 2026-05-17
 
 ### Added

{dccd-2.2.0 → dccd-2.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.2.0
+Version: 2.3.1
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -37,6 +37,7 @@ Provides-Extra: daemon
 Requires-Dist: pyyaml>=6.0; extra == "daemon"
 Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
 Requires-Dist: typer>=0.12; extra == "daemon"
+Requires-Dist: tqdm>=4.64; extra == "daemon"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.4; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
@@ -48,6 +49,7 @@ Requires-Dist: pandas-stubs>=2.0; extra == "dev"
 Requires-Dist: pyyaml>=6.0; extra == "dev"
 Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
 Requires-Dist: typer>=0.12; extra == "dev"
+Requires-Dist: tqdm>=4.64; extra == "dev"
 Provides-Extra: doc
 Requires-Dist: sphinx>=7.0; extra == "doc"
 Requires-Dist: furo; extra == "doc"
@@ -240,8 +242,11 @@ Daemon (autonomous collector) — ``config.yml``:
 
 .. code-block:: yaml
 
+    settings:
+      data_path: /data/crypto/
+      timezone: UTC
+
     storage:
-      local_path: /data/crypto/
       remotes:
         - provider: rclone
           remote: "mynas:crypto/"
@@ -252,7 +257,6 @@ Daemon (autonomous collector) — ``config.yml``:
         pairs: [BTC/USDT, ETH/USDT]
         span: 3600
         format: parquet
-        by_period: Y
 
     stream_jobs:
       - exchange: binance
@@ -260,6 +264,30 @@ Daemon (autonomous collector) — ``config.yml``:
         channels: [trades, book]
         time_step: 60
 
+CLI quick start:
+
+.. code-block:: bash
+
+    # Validate the config
+    dccd validate --config config.yml
+
+    # Backfill all OHLC history defined in config (resumable)
+    dccd backfill --config config.yml --start "2020-01-01 00:00:00"
+
+    # Dry run — estimate windows and time without downloading
+    dccd backfill --config config.yml --dry-run
+
+    # Backfill only one exchange
+    dccd backfill --config config.yml --exchange kraken
+
+    # One incremental batch per job, then exit (for cron)
+    dccd collect --config config.yml
+
+    # Continuous daemon (Ctrl-C to stop)
+    dccd start --config config.yml
+
+Python API:
+
 .. code-block:: python
 
     from dccd.daemon.config import load_config

{dccd-2.2.0 → dccd-2.3.1}/README.rst

@@ -180,8 +180,11 @@ Daemon (autonomous collector) — ``config.yml``:
 
 .. code-block:: yaml
 
+    settings:
+      data_path: /data/crypto/
+      timezone: UTC
+
     storage:
-      local_path: /data/crypto/
       remotes:
         - provider: rclone
           remote: "mynas:crypto/"
@@ -192,7 +195,6 @@ Daemon (autonomous collector) — ``config.yml``:
         pairs: [BTC/USDT, ETH/USDT]
         span: 3600
         format: parquet
-        by_period: Y
 
     stream_jobs:
       - exchange: binance
@@ -200,6 +202,30 @@ Daemon (autonomous collector) — ``config.yml``:
         channels: [trades, book]
         time_step: 60
 
+CLI quick start:
+
+.. code-block:: bash
+
+    # Validate the config
+    dccd validate --config config.yml
+
+    # Backfill all OHLC history defined in config (resumable)
+    dccd backfill --config config.yml --start "2020-01-01 00:00:00"
+
+    # Dry run — estimate windows and time without downloading
+    dccd backfill --config config.yml --dry-run
+
+    # Backfill only one exchange
+    dccd backfill --config config.yml --exchange kraken
+
+    # One incremental batch per job, then exit (for cron)
+    dccd collect --config config.yml
+
+    # Continuous daemon (Ctrl-C to stop)
+    dccd start --config config.yml
+
+Python API:
+
 .. code-block:: python
 
     from dccd.daemon.config import load_config

{dccd-2.2.0 → dccd-2.3.1}/dccd/daemon/__init__.py

@@ -10,6 +10,7 @@ Submodules
 
 .. autosummary::
 
+   backfill
    config
    health
    storage
@@ -18,6 +19,7 @@ Submodules
 
 """
 
+from dccd.daemon.backfill import KrakenBackfill, OHLCBackfill, make_job, run_backfill
 from dccd.daemon.config import CollectorConfig, load_config
 from dccd.daemon.health import HealthMonitor
 from dccd.daemon.scheduler import build_histo_scheduler, run_once
@@ -27,7 +29,11 @@ from dccd.daemon.stream_manager import StreamManager, SyncService
 __all__ = [
     'CollectorConfig',
     'HealthMonitor',
+    'KrakenBackfill',
+    'OHLCBackfill',
     'load_config',
+    'make_job',
+    'run_backfill',
     'RemoteStorage',
     'StreamManager',
     'SyncService',