dccd 2.3.1__tar.gz → 2.3.2__tar.gz

{dccd-2.3.1 → dccd-2.3.2}/CHANGELOG.md

@@ -6,10 +6,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [2.3.1] - 2026-05-24
+## [2.3.2] - 2026-05-25
+
+### Added
+
+- `dccd/daemon/cli.py` — `dccd status --json`: emit raw metrics as a JSON object on stdout, suitable for piping into Grafana / jq (#53)
+- `dccd/daemon/config.py` — `HistoJob.max_retries` (int, 1–10, default 3) and `HistoJob.retry_delay` (float ≥ 0, default 2.0): per-job retry configuration for transient network errors; delay is exponential (`retry_delay * 2^(attempt-1)`) (#53)
+- `dccd/daemon/config.py` — `resolve_config_path()` and `DEFAULT_CONFIG_PATH`: CLI commands now fall back to `$XDG_CONFIG_HOME/dccd/config.yml` (default `~/.config/dccd/config.yml`) when no `--config` option is provided and `./config.yml` does not exist (#49)
+- `dccd/daemon/cli.py` — `dccd inventory`: scans `data_path` and prints a table of all stored OHLC, trades, and orderbook data with date range, row count, and gap count per series; uses Polars for fast columnar reads (#50)
+- `dccd/daemon/cli.py` — `dccd remove --exchange X --pair Y --span N`: removes a pair from a histo_job (or the whole job if it was the last pair) and re-validates the config before writing (#50)
+
+### Changed
+
+- `dccd/storage.py`, `dccd/histo_dl/exchange.py`, `dccd/daemon/backfill.py`, `dccd/process_data.py`, `dccd/daemon/stream_manager.py` — replace pandas with polars throughout; `DataStore.save/load` accept/return `pl.DataFrame`; `get_data()` defaults to `format='polars'`; `set_marketdepth` returns a flat long-format `pl.DataFrame`; stream savers write parquet via `DataStore`; `pandas` removed from core dependencies (#52)
+- `dccd/daemon/backfill.py` — backfill progress bar now shows the current window date (`YYYY-MM-DD → YYYY-MM-DD`) instead of a raw window count (`n win`); makes it easy to see which period is being downloaded at a glance (#48)
 
 ### Fixed
 
+- `dccd/histo_dl/exchange.py` — `_sort_data` no longer raises `ColumnNotFoundError: "date"` when the exchange API returns an empty candle list; the polars migration (PR #52) had re-introduced a variant of the empty-data crash from v2.3.1; now returns early with an empty `self.df` (#54)
+
+## [2.3.1] - 2026-05-24
+
+### Fixed
 - `dccd/storage.py` — `DataStore.missing_intervals` now detects the gap **before** the first saved row when the requested `start` predates `file_min`; previously only the trailing gap (after `file_max`) was returned, causing `dccd backfill --start <early-date>` to silently skip all historical data before the first existing candle (#46)
 - `dccd/histo_dl/coinbase.py` — raise `RuntimeError` when Coinbase returns HTTP 200 with a JSON dict (e.g. `{"message": "..."}` for near-future windows) instead of silently iterating dict keys and crashing with `ValueError` (#45)
 - `dccd/histo_dl/coinbase.py` — additional guard: raise `RuntimeError` when Coinbase returns a JSON list whose first element is not itself a list/tuple (e.g. `["message"]`); previously caused `float("m")` `ValueError` (#45)

{dccd-2.3.1 → dccd-2.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 2.3.1
+Version: 2.3.2
 Summary: Download Crypto Currency Data from different exchanges.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT
@@ -22,7 +22,8 @@ Requires-Python: >=3.10
 Description-Content-Type: text/x-rst
 License-File: LICENSE.txt
 Requires-Dist: numpy>=1.26
-Requires-Dist: pandas>=2.0
+Requires-Dist: polars>=0.20
+Requires-Dist: pyarrow>=13
 Requires-Dist: requests>=2.28
 Requires-Dist: openpyxl>=3.1
 Requires-Dist: websockets>=12.0
@@ -31,8 +32,8 @@ Requires-Dist: SQLAlchemy>=2.0
 Requires-Dist: tenacity>=8.0
 Requires-Dist: pydantic>=2.0
 Provides-Extra: io
-Requires-Dist: pyarrow>=13; extra == "io"
 Requires-Dist: polars>=0.20; extra == "io"
+Requires-Dist: pyarrow>=13; extra == "io"
 Provides-Extra: daemon
 Requires-Dist: pyyaml>=6.0; extra == "daemon"
 Requires-Dist: apscheduler<4,>=3.10; extra == "daemon"
@@ -45,7 +46,6 @@ Requires-Dist: pytest-cov>=4.1; extra == "dev"
 Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: interrogate>=1.5; extra == "dev"
 Requires-Dist: mypy>=1.0; extra == "dev"
-Requires-Dist: pandas-stubs>=2.0; extra == "dev"
 Requires-Dist: pyyaml>=6.0; extra == "dev"
 Requires-Dist: apscheduler<4,>=3.10; extra == "dev"
 Requires-Dist: typer>=0.12; extra == "dev"
@@ -110,10 +110,6 @@ From pip::
 
     $ pip install dccd
 
-With optional Parquet / Polars support::
-
-    $ pip install "dccd[io]"
-
 With autonomous daemon support (APScheduler + PyYAML)::
 
     $ pip install "dccd[daemon]"
@@ -173,14 +169,14 @@ Output formats
 --------------
 
 Historical data can be saved as **CSV**, **Excel** (``.xlsx``), **SQLite**,
-**PostgreSQL** (via SQLAlchemy), or **Parquet** (requires ``dccd[io]``).
-Parquet files can be read back as either a ``pandas.DataFrame`` or a
-``polars.DataFrame``.
+**PostgreSQL** (via SQLAlchemy), or **Parquet**.
+All DataFrames are native ``polars.DataFrame``.  A ``pandas.DataFrame`` can be
+obtained via ``get_data(format='pandas')``.
 
 Quick start
 ===========
 
-Historical data (pandas):
+Historical data:
 
 .. code-block:: python
 
@@ -189,13 +185,8 @@ Historical data (pandas):
     obj = FromBinance('/path/to/data/', 'BTC', 3600, fiat='USDT')
     obj.import_data(start='2024-01-01 00:00:00', end='2024-12-31 00:00:00')
     obj.save(form='parquet')
-    df = obj.get_data()            # pandas DataFrame
-
-Polars output:
-
-.. code-block:: python
-
-    df_pl = obj.get_data(format='polars')
+    df = obj.get_data()               # polars DataFrame (default)
+    df_pd = obj.get_data(format='pandas')  # pandas DataFrame (optional)
 
 Incremental update (resume from last saved point):
 
@@ -222,7 +213,7 @@ Trades (historical or recent):
     obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
     obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
     obj.save_trades(form='csv')
-    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+    df = obj.trades_df    # polars DataFrame — columns: TS, price, amount, type, tid
 
     # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
     FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
@@ -286,6 +277,16 @@ CLI quick start:
     # Continuous daemon (Ctrl-C to stop)
     dccd start --config config.yml
 
+    # Add / remove a histo job in-place
+    dccd add --exchange kraken --pair ETH/USD --span 86400 --config config.yml
+    dccd remove --exchange kraken --pair ETH/USD --span 86400 --config config.yml
+
+    # Inspect all data on disk (OHLC, trades, orderbook)
+    dccd inventory --config config.yml
+
+Note: ``--config`` is optional — dccd searches ``./config.yml`` then
+``~/.config/dccd/config.yml`` when omitted.
+
 Python API:
 
 .. code-block:: python

{dccd-2.3.1 → dccd-2.3.2}/README.rst

@@ -48,10 +48,6 @@ From pip::
 
     $ pip install dccd
 
-With optional Parquet / Polars support::
-
-    $ pip install "dccd[io]"
-
 With autonomous daemon support (APScheduler + PyYAML)::
 
     $ pip install "dccd[daemon]"
@@ -111,14 +107,14 @@ Output formats
 --------------
 
 Historical data can be saved as **CSV**, **Excel** (``.xlsx``), **SQLite**,
-**PostgreSQL** (via SQLAlchemy), or **Parquet** (requires ``dccd[io]``).
-Parquet files can be read back as either a ``pandas.DataFrame`` or a
-``polars.DataFrame``.
+**PostgreSQL** (via SQLAlchemy), or **Parquet**.
+All DataFrames are native ``polars.DataFrame``.  A ``pandas.DataFrame`` can be
+obtained via ``get_data(format='pandas')``.
 
 Quick start
 ===========
 
-Historical data (pandas):
+Historical data:
 
 .. code-block:: python
 
@@ -127,13 +123,8 @@ Historical data (pandas):
     obj = FromBinance('/path/to/data/', 'BTC', 3600, fiat='USDT')
     obj.import_data(start='2024-01-01 00:00:00', end='2024-12-31 00:00:00')
     obj.save(form='parquet')
-    df = obj.get_data()            # pandas DataFrame
-
-Polars output:
-
-.. code-block:: python
-
-    df_pl = obj.get_data(format='polars')
+    df = obj.get_data()               # polars DataFrame (default)
+    df_pd = obj.get_data(format='pandas')  # pandas DataFrame (optional)
 
 Incremental update (resume from last saved point):
 
@@ -160,7 +151,7 @@ Trades (historical or recent):
     obj = FromBinance('/path/', 'BTC', 3600, fiat='USDT')
     obj.import_trades(start='2024-01-01 00:00:00', end='2024-01-02 00:00:00')
     obj.save_trades(form='csv')
-    df = obj.trades_df    # pandas DataFrame — columns: timestamp, price, amount, type, tid
+    df = obj.trades_df    # polars DataFrame — columns: TS, price, amount, type, tid
 
     # Kraken also supports full history; Bybit/Coinbase return recent-only snapshots
     FromKraken('/path/', 'BTC', 3600).import_trades(start='2024-01-01', end='2024-01-02').save_trades()
@@ -224,6 +215,16 @@ CLI quick start:
     # Continuous daemon (Ctrl-C to stop)
     dccd start --config config.yml
 
+    # Add / remove a histo job in-place
+    dccd add --exchange kraken --pair ETH/USD --span 86400 --config config.yml
+    dccd remove --exchange kraken --pair ETH/USD --span 86400 --config config.yml
+
+    # Inspect all data on disk (OHLC, trades, orderbook)
+    dccd inventory --config config.yml
+
+Note: ``--config`` is optional — dccd searches ``./config.yml`` then
+``~/.config/dccd/config.yml`` when omitted.
+
 Python API:
 
 .. code-block:: python

{dccd-2.3.1 → dccd-2.3.2}/dccd/continuous_dl/bitfinex.py

@@ -286,7 +286,7 @@ def get_data_bitfinex(channel: str, process_func: Any, process_params: dict[str,
                    'Oracle', 'MSSQL', 'MySQL'},
         It will create an IODataBase object to save/update the database in the
         specified format `save_method`, default is 'DataFrame' it save as
-        binary pd.DataFrame object. More informations are available into
+        binary pl.DataFrame object. More informations are available into
         :mod:`dccd.tools.io`.
     io_params : dict, optional
         Dictionary of the keyword arguments available to the

{dccd-2.3.1 → dccd-2.3.2}/dccd/daemon/backfill.py

@@ -37,9 +37,10 @@ from __future__ import annotations
 
 import time as time_mod
 from abc import ABC, abstractmethod
+from datetime import datetime, timezone
 from typing import TYPE_CHECKING
 
-import pandas as pd
+import polars as pl
 from tqdm import tqdm
 
 from dccd.tools.date_time import date_to_TS
@@ -92,6 +93,12 @@ class _BackfillBase(ABC):
     form : str
         Output format (accepted for backward compatibility; storage is
         always Parquet via :class:`~dccd.storage.DataStore`).
+    max_retries : int, optional
+        Maximum fetch attempts per window before skipping it.  Default 3.
+    retry_delay : float, optional
+        Base delay in seconds between retries.  Actual delay is
+        ``retry_delay * 2 ** (attempt - 1)`` (exponential back-off).
+        Default 2.0.
 
     """
 
@@ -100,10 +107,14 @@ class _BackfillBase(ABC):
         obj: ImportDataCryptoCurrencies,
         sleep: float,
         form: str,
+        max_retries: int = 3,
+        retry_delay: float = 2.0,
     ) -> None:
         self.obj = obj
         self.sleep = sleep
         self.form = form
+        self.max_retries = max_retries
+        self.retry_delay = retry_delay
         cls_name = type(obj).__name__[4:]  # strip leading 'From'
         self.label = f'{cls_name:8s} {obj.crypto}/{obj.fiat}'
 
@@ -184,9 +195,15 @@ class _BackfillBase(ABC):
             tqdm.write(f'[{self.label}] already up to date')
             return
 
-        total = sum(max(1, (e - s) // self.window_size + 1) for s, e in intervals)
+        def _iso(ts: int) -> str:
+            return datetime.fromtimestamp(ts, tz=timezone.utc).strftime('%Y-%m-%d')
+
+        total        = sum(max(1, (e - s) // self.window_size + 1) for s, e in intervals)
+        end_date_str = _iso(intervals[-1][1])
         bar = tqdm(
-            total=total, desc=self.label, unit='win',
+            total=total,
+            desc=f'{self.label}  {_iso(intervals[0][0])} → {end_date_str}',
+            unit='',
             position=position, leave=True, dynamic_ncols=True,
         )
 
@@ -200,22 +217,23 @@ class _BackfillBase(ABC):
 
                 last_exc: Exception | None = None
                 n = 0
-                for attempt in range(1, _MAX_RETRIES + 1):
+                for attempt in range(1, self.max_retries + 1):
                     try:
                         n = self._fetch_window(current, end)
                         break
                     except Exception as exc:
                         last_exc = exc
-                        if attempt < _MAX_RETRIES:
+                        if attempt < self.max_retries:
+                            delay = self.retry_delay * (2 ** (attempt - 1))
                             tqdm.write(
-                                f'[{self.label}] attempt {attempt}/{_MAX_RETRIES} '
-                                f'failed: {exc} — retrying in 2s'
+                                f'[{self.label}] attempt {attempt}/{self.max_retries} '
+                                f'failed: {exc} — retrying in {delay:.1f}s'
                             )
-                            time_mod.sleep(2)
+                            time_mod.sleep(delay)
                 else:
                     tqdm.write(
                         f'[{self.label}] window {current} failed after '
-                        f'{_MAX_RETRIES} attempts: {last_exc} — skipping'
+                        f'{self.max_retries} attempts: {last_exc} — skipping'
                     )
                     skipped += 1
                     current += self.window_size
@@ -227,8 +245,10 @@ class _BackfillBase(ABC):
                     self.obj.save()
                     n_candles += n
 
+                current_date = _iso(current)
                 current = self._advance(current, end)
                 bar.update(1)
+                bar.set_description(f'{self.label}  {current_date} → {end_date_str}')
                 bar.set_postfix(candles=n_candles, skipped=skipped)
                 time_mod.sleep(self.sleep)
 
@@ -256,6 +276,10 @@ class OHLCBackfill(_BackfillBase):
         Seconds to wait between requests.
     form : str
         Output format (accepted for backward compatibility).
+    max_retries : int, optional
+        See :class:`_BackfillBase`.  Default 3.
+    retry_delay : float, optional
+        See :class:`_BackfillBase`.  Default 2.0.
 
     """
 
@@ -265,8 +289,10 @@ class OHLCBackfill(_BackfillBase):
         max_candles: int,
         sleep: float,
         form: str,
+        max_retries: int = 3,
+        retry_delay: float = 2.0,
     ) -> None:
-        super().__init__(obj, sleep, form)
+        super().__init__(obj, sleep, form, max_retries=max_retries, retry_delay=retry_delay)
         self.max_candles = max_candles
 
     @property
@@ -275,7 +301,7 @@ class OHLCBackfill(_BackfillBase):
 
     def _fetch_window(self, current: int, end: int) -> int:
         self.obj.import_data(start=current, end=end)
-        if self.obj.df is None or self.obj.df.empty:
+        if self.obj.df is None or len(self.obj.df) == 0:
             return 0
         self._warn_if_suspicious(current, end)
         return len(self.obj.df)
@@ -439,28 +465,29 @@ class KrakenBackfill(_BackfillBase):
             return []
 
         span = self.obj.span
-        df = pd.DataFrame(trades)
-        df['ts'] = pd.to_datetime(df['timestamp'], unit='s', utc=True)
-        df = df.set_index('ts').sort_index()
-
-        freq = f'{span}s'
-        origin = pd.Timestamp(start_ts, unit='s', tz='UTC')
-        resample_kw = {'closed': 'left', 'label': 'left', 'origin': origin}
-        ohlc = df['price'].resample(freq, **resample_kw).ohlc()
-        vol  = df['amount'].resample(freq, **resample_kw).sum()
-        qvol = (df['price'] * df['amount']).resample(freq, **resample_kw).sum()
-
+        df = (
+            pl.DataFrame(trades)
+            .with_columns(pl.from_epoch('timestamp', time_unit='s').alias('ts'))
+            .sort('ts')
+        )
         result = (
-            ohlc.join(vol.rename('volume')).join(qvol.rename('quoteVolume'))
-            .loc[
-                pd.Timestamp(start_ts, unit='s', tz='UTC'):
-                pd.Timestamp(end_ts - 1, unit='s', tz='UTC'),
-            ]
-            .dropna(subset=['open'])
+            df.group_by_dynamic('ts', every=f'{span}s', closed='left', start_by='datapoint')
+            .agg(
+                pl.col('price').first().alias('open'),
+                pl.col('price').max().alias('high'),
+                pl.col('price').min().alias('low'),
+                pl.col('price').last().alias('close'),
+                pl.col('amount').sum().alias('volume'),
+                (pl.col('price') * pl.col('amount')).sum().alias('quoteVolume'),
+            )
+            .filter(
+                (pl.col('ts').dt.epoch(time_unit='s') >= start_ts)
+                & (pl.col('ts').dt.epoch(time_unit='s') < end_ts)
+            )
         )
 
         return [{
-            'date':            round(row_ts.timestamp()),
+            'date':            int(row['ts'].timestamp()),
             'open':            float(row['open']),
             'high':            float(row['high']),
             'low':             float(row['low']),
@@ -471,7 +498,7 @@ class KrakenBackfill(_BackfillBase):
                 float(row['quoteVolume'] / row['volume'])
                 if row['volume'] > 0 else float(row['close'])
             ),
-        } for row_ts, row in result.iterrows()]
+        } for row in result.iter_rows(named=True)]
 
 
 # ---------------------------------------------------------------------------
@@ -486,6 +513,8 @@ def make_job(
     path: str,
     tz: str,
     form: str,
+    max_retries: int = 3,
+    retry_delay: float = 2.0,
 ) -> _BackfillBase:
     """Build the appropriate backfill strategy for an (exchange, pair).
 
@@ -532,13 +561,16 @@ def make_job(
     obj = cls(path, crypto, span, fiat, form=form, tz=tz)
 
     if exchange == _KRAKEN_EXCHANGE:
-        return KrakenBackfill(obj, sleep=sleep, form=form)
+        return KrakenBackfill(obj, sleep=sleep, form=form,
+                              max_retries=max_retries, retry_delay=retry_delay)
 
     return OHLCBackfill(
         obj,
         max_candles=defaults['max_candles'],
         sleep=sleep,
         form=form,
+        max_retries=max_retries,
+        retry_delay=retry_delay,
     )
 
 
@@ -587,6 +619,8 @@ def run_backfill(
             job = make_job(
                 histo_job.exchange, crypto, fiat, histo_job.span,
                 path, tz, histo_job.format,
+                max_retries=histo_job.max_retries,
+                retry_delay=histo_job.retry_delay,
             )
             if job.obj.full_path in seen_paths:
                 tqdm.write(