dccd 3.0.0__tar.gz → 3.2.0__tar.gz

{dccd-3.0.0 → dccd-3.2.0}/CHANGELOG.md

@@ -16,6 +16,101 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Removed
 
+## [3.2.0] - 2026-06-10
+
+### Added
+
+- Dev workflow: hierarchical, file-based **plan trees** under `doc/dev/plans/`
+  (committed) with a `<plans_dir>` descriptor key. A roadmap item expands into a
+  global `00-plan.md` + precise leaf specs (adaptive depth); each leaf declares a
+  `complexity` that derives its execution model (`low→haiku`/`medium→sonnet`/
+  `high→opus`). New `/plan` (build the tree + open the plan PR first) and
+  `/execute-leaf` (spawn an agent per leaf, verify on real data) skills; `/pick-task`,
+  `/finish-task`, `/abandon-task`, `/release` and `CLAUDE.md` updated to chain
+  through it. Backward-compatible: no `plans_dir` ⇒ the old plan-mode loop. (#94)
+- Restart/reboot safety verified on a real server `systemctl reboot`: the daemon
+  auto-starts, the trades stream reconnects, the interval backfill re-arms, the
+  `RunsStore` (SQLite WAL) survives and appends, and the coverage manifest keeps the
+  resume cursor (no gap). New `test_restart.py` guards RunsStore persistence across a
+  reopen and scheduler interval re-arm from config. (#99)
+- Ops for unattended deploy: `HealthMonitor` is now wired into the daemon (CLI
+  `dccd start` and the standalone API) — it was implemented but never instantiated,
+  so webhook alerts never fired. Docker `HEALTHCHECK` on `/health`, commented
+  systemd resource limits, and journald log-rotation guidance. Verified live on a
+  server: a failing job past the threshold delivered a real webhook POST, and the
+  container reports `healthy`. (#100)
+- Docs: new `how-to/deploy` guide — a blessed, host-validated path to run dccd
+  unattended on a server (systemd + venv recommended, Docker alternative), covering
+  install, secret injection, `/health`, restart/reboot safety, logs, alerts and the
+  old-CPU caveat. Completes **Epic A** (run on a remote server). (#102)
+
+### Changed
+
+- `Dockerfile`: pin the base image to a digest (reproducible builds) and add a
+  `POLARS_VARIANT` build arg — on CPUs without AVX2 (older servers) the default
+  `polars` wheel crashes with SIGILL, so
+  `docker build --build-arg POLARS_VARIANT=polars-lts-cpu` installs the LTS-CPU
+  build instead. Verified end-to-end on a real host (build, run, `/health`, Bearer
+  auth, a backfill writing correct OHLC to the `/data` volume). (#97)
+- Docs: `how-to/protect-ui` now covers deploy-time secret injection — the token and
+  `rclone.conf` are mounted at run time, never baked into the image (verified on the
+  built image: `docker history`/filesystem show no config); the YAML loader does not
+  expand `${ENV}` placeholders, so the mounted-file pattern is the blessed one. (#101)
+
+### Fixed
+
+- `deploy/dccd.service`: `ExecStart` pointed at `/usr/local/bin/dccd` and failed
+  `systemd-analyze verify`; it now uses a venv path (`/opt/dccd/venv/bin/dccd`) with
+  `StateDirectory=dccd` (systemd owns `/var/lib/dccd`). The install spec dropped the
+  non-existent `ui` extra (`.[daemon,ui]` → `.[daemon]`, also in the `Dockerfile`).
+  Verified a real system-wide install: `systemd-analyze verify` passes, the service
+  is active, auto-restarts after SIGKILL, and a backfill writes correct OHLC under
+  the hardened `/var/lib/dccd/data` (`ProtectSystem=strict`). (#98)
+- `HealthMonitor` counted consecutive failures per `run_id`, but each backfill run
+  has a unique id (`{spec}@{ts}`), so repeated failures never accumulated (only
+  streams, with a stable `@stream` id, could alert). It now keys on the job
+  (spec id) so repeated backfill failures trip the alert. (#100)
+
+### Deprecated
+
+### Removed
+
+## [3.1.0] - 2026-06-09
+
+### Added
+
+- `dccd start` now schedules rclone remote sync: when `storage.remotes` is set,
+  the daemon mirrors the store off-box every `storage.sync_interval` seconds with
+  exponential backoff, persisted run history (`sync` runs in `RunsStore`) and a
+  live `remote-sync` EventBus status. Previously `RemoteStorage` was implemented
+  but never driven — a server synced nothing. (#86)
+- Storage page surfaces remote sync: last/next sync, status, configured remotes
+  and synced volume, plus a **Sync now** button — backed by
+  `GET`/`POST /api/storage/sync`. The shared `operations.sync_remote` primitive
+  records each cycle, so the manual button and the scheduled loop stay in sync. (#87)
+- Coverage manifest (`CoverageStore`, SQLite under `.dccd/`): backfill records each
+  dataset's `[min_ts, max_ts]` extent, and `start="last"` falls back to the
+  manifest's `max_ts` when no local file exists — so local data can be dropped to
+  free disk without forcing a re-download on the next backfill. (#88)
+- Free-space purge: `storage.min_free_gb` (default `0` = off). After each
+  successful sync the daemon drops the oldest already-synced Parquet files until
+  free space is back above the floor (the coverage manifest keeps the resume
+  cursor, `.dccd/` is never touched). (#89)
+- Read-through restore: reading a dataset whose local Parquet was purged now pulls
+  it back from the remote (`rclone copy`) before loading, so a purge is
+  transparent to readers (`Client.read`, `POST /api/read`). (#90)
+- Docs: the `how-to/sync-remote` guide now covers rclone provisioning, the
+  `min_free_gb` free-space purge, read-through restore, and restore/integrity
+  (`rclone copy`/`rclone check`) — completing Epic C (tiered storage). (#91)
+
+### Changed
+
+### Fixed
+
+### Deprecated
+
+### Removed
+
 ## [3.0.0] - 2026-06-07
 
 ### Added

{dccd-3.0.0 → dccd-3.2.0}/CLAUDE.md

@@ -79,22 +79,38 @@ away without losing unrelated good work is too big: split it. This is what makes
 
 ### Dev loop & docs of record
 
-The iterative loop is tooled by skills, with three tracked docs as the sources of
+The iterative loop is tooled by skills, with four tracked docs as the sources of
 truth:
 
 | Doc | Holds | Updated by |
 |-----|-------|-----------|
-| `doc/dev/07-roadmap.md` | open work (single source) | `/pick-task` reads · `/finish-task`, `/abandon-task` update |
+| `doc/dev/07-roadmap.md` | open work — single source *index* | `/pick-task` reads · `/finish-task`, `/abandon-task` update |
+| `doc/dev/plans/<epic>/` | open work *detail* — durable hierarchical plan trees (global + leaf specs) | `/plan` writes · `/execute-leaf` reads · `/finish-task`/`/abandon-task` archive |
 | `doc/dev/03-decisions.md` | the *why* — ADR journal (+ settled rationale) | `/finish-task` (accepted), `/abandon-task` (rejected/tombstone) |
 | `doc/dev/06-status.md` | where things stand | `/finish-task`, `/groom-docs` |
 
 `CHANGELOG.md` + git log stay authoritative for *what* shipped. The loop:
-`/pick-task` (smallest slice → branch) → plan (split big plans into small PRs) →
-`/finish-task` (tests, ADR entry, status, PR) **or** `/abandon-task` (salvage the
-lesson + close the PR); `/groom-docs` periodically keeps `doc/dev/` lean and true.
 
-**Model per task** (advisory — you set it via `/model`, or a skill spawns a
-subagent with an explicit `model`; subagents otherwise *inherit* the parent):
+`/pick-task` (smallest coherent slice; **no branch yet**) →
+`/plan` (decompose into a `doc/dev/plans/<epic>/` tree — adaptive depth: a single
+leaf for a trivial task, a global `00-plan.md` + leaves otherwise — and open the
+**plan PR** that lands the tree on `develop` first) →
+`/execute-leaf <epic> next` (cut the leaf branch, **spawn an agent at the model
+derived from the leaf's `complexity`**, which implements + tests + **verifies on
+real data**, then reports) →
+`/finish-task` (tests, ADR, CHANGELOG, leaf PR, archive the leaf, tick the global
+checklist) → … per leaf … → last leaf removes the roadmap line → `/release`.
+
+`/abandon-task` salvages the lesson + closes a bad PR (tombstones the leaf);
+`/groom-docs` periodically keeps `doc/dev/` lean and true. The full format lives in
+[`doc/dev/plans/README.md`](doc/dev/plans/README.md). The workflow is
+backward-compatible: a repo whose `.claude/workflow.json` has **no `plans_dir`**
+falls back to the older `/pick-task → plan mode → /finish-task` loop.
+
+**Model per task** (advisory — you set it via `/model`, a skill spawns a subagent
+with an explicit `model`, or a plan **leaf's `complexity` derives it**:
+`low→haiku`, `medium→sonnet`, `high→opus`; subagents otherwise *inherit* the
+parent):
 
 | Model | For |
 |-------|-----|

{dccd-3.0.0 → dccd-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 3.0.0
+Version: 3.2.0
 Summary: Download Crypto Currency Data — hexagonal architecture, async-first.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT

{dccd-3.0.0 → dccd-3.2.0}/dccd/__init__.py

@@ -70,6 +70,8 @@ class Client:
         self._config: AppConfig | None = None
         self._store: ParquetStore | None = None
         self._registry: SourceRegistry | None = None
+        self._coverage_store: Any = None
+        self._remote: Any = None
 
     def _require_ready(self) -> tuple["SourceRegistry", "ParquetStore"]:
         if self._registry is None or self._store is None:
@@ -78,7 +80,12 @@ class Client:
 
     async def __aenter__(self) -> "Client":
         from dccd.application.config import AppConfig, load_config, resolve_config_path
-        from dccd.application.service_factory import build_registry, build_store
+        from dccd.application.service_factory import (
+            build_coverage_store,
+            build_registry,
+            build_remote,
+            build_store,
+        )
 
         try:
             path = resolve_config_path(self._config_path)
@@ -88,6 +95,8 @@ class Client:
 
         # Single source of truth for adapter wiring — same as CLI and API.
         self._store = build_store(self._config.settings.data_path)
+        self._coverage_store = build_coverage_store(self._config.settings.data_path)
+        self._remote = build_remote(self._config)
         self._registry = build_registry()
         return self
 
@@ -156,7 +165,8 @@ class Client:
             origin="runtime",
         )
         registry, store = self._require_ready()
-        return await do_backfill(spec, registry=registry, store=store)
+        return await do_backfill(spec, registry=registry, store=store,
+                                 coverage_store=self._coverage_store)
 
     async def stream(self, exchange: str, symbol: str, data_type: str = "trades",
                      span: int | None = None, depth: int | None = None,
@@ -249,7 +259,8 @@ class Client:
         _, store = self._require_ready()
         target = JobTarget(exchange=exchange, symbol=Symbol.parse(symbol),
                            data_type=DataType(data_type), span=span)
-        return cast("pl.DataFrame", do_read(target, store=store, start_ns=start_ns, end_ns=end_ns))
+        return cast("pl.DataFrame", do_read(target, store=store, start_ns=start_ns,
+                                            end_ns=end_ns, remote=self._remote))
 
     def inventory(self) -> list[dict[str, Any]]:
         """List every stored dataset with its coverage.

{dccd-3.0.0 → dccd-3.2.0}/dccd/application/config.py

@@ -70,10 +70,12 @@ class RemoteConfig(BaseModel):
 
 
 class StorageConfig(BaseModel):
-    """Storage settings: local path, rclone remotes, and sync interval."""
+    """Storage settings: local path, rclone remotes, sync interval, and the
+    free-space floor (GiB) that triggers purging already-synced local files."""
     local_path: str = ""
     remotes: list[RemoteConfig] = Field(default_factory=list)
     sync_interval: int = 3600
+    min_free_gb: float = 0.0
 
 
 class AlertConfig(BaseModel):

{dccd-3.0.0 → dccd-3.2.0}/dccd/application/monitor.py

@@ -41,13 +41,18 @@ class HealthMonitor:
     def _on_event(self, event: Event) -> None:
         if not isinstance(event, StatusEvent):
             return
+        # Count failures per *job*, not per run: a run_id is `{spec_id}@{run}` and
+        # each backfill run is unique, so keying on run_id would never accumulate
+        # across runs (only streams reuse `{spec_id}@stream`). Key on the spec_id
+        # prefix so repeated failures of the same job trip the alert.
+        key = event.run_id.split("@", 1)[0]
         if event.state == "failed":
-            self._consecutive[event.run_id] += 1
-            count = self._consecutive[event.run_id]
+            self._consecutive[key] += 1
+            count = self._consecutive[key]
             if count >= self._max_errors:
-                self._alert(event.run_id, count)
+                self._alert(key, count)
         elif event.state == "succeeded":
-            self._consecutive[event.run_id] = 0
+            self._consecutive[key] = 0
 
     def _alert(self, run_id: str, count: int) -> None:
         msg = f"dccd alert: {run_id} failed {count} times consecutively."

{dccd-3.0.0 → dccd-3.2.0}/dccd/application/operations.py

@@ -27,10 +27,12 @@ from dccd.domain.timeutils import NS, ns_now, ns_to_dt
 from dccd.domain.types import DataType
 from dccd.sources.base import OHLCHistory, OrderBookSnapshotREST, TradesHistory
 from dccd.sources.registry import SourceRegistry
+from dccd.storage.coverage_sqlite import CoverageStore
 from dccd.storage.parquet import ParquetStore
+from dccd.storage.remote import RemoteStorage
 from dccd.storage.runs_sqlite import RunsStore
 
-__all__ = ["backfill", "stream", "read", "inventory"]
+__all__ = ["backfill", "stream", "read", "inventory", "sync_remote"]
 
 logger = logging.getLogger(__name__)
 
@@ -112,6 +114,7 @@ async def backfill(
     registry: SourceRegistry,
     store: ParquetStore,
     runs_store: RunsStore | None = None,
+    coverage_store: CoverageStore | None = None,
     events: RunEvents | None = None,
     stop_event: asyncio.Event | None = None,
     run_id: str | None = None,
@@ -136,6 +139,10 @@ async def backfill(
     registry : SourceRegistry
     store : ParquetStore
     runs_store : RunsStore or None
+    coverage_store : CoverageStore or None
+        When set, ``start="last"`` falls back to the manifest's recorded
+        ``max_ts`` if no local file exists (so a dropped store doesn't trigger a
+        re-download), and the dataset's extent is recorded on success.
     events : RunEvents or None
     stop_event : asyncio.Event or None
         Set externally to cancel mid-run cleanly.
@@ -169,6 +176,11 @@ async def backfill(
 
     if params.start == "last":
         last = store.last_timestamp(ds)
+        if last is None and coverage_store is not None:
+            # Local files may have been dropped to free disk; the coverage
+            # manifest remembers how far we got, so we resume from there instead
+            # of re-downloading from the bounded default lookback.
+            last = coverage_store.get_max_ts(ds)
         if last is not None:
             start_ns: int = last + 1
         else:
@@ -200,6 +212,17 @@ async def backfill(
     # Counts every item received from the paginator, including unflushed ones.
     _collected: list[int] = [0]
 
+    # Min/max timestamp seen this run, fed to the coverage manifest on success so
+    # the dataset's extent survives a local-data drop (see CoverageStore).
+    _run_min: list[int | None] = [None]
+    _run_max: list[int | None] = [None]
+
+    def _track_ts(ts: int) -> None:
+        if _run_min[0] is None or ts < _run_min[0]:
+            _run_min[0] = ts
+        if _run_max[0] is None or ts > _run_max[0]:
+            _run_max[0] = ts
+
     # Progress is reported by *time covered* of the requested window, which gives
     # a real, smooth bar for both OHLC and cursor-paginated trades (the latter
     # have no page total). ``at`` is the timestamp reached. The window is read
@@ -257,6 +280,7 @@ async def backfill(
                     break
                 bars.append(bar)
                 _collected[0] += 1
+                _track_ts(bar.ts)
                 if _collected[0] % 200 == 0:
                     _emit_time(bar.ts)
                 if len(bars) >= _FLUSH_BATCH:
@@ -295,6 +319,7 @@ async def backfill(
                     break
                 batch.append(trade)
                 _collected[0] += 1
+                _track_ts(trade.ts)
                 if _collected[0] % 1000 == 0:
                     _emit_time(trade.ts)  # progress by time covered, not page count
                 if len(batch) >= _FLUSH_BATCH:
@@ -309,6 +334,7 @@ async def backfill(
                 raise NoCapability(target.exchange, "orderbook", "snapshot")
             depth = params.depth or 50
             snap = await adapter.fetch_orderbook(target.symbol, depth)
+            _track_ts(snap.ts)
             total_written += await _flush(store, ds, [snap], prov_src)
 
     except Exception as exc:
@@ -328,6 +354,12 @@ async def backfill(
     if runs_store:
         runs_store.finish_run(run_id, state, rows_written=total_written)
 
+    # Record coverage so this dataset's extent survives a later local-data drop.
+    if coverage_store is not None and _run_max[0] is not None:
+        coverage_store.record(
+            ds, min_ts=_run_min[0], max_ts=_run_max[0], rows_added=total_written
+        )
+
     return {"run_id": run_id, "rows_written": total_written, "start_ns": start_ns, "end_ns": end_ns}
 
 
@@ -461,12 +493,87 @@ def read(
     store: ParquetStore,
     start_ns: int | None = None,
     end_ns: int | None = None,
+    remote: RemoteStorage | None = None,
 ) -> Any:
-    """Read stored data for *target* in the given nanosecond range."""
+    """Read stored data for *target* in the given nanosecond range.
+
+    Read-through restore: when *remote* is set and the dataset has no local
+    Parquet (e.g. it was purged to free disk), the dataset directory is pulled
+    back from the remote (``rclone copy``) before loading, so a purge is
+    transparent to readers.
+    """
     ds = _make_dataset_id(target)
+    if remote is not None:
+        directory = store.directory(ds)
+        if not any(directory.glob("*.parquet")):
+            rel = directory.relative_to(store.root)
+            remote.restore(str(rel))
     return store.load(ds, start_ns, end_ns)
 
 
 def inventory(*, store: ParquetStore) -> list[dict[str, Any]]:
     """Return a list of dataset descriptors for all stored data."""
     return store.inventory()
+
+
+async def sync_remote(
+    remote: RemoteStorage,
+    *,
+    runs_store: RunsStore | None = None,
+    events: RunEvents | None = None,
+    run_id: str | None = None,
+) -> dict[str, Any]:
+    """Run one remote-sync cycle: mirror the local store to all rclone remotes.
+
+    Records the cycle as a ``sync`` run in *runs_store* (so the Storage UI can
+    show "last sync") and emits ``status``/``log`` on *events*. Shared by the
+    scheduler's periodic loop and the manual "Sync now" endpoint, so the
+    run-recording lives in exactly one place.
+
+    Parameters
+    ----------
+    remote : RemoteStorage
+    runs_store : RunsStore or None
+    events : RunEvents or None
+    run_id : str or None
+        Override the auto-generated run id.
+
+    Returns
+    -------
+    dict
+        ``{'run_id', 'results', 'ok'}`` — ``results`` maps remote → success;
+        ``ok`` is True only when every configured remote synced.
+    """
+    if run_id is None:
+        run_id = f"remote-sync@{time.time_ns()}"
+    if runs_store:
+        runs_store.create_run(run_id, "remote-sync", "sync", "-", "all", "-")
+    if events:
+        events.status("running")
+    try:
+        results = await remote.sync_all()
+    except Exception as exc:
+        msg = f"Remote sync error: {exc}"
+        if events:
+            events.log(msg, "error")
+            events.status("failed")
+        if runs_store:
+            runs_store.finish_run(run_id, "failed", error=str(exc))
+        return {"run_id": run_id, "results": {}, "ok": False}
+
+    failed = [r for r, ok in results.items() if not ok]
+    if failed:
+        msg = f"Remote sync failed for: {', '.join(failed)}"
+        if events:
+            events.log(msg, "error")
+            events.status("failed")
+        if runs_store:
+            runs_store.finish_run(run_id, "failed", error=msg)
+        return {"run_id": run_id, "results": results, "ok": False}
+
+    if events:
+        events.log(f"Synced {len(results)} remote(s)")
+        events.status("succeeded")
+    if runs_store:
+        runs_store.finish_run(run_id, "succeeded", rows_written=len(results))
+    return {"run_id": run_id, "results": results, "ok": True}

{dccd-3.0.0 → dccd-3.2.0}/dccd/application/scheduler.py

@@ -6,10 +6,12 @@ import asyncio
 import logging
 import time
 
-from dccd.application.events import EventBus
+from dccd.application.events import EventBus, RunEvents
 from dccd.application.jobs import JobSpec
 from dccd.sources.registry import SourceRegistry
+from dccd.storage.coverage_sqlite import CoverageStore
 from dccd.storage.parquet import ParquetStore
+from dccd.storage.remote import RemoteStorage
 from dccd.storage.runs_sqlite import RunsStore
 
 __all__ = ["Scheduler"]
@@ -90,6 +92,11 @@ class Scheduler:
     store : ParquetStore
     runs_store : RunsStore or None
     events : EventBus
+    remote : RemoteStorage or None
+        When set (rclone remotes configured), :meth:`start` launches a periodic
+        loop that mirrors the local store off-box every ``sync_interval`` seconds.
+    sync_interval : int
+        Seconds between remote-sync cycles (default 3600).
     """
 
     def __init__(
@@ -98,11 +105,22 @@ class Scheduler:
         store: ParquetStore,
         runs_store: RunsStore | None = None,
         events: EventBus | None = None,
+        remote: RemoteStorage | None = None,
+        sync_interval: int = 3600,
+        coverage_store: CoverageStore | None = None,
+        data_path: str | None = None,
+        min_free_gb: float = 0.0,
     ) -> None:
         self._registry = registry
         self._store = store
         self._runs_store = runs_store
         self._events = events or EventBus()
+        self._remote = remote
+        self._sync_interval = sync_interval
+        self._coverage_store = coverage_store
+        self._data_path = data_path
+        self._min_free_gb = min_free_gb
+        self._sync_task: asyncio.Task[None] | None = None
         self._streams: dict[str, _StreamWorker] = {}
         self._interval_tasks: list[asyncio.Task[None]] = []
         # Per-spec recurring backfill loops, keyed by spec id, with the interval
@@ -186,6 +204,8 @@ class Scheduler:
     async def start(self, specs: list[JobSpec]) -> None:
         """Start all enabled specs (full daemon mode)."""
         self._running = True
+        if self._remote is not None and self._sync_task is None:
+            self._sync_task = asyncio.create_task(self._sync_loop())
         for spec in specs:
             if not spec.enabled:
                 continue
@@ -207,6 +227,13 @@ class Scheduler:
     async def stop(self) -> None:
         """Stop all running jobs."""
         self._running = False
+        if self._sync_task is not None:
+            self._sync_task.cancel()
+            try:
+                await self._sync_task
+            except (asyncio.CancelledError, Exception):
+                pass
+            self._sync_task = None
         for task in self._interval_tasks:
             task.cancel()
         for task, _ in self._interval_loops.values():
@@ -216,6 +243,64 @@ class Scheduler:
         self._interval_tasks.clear()
         self._interval_loops.clear()
 
+    async def _sync_loop(self) -> None:
+        """Periodically mirror the local store to the configured rclone remotes.
+
+        Runs only when a :class:`~dccd.storage.remote.RemoteStorage` was wired in
+        (``storage.remotes`` non-empty). Each cycle is delegated to
+        :func:`dccd.application.operations.sync_remote` (which records a ``sync``
+        run in :class:`RunsStore` and emits live ``remote-sync`` EventBus status);
+        this loop only owns the cadence and the exponential backoff on failure
+        (30s → capped at ``sync_interval``) so a flapping remote doesn't hammer
+        rclone.
+        """
+        from dccd.application.operations import sync_remote
+        assert self._remote is not None
+        run_events = self._events.for_run("remote-sync")
+        backoff = 30.0
+        try:
+            while self._running:
+                result = await sync_remote(
+                    self._remote, runs_store=self._runs_store, events=run_events
+                )
+                if result["ok"]:
+                    # Remote is now up to date → safe to free disk by dropping the
+                    # oldest already-synced files (the coverage manifest keeps the
+                    # resume cursor). Runs only when a floor is configured.
+                    await self._maybe_purge(run_events)
+                    backoff = 30.0
+                    await asyncio.sleep(self._sync_interval)
+                else:
+                    logger.warning("Remote sync failed — retry in %ds", int(backoff))
+                    await asyncio.sleep(min(backoff, self._sync_interval))
+                    backoff = min(backoff * 2, float(self._sync_interval))
+        except asyncio.CancelledError:
+            return
+
+    async def _maybe_purge(self, run_events: RunEvents) -> None:
+        """Free disk by dropping oldest synced files when below the floor.
+
+        Called right after a successful sync (remote is current), so dropped
+        files are recoverable from the remote. No-op unless ``min_free_gb`` and a
+        ``data_path`` are configured.
+        """
+        if self._min_free_gb <= 0 or not self._data_path:
+            return
+        from dccd.storage.purge import purge_to_free_space
+        try:
+            res = await asyncio.to_thread(
+                purge_to_free_space, self._data_path, self._min_free_gb
+            )
+        except Exception as exc:
+            logger.warning("Purge failed: %s", exc)
+            return
+        if res["removed"]:
+            run_events.log(
+                f"Purged {len(res['removed'])} file(s), "
+                f"freed ~{res['freed_bytes'] / (1024 ** 3):.2f} GiB to stay above "
+                f"{self._min_free_gb} GiB free"
+            )
+
     async def _interval_loop(self, spec: JobSpec) -> None:
         every = spec.trigger.every or spec.target.span or 3600
         while self._running:
@@ -231,6 +316,7 @@ class Scheduler:
                 registry=self._registry,
                 store=self._store,
                 runs_store=self._runs_store,
+                coverage_store=self._coverage_store,
                 events=run_events,
             )
         except Exception as exc:

{dccd-3.0.0 → dccd-3.2.0}/dccd/application/service_factory.py

@@ -11,11 +11,20 @@ import pathlib
 from typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
+    from dccd.application.config import AppConfig
     from dccd.sources.registry import SourceRegistry
+    from dccd.storage.coverage_sqlite import CoverageStore
     from dccd.storage.parquet import ParquetStore
+    from dccd.storage.remote import RemoteStorage
     from dccd.storage.runs_sqlite import RunsStore
 
-__all__ = ["build_registry", "build_store", "build_runs_store"]
+__all__ = [
+    "build_registry",
+    "build_store",
+    "build_runs_store",
+    "build_remote",
+    "build_coverage_store",
+]
 
 
 def build_registry() -> "SourceRegistry":
@@ -77,3 +86,50 @@ def build_runs_store(data_path: str | pathlib.Path) -> "RunsStore":
     from dccd.storage.runs_sqlite import RunsStore
 
     return RunsStore(pathlib.Path(data_path) / ".dccd" / "runs.db")
+
+
+def build_coverage_store(data_path: str | pathlib.Path) -> "CoverageStore":
+    """Return a :class:`~dccd.storage.coverage_sqlite.CoverageStore`.
+
+    The database lives at ``{data_path}/.dccd/coverage.db`` — the manifest that
+    lets local data be dropped without forcing a re-download on the next
+    backfill.
+
+    Parameters
+    ----------
+    data_path : str or Path
+
+    Returns
+    -------
+    CoverageStore
+    """
+    from dccd.storage.coverage_sqlite import CoverageStore
+
+    return CoverageStore(pathlib.Path(data_path) / ".dccd" / "coverage.db")
+
+
+def build_remote(cfg: "AppConfig") -> "RemoteStorage | None":
+    """Return a :class:`~dccd.storage.remote.RemoteStorage`, or ``None``.
+
+    Returns ``None`` when no rclone remotes are configured (``storage.remotes``
+    empty) — there is nothing to sync, so the daemon skips the sync loop. The
+    local root is ``settings.data_path`` (the canonical store root used by
+    :func:`build_store`).
+
+    Parameters
+    ----------
+    cfg : AppConfig
+
+    Returns
+    -------
+    RemoteStorage or None
+    """
+    if not cfg.storage.remotes:
+        return None
+
+    from dccd.storage.remote import RemoteStorage
+
+    return RemoteStorage(
+        cfg.settings.data_path,
+        [r.model_dump() for r in cfg.storage.remotes],
+    )