dccd 3.1.0__tar.gz → 3.2.0__tar.gz

{dccd-3.1.0 → dccd-3.2.0}/CHANGELOG.md

@@ -16,6 +16,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Removed
 
+## [3.2.0] - 2026-06-10
+
+### Added
+
+- Dev workflow: hierarchical, file-based **plan trees** under `doc/dev/plans/`
+  (committed) with a `<plans_dir>` descriptor key. A roadmap item expands into a
+  global `00-plan.md` + precise leaf specs (adaptive depth); each leaf declares a
+  `complexity` that derives its execution model (`low→haiku`/`medium→sonnet`/
+  `high→opus`). New `/plan` (build the tree + open the plan PR first) and
+  `/execute-leaf` (spawn an agent per leaf, verify on real data) skills; `/pick-task`,
+  `/finish-task`, `/abandon-task`, `/release` and `CLAUDE.md` updated to chain
+  through it. Backward-compatible: no `plans_dir` ⇒ the old plan-mode loop. (#94)
+- Restart/reboot safety verified on a real server `systemctl reboot`: the daemon
+  auto-starts, the trades stream reconnects, the interval backfill re-arms, the
+  `RunsStore` (SQLite WAL) survives and appends, and the coverage manifest keeps the
+  resume cursor (no gap). New `test_restart.py` guards RunsStore persistence across a
+  reopen and scheduler interval re-arm from config. (#99)
+- Ops for unattended deploy: `HealthMonitor` is now wired into the daemon (CLI
+  `dccd start` and the standalone API) — it was implemented but never instantiated,
+  so webhook alerts never fired. Docker `HEALTHCHECK` on `/health`, commented
+  systemd resource limits, and journald log-rotation guidance. Verified live on a
+  server: a failing job past the threshold delivered a real webhook POST, and the
+  container reports `healthy`. (#100)
+- Docs: new `how-to/deploy` guide — a blessed, host-validated path to run dccd
+  unattended on a server (systemd + venv recommended, Docker alternative), covering
+  install, secret injection, `/health`, restart/reboot safety, logs, alerts and the
+  old-CPU caveat. Completes **Epic A** (run on a remote server). (#102)
+
+### Changed
+
+- `Dockerfile`: pin the base image to a digest (reproducible builds) and add a
+  `POLARS_VARIANT` build arg — on CPUs without AVX2 (older servers) the default
+  `polars` wheel crashes with SIGILL, so
+  `docker build --build-arg POLARS_VARIANT=polars-lts-cpu` installs the LTS-CPU
+  build instead. Verified end-to-end on a real host (build, run, `/health`, Bearer
+  auth, a backfill writing correct OHLC to the `/data` volume). (#97)
+- Docs: `how-to/protect-ui` now covers deploy-time secret injection — the token and
+  `rclone.conf` are mounted at run time, never baked into the image (verified on the
+  built image: `docker history`/filesystem show no config); the YAML loader does not
+  expand `${ENV}` placeholders, so the mounted-file pattern is the blessed one. (#101)
+
+### Fixed
+
+- `deploy/dccd.service`: `ExecStart` pointed at `/usr/local/bin/dccd` and failed
+  `systemd-analyze verify`; it now uses a venv path (`/opt/dccd/venv/bin/dccd`) with
+  `StateDirectory=dccd` (systemd owns `/var/lib/dccd`). The install spec dropped the
+  non-existent `ui` extra (`.[daemon,ui]` → `.[daemon]`, also in the `Dockerfile`).
+  Verified a real system-wide install: `systemd-analyze verify` passes, the service
+  is active, auto-restarts after SIGKILL, and a backfill writes correct OHLC under
+  the hardened `/var/lib/dccd/data` (`ProtectSystem=strict`). (#98)
+- `HealthMonitor` counted consecutive failures per `run_id`, but each backfill run
+  has a unique id (`{spec}@{ts}`), so repeated failures never accumulated (only
+  streams, with a stable `@stream` id, could alert). It now keys on the job
+  (spec id) so repeated backfill failures trip the alert. (#100)
+
+### Deprecated
+
+### Removed
+
 ## [3.1.0] - 2026-06-09
 
 ### Added

{dccd-3.1.0 → dccd-3.2.0}/CLAUDE.md

@@ -79,22 +79,38 @@ away without losing unrelated good work is too big: split it. This is what makes
 
 ### Dev loop & docs of record
 
-The iterative loop is tooled by skills, with three tracked docs as the sources of
+The iterative loop is tooled by skills, with four tracked docs as the sources of
 truth:
 
 | Doc | Holds | Updated by |
 |-----|-------|-----------|
-| `doc/dev/07-roadmap.md` | open work (single source) | `/pick-task` reads · `/finish-task`, `/abandon-task` update |
+| `doc/dev/07-roadmap.md` | open work — single source *index* | `/pick-task` reads · `/finish-task`, `/abandon-task` update |
+| `doc/dev/plans/<epic>/` | open work *detail* — durable hierarchical plan trees (global + leaf specs) | `/plan` writes · `/execute-leaf` reads · `/finish-task`/`/abandon-task` archive |
 | `doc/dev/03-decisions.md` | the *why* — ADR journal (+ settled rationale) | `/finish-task` (accepted), `/abandon-task` (rejected/tombstone) |
 | `doc/dev/06-status.md` | where things stand | `/finish-task`, `/groom-docs` |
 
 `CHANGELOG.md` + git log stay authoritative for *what* shipped. The loop:
-`/pick-task` (smallest slice → branch) → plan (split big plans into small PRs) →
-`/finish-task` (tests, ADR entry, status, PR) **or** `/abandon-task` (salvage the
-lesson + close the PR); `/groom-docs` periodically keeps `doc/dev/` lean and true.
 
-**Model per task** (advisory — you set it via `/model`, or a skill spawns a
-subagent with an explicit `model`; subagents otherwise *inherit* the parent):
+`/pick-task` (smallest coherent slice; **no branch yet**) →
+`/plan` (decompose into a `doc/dev/plans/<epic>/` tree — adaptive depth: a single
+leaf for a trivial task, a global `00-plan.md` + leaves otherwise — and open the
+**plan PR** that lands the tree on `develop` first) →
+`/execute-leaf <epic> next` (cut the leaf branch, **spawn an agent at the model
+derived from the leaf's `complexity`**, which implements + tests + **verifies on
+real data**, then reports) →
+`/finish-task` (tests, ADR, CHANGELOG, leaf PR, archive the leaf, tick the global
+checklist) → … per leaf … → last leaf removes the roadmap line → `/release`.
+
+`/abandon-task` salvages the lesson + closes a bad PR (tombstones the leaf);
+`/groom-docs` periodically keeps `doc/dev/` lean and true. The full format lives in
+[`doc/dev/plans/README.md`](doc/dev/plans/README.md). The workflow is
+backward-compatible: a repo whose `.claude/workflow.json` has **no `plans_dir`**
+falls back to the older `/pick-task → plan mode → /finish-task` loop.
+
+**Model per task** (advisory — you set it via `/model`, a skill spawns a subagent
+with an explicit `model`, or a plan **leaf's `complexity` derives it**:
+`low→haiku`, `medium→sonnet`, `high→opus`; subagents otherwise *inherit* the
+parent):
 
 | Model | For |
 |-------|-----|

{dccd-3.1.0 → dccd-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 3.1.0
+Version: 3.2.0
 Summary: Download Crypto Currency Data — hexagonal architecture, async-first.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT

{dccd-3.1.0 → dccd-3.2.0}/dccd/application/monitor.py

@@ -41,13 +41,18 @@ class HealthMonitor:
     def _on_event(self, event: Event) -> None:
         if not isinstance(event, StatusEvent):
             return
+        # Count failures per *job*, not per run: a run_id is `{spec_id}@{run}` and
+        # each backfill run is unique, so keying on run_id would never accumulate
+        # across runs (only streams reuse `{spec_id}@stream`). Key on the spec_id
+        # prefix so repeated failures of the same job trip the alert.
+        key = event.run_id.split("@", 1)[0]
         if event.state == "failed":
-            self._consecutive[event.run_id] += 1
-            count = self._consecutive[event.run_id]
+            self._consecutive[key] += 1
+            count = self._consecutive[key]
             if count >= self._max_errors:
-                self._alert(event.run_id, count)
+                self._alert(key, count)
         elif event.state == "succeeded":
-            self._consecutive[event.run_id] = 0
+            self._consecutive[key] = 0
 
     def _alert(self, run_id: str, count: int) -> None:
         msg = f"dccd alert: {run_id} failed {count} times consecutively."

{dccd-3.1.0 → dccd-3.2.0}/dccd/interfaces/api/app.py

@@ -37,6 +37,7 @@ from pydantic import BaseModel
 from dccd.application.config import AppConfig, load_config, resolve_config_path
 from dccd.application.events import EventBus
 from dccd.application.jobs import JobParams, JobSpec, JobTarget, Trigger
+from dccd.application.monitor import HealthMonitor
 from dccd.application.registry import REGISTRY
 from dccd.application.scheduler import Scheduler
 from dccd.application.service_factory import (
@@ -179,7 +180,10 @@ def create_app(
         app.state.remote = build_remote(cfg)
 
         if scheduler is not None:
+            # `dccd start` owns the scheduler *and* its HealthMonitor (wired on the
+            # scheduler's bus in cmd_start) — don't double-wire here.
             app.state.scheduler = scheduler
+            app.state.monitor = None
         else:
             app.state.scheduler = Scheduler(
                 app.state.registry,
@@ -188,6 +192,14 @@ def create_app(
                 app.state.event_bus,
                 coverage_store=app.state.coverage_store,
             )
+            # Standalone (`dccd ui`): wire alerts on this app's bus, which the
+            # standalone scheduler publishes to.
+            app.state.monitor = HealthMonitor(
+                app.state.runs_store,
+                app.state.event_bus,
+                webhook_url=cfg.alerts.webhook_url,
+                max_consecutive_errors=cfg.alerts.max_consecutive_errors,
+            )
 
         # Register stream workers from config so they can be started/stopped
         # from the UI even in standalone dccd-ui mode (without dccd start).

{dccd-3.1.0 → dccd-3.2.0}/dccd/interfaces/cli/main.py

@@ -146,6 +146,7 @@ def cmd_start(
     import uvicorn
 
     from dccd.application.events import EventBus
+    from dccd.application.monitor import HealthMonitor
     from dccd.application.scheduler import Scheduler
     from dccd.application.service_factory import (
         build_coverage_store,
@@ -162,6 +163,14 @@ def cmd_start(
     coverage_store = build_coverage_store(cfg.settings.data_path)
     registry = build_registry()
     bus = EventBus()
+    # Fire webhook alerts on repeated job failures. Subscribes to the same bus the
+    # scheduler publishes to; kept referenced for the daemon's lifetime (it was
+    # never instantiated before, so alerts never fired).
+    monitor = HealthMonitor(
+        runs_store, bus,
+        webhook_url=cfg.alerts.webhook_url,
+        max_consecutive_errors=cfg.alerts.max_consecutive_errors,
+    )
     remote = build_remote(cfg)
     scheduler = Scheduler(
         registry, store, runs_store, bus,
@@ -185,7 +194,7 @@ def cmd_start(
     ui_host = host or cfg.settings.ui_host
     ui_port = port or cfg.settings.ui_port
 
-    async def _run():
+    async def _run(_monitor: object = monitor):  # keep the HealthMonitor alive
         await scheduler.start(all_specs)
         typer.echo(f"Daemon running — UI at http://{ui_host}:{ui_port}")
         server = uvicorn.Server(uvicorn.Config(fastapi_app, host=ui_host, port=ui_port, log_level="warning"))

{dccd-3.1.0 → dccd-3.2.0}/dccd/tests/v3/test_application.py

@@ -358,3 +358,65 @@ class TestOperationRegistry:
         """Each operation must be accessible from both CLI and API (parity test)."""
         required_ops = {"backfill", "stream", "read", "inventory"}
         assert required_ops.issubset(set(REGISTRY.operations))
+
+
+class TestHealthMonitor:
+    """HealthMonitor fires a webhook on N consecutive failures and resets on success."""
+
+    def test_alerts_after_threshold_and_resets(self, monkeypatch):
+        import urllib.request
+
+        from dccd.application.events import EventBus, StatusEvent
+        from dccd.application.monitor import HealthMonitor
+
+        calls: list[str] = []
+
+        class _Resp:
+            def __enter__(self):
+                return self
+
+            def __exit__(self, *a):
+                return False
+
+        def fake_urlopen(req, timeout=0):
+            calls.append(getattr(req, "full_url", str(req)))
+            return _Resp()
+
+        monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
+
+        bus = EventBus()
+        HealthMonitor(None, bus, webhook_url="http://hook.test", max_consecutive_errors=3)
+
+        # Distinct run ids that share one job (spec id) — each backfill run is
+        # unique (`{spec}@{ts}`), so the monitor must accumulate across runs.
+        job = "backfill:binance:BTC/USDT:ohlc:3600s"
+        # Below threshold: no alert.
+        bus.emit(StatusEvent(run_id=f"{job}@1", state="failed"))
+        bus.emit(StatusEvent(run_id=f"{job}@2", state="failed"))
+        assert calls == []
+
+        # Threshold reached (3rd failure, different run): one alert.
+        bus.emit(StatusEvent(run_id=f"{job}@3", state="failed"))
+        assert len(calls) == 1
+
+        # A success resets the counter; failures must re-accumulate.
+        bus.emit(StatusEvent(run_id=f"{job}@4", state="succeeded"))
+        bus.emit(StatusEvent(run_id=f"{job}@5", state="failed"))
+        bus.emit(StatusEvent(run_id=f"{job}@6", state="failed"))
+        assert len(calls) == 1
+        bus.emit(StatusEvent(run_id=f"{job}@7", state="failed"))
+        assert len(calls) == 2
+
+        # A different job keeps its own independent counter.
+        bus.emit(StatusEvent(run_id="backfill:kraken:BTC/USD:ohlc:3600s@1", state="failed"))
+        assert len(calls) == 2
+
+    def test_no_webhook_no_crash(self):
+        from dccd.application.events import EventBus, StatusEvent
+        from dccd.application.monitor import HealthMonitor
+
+        bus = EventBus()
+        HealthMonitor(None, bus, webhook_url=None, max_consecutive_errors=1)
+        # Must not raise even past threshold when no webhook is configured.
+        bus.emit(StatusEvent(run_id="r1", state="failed"))
+        bus.emit(StatusEvent(run_id="r1", state="failed"))

dccd-3.2.0/dccd/tests/v3/test_restart.py

@@ -0,0 +1,77 @@
+"""Restart safety regression guards.
+
+Durable state must survive a process restart and the scheduler must re-arm its
+recurring work from config alone (so a reboot resumes with no manual step and no
+gap). Verified live on a real `systemctl reboot` of a server in PR #99 — these are
+the cheap regression guards for that behaviour.
+"""
+
+import pytest
+
+from dccd.application.jobs import JobSpec, JobTarget, Trigger
+from dccd.application.scheduler import Scheduler
+from dccd.domain.symbol import Symbol
+from dccd.domain.types import DataType
+from dccd.storage.runs_sqlite import RunsStore
+
+
+def _interval_spec(every: int = 120) -> JobSpec:
+    target = JobTarget(
+        exchange="binance",
+        symbol=Symbol(base="BTC", quote="USDT"),
+        data_type=DataType.OHLC,
+        span=3600,
+    )
+    return JobSpec(
+        id=JobSpec.make_id("backfill", target),
+        operation="backfill",
+        target=target,
+        trigger=Trigger(kind="interval", every=every),
+    )
+
+
+def test_runsstore_survives_reopen(tmp_path):
+    """A fresh RunsStore at the same path keeps prior runs (append, not truncate)."""
+    db = tmp_path / "runs.db"
+    s1 = RunsStore(db)
+    s1.create_run("r1", "spec-1", "backfill", "binance", "BTC/USDT", "ohlc")
+    s1.finish_run("r1", "succeeded", rows_written=10)
+    s1.create_run("r2", "spec-1", "backfill", "binance", "BTC/USDT", "ohlc")
+    s1.finish_run("r2", "succeeded", rows_written=5)
+
+    # New instance at the same path == a daemon restart.
+    s2 = RunsStore(db)
+    assert {r["run_id"] for r in s2.list_runs()} == {"r1", "r2"}
+
+    # A post-restart run appends; the history is not reset.
+    s2.create_run("r3", "spec-1", "backfill", "binance", "BTC/USDT", "ohlc")
+    assert len(RunsStore(db).list_runs()) == 3
+
+
+@pytest.mark.asyncio
+async def test_scheduler_rearms_intervals_from_specs():
+    """A fresh Scheduler re-arms the same interval loop from the same specs.
+
+    This is the boot reconstruction path: `cmd_start` rebuilds everything from
+    config and calls `scheduler.start(cfg.all_job_specs())`. No cross-process
+    in-memory state — the spec id is the only thing that carries over.
+    """
+    spec = _interval_spec()
+
+    async def _noop(_spec):
+        return None
+
+    sched = Scheduler(registry=None, store=None)  # type: ignore[arg-type]
+    sched._run_once = _noop  # type: ignore[assignment]
+    sched._running = True
+    await sched.sync_intervals([spec])
+    assert spec.id in sched._interval_loops
+    await sched.stop()
+
+    # A second, independent Scheduler from the same spec arms it identically.
+    sched2 = Scheduler(registry=None, store=None)  # type: ignore[arg-type]
+    sched2._run_once = _noop  # type: ignore[assignment]
+    sched2._running = True
+    await sched2.sync_intervals([spec])
+    assert spec.id in sched2._interval_loops
+    await sched2.stop()

{dccd-3.1.0 → dccd-3.2.0}/dccd.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dccd
-Version: 3.1.0
+Version: 3.2.0
 Summary: Download Crypto Currency Data — hexagonal architecture, async-first.
 Author-email: Arthur Bernard <arthur.bernard.92@gmail.com>
 License: MIT

{dccd-3.1.0 → dccd-3.2.0}/dccd.egg-info/SOURCES.txt

@@ -74,6 +74,7 @@ dccd/tests/v3/test_domain_extended.py
 dccd/tests/v3/test_network.py
 dccd/tests/v3/test_purge.py
 dccd/tests/v3/test_remote_sync.py
+dccd/tests/v3/test_restart.py
 dccd/tests/v3/test_restore.py
 dccd/tests/v3/test_sources.py
 dccd/tests/v3/test_storage.py

{dccd-3.1.0 → dccd-3.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "dccd"
-version = "3.1.0"
+version = "3.2.0"
 description = "Download Crypto Currency Data — hexagonal architecture, async-first."
 readme = "README.md"
 license = { text = "MIT" }