@smilintux/skcapstone 0.10.0 → 0.12.5

package/docs/sk-integration-HANDOFF.md

@@ -0,0 +1,117 @@
+# sk* ⇄ skcapstone Integration — Orchestration Handoff
+
+**Last updated:** 2026-06-09 (Lumina closeout session — EPIC COMPLETE)
+**Epic:** coord `fca7f138` — "EPIC: sk* ⇄ skcapstone optional integration backbone"
+**Design:** [`docs/ADR-optional-integration-backbone.md`](./ADR-optional-integration-backbone.md)
+**Goal:** every sk* service uses skcapstone's **sk-alert** + **skscheduler** by default *when skcapstone is installed*, and runs fully standalone when it is not. Default-on by presence; `SK_STANDALONE=1` forces native.
+
+> **EPIC CLOSED 2026-06-09:** Backbone + 8 adapters + dual-mode harness (114 tests green) + README docs DONE.
+> Only skchat (`ad4f721a`) remains, owned by a separate thread.
+
+---
+
+## 1. Status at a glance
+
+| Layer | State |
+|---|---|
+| Backbone (skcapstone) | ✅ **DONE** — 4/4 tasks, 208 tests green, pushed to `github/main` |
+| Reference adapter (skmemory) | ✅ **DONE** — pushed `skmemory@docs-first-principles` |
+| sksecurity adapter | ✅ **DONE** — pushed `sksecurity@main` |
+| skgateway adapter (Node) | ✅ **DONE** — pushed `skgateway@main` |
+| skcomms / capauth / skvoice / skseed / cloud9 adapters | ✅ **DONE** — all landed |
+| skchat adapter | ⬜ OPEN — **owned by a separate thread**, do not touch |
+| Dual-mode test harness (`71186ebb`) | ✅ **DONE** — `tests/test_integration_backbone.py` 114 green |
+| Per-repo README docs (`4065db2b`) | ✅ **DONE** — all 8 repos have "Integration modes" section |
+
+coord: All sk-integration tasks DONE except skchat (separate thread). EPIC `fca7f138` CLOSED.
+
+---
+
+## 2. The backbone (DONE — this is the stable contract everything builds on)
+
+All in `skcapstone` (`src/skcapstone/`), committed `c5d8d7c`, `9d9f93d`, `d90dfbf` on `main` (pushed to `github`):
+
+- **`sdk.py`** — the ONLY public surface consumers import. Semver-frozen:
+  - `is_available() -> bool`
+  - `alert(topic, payload, *, level='info', notify=None, ttl_seconds=86400) -> bool`
+  - `register_job(spec, home=None) -> str` / `unregister_job(name, home=None) -> bool`
+  - `coord_create(title, **kw) -> str`
+  - `register_service(name, health_url=None, pid_file=None, home=None) -> str`
+- **`scheduler_jobs.py`** — `load_jobs_with_dropins()` merges `jobs.yaml` + `jobs.d/*.yaml`; `register_job()`/`unregister_job()` write atomic per-job fragments. Runtime callers (daemon `scheduled_tasks.py`, `cli/scheduler_cmd.py`, `doctor.py`) all repointed to the merged loader. **Honours `SKCAPSTONE_HOME`** (was a bug, fixed in `9d9f93d`).
+- **`cli/alerts.py`** — `skcapstone alerts` subscribes to `*.critical`/`*.error`/`*.warn` and styles consumer topics by severity suffix.
+- **`service_health.py`** — `check_all_services()` unions `~/.skcapstone/registry/*.json` (written by `register_service`) with the built-in checks.
+- Tests: `tests/test_sdk.py`, `test_jobs_dropins.py`, `test_alerts_consumer_topics.py`, `test_service_registry.py`.
+
+---
+
+## 3. The adapter pattern (copy this for every remaining consumer)
+
+**Canonical reference: `skmemory/skmemory/integration.py`** (commit `be33179`). Each adapter:
+
+1. Add `<pkg>/integration.py` with the optional-import guard:
+   ```python
+   try:
+       from skcapstone import sdk as _sdk
+   except Exception:
+       _sdk = None
+   def is_present() -> bool:
+       if os.environ.get("SK_STANDALONE"): return False
+       if _sdk is None: return False
+       try: return bool(_sdk.is_available())
+       except Exception: return False
+   ```
+2. `alert(event, payload, level)` → `_sdk.alert(f"{SERVICE}.{level}", {"event": event, **payload}, level=level, notify=level in {"warn","error","critical"})` when present, else structured log.
+   **CRITICAL CONVENTION:** topic is `<service>.<severity>` (e.g. `skvoice.error`), and the semantic **event name goes in the payload `event` field — NOT the topic suffix.** Otherwise `skcapstone alerts`' `*.error`/`*.critical`/`*.warn` wildcards never match it. (This was a real bug caught building skmemory.)
+3. `ensure_schedule()` → `_sdk.register_job({...})` (a `type: shell` job running the service's periodic CLI command) when present, else rely on the service's native systemd timer / thread loop.
+4. `register_self(pid_file=None)` → `_sdk.register_service(SERVICE, pid_file=...)`.
+5. **Wire into real call sites:** alert into the service's failure path; `ensure_schedule()` + `register_self()` into its startup / post-install lifecycle.
+6. Add an optional `[skcapstone]` extra to `pyproject.toml` (`skcapstone = ["skcapstone>=0.6.8"]`). **Never a hard dependency.**
+7. Tests `tests/test_integration_adapter.py`: standalone (`SK_STANDALONE=1`), absent (`monkeypatch.setattr(integration, "_sdk", None)`), integrated (sandbox `SKCAPSTONE_HOME` to `tmp_path` + `monkeypatch.setattr(skcapstone, "AGENT_HOME", str(tmp_path))`).
+
+**Node/non-Python services (reference: `skgateway/src/integration.mjs`, commit `cc7bf1a`):** can't import the SDK — integrate **file-based** by writing the same `~/.skcapstone/pubsub/topics/<topic>/msg-*.json` and `~/.skcapstone/registry/<name>.json` formats. Validated round-trip: Node `alert()` → Python `PubSub.poll()` reads it back intact. Presence = shared home exists + `SK_STANDALONE` unset.
+
+---
+
+## 4. What's left (the next session's work-list)
+
+All tagged `sk-integration` on the coord board. Run `skcapstone coord status` / inspect `~/.skcapstone/coordination/tasks/`.
+
+### Consumer adapters (mechanical mirrors of skmemory — for sonnet)
+| Task | Repo | Native fallback to preserve | Suggested scheduled job |
+|---|---|---|---|
+| `eae9b815` | **skcomms** (`skcapstone-repos/skcomms`) | peer `_notify_others()` / log; daemon heartbeat thread + systemd | heartbeat/health; note skcomms is folding INTO skcomms (canonical) |
+| `44b11628` | **capauth** | log (no native alerting yet) | key-rotation check (signing-daemon TODO — ensure_schedule may be a stub) |
+| `66881a86` | **skvoice** | log; `while True` loop + skvoice systemd | service health / TTS-cache prune |
+| `aaafe0d8` | **skseed** | log; pure kernel (periodic task type only) | belief-audit / germination; already duck-types AdaptedPrompt at `llm.py:40` |
+| `fb925612` | **cloud9** | log; systemd `cloud9-daemon.timer` + launchd plist | rehydration/FEB-state check |
+| `ad4f721a` | **skchat** | **OWNED BY ANOTHER THREAD — leave it.** Note: it already soft-bridges skcapstone MCP at `memory_bridge.py:24`; fold that into the one adapter. |
+
+### Cross-cutting
+- `71186ebb` — **dual-mode test harness**: parametrized over all consumers, asserts standalone AND integrated mode. This is the system acceptance gate for the EPIC.
+- `4065db2b` — per-repo README "Integration modes" + `~/.skcapstone/` filesystem-contract section.
+- `6b9a41a1` — note task (reference-pattern pointer); close once all adapters land.
+
+### Final step
+Once all adapters + cross-cutting are done, **complete the EPIC `fca7f138`** and consider squashing the `skcomms` superseded task notes.
+
+---
+
+## 5. Gotchas / things to know
+
+- **coord CLI:** `skcapstone coord claim <id> --agent <name>` and `complete <id> --agent <name>` (the `--agent` flag is required). `create` uses `--by`. Tasks are immutable after creation (no edit/update subcommand) — supersede by completing + creating a replacement.
+- **Don't commit the other threads' work.** In `skcapstone` the skcomms→skcomms migration shares the repo; in `skmemory` the mxbai-cutover changes (`backends/pgvector_backend.py`, `cli.py`) are unstaged from another thread. Stage only your adapter files explicitly — never `git add -A`.
+- **skcapstone remotes:** `github` is canonical (per `~/clawd/scripts/push-pending.sh`); `origin` and `forgejo` also exist and may be stale (the local `@{u}` tracks `forgejo`, which lags — verify against `github/main`).
+- **Test sandboxing:** consumers without a conftest that sets `SKCAPSTONE_HOME` must sandbox it per-test, AND `monkeypatch.setattr(skcapstone, "AGENT_HOME", str(tmp_path))` because skcapstone captures `AGENT_HOME` at import.
+- **Pre-existing broken tests (NOT yours):** `sksecurity/tests/test_truth_engine.py` fails to collect (imports a missing `_check_skmemory`); `skgateway/tests/classifier.test.mjs` has 2 pre-existing failures. Run adapter tests by file to avoid these.
+- **Leak check after integrated tests:** `ls ~/.skcapstone/config/jobs.d/<svc>_*.yaml ~/.skcapstone/registry/<svc>.json` should be clean — if a fragment leaks to the real home, a test isn't sandboxing `SKCAPSTONE_HOME`.
+
+---
+
+## 6. Commits / branches (all pushed)
+
+| Repo | Branch | Commits |
+|---|---|---|
+| skcapstone | `main` (→ `github`) | `c5d8d7c` backbone · `9d9f93d` home-fix+convention · `d90dfbf` ADR §3.5 |
+| skmemory | `docs-first-principles` | `be33179` reference adapter |
+| sksecurity | `main` | `e65979b` threat-sharing adapter |
+| skgateway | `main` | `cc7bf1a` Node file-based bridge |

package/docs/skscheduler.md

@@ -0,0 +1,155 @@
+# skscheduler — Unified Fleet Job Scheduler
+
+> The single declarative registry + per-node runner for all recurring jobs across the
+> SK fleet. One `jobs.yaml`, synced everywhere; each node runs its own scheduler thread
+> (inside the skcapstone daemon) and fires only the jobs whose affinity includes it.
+> Design spec: [`docs/superpowers/specs/2026-06-08-skscheduler-design.md`](superpowers/specs/2026-06-08-skscheduler-design.md).
+
+## Why it exists
+Scheduling was fragmented across **four** mechanisms (skcapstone `TaskScheduler` interval
+callbacks, legacy crontab, systemd user timers, Claude-Code crons) with no single place to
+define, run, or observe jobs — and `service_health` running on multiple nodes caused
+Syncthing `.sync-conflict-*` files (the root incident). skscheduler unifies this: **one
+registry, per-job node-affinity** (so a job runs on exactly the intended node — preventing
+the multi-writer class), cron **and** interval, and agent-judgment jobs (not just Python).
+
+## Architecture
+
+```mermaid
+flowchart TD
+    Y["~/.skcapstone/config/jobs.yaml<br/>(Syncthing-synced registry)"] -->|read by every node| D
+    subgraph node["each fleet node (.158, .41, …)"]
+      D["skcapstone daemon<br/>(systemd user service)"] --> TS["TaskScheduler<br/>tick loop (5s)"]
+      TS -->|interval callbacks| IC["heartbeat · memory_promotion<br/>dreaming_reflection · reprobe"]
+      TS -->|config jobs| DUE{"due? (cron|interval)<br/>AND node in affinity?"}
+      DUE -->|yes| LK["per-job overlap lock<br/>(singleton; skip if running)"]
+      LK --> JIT["jitter splay"] --> RUN["JobRunner.run + retries/backoff"]
+      RUN --> DISP{type}
+      DISP -->|python| PY["import module:fn()"]
+      DISP -->|shell| SH["subprocess (timeout)"]
+      DISP -->|agent| AG["claude -p --agent NAME"]
+      RUN --> STATE["SchedulerState.record_run<br/>(node-local, NEVER synced)"]
+      RUN --> NOTIFY["notify? → sk-alert (Telegram)"]
+    end
+    STATE --> SJ["~/.skcapstone/scheduler/&lt;host&gt;/state.json"]
+    RUN --> LOG["~/.skcapstone/scheduler/&lt;host&gt;/logs/&lt;job&gt;-&lt;ts&gt;.log"]
+```
+
+**Sync-safety invariant:** the registry (`jobs.yaml`) is synced; **run-state and logs are
+node-local and never synced** (`~/.skcapstone/.stignore` excludes `scheduler/`). The
+scheduler can never become a sync-conflict source.
+
+## Job lifecycle (one due job)
+
+```mermaid
+sequenceDiagram
+    participant Loop as TaskScheduler tick
+    participant Due as is_due()
+    participant Aff as job_runs_here()
+    participant Lock as overlap lock
+    participant Run as JobRunner
+    participant St as SchedulerState
+    participant Al as sk-alert
+    Loop->>Due: cron slot passed / interval elapsed since last_run?
+    Due-->>Loop: due
+    Loop->>Aff: this node's alias in job.nodes?
+    Aff-->>Loop: yes
+    Loop->>Lock: acquire (non-blocking)
+    alt previous run still going
+        Lock-->>Loop: busy → skip (no double-run)
+    else free
+        Lock-->>Loop: got it (in a worker thread)
+        Loop->>Run: jitter splay, then attempt 1..N
+        loop until ok or retries exhausted
+            Run->>Run: dispatch(type) with timeout
+            Run-->>Loop: ok? else backoff + retry
+        end
+        Loop->>St: record_run(ok, error, fire_time)
+        Loop->>Al: notify policy met? → DM Chef
+    end
+```
+
+## Due-check logic
+
+```mermaid
+flowchart LR
+    A{schedule set?} -->|cron| C["most-recent cron slot ≤ now"]
+    C --> C2{"last_run < that slot?"}
+    C2 -->|yes| DUE([due])
+    C2 -->|no| NO([not due])
+    A -->|every_seconds| I{"now − last_run ≥ interval?"}
+    I -->|yes| DUE
+    I -->|no| NO
+    A -->|neither| NO
+```
+Cron uses `croniter` (a declared dependency — if missing, **cron jobs silently can't fire**;
+`pip install -e .` in the venv fixes it). Missed slots are caught up once on next tick
+(misfire catch-up); per-job `catchup: false` opts out.
+
+## `jobs.yaml` schema (complete)
+
+```yaml
+jobs:
+  <job-name>:                 # YAML key = unique job id
+    # — scheduling (exactly one) —
+    schedule: "0 6 * * *"     # cron (m h dom mon dow)
+    every: 5m                 # OR interval: 30s | 5m | 1h | 1d | <seconds>
+    # — what to run (by type) —
+    type: agent               # agent | shell | python   (default python)
+    agent: lumina             # agent type  → runs `claude -p "<prompt>" --agent <name>`
+    prompt: "…"               # agent prompt
+    command: "skmem-pg-backup"# shell type  → subprocess
+    callback: "module.path:fn"# python type → import & call
+    # — placement —
+    nodes: [".41"]            # affinity: "all" | [host aliases]  (alias = SK_NODE_ALIAS)
+    # — limits / reliability (added 2026-06-09) —
+    timeout: 900              # hard-kill seconds (default 900)
+    retries: 0                # extra attempts on failure (linear)
+    retry_backoff: 0          # seconds between attempts
+    jitter: 0                 # max random splay (s) before dispatch — fleet anti-stampede
+    notify: off               # off | on_failure | on_success | always  → sk-alert (Telegram)
+    notify_level: warn        # sk-alert level for failures (info|warn|crit)
+    catchup: true             # run once on a missed slot (default true)
+    enabled: true
+```
+
+## Reliability features (2026-06-09)
+| field | purpose | infra it serves |
+|---|---|---|
+| `retries` + `retry_backoff` | re-attempt on transient failure | flaky network/registry/LLM endpoints |
+| `jitter` | random splay before dispatch | many nodes sharing a cron slot hitting one resource |
+| `notify` (+`notify_level`) | sk-alert on `on_failure`/`on_success`/`always` with output tail | unattended jobs; result delivery (e.g. sktrip) |
+| `catchup` | catch a missed slot once | laptops/asleep nodes (vs strict real-time jobs) |
+| overlap lock | singleton per job | long runs that exceed their interval |
+| node affinity | exactly-one-node execution | stateful/single-writer jobs (the original conflict fix) |
+
+## State, logs, observability
+- **State** (node-local): `~/.skcapstone/scheduler/<host>/state.json` — `last_run`, `ok`, `error`, run counts. Never synced.
+- **Logs** (node-local): `~/.skcapstone/scheduler/<host>/logs/<job>-<ts>.log` — captured stdout+stderr.
+- **CLI**: `skcapstone scheduler …` (list / run-now / status). `skcapstone doctor` validates scheduler health.
+
+## Operating it
+```bash
+systemctl --user status skcapstone.service     # the daemon that runs the scheduler
+systemctl --user start  skcapstone.service     # ⚠ if inactive, NO jobs.yaml jobs fire
+echo "$SK_NODE_ALIAS"                            # must match a value in a job's nodes: list
+skcapstone scheduler status                      # last run / ok / errors per job
+```
+
+## Troubleshooting
+| symptom | cause | fix |
+|---|---|---|
+| no jobs fire | `skcapstone.service` inactive | `systemctl --user start skcapstone.service` |
+| cron jobs never fire (interval ok) | `croniter` not installed in venv | `~/.skenv/bin/pip install -e .` |
+| job runs on wrong/extra node | `nodes:` affinity / `SK_NODE_ALIAS` mismatch | set the env / fix the list |
+| job overlaps itself | run time > interval | raise interval, or rely on the overlap lock (it skips) |
+| no failure alerts | `notify: off` | set `notify: on_failure` |
+
+## Relationship to other schedulers
+- **systemd user timers** — keep for OS-level/boot jobs and as the process supervisor for the daemon itself; migrate *application* recurring jobs here.
+- **Claude-Code crons** — separate system (cloud agents); documented, not migrated.
+- **legacy crontab** — retire into `jobs.yaml`.
+
+## Roadmap (not yet implemented)
+job dependencies (`after:`), dead-man's-switch staleness alerts, per-job env/secrets injection,
+a web/TUI run-history view, and distributed locking (currently affinity-by-declaration, not election).

package/docs/superpowers/examples/jobs.yaml

@@ -0,0 +1,31 @@
+# Reference for ~/.skcapstone/config/jobs.yaml — the skscheduler registry.
+#
+# This file is Syncthing-synced across the fleet. Each node runs its own
+# scheduler (inside the skcapstone daemon) and fires ONLY the jobs whose
+# `nodes` affinity includes that host. Scheduler run-state is node-local and
+# never synced (see ~/.skcapstone/.stignore -> `scheduler`).
+#
+# Job fields:
+#   schedule: "<cron>"   OR   every: <30s|5m|1h|1d|seconds>   (one of the two)
+#   type:    agent | shell | python
+#   nodes:   all | [<host alias>, ...]      (alias via SK_NODE_ALIAS env)
+#   agent:   <name>      (agent type) — runs `claude -p "<prompt>" --agent <name>`
+#   prompt:  "<text>"    (agent type)
+#   command: "<cmd>"     (shell type)
+#   callback: module:fn  (python type)
+#   timeout: <seconds>   (default 900)
+#   enabled: true|false
+
+jobs:
+  gtd-inbox-triage:
+    schedule: "0 6 * * *"      # daily 06:00
+    type: agent
+    nodes: [".41"]
+    agent: lumina
+    prompt: >
+      Triage the GTD inbox: for each item, clarify into next-action /
+      project / someday-maybe, or archive noise; move resolved-ITIL items
+      to done; surface stale projects. Use the gtd_* and itil_* MCP tools.
+      Keep it concise.
+    timeout: 900
+    enabled: true