@aria_asi/cli 0.2.40 → 0.2.41

package/skills/deno-kv-raft-pubsub/SKILL.md

@@ -0,0 +1,561 @@
+---
+name: deno-kv-raft-pubsub
+description: Design and build a replicated, Raft-consensus-backed PubSub system on Deno KV with Jepsen/Maelstrom correctness testing. Use when building distributed pubsub, implementing Raft from first principles, creating Maelstrom test workloads, or designing linearizable message delivery on Deno. Do NOT use for simple single-node Deno KV usage, non-consensus messaging, or when a ready-made message broker (NATS, Redis, Kafka) would suffice.
+license: MIT
+metadata:
+  author: system-architect
+  version: "1.0.0"
+  domains: distributed-systems, consensus, pubsub
+allowed-tools:
+  - Bash(deno *)
+  - Bash(mkdir *)
+  - Bash(ls *)
+  - Bash(cat *)
+  - Read(*)
+  - Write(*)
+  - Edit(*)
+  - Glob(*)
+  - Grep(*)
+  - Task(*)
+---
+
+# Deno KV Raft PubSub
+
+A complete guide for building a production-grade, Raft-consensus-backed PubSub system on top of Deno KV, verified with the Jepsen/Maelstrom distributed systems workbench. Every design decision is justified from first principles.
+
+---
+
+## What I Do
+
+I guide the construction of a replicated PubSub system where:
+
+1. **The Raft replicated log IS the pubsub message bus** — every publish, subscribe, and unsubscribe is a deterministic command in the Raft log. When committed, the state machine applies it and triggers local subscriber delivery via `Deno.Kv.watch()`.
+
+2. **Deno KV IS the persistent storage layer** — Raft persistent state (currentTerm, votedFor, log entries, snapshots) is stored in Deno KV per-node. Atomic operations (`kv.atomic()`) enforce Raft invariants at the storage level.
+
+3. **Maelstrom IS the correctness gate** — custom Maelstrom workloads test linearizability, durability, partition tolerance, and crash recovery. No production deployment without green Jepsen tests.
+
+4. **The architecture IS layered** — clean separation between Raft core, state machine, PubSub API, transport, and test harness. Each layer has a well-defined contract and can be tested in isolation.
+
+---
+
+## When to Use Me
+
+**Trigger phrases:**
+- "build a distributed pubsub system"
+- "implement Raft consensus"
+- "replicated message broker on Deno"
+- "linearizable pubsub"
+- "Maelstrom testing for consensus"
+- "Jepsen verification"
+- "Deno KV pubsub distributed"
+- "from first principles distributed systems"
+
+**Task patterns:**
+- Designing a new replicated service on Deno KV
+- Adding consensus to an existing Deno KV pubsub
+- Building a Maelstrom test suite for a node
+- Learning Raft by building it
+
+---
+
+## When NOT to Use Me
+
+- **Single-node pubsub** — if you only need in-process pubsub on one Deno KV instance, use `kv.watch()` directly. Raft adds ~3x complexity.
+- **Existing message brokers** — if NATS, Redis Streams, or Apache Kafka meet your needs, use them. Building Raft yourself is for when you need embedded consensus with no external dependencies.
+- **Non-Deno runtimes** — this design is Deno KV-specific. For Node.js, Python, or Go, use a different storage backend (etcd's Raft, etc.).
+- **Already-greenfield** — if you have a working consensus system, don't rebuild. This skill is for new construction.
+- **Performance-critical hot paths** — Raft consensus adds latency (at least 1 RTT for commit). If you need microsecond pubsub, this isn't the right pattern. Use in-memory channels with eventual consistency instead.
+
+---
+
+## Architecture: First Principles
+
+### Core Insight: The Raft Log IS the PubSub Bus
+
+```
+Client publishes "hello" to topic "orders"
+          │
+          ▼
+    ┌──────────────┐
+    │ Raft Leader  │  ← receives publish request
+    │ appends log  │
+    │ entry:       │
+    │ {term: 5,    │
+    │  index: 42,  │
+    │  cmd: {      │
+    │   type:      │
+    │    "publish",│
+    │   topic:     │
+    │    "orders", │
+    │   payload:   │
+    │    "hello"}  │
+    │ }            │
+    └──────┬───────┘
+           │ AppendEntries RPC
+           ▼
+    ┌──────────────┐     ┌──────────────┐
+    │  Follower 1  │     │  Follower 2  │
+    │  replicates  │     │  replicates  │
+    │  log entry   │     │  log entry   │
+    └──────┬───────┘     └──────┬───────┘
+           │                    │
+           ▼                    ▼
+    ┌──────────────────────────────────┐
+    │ All nodes commit entry at idx 42 │
+    │ State machine applies command    │
+    │ → writes to local KV             │
+    │ → kv.watch() fires on each node  │
+    │ → subscribers notified           │
+    └──────────────────────────────────┘
+```
+
+**Why this works:**
+- Raft guarantees: committed entries are identical across all nodes, in the same order.
+- If entry 42 is committed on node A, every node that commits 42 has identical content.
+- Subscribers on any node see the same message stream for a given topic.
+
+**Safety property:** If a publish returns success, every subsequent subscribe on any node sees that message (in commit order).
+
+### System Components
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│                     RaftPubSubNode                            │
+│                                                               │
+│  ┌──────────────────┐    ┌──────────────────┐                │
+│  │   Raft Core      │    │   PubSub API     │                │
+│  │                  │    │                  │                │
+│  │ • Leader Election│    │ • Publish(t,m)   │                │
+│  │ • AppendEntries  │    │ • Subscribe(t)   │                │
+│  │ • Log Replication│    │ • Unsubscribe(t) │                │
+│  │ • Commit/Apply   │    │ • ListTopics()   │                │
+│  │ • Snapshots      │    │ • Ack/Cursor     │                │
+│  └────────┬─────────┘    └────────┬─────────┘                │
+│           │                       │                           │
+│           ▼                       ▼                           │
+│  ┌────────────────────────────────────────────────┐          │
+│  │            State Machine                       │          │
+│  │  ┌──────────────┐  ┌──────────────────────┐   │          │
+│  │  │ Topic        │  │ Subscription         │   │          │
+│  │  │ Registry     │  │ Manager              │   │          │
+│  │  │ (topics      │  │ (kv.watch based      │   │          │
+│  │  │  as KV keys) │  │  per-subscriber)     │   │          │
+│  │  └──────────────┘  └──────────────────────┘   │          │
+│  └───────────────────────┬────────────────────────┘          │
+│                          │                                    │
+│                          ▼                                    │
+│  ┌────────────────────────────────────────────────┐          │
+│  │              Deno KV (per-node)                │          │
+│  │                                                │          │
+│  │  /raft/current_term    → number                │          │
+│  │  /raft/voted_for       → string|null           │          │
+│  │  /raft/log/[index]     → LogEntry              │          │
+│  │  /raft/commit_index    → number                │          │
+│  │  /raft/last_applied    → number                │          │
+│  │  /raft/snapshot/[idx]  → Snapshot              │          │
+│  │                                                │          │
+│  │  /pubsub/topics/[name] → TopicMeta             │          │
+│  │  /pubsub/messages/     →                       │          │
+│  │     [topic]/[seq]      → Message               │          │
+│  │  /pubsub/subscriptions/                        │          │
+│  │     [sub_id]           → Subscription          │          │
+│  └────────────────────────────────────────────────┘          │
+│                          │                                    │
+│  ┌───────────────────────┴────────────────────────┐          │
+│  │          Transport Layer                       │          │
+│  │  ┌──────────────────┐  ┌──────────────────┐   │          │
+│  │  │ WebSocket RPC    │  │ HTTP/WSS API     │   │          │
+│  │  │ (inter-node)     │  │ (client-facing)  │   │          │
+│  │  │ • RequestVote    │  │ • POST /publish  │   │          │
+│  │  │ • AppendEntries  │  │ • WS  /subscribe │   │          │
+│  │  │ • InstallSnapshot│  │ • GET  /topics   │   │          │
+│  │  └──────────────────┘  └──────────────────┘   │          │
+│  └────────────────────────────────────────────────┘          │
+│                                                               │
+│  ┌────────────────────────────────────────────────┐          │
+│  │       Maelstrom Harness (stdin/stdout)         │          │
+│  │  JSON-RPC protocol for Jepsen testing          │          │
+│  └────────────────────────────────────────────────┘          │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Key Design Decisions
+
+| Decision | Rationale | Tradeoff |
+|----------|-----------|----------|
+| **Raft log in Deno KV** (not in-memory) | Durability across restarts; atomic ops enforce invariants | ~1ms KV read latency per log access vs ~ns for memory |
+| **kv.watch() for subscriber delivery** | Leverages Deno's built-in change notification; subscribers get push-based delivery | Each subscriber opens a watch; bounded by Deno's watch limit |
+| **All state machine commands via Raft log** | Guarantees identical state across all nodes; enables Maelstrom verification | Write latency = 1 RTT for consensus |
+| **WebSocket for inter-node RPC** | Bidirectional, low-overhead, Deno-native | Requires connection management; less tooling than HTTP |
+| **Client routes to leader** (with redirect) | Only leader can propose; simplifies consistency | Clients need leader discovery; extra redirect hop |
+| **Snapshots at configurable thresholds** | Prevents unbounded log growth; enables fast restart | Snapshot creation is CPU/memory intensive; choose threshold carefully |
+
+---
+
+## Workflow
+
+Follow these phases in order. Each phase builds on the previous. Do NOT skip ahead — the system will not be testable until Phase 2 is complete.
+
+### Phase 0: Environment Setup
+
+**Goal:** Deno runtime + Maelstrom installed + project scaffolded.
+
+1. Verify Deno installation: `deno --version` (≥1.40 required for KV)
+2. Install Maelstrom: download from [Jepsen Maelstrom releases](https://github.com/jepsen-io/maelstrom/releases)
+3. Verify Maelstrom: `java -jar maelstrom.jar test --help`
+4. Scaffold project structure:
+
+```
+deno-kv-raft-pubsub/
+├── src/
+│   ├── raft/
+│   │   ├── mod.ts          # Public Raft API
+│   │   ├── types.ts        # Shared types (Term, Index, LogEntry, etc.)
+│   │   ├── persistent.ts   # Deno KV persistence layer
+│   │   ├── election.ts     # Leader election logic
+│   │   ├── replication.ts  # AppendEntries + commit
+│   │   ├── snapshot.ts     # Log compaction
+│   │   └── rpc.ts          # Inter-node RPC transport
+│   ├── pubsub/
+│   │   ├── mod.ts          # Public PubSub API
+│   │   ├── types.ts        # PubSub types
+│   │   ├── state_machine.ts # Deterministic state machine
+│   │   ├── delivery.ts     # kv.watch() based delivery
+│   │   └── api.ts          # HTTP/WS client API
+│   ├── maelstrom/
+│   │   ├── mod.ts          # Maelstrom protocol handler
+│   │   ├── types.ts        # Maelstrom message types
+│   │   └── node.ts         # Maelstrom worker entrypoint
+│   └── server.ts           # Main server entrypoint
+├── tests/
+│   ├── raft_test.ts        # Unit tests for Raft
+│   ├── pubsub_test.ts      # Unit tests for PubSub
+│   └── integration_test.ts # Multi-node integration
+├── maelstrom/
+│   ├── workloads/
+│   │   ├── pubsub_append.clj   # Custom workload
+│   │   └── pubsub_total.clj    # Custom workload
+│   └── run_tests.sh        # Test runner script
+├── deno.json               # Deno config
+└── README.md
+```
+
+5. Create `deno.json`:
+```json
+{
+  "tasks": {
+    "dev": "deno run --allow-net --allow-read --allow-write --unstable-kv src/server.ts",
+    "test": "deno test --allow-net --allow-read --allow-write --unstable-kv",
+    "maelstrom": "./maelstrom/run_tests.sh"
+  },
+  "imports": {
+    "std/": "https://deno.land/std@0.224.0/"
+  }
+}
+```
+
+### Phase 1: Raft Core (No PubSub)
+
+**Goal:** A working Raft implementation that passes Maelstrom's `lin-kv` workload (linearizable KV store). This validates consensus before adding pubsub complexity.
+
+Read these references before starting:
+- [reference/raft-spec.md](reference/raft-spec.md) — Complete Raft spec adapted for Deno KV
+- [reference/state-machine.md](reference/state-machine.md) — Deterministic state machine design
+
+**Sub-phase 1.1: Persistent State Layer**
+
+Implement `persistent.ts`:
+- Initialize Deno KV connection (open with path parameter for per-node isolation)
+- Define KV key schemas for all Raft persistent state
+- Implement atomic read-modify-write for `currentTerm`, `votedFor`
+- Implement log entry append with atomic index tracking
+- Implement commit index and last_applied tracking
+
+**Validation:** Write a test that starts two nodes, has one become leader, commits entries, and verifies followers replicate them.
+
+**Sub-phase 1.2: Leader Election**
+
+Implement `election.ts`:
+- Election timer with randomized timeout (150–300ms)
+- RequestVote RPC: send and handle
+- Follower grants vote if: candidate's term ≥ currentTerm AND (votedFor is null OR votedFor is candidate) AND candidate's log is at least as up-to-date
+- Candidate becomes leader upon majority votes
+- Leader sends heartbeats (AppendEntries with no entries) every 50ms
+- Step-down: if a node receives an RPC with term > currentTerm, revert to follower
+
+**Validation:** Run a 3-node cluster without any client commands. Kill the leader. Verify election completes within 1 second.
+
+**Sub-phase 1.3: Log Replication**
+
+Implement `replication.ts`:
+- Leader appends client command to local log
+- Sends AppendEntries to all followers with prevLogIndex, prevLogTerm
+- Follower checks: log contains entry at prevLogIndex with matching term
+  - If yes: append new entries, respond success
+  - If no: respond failure, leader decrements nextIndex and retries
+- Leader commits when entry is replicated to majority
+- Apply committed entries to state machine in order
+
+**Sub-phase 1.4: Maelstrom lin-kv Test**
+
+Implement `maelstrom/mod.ts` and `maelstrom/node.ts`:
+- Parse stdin JSON lines → MaelstromMessage
+- Handle `init` message: store node_id, node_ids
+- Handle `read`, `write`, `cas` messages → route to Raft leader as state machine commands
+- Return responses to stdout
+- Build a trivial KV state machine (just a Map) for testing
+
+Run: `java -jar maelstrom.jar test -w lin-kv --bin deno run --allow-all src/maelstrom/node.ts --node-count 3 --time-limit 30 --rate 10`
+
+**Gate:** Do NOT proceed to Phase 2 until `lin-kv` passes with 0 anomalies. If Maelstrom reports consistency violations, debug the Raft implementation. Common issues:
+- Incorrect log matching check in AppendEntries handler
+- Not persisting votedFor before responding to RequestVote
+- Committing entries from previous terms (use leader's term for commit)
+- Leader not waiting for majority before responding to client
+
+### Phase 2: PubSub State Machine
+
+**Goal:** Replace the trivial KV state machine with a PubSub state machine that handles topic management and message delivery.
+
+Read: [reference/pubsub-api.md](reference/pubsub-api.md) — Complete API contract
+
+**Sub-phase 2.1: State Machine Commands**
+
+Define command types in `pubsub/types.ts`:
+```typescript
+type PubSubCommand =
+  | { type: "create_topic"; topic: string; metadata?: Record<string, unknown> }
+  | { type: "delete_topic"; topic: string }
+  | { type: "publish"; topic: string; payload: unknown; id: string }
+  | { type: "subscribe"; topic: string; subscriber_id: string }
+  | { type: "unsubscribe"; topic: string; subscriber_id: string };
+```
+
+**Sub-phase 2.2: Deterministic State Machine**
+
+Implement `pubsub/state_machine.ts`:
+- Maintain topic registry (Set of active topics)
+- For `publish`: assign monotonically increasing sequence number per topic, write message to KV at `/pubsub/messages/{topic}/{seq}`
+- For `subscribe`/`unsubscribe`: track subscriptions in KV (for crash recovery)
+- All mutations go through the Raft log (leader proposes, all apply)
+
+**Critical: Determinism** — The state machine must produce identical results on every node:
+- No random number generation in apply()
+- No wall-clock time in apply()
+- No node-ID-dependent logic in apply()
+- All inputs come from the command itself
+
+**Sub-phase 2.3: Subscriber Delivery**
+
+Implement `pubsub/delivery.ts`:
+- On state machine apply of a `publish` command, write to KV key `/pubsub/messages/{topic}/{seq}`
+- Subscribers use `kv.watch([prefix("/pubsub/messages/{topic}/")])` to receive new messages
+- On `subscribe` command applied, start a new kv.watch for that topic
+- Handle reconnection: after crash recovery, subscribers restart from last committed sequence
+
+**Sub-phase 2.4: Client API**
+
+Implement `pubsub/api.ts`:
+- `POST /publish/:topic` — forward to leader; leader proposes to Raft; respond after commit (or with future index for async)
+- `GET /subscribe/:topic` (WebSocket upgrade) — establish persistent connection; stream messages from kv.watch
+- `GET /topics` — read from local state machine (all nodes have identical state)
+- Leader discovery: any node returns `{ leader: "node-2:8080" }` on requests
+
+### Phase 3: Maelstrom PubSub Testing
+
+**Goal:** Create custom Maelstrom workloads that verify PubSub safety properties.
+
+Read: [reference/maelstrom-integration.md](reference/maelstrom-integration.md) — Complete Maelstrom integration guide
+
+**Custom Workloads:**
+
+1. **pubsub-append** — Append-only workload:
+   - Clients publish to topics, subscribers verify they receive all messages in order
+   - Network partitions injected; verify minority partition makes no progress
+   - Verify: no gaps, no duplicates, ordering preserved
+
+2. **pubsub-total** — Linearizability workload:
+   - Multiple publishers on same topic, multiple subscribers
+   - After each publish acknowledged, subscriber on ANY node must see it
+   - Verify: real-time ordering (if publish A completes before publish B starts, A appears before B in all subscriber streams)
+
+3. **pubsub-crash** — Crash recovery:
+   - Kill leaders during publish, verify no lost messages
+   - Kill followers, verify they catch up on restart
+   - Kill all nodes, restart, verify all committed messages are deliverable
+
+4. **pubsub-set** — Set-based safety:
+   - Track set of all published messages across all nodes
+   - Verify: no message appears in subscriber stream that hasn't been published
+   - Verify: all published-and-acknowledged messages appear in all subscriber streams
+
+**Testing cadence:**
+```
+Phase 1 Gate: lin-kv passes (30s, 3 nodes, 10 req/s) ← MUST PASS
+Phase 2 Gate: pubsub-append passes (60s, 3 nodes, 20 req/s)
+Phase 3 Gate: pubsub-total passes (60s, 3 nodes, 20 req/s)
+Release Gate: all workloads pass (120s, 5 nodes, 50 req/s)
+```
+
+### Phase 4: Production Hardening
+
+Only after all Maelstrom tests pass:
+
+1. **Snapshotting** — Implement log compaction when log exceeds 10,000 entries:
+   - Create snapshot of state machine
+   - Truncate log up to snapshot index
+   - Send snapshots to slow followers via InstallSnapshot RPC
+
+2. **Membership changes** — Implement joint consensus for adding/removing nodes:
+   - Enter joint configuration (Cold,new)
+   - Commit joint config
+   - Transition to new config (Cnew)
+   - Commit new config
+
+3. **Backpressure** — Limit in-flight AppendEntries per follower; pause proposals when log grows too far ahead of commit index
+
+4. **Observability** — Expose metrics:
+   - `raft_term` (gauge), `raft_commit_index` (gauge), `raft_log_size` (gauge)
+   - `pubsub_messages_published_total` (counter), `pubsub_subscribers_active` (gauge)
+   - `raft_elections_total` (counter), `raft_leader_changes_total` (counter)
+
+5. **Client libraries** — Generate from API spec; provide at-least-once delivery with client-side dedup
+
+---
+
+## Safety Properties (The Contract)
+
+| Property | What it means | How Raft guarantees it |
+|----------|---------------|----------------------|
+| **Linearizability** | After publish returns, all reads see it | Write goes through leader → committed → applied → visible |
+| **Ordering** | Messages in a topic are in commit order | Raft log is totally ordered; state machine applies sequentially |
+| **Exactly-once** | No duplicate message delivery | Monotonic sequence numbers per topic; clients track last received |
+| **Durability** | Committed messages survive crashes | Raft log persisted in Deno KV; majority replication |
+| **Partition tolerance** | Minority partition doesn't corrupt | No commit without majority; split-brain prevented by election restriction |
+
+---
+
+## References
+
+These reference files provide detailed specifications. Read them in order:
+
+1. **[reference/raft-spec.md](reference/raft-spec.md)** — Complete Raft protocol specification adapted for Deno KV storage. Includes persistent state key schemas, RPC message formats, leader election algorithm, log replication protocol, and snapshot installation.
+
+2. **[reference/state-machine.md](reference/state-machine.md)** — Deterministic state machine design for PubSub. Includes command types, apply() semantics, KV key layout, and determinism constraints.
+
+3. **[reference/pubsub-api.md](reference/pubsub-api.md)** — Client-facing API contract. HTTP endpoints, WebSocket protocol, error codes, leader discovery, and delivery semantics (at-least-once vs exactly-once).
+
+4. **[reference/maelstrom-integration.md](reference/maelstrom-integration.md)** — Maelstrom test harness integration. stdin/stdout protocol, custom workload design, Jepsen checker integration, and fault injection patterns.
+
+---
+
+## Output & Organization
+
+### Source File Structure
+
+All implementation goes under `src/` with strict module boundaries:
+
+```
+src/
+├── raft/           # Pure Raft — no pubsub awareness
+│   ├── mod.ts      # Exports: RaftNode class
+│   ├── types.ts    # Term, Index, LogEntry, Role, RpcMessage
+│   ├── persistent.ts  # KV read/write for Raft state
+│   ├── election.ts    # Timeout management, RequestVote
+│   ├── replication.ts # AppendEntries, commit tracking
+│   ├── snapshot.ts    # Snapshot creation and installation
+│   └── rpc.ts         # WebSocket transport, serialization
+├── pubsub/         # PubSub on top of Raft
+│   ├── mod.ts      # Exports: PubSubNode class
+│   ├── types.ts    # PubSubCommand, TopicMeta, Message, Subscription
+│   ├── state_machine.ts  # Deterministic apply() for all commands
+│   ├── delivery.ts       # kv.watch() subscriber delivery
+│   └── api.ts            # HTTP/WS server for client connections
+├── maelstrom/      # Maelstrom test harness
+│   ├── mod.ts      # Exports: MaelstromNode class
+│   ├── types.ts    # MaelstromMessage, InitBody, RpcBody
+│   └── node.ts     # Entrypoint: stdin→parse→route→stdout
+└── server.ts       # Production entrypoint (HTTP + WS + Raft)
+```
+
+### Naming Conventions
+
+- **Files**: lowercase with underscores, `.ts` extension
+- **Modules**: one public `mod.ts` per directory that re-exports the public API
+- **Types**: PascalCase for types (`LogEntry`), UPPER_SNAKE for constants (`HEARTBEAT_INTERVAL_MS`)
+- **Functions**: camelCase for pure functions (`becomeLeader`), PascalCase for constructors
+- **KV Keys**: `/prefix/subprefix/identifier` format (e.g., `/raft/log/0000000042`)
+- **Test files**: mirror source structure with `_test.ts` suffix
+
+### Build & Run Commands
+
+```bash
+# Development
+deno task dev                    # Start a single node
+
+# Multi-node local cluster (for integration testing)
+deno run --allow-all src/server.ts --node-id node-1 --port 8081 --peers node-2:8082,node-3:8083 &
+deno run --allow-all src/server.ts --node-id node-2 --port 8082 --peers node-1:8081,node-3:8083 &
+deno run --allow-all src/server.ts --node-id node-3 --port 8083 --peers node-1:8081,node-2:8082 &
+
+# Maelstrom testing (lin-kv gate)
+java -jar maelstrom.jar test \
+  -w lin-kv \
+  --bin deno run --allow-all src/maelstrom/node.ts \
+  --node-count 3 \
+  --time-limit 30 \
+  --rate 10
+
+# Maelstrom testing (custom pubsub workloads)
+java -jar maelstrom.jar test \
+  -w pubsub-append \
+  --bin deno run --allow-all src/maelstrom/node.ts \
+  --node-count 3 \
+  --time-limit 60 \
+  --rate 20
+
+# Run all tests
+deno task test
+deno task maelstrom
+```
+
+### Kv Key Schemas
+
+All raft state in Deno KV uses the `/raft/` prefix:
+```
+/raft/current_term       → number (u64)
+/raft/voted_for          → string | null
+/raft/log/{index:016d}   → { term: number, command: PubSubCommand }
+/raft/commit_index       → number (u64)
+/raft/last_applied       → number (u64)
+/raft/snapshot/{index}   → { data: Uint8Array, last_included_index: number, last_included_term: number }
+/raft/configuration      → string[] (node IDs)
+```
+
+All pubsub state uses the `/pubsub/` prefix:
+```
+/pubsub/topics/{name}              → { created_at: number, metadata?: object }
+/pubsub/messages/{topic}/{seq:016d} → { id: string, payload: unknown, timestamp: number }
+/pubsub/subscriptions/{sub_id}     → { subscriber_id: string, topic: string, last_delivered: number }
+/pubsub/sequences/{topic}          → number (last sequence number)
+```
+
+Zen: the raft log is the truth. the state machine is its shadow. pubsub is what the shadow casts.
+## First-Class Production Contract
+
+This skill must deepen the work, not decorate it. When loaded, apply it as a production requirement:
+
+- no placeholders, no fake fallbacks, no vague future wiring, and no cosmetic compliance
+- real proof from the repo, runtime, tests, logs, endpoint, or owner instruction before any completion claim
+- a redo contract whenever the skill blocks or finds a gap: what failed, what to change, how to re-test, and when to re-submit
+- architect/ARIA console escalation when the same failure repeats twice, crosses system boundaries, or cannot be verified locally
+- owner-visible output that states the exact state of the work: changed, verified, blocked, or needing a decision
+## Self-Executing Recovery Contract
+
+When this skill blocks, fails verification, or finds a gap, the agent must execute recovery rather than hand work back as a user-only decision.
+
+- Make exactly one self-executed recovery attempt using the strongest safe local, repository, runtime, test, log, endpoint, or owner-provided evidence available.
+- If the same failure repeats after that one recovery attempt, switch to architect execution mode: operate under the architect/ARIA harness to diagnose, choose the next action, execute the repair path, and verify the result.
+- If Aria or the architect service is unavailable, fall back to the strongest available client LLM under the architect harness or strongest available local harness substrate; mark the fallback explicitly and continue recovery instead of stopping.
+- Do not end with user-handoff-only behavior. If blocked after the recovery path, report the executed recovery attempt, evidence, remaining blocker, and the next concrete action or owner decision needed; do not merely ask the user to take over.