@aria_asi/cli 0.2.40 → 0.2.41

package/skills/deno-kv-raft-pubsub/reference/raft-spec.md

@@ -0,0 +1,402 @@
+# Raft Protocol Specification for Deno KV
+
+This is the complete Raft consensus protocol specification, adapted for Deno KV as the persistent storage layer. Follow this spec exactly — deviations cause Maelstrom failures.
+
+---
+
+## 1. Persistent State (Deno KV)
+
+All persistent state is stored in Deno KV using the `/raft/` key prefix. Every mutation uses `kv.atomic()` for atomicity.
+
+### Key Schema
+
+| KV Key | Type | Description |
+|--------|------|-------------|
+| `/raft/current_term` | `number` (u64) | Latest term server has seen. Initialized to 0 on first boot. |
+| `/raft/voted_for` | `string \| null` | CandidateId that received vote in current term. Null if none. |
+| `/raft/log/{index}` | `LogEntry` | Log entry at position `index`. Index is zero-padded to 16 digits for correct lexicographic ordering. |
+| `/raft/commit_index` | `number` (u64) | Index of highest log entry known to be committed. Initialized to 0. |
+| `/raft/last_applied` | `number` (u64) | Index of highest log entry applied to state machine. Initialized to 0. |
+| `/raft/snapshot/{index}` | `Snapshot` | Compressed snapshot at index. Presence indicates log truncated before this index. |
+| `/raft/configuration` | `string[]` | Current cluster member node IDs. |
+
+### LogEntry Type
+
+```typescript
+interface LogEntry {
+  term: number;    // Term when entry was received by leader
+  index: number;   // Position in the log (redundant with key, stored for snapshot recovery)
+  command: PubSubCommand;  // The state machine command (opaque to Raft)
+}
+```
+
+### Snapshot Type
+
+```typescript
+interface Snapshot {
+  data: Uint8Array;         // Serialized state machine state
+  last_included_index: number;  // Log index covered by this snapshot
+  last_included_term: number;   // Term of entry at last_included_index
+}
+```
+
+### Atomic Operations
+
+**Updating currentTerm (must be atomic):**
+```
+kv.atomic()
+  .check({ key: "/raft/current_term", versionstamp: currentVersion })
+  .set("/raft/current_term", newTerm)
+  .commit()
+```
+
+**Voting (currentTerm and votedFor must be updated together):**
+```
+kv.atomic()
+  .check({ key: "/raft/current_term", versionstamp: currentVersion })
+  .check({ key: "/raft/voted_for", versionstamp: votedForVersion })
+  .set("/raft/current_term", newTerm)
+  .set("/raft/voted_for", candidateId)
+  .commit()
+```
+
+**Appending log entries (index must be monotonically increasing):**
+```
+kv.atomic()
+  .check({ key: "/raft/log/" + pad(index - 1), versionstamp: prevVersion })
+  .set("/raft/log/" + pad(index), entry)
+  .commit()
+```
+
+### Initialization
+
+On first boot (no KV keys exist):
+1. Set `/raft/current_term` = 0
+2. Set `/raft/voted_for` = null
+3. Set `/raft/commit_index` = 0
+4. Set `/raft/last_applied` = 0
+5. Set `/raft/configuration` = initial cluster nodes
+
+---
+
+## 2. Volatile State (In-Memory)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `role` | `"follower" \| "candidate" \| "leader"` | Current role. Starts as follower. |
+| `leaderId` | `string \| null` | ID of current leader (known to followers) |
+| `commitIndex` | `number` | Mirrors persistent commit_index for fast access |
+| `lastApplied` | `number` | Mirrors persistent last_applied |
+| `nextIndex` | `Map<string, number>` | (Leader only) Next index to send to each follower. Init to `lastLogIndex + 1`. |
+| `matchIndex` | `Map<string, number>` | (Leader only) Highest log entry known replicated on each follower. Init to 0. |
+| `electionTimer` | `number` | Timeout handle for election. Randomized 150-300ms. |
+| `heartbeatTimer` | `number` | (Leader only) Periodic heartbeat interval. Fixed 50ms. |
+
+---
+
+## 3. RPC Specifications
+
+All RPCs are sent over WebSocket connections between nodes. Messages are JSON-serialized.
+
+### 3.1 RequestVote RPC
+
+**Invoked by:** Candidate, to gather votes.
+
+**Arguments:**
+```typescript
+interface RequestVoteRequest {
+  type: "request_vote";
+  term: number;          // Candidate's term
+  candidateId: string;   // Candidate requesting vote
+  lastLogIndex: number;  // Index of candidate's last log entry
+  lastLogTerm: number;   // Term of candidate's last log entry
+}
+```
+
+**Results:**
+```typescript
+interface RequestVoteResponse {
+  type: "request_vote_response";
+  term: number;          // currentTerm, for candidate to update itself
+  voteGranted: boolean;  // true means candidate received vote
+}
+```
+
+**Receiver implementation (follower):**
+1. If `args.term < currentTerm` → reply `{term: currentTerm, voteGranted: false}`
+2. If `args.term > currentTerm` → set `currentTerm = args.term`, convert to follower, clear `votedFor` (via atomic KV update)
+3. If `votedFor` is null OR `votedFor == args.candidateId`:
+   a. AND candidate's log is at least as up-to-date (see §3.1.1):
+      - Set votedFor = args.candidateId (atomic with currentTerm)
+      - Reply `{term: currentTerm, voteGranted: true}`
+   b. ELSE: reply `{term: currentTerm, voteGranted: false}`
+4. Reset election timer (we heard from a valid candidate)
+
+**§3.1.1 Log Up-To-Date Check:**
+Candidate's log is at least as up-to-date if:
+- `args.lastLogTerm > lastLogTerm` OR
+- `args.lastLogTerm == lastLogTerm` AND `args.lastLogIndex >= lastLogIndex`
+
+Where `lastLogTerm` and `lastLogIndex` refer to the receiver's last log entry.
+
+### 3.2 AppendEntries RPC
+
+**Invoked by:** Leader, to replicate log entries. Also serves as heartbeat.
+
+**Arguments:**
+```typescript
+interface AppendEntriesRequest {
+  type: "append_entries";
+  term: number;           // Leader's term
+  leaderId: string;       // So follower can redirect clients
+  prevLogIndex: number;   // Index of log entry immediately preceding new ones
+  prevLogTerm: number;    // Term of prevLogIndex entry
+  entries: LogEntry[];    // Log entries to store (empty for heartbeat)
+  leaderCommit: number;   // Leader's commitIndex
+}
+```
+
+**Results:**
+```typescript
+interface AppendEntriesResponse {
+  type: "append_entries_response";
+  term: number;           // currentTerm, for leader to update
+  success: boolean;       // true if follower contained matching entry at prevLogIndex
+  lastLogIndex: number;   // For leader to optimize nextIndex backtracking
+}
+```
+
+**Receiver implementation (follower):**
+1. If `args.term < currentTerm` → reply `{term: currentTerm, success: false}`
+2. If `args.term >= currentTerm` → set `currentTerm = args.term`, convert to follower, set `leaderId = args.leaderId`
+3. If log does NOT contain entry at `prevLogIndex` with term matching `prevLogTerm`:
+   - Reply `{term: currentTerm, success: false, lastLogIndex: lastLogIndex}`
+   - (Leader will decrement nextIndex and retry)
+4. If existing entry conflicts with new entry (same index, different term):
+   - Delete the existing entry and all that follow it
+5. Append any new entries not already in the log
+6. If `leaderCommit > commitIndex`:
+   - Set `commitIndex = min(leaderCommit, index of last new entry)`
+   - Apply all entries from `lastApplied + 1` through `commitIndex` to state machine
+7. Reply `{term: currentTerm, success: true}`
+
+**Leader implementation (send side):**
+1. For each follower, send AppendEntries with `prevLogIndex = nextIndex[follower] - 1`
+2. On success: set `nextIndex[follower] += entries.length`, `matchIndex[follower] = prevLogIndex + entries.length`
+3. On failure: set `nextIndex[follower] -= 1` (or use `lastLogIndex` from response for faster backtracking)
+4. After each AppendEntries round, check if any N > commitIndex exists where:
+   - `N > commitIndex`
+   - A majority of `matchIndex[follower] >= N`
+   - `log[N].term == currentTerm`
+   - If so: set `commitIndex = N`
+
+**IMPORTANT — Leader Completeness:** Leaders commit only entries from their own term in any single step (this is the "commit from current term" rule). An entry from a previous term is committed indirectly when a later entry from the current term is committed.
+
+### 3.3 InstallSnapshot RPC
+
+**Invoked by:** Leader, when a follower is too far behind (nextIndex points to a log entry that has been compacted/snapshotted).
+
+**Arguments:**
+```typescript
+interface InstallSnapshotRequest {
+  type: "install_snapshot";
+  term: number;                    // Leader's term
+  leaderId: string;
+  lastIncludedIndex: number;
+  lastIncludedTerm: number;
+  offset: number;                  // Byte offset (for chunked transfer)
+  data: Uint8Array;               // Snapshot chunk
+  done: boolean;                   // True if this is the last chunk
+}
+```
+
+**Results:**
+```typescript
+interface InstallSnapshotResponse {
+  type: "install_snapshot_response";
+  term: number;
+}
+```
+
+---
+
+## 4. Leader Election Algorithm
+
+### State Transitions
+
+```
+                    timeout, start election
+        ┌─────────┐ ──────────────────────► ┌───────────┐
+        │ Follower│                          │ Candidate  │
+        └─────────┘ ◄────────────────────── └───────────┘
+              ▲       discover higher term,        │
+              │       or election timeout          │
+              │                                    │
+              │                              receives votes
+              │                              from majority
+              │                                    │
+              │                                    ▼
+              │                            ┌───────────┐
+              └────────────────────────────│  Leader   │
+                    discover higher term   └───────────┘
+```
+
+### Follower → Candidate
+
+Trigger: election timer fires (150-300ms randomized, no AppendEntries received)
+
+1. Increment `currentTerm` (atomic KV update, clearing votedFor)
+2. Vote for self (set `votedFor = self_id`)
+3. Reset election timer
+4. Send RequestVote RPC to all other nodes (parallel, non-blocking)
+5. Count votes:
+   - If vote granted: increment vote count
+   - If response term > currentTerm: step down to follower
+   - If majority votes received → become leader
+6. If election timer fires again before majority → restart election (increment term, new election)
+
+### Candidate → Leader
+
+1. Set `role = "leader"`
+2. Initialize `nextIndex[follower] = lastLogIndex + 1` for all followers
+3. Initialize `matchIndex[follower] = 0` for all followers
+4. Send immediate heartbeat (AppendEntries with empty entries) to all followers
+5. Start heartbeat timer (50ms)
+
+### Any → Follower
+
+Trigger: receive RPC with `term > currentTerm`
+
+1. Set `currentTerm = rpcTerm` (atomic KV update, clearing votedFor)
+2. Set `role = "follower"`
+3. Set `votedFor = null` (if not already)
+4. Reset election timer
+
+### Timer Configuration
+
+```typescript
+const ELECTION_TIMEOUT_MIN_MS = 150;
+const ELECTION_TIMEOUT_MAX_MS = 300;
+const HEARTBEAT_INTERVAL_MS = 50;
+
+function randomElectionTimeout(): number {
+  return ELECTION_TIMEOUT_MIN_MS +
+    Math.random() * (ELECTION_TIMEOUT_MAX_MS - ELECTION_TIMEOUT_MIN_MS);
+}
+```
+
+### Split-Vote Prevention
+
+If an election fails (no majority), the candidate waits for a new randomized timeout. The randomization ensures that in the next round, one candidate is likely to timeout before others and win.
+
+---
+
+## 5. Log Replication
+
+### Normal Operation
+
+1. Client sends command to leader (or follower redirects to leader)
+2. Leader appends command as new log entry (local KV write)
+3. Leader sends AppendEntries to all followers (in parallel)
+4. When entry replicated to majority:
+   - Leader increments `commitIndex`
+   - Leader applies entry to state machine
+   - Leader responds to client
+5. Followers learn of committed entries via next AppendEntries (heartbeat carries `leaderCommit`)
+
+### Log Matching Property
+
+**Invariant:** If two logs contain an entry with the same index and term, then the logs are identical up to and including that index.
+
+Maintained by: AppendEntries consistency check (prevLogIndex + prevLogTerm must match).
+
+### Conflict Resolution
+
+When a follower rejects an AppendEntries (log inconsistency):
+
+1. Leader decrements `nextIndex[follower]` and retries
+2. Optimization: follower returns `lastLogIndex` in failure response
+   - Leader can skip directly to that index instead of stepping one at a time
+   - This converges in O(log N) retries instead of O(N)
+
+### Commitment Rules
+
+An entry is committed when:
+1. It is stored on a majority of nodes, AND
+2. At least one entry from the leader's current term is also stored on a majority
+
+This is the "Raft commit rule" that prevents committing entries from previous terms that could be overwritten.
+
+---
+
+## 6. Snapshotting (Log Compaction)
+
+### When to Snapshot
+
+Trigger when `lastApplied - snapshotIndex > SNAPSHOT_THRESHOLD` (default: 10,000 entries).
+
+### Creating a Snapshot
+
+1. Serialize the state machine state at `lastApplied`
+2. Create snapshot entry in KV: `/raft/snapshot/{lastApplied}`
+3. Delete log entries 1 through `lastApplied` (use `kv.atomic()` with range delete)
+
+### Installing a Snapshot (Follower)
+
+1. Leader detects follower needs snapshot (nextIndex points before snapshot)
+2. Leader sends InstallSnapshot RPC (possibly chunked)
+3. Follower:
+   - Save snapshot data
+   - Discard any existing log entries
+   - Reset state machine to snapshot contents
+   - Load snapshot's last included index/term
+   - Respond
+
+### Restarting from Snapshot
+
+On node restart:
+1. Check for latest snapshot in KV (`/raft/snapshot/` prefix, find max index)
+2. If snapshot exists:
+   - Load state machine from snapshot
+   - Set `lastApplied = snapshot.last_included_index`
+   - Set `commitIndex = snapshot.last_included_index`
+3. Load remaining log entries after snapshot index
+4. Apply any unapplied committed entries
+
+---
+
+## 7. Consistency Guarantees
+
+| Property | Implementation |
+|----------|---------------|
+| **Election Safety** | At most one leader per term (votedFor prevents double-voting) |
+| **Leader Append-Only** | Leader never overwrites or deletes entries |
+| **Log Matching** | If two logs have entries at same index with same term, all prior entries match |
+| **Leader Completeness** | Committed entries survive elections (election restriction: voter checks log currency) |
+| **State Machine Safety** | All nodes apply same commands in same order (identical replicated log) |
+
+---
+
+## 8. Common Bugs (from Raft paper and Maelstrom experience)
+
+1. **Not persisting votedFor before responding to RequestVote** — must atomically update currentTerm + votedFor, THEN respond. Otherwise a crashed-then-restarted node might vote twice in the same term.
+
+2. **Committing entries from previous terms** — use the "commit from current term" rule. If leader tries to commit old entries directly, they might be overwritten after a leader change.
+
+3. **Log truncation on AppendEntries rejection** — follower must delete conflicting entries AND ALL FOLLOWING ENTRIES, not just the conflicting one.
+
+4. **Not resetting election timer** — when a follower receives a valid AppendEntries OR a valid RequestVote (even if it doesn't grant the vote), reset the timer. Otherwise unnecessary elections.
+
+5. **Using wall-clock time in state machine** — never call `Date.now()` in `apply()`. Pass timestamps in commands instead.
+
+6. **Non-atomic KV updates** — always use `kv.atomic()` with versionstamp checks when updating `currentTerm` and `votedFor` together. A non-atomic update could leave them inconsistent.
+
+7. **Heartbeat too fast** — 50ms is aggressive but necessary for fast failover in Maelstrom tests (where timeouts are 150-300ms). In production, consider 100-200ms.
+
+---
+
+## References
+
+- [Raft Paper (Extended)](https://raft.github.io/raft.pdf) — Ongaro, Ousterhout — the canonical spec
+- [Raft Website](https://raft.github.io/) — Visualizations and thesis
+- [Maelstrom Raft Guide](https://github.com/jepsen-io/maelstrom/blob/main/doc/06-raft.md) — Maelstrom-specific Raft implementation notes

package/skills/deno-kv-raft-pubsub/reference/state-machine.md

@@ -0,0 +1,182 @@
+# Deterministic State Machine Design
+
+The state machine transforms committed Raft log entries into PubSub state. It must be **deterministic**: given the same log of commands, any node must produce identical state.
+
+---
+
+## 1. Command Types
+
+All state changes go through these command types, serialized in the Raft log:
+
+```typescript
+type PubSubCommand =
+  | CreateTopicCommand
+  | DeleteTopicCommand
+  | PublishCommand
+  | SubscribeCommand
+  | UnsubscribeCommand;
+
+interface CreateTopicCommand {
+  type: "create_topic";
+  topic: string;           // Topic name (unique)
+  metadata?: Record<string, unknown>;
+}
+
+interface DeleteTopicCommand {
+  type: "delete_topic";
+  topic: string;
+}
+
+interface PublishCommand {
+  type: "publish";
+  id: string;              // Unique message ID (crypto.randomUUID() by client)
+  topic: string;
+  payload: unknown;        // JSON-serializable payload
+  timestamp: number;       // Client-provided timestamp (NOT Date.now() in state machine)
+}
+
+interface SubscribeCommand {
+  type: "subscribe";
+  topic: string;
+  subscriber_id: string;   // Unique subscriber identifier
+}
+
+interface UnsubscribeCommand {
+  type: "unsubscribe";
+  topic: string;
+  subscriber_id: string;
+}
+```
+
+---
+
+## 2. State Machine State
+
+### In-Memory State
+
+```typescript
+class PubSubStateMachine {
+  // Active topics
+  private topics: Set<string>;
+
+  // Per-topic message sequence counter
+  private sequences: Map<string, number>;
+
+  // Active subscriptions (subscriber_id → topic)
+  private subscriptions: Map<string, Set<string>>;
+
+  // Last applied index (for idempotent replay)
+  private lastAppliedIndex: number;
+}
+```
+
+### Persistent State (Deno KV)
+
+| KV Key | Value | Purpose |
+|--------|-------|---------|
+| `/pubsub/topics/{name}` | `{ created_at: number, deleted?: boolean }` | Topic registry with tombstone support |
+| `/pubsub/messages/{topic}/{seq:016d}` | `{ id: string, payload: unknown, timestamp: number }` | Message storage, zero-padded sequence |
+| `/pubsub/sequences/{topic}` | `number` | Current sequence number for topic |
+| `/pubsub/subscriptions/{sub_id}` | `{ subscriber_id: string, topic: string, last_delivered: number }` | Subscription with cursor |
+
+---
+
+## 3. apply() Implementation
+
+The `apply(command: PubSubCommand, index: number)` method is called sequentially for each committed log entry. It must be **fully deterministic**.
+
+### Determinism Rules
+
+1. **No random number generation** — all randomness must come from the command (client generates UUIDs)
+2. **No wall-clock time** — use `command.timestamp` (provided by client), not `Date.now()`
+3. **No node-ID-dependent logic** — all nodes must produce identical KV state
+4. **No external I/O** — no network calls, no file system reads
+5. **Idempotent replay** — if called twice with same command at same index, no side effects beyond the first call (checked via `lastAppliedIndex`)
+
+### apply() Pseudocode
+
+```
+function apply(command: PubSubCommand, index: number):
+    if index <= lastAppliedIndex:
+        return  // Idempotent replay guard
+
+    switch command.type:
+        case "create_topic":
+            kv.set("/pubsub/topics/" + command.topic, {
+                created_at: command.timestamp || Date.now()
+            })
+            topics.add(command.topic)
+            sequences.set(command.topic, 0)
+
+        case "delete_topic":
+            kv.set("/pubsub/topics/" + command.topic, {
+                created_at: existing.created_at,
+                deleted: true
+            })
+            topics.delete(command.topic)
+
+        case "publish":
+            if !topics.has(command.topic):
+                return  // Topic doesn't exist, silent no-op
+            seq = sequences.get(command.topic) + 1
+            sequences.set(command.topic, seq)
+            kv.atomic()
+                .set("/pubsub/messages/" + command.topic + "/" + pad16(seq), {
+                    id: command.id,
+                    payload: command.payload,
+                    timestamp: command.timestamp
+                })
+                .set("/pubsub/sequences/" + command.topic, seq)
+                .commit()
+            // kv.watch() on /pubsub/messages/{topic}/ will fire for subscribers
+
+        case "subscribe":
+            kv.set("/pubsub/subscriptions/" + command.subscriber_id, {
+                subscriber_id: command.subscriber_id,
+                topic: command.topic,
+                last_delivered: sequences.get(command.topic) || 0
+            })
+            if !subscriptions.has(command.subscriber_id):
+                subscriptions.set(command.subscriber_id, new Set())
+            subscriptions.get(command.subscriber_id).add(command.topic)
+            // Start kv.watch for this subscriber on this topic
+
+        case "unsubscribe":
+            kv.delete("/pubsub/subscriptions/" + command.subscriber_id)
+            if subscriptions.has(command.subscriber_id):
+                subscriptions.get(command.subscriber_id).delete(command.topic)
+            // Stop kv.watch for this subscriber on this topic
+
+    lastAppliedIndex = index
+```
+
+---
+
+## 4. Snapshot Format
+
+When the Raft layer takes a snapshot, the state machine serializes its state:
+
+```typescript
+interface PubSubSnapshot {
+  topics: Array<{ name: string; created_at: number; deleted?: boolean }>;
+  sequences: Record<string, number>;  // topic → last sequence number
+  subscriptions: Record<string, Array<string>>;  // subscriber_id → topics
+}
+```
+
+**Restoring from snapshot:**
+1. Deserialize snapshot data
+2. Repopulate in-memory state (topics, sequences, subscriptions)
+3. All persistent KV state for messages remains (snapshot doesn't delete message history — only the Raft log entries are compacted)
+
+---
+
+## 5. Lock-Free Delivery Model
+
+The state machine doesn't directly deliver messages. It writes to KV, and the Delivery layer reads from KV via `kv.watch()`.
+
+This separation means:
+- State machine apply() is fast (just KV writes)
+- Delivery is async and non-blocking
+- Subscribers on any node see messages as they are committed
+- If a subscriber disconnects, it can reconnect and resume from its last cursor