@aria_asi/cli 0.2.39 → 0.2.41

package/skills/deno-kv-raft-pubsub/reference/maelstrom-integration.md

@@ -0,0 +1,393 @@
+# Maelstrom Integration Guide
+
+Maelstrom is Jepsen's workbench for testing distributed systems. This guide covers integrating your Raft PubSub node with Maelstrom's test harness.
+
+---
+
+## 1. Maelstrom Protocol
+
+Maelstrom communicates with your node via **stdin/stdout JSON lines**. One JSON object per line, newline-delimited.
+
+### Message Format
+
+```json
+{
+  "src": "c1",
+  "dest": "n0",
+  "body": {
+    "type": "read",
+    "msg_id": 1,
+    "key": 5
+  }
+}
+```
+
+- `src`: Source node/client ID
+- `dest`: Destination node ID (your node)
+- `body`: Message body with `type` field
+
+### Required Message Types
+
+Your node MUST handle:
+
+#### init
+```json
+{
+  "src": "c1", "dest": "n0",
+  "body": {
+    "type": "init",
+    "msg_id": 1,
+    "node_id": "n0",
+    "node_ids": ["n0", "n1", "n2"]
+  }
+}
+```
+Response:
+```json
+{
+  "src": "n0", "dest": "c1",
+  "body": {
+    "type": "init_ok",
+    "in_reply_to": 1
+  }
+}
+```
+
+#### echo (health check)
+```json
+{
+  "src": "c1", "dest": "n0",
+  "body": {
+    "type": "echo",
+    "msg_id": 2,
+    "echo": "hello"
+  }
+}
+```
+Response:
+```json
+{
+  "src": "n0", "dest": "c1",
+  "body": {
+    "type": "echo_ok",
+    "in_reply_to": 2,
+    "echo": "hello"
+  }
+}
+```
+
+---
+
+## 2. Node Entrypoint
+
+Your Maelstrom binary (specified via `--bin`) is a Deno script that:
+
+1. Reads lines from `Deno.stdin`
+2. Parses each line as JSON
+3. Routes to appropriate handler
+4. Writes JSON response to `Deno.stdout`
+
+```typescript
+// src/maelstrom/node.ts
+const decoder = new TextDecoder();
+const encoder = new TextEncoder();
+let buffer = "";
+
+// Read stdin line by line
+const reader = Deno.stdin.readable.getReader();
+while (true) {
+  const { done, value } = await reader.read();
+  if (done) break;
+
+  buffer += decoder.decode(value);
+  const lines = buffer.split("\n");
+  buffer = lines.pop() || "";  // Keep incomplete line
+
+  for (const line of lines) {
+    if (!line.trim()) continue;
+    const msg = JSON.parse(line);
+    const response = await handleMessage(msg);
+    await Deno.stdout.write(encoder.encode(JSON.stringify(response) + "\n"));
+  }
+}
+```
+
+### Maelstrom-Specific Requirements
+
+1. **No network listening** — Maelstrom handles the network. Your node communicates ONLY via stdin/stdout.
+2. **All inter-node messages go through Maelstrom** — send a message to another node by writing to stdout:
+   ```json
+   {"src": "n0", "dest": "n1", "body": {"type": "request_vote", "msg_id": 10, ...}}
+   ```
+   Maelstrom delivers it to the target node's stdin.
+3. **msg_id tracking** — every message must have a unique `msg_id`. Responses include `in_reply_to`.
+4. **No persistent storage assumed** — Maelstrom may wipe process state between restarts (unless you use the `--no-fresh` flag). Use Deno KV with a temp directory.
+
+---
+
+## 3. Building lin-kv (Gate Test)
+
+The `lin-kv` workload tests a linearizable key-value store. This is your first milestone.
+
+### Required Operations
+
+#### read
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "read", "msg_id": 1, "key": 5}}
+```
+Response: `{"type": "read_ok", "in_reply_to": 1, "value": 42}` (or `null` if key doesn't exist)
+
+#### write
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "write", "msg_id": 2, "key": 5, "value": 42}}
+```
+Response: `{"type": "write_ok", "in_reply_to": 2}`
+
+#### cas (compare-and-swap)
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "cas", "msg_id": 3, "key": 5, "from": 42, "to": 99}}
+```
+Response (success): `{"type": "cas_ok", "in_reply_to": 3}`
+Response (failure): `{"type": "error", "in_reply_to": 3, "code": 20, "text": "expected 42 but had 21"}`
+
+### Implementation Strategy
+
+1. Wrap your Raft node: MaelstromNode → RaftNode → Deno KV
+2. Read/write/cas are state machine commands
+3. Wait for commit before responding (blocks the stdin loop — use async)
+4. Forward to leader if not leader (or redirect client via error)
+
+### Running lin-kv
+
+```bash
+java -jar maelstrom.jar test \
+  -w lin-kv \
+  --bin "deno run --allow-read --allow-write --allow-env --unstable-kv src/maelstrom/node.ts" \
+  --node-count 3 \
+  --time-limit 30 \
+  --rate 10 \
+  --concurrency 2n \
+  --nemesis partition \
+  --test-count 1
+```
+
+**Success criteria:** "Everything looks good! ヽ(´ー｀)ノ" with no anomalies.
+
+---
+
+## 4. Custom PubSub Workloads
+
+After `lin-kv` passes, build these custom Maelstrom workloads.
+
+### 4.1 Workload: pubsub-append
+
+**Goal:** Verify append-only pubsub with no gaps, no duplicates, in-order delivery.
+
+**Workload Definition (pubsub_append.clj/edn):**
+```clojure
+{:name "pubsub-append"
+ :net {:client {:class maelstrom.workload.pubsub_append/Client}
+       :server {:class maelstrom.workload.pubsub_append/Server}}
+ :ops {:publish {:type :publish, :topic "test-topic"}
+       :subscribe {:type :subscribe, :topic "test-topic"}
+       :poll {:type :poll, :subscriber "client-sub"}}
+ :client {:op-count 100
+          :topics ["topic-a" "topic-b" "topic-c"]
+          :subscriber-count 3}}
+```
+
+**Checker:** Verifies that for each subscriber, received messages form a contiguous prefix of all published messages for that topic. No holes, no out-of-order.
+
+### 4.2 Workload: pubsub-total
+
+**Goal:** Verify real-time ordering (linearizability).
+
+**Checker logic:**
+- Record timestamps of publish operations
+- If publish A returns (is acknowledged) before publish B starts, then A must appear before B in every subscriber stream
+- Jempson checker tracks the partial order of operations and verifies the observed order satisfies it
+
+### 4.3 Workload: pubsub-crash
+
+**Goal:** Verify durability and recovery.
+
+**Nemesis:** `--nemesis kill` — kills random nodes at random intervals.
+
+**Checker:** After recovery, all acknowledged publishes must be deliverable. Total message count across all subscribers matches total published (acknowledged) messages.
+
+### 4.4 Implementing Custom Workloads in Maelstrom
+
+Maelstrom workloads are Clojure files executed by the Jepsen test runner. Place them in:
+
+```
+maelstrom/workloads/pubsub_append.clj
+maelstrom/workloads/pubsub_total.clj
+maelstrom/workloads/pubsub_crash.clj
+```
+
+And register them with Maelstrom's workload registry. Alternatively, you can implement the client logic within your node and use a generic append workload, but custom workloads give better safety properties.
+
+---
+
+## 5. Node Implementation for PubSub Workloads
+
+Your node needs to handle these additional message types:
+
+### publish
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "publish", "msg_id": 10, "topic": "orders", "value": "hello"}}
+```
+Response: `{"type": "publish_ok", "in_reply_to": 10, "sequence": 42}`
+
+### subscribe
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "subscribe", "msg_id": 11, "topic": "orders", "subscriber_id": "sub-1"}}
+```
+Response: `{"type": "subscribe_ok", "in_reply_to": 11, "cursor": 0}`
+
+### poll (get next messages)
+```json
+{"src": "c1", "dest": "n0", "body": {"type": "poll", "msg_id": 12, "subscriber_id": "sub-1", "max": 10}}
+```
+Response: `{"type": "poll_ok", "in_reply_to": 12, "messages": [{"sequence": 1, "value": "hello"}, {"sequence": 2, "value": "world"}]}`
+
+---
+
+## 6. Fault Injection Patterns
+
+Maelstrom injects faults via nemeses:
+
+| Nemesis | What it does | What you test |
+|---------|-------------|---------------|
+| `partition` | Splits network into two groups | Election safety, no split-brain |
+| `kill` | Kills random processes | Crash recovery, log durability |
+| `pause` | Pauses processes (SIGSTOP) | Timeout handling, heartbeat resilience |
+| `clock-skew` | Skews system clock | Clock-independent correctness |
+
+### Test Matrix (what to run before release)
+
+```
+┌──────────────────┬───────────┬───────────┬───────────┬───────────┐
+│ Workload / Nemesis│ partition │   kill    │   pause   │clock-skew │
+├──────────────────┼───────────┼───────────┼───────────┼───────────┤
+│ lin-kv           │    ✓      │    ✓      │    ✓      │    ✓      │
+│ pubsub-append    │    ✓      │    ✓      │    ✓      │    ✓      │
+│ pubsub-total     │    ✓      │    ✓      │    ✓      │    ✓      │
+│ pubsub-crash     │    ✓      │    ✓      │    ✓      │    ✓      │
+└──────────────────┴───────────┴───────────┴───────────┴───────────┘
+```
+
+All 16 cells must pass (✓ = no anomalies).
+
+---
+
+## 7. Debugging Maelstrom Failures
+
+### Common Failure Modes
+
+| Symptom | Likely Cause | Fix |
+|---------|-------------|-----|
+| "lost messages" | Leader crashed before commit, entries not persisted | Ensure entries are fsynced to KV before AppendEntries response |
+| "stale reads" | Follower serving reads from stale state | Ensure follower only serves committed+applied reads |
+| "split-brain" | Election restriction not enforced | Verify voter checks `lastLogTerm >= myLastLogTerm` |
+| "non-linearizable" | Committing entries from previous terms | Use "commit from current term" rule |
+| "timeout" | Node stuck; RPC not processed | Check async I/O; verify stdin reader doesn't block |
+| "key not found" | State machine not deterministic | Check for `Date.now()` or `Math.random()` in apply() |
+
+### Maelstrom Output Interpretation
+
+```
+ :anomalies
+ {:G1 {:valid? true},              ; Read committed — PASS
+  :G2 {:valid? false,              ; Monotonic reads — FAIL
+       :anomaly-count 3
+       :anomalies [...]}}
+```
+
+- **G1 (Read Committed):** No dirty reads
+- **G2 (Monotonic Reads):** Read values don't go backward in time
+- **G-single (Linearizability):** Full linearizability for single-key ops
+- **internal (Internal Consistency):** State machine consistency
+
+---
+
+## 8. Test Runner Script
+
+`maelstrom/run_tests.sh`:
+
+```bash
+#!/bin/bash
+set -euo pipefail
+
+MAELSTROM_JAR="./maelstrom.jar"
+DENO_BIN="deno run --allow-read --allow-write --allow-env --unstable-kv"
+NODE_SCRIPT="src/maelstrom/node.ts"
+
+echo "=== Phase 1: lin-kv gate ==="
+java -jar "$MAELSTROM_JAR" test \
+  -w lin-kv \
+  --bin "$DENO_BIN $NODE_SCRIPT" \
+  --node-count 3 \
+  --time-limit 30 \
+  --rate 10 \
+  --nemesis partition
+
+echo "=== Phase 2: pubsub-append ==="
+java -jar "$MAELSTROM_JAR" test \
+  -w pubsub-append \
+  --bin "$DENO_BIN $NODE_SCRIPT" \
+  --node-count 3 \
+  --time-limit 60 \
+  --rate 20 \
+  --nemesis partition
+
+echo "=== Phase 3: pubsub-total ==="
+java -jar "$MAELSTROM_JAR" test \
+  -w pubsub-total \
+  --bin "$DENO_BIN $NODE_SCRIPT" \
+  --node-count 3 \
+  --time-limit 60 \
+  --rate 20
+
+echo "=== Phase 4: pubsub-crash ==="
+java -jar "$MAELSTROM_JAR" test \
+  -w pubsub-crash \
+  --bin "$DENO_BIN $NODE_SCRIPT" \
+  --node-count 5 \
+  --time-limit 120 \
+  --rate 50 \
+  --nemesis kill
+
+echo "=== All tests passed ==="
+```
+
+---
+
+## 9. Running Without Maelstrom (Local Multi-Node)
+
+For development, run nodes directly with explicit configuration:
+
+```bash
+# Terminal 1
+deno run --allow-all --unstable-kv src/server.ts \
+  --node-id node-1 --port 8081 --kv-path ./data/node-1 \
+  --peers node-2:8082,node-3:8083
+
+# Terminal 2
+deno run --allow-all --unstable-kv src/server.ts \
+  --node-id node-2 --port 8082 --kv-path ./data/node-2 \
+  --peers node-1:8081,node-3:8083
+
+# Terminal 3
+deno run --allow-all --unstable-kv src/server.ts \
+  --node-id node-3 --port 8083 --kv-path ./data/node-3 \
+  --peers node-1:8081,node-2:8082
+```
+
+Then test with curl:
+```bash
+curl -X POST http://localhost:8081/topics/test-topic
+curl -X POST http://localhost:8081/topics/test-topic/messages \
+  -H "Content-Type: application/json" \
+  -d '{"id": "msg-1", "payload": "hello", "timestamp": 1714000000000}'
+```