loki-mode 5.51.0 → 5.52.1

package/docs/certification/04-production-deployment/quiz.md

@@ -0,0 +1,93 @@
+# Module 4 Quiz: Production Deployment
+
+Answer each question by selecting the best option (A, B, C, or D).
+
+---
+
+**Question 1:** What base image does the Loki Mode Dockerfile use?
+
+A) Alpine Linux 3.19
+B) Node.js 20 official image
+C) Ubuntu 24.04
+D) Debian Bookworm
+
+---
+
+**Question 2:** Which volume mount in docker-compose.yml gives the container read-write access?
+
+A) `~/.gitconfig:/home/loki/.gitconfig`
+B) `.:/workspace:rw`
+C) `~/.ssh:/home/loki/.ssh`
+D) `~/.config/gh:/home/loki/.config/gh`
+
+---
+
+**Question 3:** What does `LOKI_STAGED_AUTONOMY=true` do?
+
+A) Enables parallel agent execution in stages
+B) Requires human approval before execution
+C) Stages deployment across multiple environments
+D) Enables incremental feature rollout
+
+---
+
+**Question 4:** What is the default maximum number of parallel agents?
+
+A) 3
+B) 5
+C) 10
+D) 20
+
+---
+
+**Question 5:** How do you set a cost budget limit for a Loki Mode session?
+
+A) `loki start --max-cost 10`
+B) `loki start --budget 10.00 ./prd.md`
+C) `LOKI_COST_LIMIT=10 loki start`
+D) `loki config set budget 10.00`
+
+---
+
+**Question 6:** What does the completion council do?
+
+A) Reviews all code changes before they are committed
+B) Votes on whether the project is truly complete to prevent premature termination
+C) Manages the deployment pipeline approval process
+D) Assigns tasks to available agents
+
+---
+
+**Question 7:** What is the default dashboard port?
+
+A) 3000
+B) 8080
+C) 57374
+D) 9090
+
+---
+
+**Question 8:** Which environment variables enable TLS for the dashboard?
+
+A) `LOKI_HTTPS=true` and `LOKI_HTTPS_PORT=443`
+B) `LOKI_TLS_CERT` and `LOKI_TLS_KEY`
+C) `LOKI_SSL_CERT` and `LOKI_SSL_KEY`
+D) `LOKI_DASHBOARD_TLS=true`
+
+---
+
+**Question 9:** What does `LOKI_COUNCIL_STAGNATION_LIMIT=5` mean?
+
+A) The council can only reject completion 5 times
+B) After 5 iterations with no git changes, stagnation is flagged
+C) The council checks every 5 minutes
+D) Maximum 5 council members can vote
+
+---
+
+**Question 10:** How do you restrict which directories agents can modify?
+
+A) `LOKI_READ_ONLY_PATHS=/etc,/usr`
+B) `LOKI_ALLOWED_PATHS=/workspace/src,/workspace/tests`
+C) `LOKI_SANDBOX_PATHS=/safe/dir`
+D) `LOKI_WRITE_DIRS=src,tests`

package/docs/certification/05-troubleshooting/lab.md

@@ -0,0 +1,254 @@
+# Module 5 Lab: Diagnose and Troubleshoot
+
+## Objective
+
+Practice inspecting Loki Mode state files, interpreting circuit breaker status, examining the dead-letter queue, and using recovery procedures.
+
+## Prerequisites
+
+- Loki Mode installed (`npm install -g loki-mode`)
+- `jq` installed for JSON inspection
+- Familiarity with the `.loki/` directory structure (Module 1)
+
+## Step 1: Create a Simulated `.loki/` State
+
+Create a mock `.loki/` directory with sample state files to practice inspection:
+
+```bash
+mkdir -p /tmp/troubleshoot-lab && cd /tmp/troubleshoot-lab
+git init
+
+# Create the .loki directory structure
+mkdir -p .loki/{state,queue,signals,memory/episodic,memory/semantic,logs}
+```
+
+Create a sample orchestrator state:
+
+```bash
+cat > .loki/state/orchestrator.json << 'EOF'
+{
+  "currentPhase": "DEVELOPMENT",
+  "tasksCompleted": 12,
+  "tasksFailed": 3,
+  "totalTasks": 20,
+  "startedAt": "2026-02-20T10:00:00Z",
+  "lastUpdated": "2026-02-20T14:30:00Z"
+}
+EOF
+```
+
+Create a sample circuit breaker file:
+
+```bash
+cat > .loki/state/circuit-breakers.json << 'EOF'
+{
+  "api/claude": {
+    "state": "CLOSED",
+    "failure_count": 0,
+    "success_count": 0,
+    "last_failure_time": null,
+    "last_state_change": "2026-02-20T10:00:00Z",
+    "cooldown_until": null,
+    "failure_window_start": null
+  },
+  "api/openai": {
+    "state": "OPEN",
+    "failure_count": 3,
+    "success_count": 0,
+    "last_failure_time": "2026-02-20T14:25:42Z",
+    "last_state_change": "2026-02-20T14:25:42Z",
+    "cooldown_until": "2026-02-20T14:30:42Z",
+    "failure_window_start": "2026-02-20T14:24:50Z"
+  },
+  "api/gemini": {
+    "state": "HALF_OPEN",
+    "failure_count": 0,
+    "success_count": 1,
+    "last_failure_time": "2026-02-20T14:10:00Z",
+    "last_state_change": "2026-02-20T14:15:00Z",
+    "cooldown_until": null,
+    "failure_window_start": null
+  }
+}
+EOF
+```
+
+Create a sample dead-letter queue:
+
+```bash
+cat > .loki/queue/dead-letter.json << 'EOF'
+{
+  "tasks": [
+    {
+      "task_id": "task-007",
+      "original_queue": "in-progress",
+      "failure_count": 5,
+      "first_failure": "2026-02-20T11:00:00Z",
+      "last_failure": "2026-02-20T14:00:00Z",
+      "error_summary": "Database migration script fails on foreign key constraint",
+      "attempts": [
+        {
+          "attempt_number": 1,
+          "timestamp": "2026-02-20T11:00:00Z",
+          "approach": "Direct ALTER TABLE with constraint",
+          "error_type": "validation",
+          "error_message": "ERROR: cannot add foreign key constraint - referenced table not yet created",
+          "agent_id": "eng-database-001"
+        },
+        {
+          "attempt_number": 5,
+          "timestamp": "2026-02-20T14:00:00Z",
+          "approach": "Deferred constraint with migration ordering",
+          "error_type": "validation",
+          "error_message": "ERROR: circular dependency between users and organizations tables",
+          "agent_id": "eng-database-001"
+        }
+      ],
+      "recovery_strategy": "requires_human_review",
+      "task_data": {
+        "title": "Create database migration for user-organization relationship",
+        "description": "Add foreign keys between users and organizations tables",
+        "dependencies": ["task-005"],
+        "priority": "high"
+      }
+    }
+  ],
+  "metadata": {
+    "last_reviewed": "2026-02-20T08:00:00Z",
+    "total_abandoned": 0,
+    "total_recovered": 2
+  }
+}
+EOF
+```
+
+## Step 2: Inspect Circuit Breaker State
+
+Practice reading circuit breaker state:
+
+```bash
+# List all circuit breaker states
+cat .loki/state/circuit-breakers.json | jq 'to_entries[] | {api: .key, state: .value.state}'
+
+# Find which APIs are in OPEN state
+cat .loki/state/circuit-breakers.json | jq 'to_entries[] | select(.value.state == "OPEN") | .key'
+
+# Check the cooldown time for the OPEN circuit
+cat .loki/state/circuit-breakers.json | jq '.["api/openai"].cooldown_until'
+
+# Check how many successes the HALF_OPEN circuit needs
+cat .loki/state/circuit-breakers.json | jq '.["api/gemini"].success_count'
+```
+
+**Questions to answer:**
+1. Which API is currently blocked (OPEN)?
+2. When does the cooldown expire?
+3. How many more successes does the HALF_OPEN circuit need to return to CLOSED? (Answer: 2 more, needs 3 total)
+
+## Step 3: Analyze the Dead-Letter Queue
+
+```bash
+# Count tasks in dead-letter
+cat .loki/queue/dead-letter.json | jq '.tasks | length'
+
+# View the error summary
+cat .loki/queue/dead-letter.json | jq '.tasks[0].error_summary'
+
+# View all attempts for the first task
+cat .loki/queue/dead-letter.json | jq '.tasks[0].attempts'
+
+# Check the recovery strategy
+cat .loki/queue/dead-letter.json | jq '.tasks[0].recovery_strategy'
+
+# Check when the queue was last reviewed
+cat .loki/queue/dead-letter.json | jq '.metadata.last_reviewed'
+```
+
+**Questions to answer:**
+1. What is the root cause of the failure?
+2. What recovery strategy is assigned?
+3. Is the `last_reviewed` timestamp more than 24 hours old? (If so, the queue should be processed before new work.)
+
+## Step 4: Simulate Signal Files
+
+Create signal files and understand their purpose:
+
+```bash
+# Simulate a PAUSE signal
+touch .loki/signals/PAUSE
+ls .loki/signals/
+# In a real session, the agent would stop after the current iteration
+
+# Remove PAUSE
+rm .loki/signals/PAUSE
+
+# Simulate a DRIFT_DETECTED signal
+cat >> .loki/signals/DRIFT_DETECTED << 'EOF'
+{"timestamp":"2026-02-20T14:30:00Z","task_id":"task-012","severity":"medium","detected_drift":"Started optimizing CSS instead of implementing API endpoint"}
+{"timestamp":"2026-02-20T14:35:00Z","task_id":"task-012","severity":"medium","detected_drift":"Switched to refactoring tests instead of implementing API endpoint"}
+EOF
+
+# Read the drift log
+cat .loki/signals/DRIFT_DETECTED | jq -s '.'
+
+# Simulate HUMAN_REVIEW_NEEDED
+cat > .loki/signals/HUMAN_REVIEW_NEEDED << 'EOF'
+{
+  "timestamp": "2026-02-20T14:40:00Z",
+  "reason": "security_decision",
+  "task_id": "task-015",
+  "context": "Requires AWS production credentials for deployment",
+  "severity": "critical",
+  "blocking": true
+}
+EOF
+
+cat .loki/signals/HUMAN_REVIEW_NEEDED | jq .
+```
+
+## Step 5: Practice Recovery Commands
+
+Use the Loki Mode CLI recovery commands:
+
+```bash
+# Check current status
+loki status
+
+# Reset commands (these work on actual .loki/ state):
+# loki reset retries    -- Reset retry counters
+# loki reset failed     -- Reset failed task status
+# loki reset all        -- Reset all session state
+
+# View logs
+loki logs
+```
+
+## Step 6: Inspect Orchestrator State
+
+```bash
+# View current phase
+cat .loki/state/orchestrator.json | jq '.currentPhase'
+
+# Calculate progress
+cat .loki/state/orchestrator.json | jq '{
+  phase: .currentPhase,
+  progress: "\(.tasksCompleted)/\(.totalTasks)",
+  failed: .tasksFailed
+}'
+```
+
+## Verification Checklist
+
+- [ ] You can read and interpret circuit breaker states (CLOSED, OPEN, HALF_OPEN)
+- [ ] You can calculate when an OPEN circuit will transition to HALF_OPEN
+- [ ] You can inspect dead-letter queue tasks and identify recovery strategies
+- [ ] You understand the drift detection signal and its accumulation thresholds
+- [ ] You know which signal files exist and what they trigger
+- [ ] You can use `loki status`, `loki logs`, and `loki reset` for recovery
+
+## Cleanup
+
+```bash
+cd ~
+rm -rf /tmp/troubleshoot-lab
+```

package/docs/certification/05-troubleshooting/lesson.md

@@ -0,0 +1,266 @@
+# Module 5: Troubleshooting
+
+## Overview
+
+This module covers diagnosing and resolving common issues in Loki Mode: gate failures, session conflicts, circuit breakers, dead-letter queue processing, signal handling, and recovery procedures. The primary reference is `skills/troubleshooting.md`.
+
+## Common Issues
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| Agent stuck / no progress | Lost context | Read `.loki/CONTINUITY.md` at session start |
+| Task repeating | Not checking queue state | Check `.loki/queue/*.json` before claiming |
+| Code review failing | Skipped static analysis | Run static analysis BEFORE AI reviewers |
+| Tests failing after merge | Skipped quality gates | Never bypass severity-based blocking |
+| Rate limit hit | Too many parallel agents | Check circuit breakers, use exponential backoff |
+| Cannot find what to do | Not following RARV cycle | Check `orchestrator.json`, follow decision tree |
+
+## Quality Gate Failures
+
+When a quality gate fails, identify which gate triggered the failure:
+
+**Gates 1-6 (Review gates):**
+- Check the review output for severity levels
+- Critical/High/Medium = BLOCK (must fix)
+- Low/Cosmetic = TODO (informational)
+- If all 3 reviewers pass unanimously, Gate 4 runs Devil's Advocate
+
+**Gate 7 (Test coverage):**
+- Unit tests must have 100% pass rate and >80% coverage
+- Integration tests must have 100% pass rate
+- Fix failing tests before proceeding (never delete or skip tests)
+
+**Gate 8 (Mock detector):**
+- Runs `tests/detect-mock-problems.sh`
+- Flags tests that mock internal modules instead of using real code
+- Flags tautological assertions and high internal mock ratios
+- Disable with `LOKI_GATE_MOCK_DETECTOR=false` (not recommended)
+
+**Gate 9 (Test mutation detector):**
+- Runs `tests/detect-test-mutations.sh`
+- Detects assertion values changed alongside implementation (test fitting)
+- Detects low assertion density and missing pass/fail tracking
+- Disable with `LOKI_GATE_MUTATION_DETECTOR=false` (not recommended)
+
+## Circuit Breaker System
+
+The circuit breaker prevents cascading failures when API providers are unavailable. State is tracked in `.loki/state/circuit-breakers.json`.
+
+### States
+
+| State | Behavior | Transitions |
+|-------|----------|-------------|
+| **CLOSED** | Normal operation, all requests pass | -> OPEN after 3 failures in 60s |
+| **OPEN** | All requests blocked | -> HALF_OPEN after 300s cooldown |
+| **HALF_OPEN** | Limited probe requests | -> CLOSED after 3 successes; -> OPEN on any failure |
+
+### Inspecting Circuit Breaker State
+
+```bash
+cat .loki/state/circuit-breakers.json | jq .
+```
+
+Example output:
+
+```json
+{
+  "api/claude": {
+    "state": "CLOSED",
+    "failure_count": 0,
+    "last_failure_time": null,
+    "cooldown_until": null
+  },
+  "api/openai": {
+    "state": "OPEN",
+    "failure_count": 3,
+    "last_failure_time": "2025-01-20T10:35:42Z",
+    "cooldown_until": "2025-01-20T10:40:42Z"
+  }
+}
+```
+
+### Recovery Protocol
+
+When a circuit breaker is OPEN:
+1. Check the `cooldown_until` timestamp
+2. Reduce parallel agent count (e.g., from 10 to 2)
+3. Disable non-critical background operations
+4. Wait for HALF_OPEN state
+5. Monitor probe request results
+6. After CLOSED state is restored, gradually increase parallelism
+
+## Dead-Letter Queue
+
+Tasks that fail 5+ times are moved to `.loki/queue/dead-letter.json`. This prevents infinite retry loops.
+
+### Inspecting the Dead-Letter Queue
+
+```bash
+cat .loki/queue/dead-letter.json | jq '.tasks | length'    # Count failed tasks
+cat .loki/queue/dead-letter.json | jq '.tasks[0]'          # View first failed task
+```
+
+### Recovery Strategies
+
+| Strategy | When to Use |
+|----------|-------------|
+| `retry_with_simpler_approach` | Complex implementation failed multiple times |
+| `dependency_blocked` | Task needs output from another failed task |
+| `requires_human_review` | Security decision, unclear spec, or irreversible action |
+| `permanent_abandon` | 10+ attempts, or same error across 3 different approaches |
+
+### Retry Conditions
+
+A dead-letter task can be retried when:
+- A dependency that was blocking it is now available
+- A new approach has been identified
+- A simpler scope has been defined
+- A blocking bug has been fixed
+
+### Permanent Abandon Criteria
+
+Move to `.loki/queue/abandoned.json` when:
+- 10+ total attempts across all strategies
+- Same error with 3 different approaches
+- Dependency will never be available
+- Scope is no longer relevant
+
+## Signal Processing
+
+Signals in `.loki/signals/` are inter-process communication files. Key signals:
+
+### PAUSE and STOP
+
+```bash
+# Pause after current iteration
+touch .loki/PAUSE
+
+# Stop immediately
+touch .loki/STOP
+```
+
+Or use CLI commands:
+
+```bash
+loki pause
+loki stop
+```
+
+### DRIFT_DETECTED
+
+Recorded when an agent's actions diverge from the task goal. The file is append-only (JSON lines format).
+
+```json
+{
+  "timestamp": "2026-01-25T10:30:00Z",
+  "task_id": "task-042",
+  "severity": "medium",
+  "detected_drift": "Started refactoring database schema instead of implementing auth"
+}
+```
+
+Processing rules:
+- 1 drift: Log warning, continue with correction
+- 2 drifts on same task: Escalate to orchestrator
+- 3+ accumulated drifts: Trigger context clear and full state reload
+
+### CONTEXT_CLEAR_REQUESTED
+
+Created when the context window becomes heavy. Can be triggered by:
+- Agent self-assessment ("context feels heavy")
+- After 25+ iterations
+- 3+ accumulated DRIFT_DETECTED events
+- Same error occurring 3+ times
+
+The wrapper (`run.sh`) handles this by starting a fresh session with injected state from `.loki/CONTINUITY.md`.
+
+### HUMAN_REVIEW_NEEDED
+
+Created when autonomous action is inappropriate:
+- Confidence below 0.40 on a critical decision
+- Security-critical operations
+- Irreversible operations without rollback
+- 3+ consecutive failures on the same task
+
+The task is blocked until a human provides input.
+
+## Rationalization Detection
+
+Agents can rationalize failures to avoid acknowledging mistakes. Common patterns to watch for:
+
+| Rationalization | Required Action |
+|-----------------|-----------------|
+| "I'll refactor later" | Refactor now or reduce scope |
+| "This is just an edge case" | Handle the edge case |
+| "The tests are flaky" | Fix the flaky test first |
+| "It works on my machine" | Must pass in CI |
+| "This is good enough" | Run full test suite before claiming completion |
+
+**Red flag language patterns:**
+- Hedging: "probably", "should be fine", "most likely"
+- Minimization: "just a small change", "simple fix", "minor update"
+- Verification skipping: Moving to next task without running tests
+
+When rationalization is detected: stop, identify the specific rationalization, apply the required action, and log the attempt to `.loki/memory/episodic/`.
+
+## Recovery Procedures
+
+### Context Loss Recovery
+
+1. Read `.loki/CONTINUITY.md` for current state
+2. Check `.loki/state/orchestrator.json` for current phase
+3. Review `.loki/queue/in-progress.json` for interrupted tasks
+4. Resume from last checkpoint
+
+### Rate Limit Recovery
+
+1. Check circuit breaker state in `.loki/state/circuit-breakers.json`
+2. Wait for cooldown period to expire
+3. Reduce `LOKI_MAX_PARALLEL_AGENTS`
+4. Resume with exponential backoff (base: 5s, max: 300s, multiplier: 2)
+
+### Test Failure Recovery
+
+1. Read test output carefully
+2. Determine if the test is flaky or a real failure
+3. Roll back to last passing commit if needed (`loki checkpoint` can help)
+4. Fix the code (never the test) and re-run the full suite
+
+### Session Reset
+
+If the session state becomes corrupted:
+
+```bash
+loki reset all        # Reset all session state
+loki reset retries    # Reset retry counters only
+loki reset failed     # Reset failed task status only
+```
+
+## Debugging Tools
+
+### Logs
+
+```bash
+loki logs             # Show recent log output
+loki status           # Show current session status
+loki status --json    # Machine-readable status
+```
+
+### Audit Trail
+
+```bash
+loki audit            # View recent agent actions
+```
+
+### State Inspection
+
+```bash
+cat .loki/state/orchestrator.json | jq .    # Current phase and progress
+cat .loki/queue/pending.json | jq .         # Pending tasks
+cat .loki/queue/dead-letter.json | jq .     # Failed tasks
+cat .loki/state/circuit-breakers.json | jq . # API health
+```
+
+## Summary
+
+Troubleshooting Loki Mode involves inspecting the `.loki/` directory state: orchestrator phase, task queues, circuit breakers, signals, and memory. The circuit breaker system prevents cascading API failures. The dead-letter queue captures persistently failing tasks. Signals coordinate between processes. Rationalization detection helps identify when agents are avoiding real problems. Recovery procedures exist for context loss, rate limits, test failures, and corrupted state.

package/docs/certification/05-troubleshooting/quiz.md

@@ -0,0 +1,93 @@
+# Module 5 Quiz: Troubleshooting
+
+Answer each question by selecting the best option (A, B, C, or D).
+
+---
+
+**Question 1:** What are the three states of the circuit breaker system?
+
+A) Active, Inactive, Standby
+B) Closed, Open, Half-Open
+C) Running, Paused, Stopped
+D) Green, Yellow, Red
+
+---
+
+**Question 2:** How many failures within 60 seconds trigger a circuit breaker to OPEN?
+
+A) 1
+B) 2
+C) 3
+D) 5
+
+---
+
+**Question 3:** What is the default cooldown period when a circuit breaker is in the OPEN state?
+
+A) 30 seconds
+B) 60 seconds
+C) 300 seconds (5 minutes)
+D) 600 seconds (10 minutes)
+
+---
+
+**Question 4:** After how many failures is a task moved to the dead-letter queue?
+
+A) 3
+B) 5
+C) 7
+D) 10
+
+---
+
+**Question 5:** What happens when 3 or more DRIFT_DETECTED signals accumulate?
+
+A) The session terminates immediately
+B) A context clear is triggered and state is reloaded from scratch
+C) The task is moved to the dead-letter queue
+D) All agents are stopped and restarted
+
+---
+
+**Question 6:** Which file should an agent read first when recovering from context loss?
+
+A) `.loki/queue/pending.json`
+B) `.loki/CONTINUITY.md`
+C) `.loki/session.json`
+D) `.loki/memory/index.json`
+
+---
+
+**Question 7:** What does the `loki reset retries` command do?
+
+A) Deletes all tasks from the queue
+B) Restarts the AI provider CLI
+C) Resets retry counters only
+D) Removes the entire `.loki/` directory
+
+---
+
+**Question 8:** Which environment variable disables Gate 8 (Mock Detector)?
+
+A) `LOKI_SKIP_MOCK_CHECK=true`
+B) `LOKI_GATE_MOCK_DETECTOR=false`
+C) `LOKI_DISABLE_GATE_8=true`
+D) `LOKI_NO_MOCK_DETECTION=true`
+
+---
+
+**Question 9:** When should a dead-letter task be permanently abandoned?
+
+A) After 3 failed attempts
+B) After 5 failed attempts
+C) After 10+ total attempts, or same error with 3 different approaches
+D) Only when manually deleted by the user
+
+---
+
+**Question 10:** What is a red flag indication that an agent is rationalizing a failure?
+
+A) The agent requests a model upgrade
+B) The agent uses language like "probably", "should be fine", or "just a small change"
+C) The agent creates a new branch for the fix
+D) The agent runs additional tests