@seanyao/roll 4.630.2 → 4.702.2

package/guide/en/practices/engineering-common-sense.md

@@ -0,0 +1,395 @@
+# Roll Engineering Common Sense Checklist
+
+> **These are not best practices — they are baseline requirements.** Violations are Bugs.
+
+## 1. Idempotency 🔁
+
+**Definition:** Executing the same operation N times produces the same result as executing it once.
+
+**Must test:**
+```typescript
+it('should be idempotent', async () => {
+  await operation(data)  // 1st
+  const result1 = await getState()
+  
+  await operation(data)  // 2nd
+  const result2 = await getState()
+  
+  await operation(data)  // 3rd
+  const result3 = await getState()
+  
+  expect(result1).toEqual(result2)
+  expect(result2).toEqual(result3)
+})
+```
+
+**Common scenarios:**
+- [ ] Import/ingest operations
+- [ ] Configuration updates
+- [ ] State changes
+- [ ] API calls
+- [ ] File writes
+
+**Anti-pattern (this time):** Running ingest repeatedly -> files duplicated 7 times
+
+---
+
+## 2. Cross-Module Contract Consistency 🔗
+
+**Definition:** Data/IDs/formats shared across multiple modules must be exactly consistent.
+
+**Must check:**
+```typescript
+// Checklist
+- [ ] Is the ID generation algorithm consistent?
+- [ ] Is the data serialization format consistent?
+- [ ] Is path handling consistent? (e.g., / vs -)
+- [ ] Has it been extracted into a shared function/constant?
+```
+
+**Test template:**
+```typescript
+it('should generate same ID across modules', () => {
+  const scannerId = generateScannerId('articles/test.md')
+  const inboxId = generateInboxId('articles/test.md')
+  expect(scannerId).toEqual(inboxId)
+})
+```
+
+**Anti-pattern (this time):** Scanner used `-` to replace `/`, inbox used raw path -> deduplication failed
+
+---
+
+## 3. Data Flow Integrity 🌊
+
+**Definition:** The complete pipeline from producer to consumer must be connected end-to-end.
+
+**Must verify:**
+```typescript
+// Integration test - must exist
+describe('Data Flow: Producer -> Consumer', () => {
+  it('should write data that consumer can read', async () => {
+    await producer.write(testData)
+    const result = await consumer.read()
+    expect(result).toEqual(testData)
+  })
+})
+```
+
+**Checklist:**
+- [ ] Who writes the data? (Producer)
+- [ ] Who reads the data? (Consumer)
+- [ ] What is the intermediate storage? (state/file/cache)
+- [ ] Is there an integration test to verify?
+
+**Anti-pattern (this time):** Ingest didn't write state, status couldn't read it -> showed 0
+
+---
+
+## 4. Atomicity ⚛️
+
+**Definition:** An operation either fully succeeds or does not execute at all (no intermediate state).
+
+**Must consider:**
+- [ ] How to roll back on partial failure?
+- [ ] Is there a transaction mechanism?
+- [ ] How is data consistency guaranteed after a crash?
+
+**Test template:**
+```typescript
+it('should be atomic', async () => {
+  try {
+    await operation([item1, item2, INVALID_ITEM, item4])
+  } catch (e) {
+    // After failure, processed items should be rolled back
+    const state = await getState()
+    expect(state).toEqual(initialState)
+  }
+})
+```
+
+---
+
+## 5. Input Validation 🛡️
+
+**Definition:** Never trust any external input — it must be validated.
+
+**Must check:**
+- [ ] Null/undefined handling
+- [ ] Type checking
+- [ ] Range checking (array length, numeric range)
+- [ ] Special character/injection attack protection
+- [ ] File path traversal protection
+
+**Test template:**
+```typescript
+it('should handle invalid inputs gracefully', async () => {
+  await expect(operation(null)).rejects.toThrow()
+  await expect(operation('')).rejects.toThrow()
+  await expect(operation({})).rejects.toThrow()
+})
+```
+
+---
+
+## 6. Graceful Degradation 🪂
+
+**Definition:** When a dependency fails, the system should still provide limited functionality.
+
+**Must consider:**
+- [ ] What if an external API fails?
+- [ ] What if the database connection drops?
+- [ ] Is there a fallback mechanism?
+- [ ] What feedback does the user get?
+
+**Test template:**
+```typescript
+it('should degrade gracefully when dependency fails', async () => {
+  mockDependency.toThrow('Network error')
+  
+  // Should not crash
+  const result = await operation()
+  
+  // Should return fallback value or partial result
+  expect(result).toEqual(fallbackValue)
+})
+```
+
+---
+
+## 7. Observability 👁️
+
+**Definition:** System state must be visible and traceable.
+
+**Must provide:**
+- [ ] Progress feedback (for long-running operations)
+- [ ] Status query interface (e.g., status command)
+- [ ] Error logs (failure reasons)
+- [ ] Key metrics (counts, durations)
+
+**Improvements this time:**
+- Added `kkb status` showing raw files statistics ✅
+- Added `kkb compile` progress feedback ✅
+
+---
+
+## 8. Concurrency Safety 🧵
+
+**Definition:** Shared resource access across multiple threads/processes must be safe.
+
+**Must consider:**
+- [ ] File read/write conflicts
+- [ ] Database transaction isolation levels
+- [ ] Locking for shared in-memory state
+- [ ] Race conditions
+
+**Test template:**
+```typescript
+it('should handle concurrent writes', async () => {
+  await Promise.all([
+    operation(data1),
+    operation(data2),
+    operation(data3)
+  ])
+  
+  // Verify final state consistency
+  const state = await getState()
+  expect(state).toBeValid()
+})
+```
+
+---
+
+## Mandatory Check Process
+
+At the **Test Design Review** phase of each Story, the following must be answered:
+
+```markdown
+### Engineering Common Sense Checklist
+- [ ] **Idempotency**: Can it be run repeatedly? Are there tests?
+- [ ] **Cross-Module Contract**: Are IDs/formats/algorithms consistent?
+- [ ] **Data Flow**: Is the producer -> consumer pipeline complete?
+- [ ] **Atomicity**: Will partial failures roll back?
+- [ ] **Input Validation**: Are all inputs validated?
+- [ ] **Graceful Degradation**: What happens when a dependency fails?
+- [ ] **Observability**: Can the user see progress/status?
+- [ ] **Concurrency Safety**: Is multi-threaded access safe?
+
+**If any item is not met, tests/design must be added before writing implementation code.**
+```
+
+---
+
+## 9. Shell Script Performance 🐚
+
+**Definition:** `$()` command substitution forks a subshell. In hot paths (functions called per-test, per-catalog-entry, or per-message-lookup) this is the dominant cost.
+
+**Rule of thumb:**
+- 1 subshell fork ≈ 2–3 ms on Linux
+- 1000 forks in a test suite setup = +2–3 s per test run
+- 1739 tests × 2.3 s = **~67 minutes of wasted CI time** (observed in this repo)
+
+**Must avoid in hot paths:**
+```bash
+# ❌ Subshell fork — slow when called thousands of times
+upper="$(echo "$lang" | tr '[:lower:]' '[:upper:]')"
+upper="$(_some_helper_function "$lang")"
+safe="$(_sanitize_key "$key")"
+
+# ✅ Inline equivalents — zero fork
+case "$lang" in
+  en|EN) upper=EN ;;
+  zh|ZH) upper=ZH ;;
+  *)     upper="$(printf '%s' "$lang" | tr '[:lower:]' '[:upper:]')" ;;  # rare path OK
+esac
+safe="${key//[^A-Za-z0-9_]/_}"   # parameter expansion, no fork
+printf -v "$varname" '%s' "$val"  # printf -v instead of subshell assignment
+```
+
+**Compatibility trap — the "fixed twice" anti-pattern:**
+```
+PR #211: fixed $(echo | tr) → ${lang^^}      ✅ fast, but bash 4+ only
+PR #213: fixed bash 3.2 compat → $(_helper)  ❌ reintroduced the subshell
+PR #218: fixed both → inline case             ✅ fast + bash 3.2 safe
+```
+When fixing for compatibility, verify the fix does not reintroduce the original problem.
+Add a timing assertion or benchmark test to guard the regression.
+
+**Dead code from inlining:**
+When inlining a helper function's logic, remove the original function and grep for all callers.
+A leftover function with zero callers misleads future contributors into calling it via `$()`
+and reintroducing the fork.
+
+**Must check:**
+- [ ] Is this function called inside a loop, setup(), or per-entry catalog loading?
+- [ ] Does it use `$()` anywhere? Can it be replaced with `case`, `${var//...}`, or `printf -v`?
+- [ ] After inlining, is the original helper now dead code? Remove it.
+- [ ] Is there a timing test or CI cap to catch regressions? (see `ROLL_TEST_TIME_CAP`)
+
+---
+
+## 10. Shell Resource Cleanup 🧹
+
+**Definition:** Every `trap` set must be explicitly reset. Temporary files and lock files must never outlive their owning process.
+
+**trap - EXIT pattern:**
+```bash
+# ❌ Dangling trap — EXIT handler persists in the calling shell after return
+local tmp; tmp=$(mktemp)
+trap "rm -f '$tmp'" EXIT
+# ... do work ...
+mv "$tmp" "$dst"
+return 0   # EXIT trap still armed — any later exit fires rm on a stale path
+
+# ✅ Always reset after use
+local tmp; tmp=$(mktemp)
+trap "rm -f '$tmp'" EXIT
+# ... do work ...
+mv "$tmp" "$dst" 2>/dev/null || rm -f "$tmp"
+trap - EXIT   # disarm before return
+return 0
+```
+
+**Directory scope in cleanup helpers:**
+```bash
+# ❌ Wrong — scans the env-var default, not the actual path in use
+_cleanup_tmp() {
+  local dir; dir=$(dirname "${MY_PATH:-$HOME/.default/path}")
+  ...
+}
+_do_work() {
+  local custom_path="$1"   # may differ from MY_PATH
+  _cleanup_tmp             # scans wrong directory!
+}
+
+# ✅ Pass the actual directory explicitly
+_cleanup_tmp() {
+  local dir="${1:-$(dirname "${MY_PATH:-$HOME/.default/path}")}"
+  ...
+}
+_do_work() {
+  local custom_path="$1"
+  _cleanup_tmp "$(dirname "$custom_path")"
+}
+```
+
+**Must check:**
+- [ ] Every `trap "…" EXIT` has a matching `trap - EXIT` before the function returns.
+- [ ] Cleanup helpers accept the target directory as an argument — no implicit env-var path.
+- [ ] Temporary file names are quoted in trap strings: `"rm -f '$tmp'"` not `"rm -f $tmp"`.
+
+---
+
+## 11. Test Reliability 🧪
+
+**Definition:** Tests must not rely on environmental assumptions that can silently fail on different hosts.
+
+**PID assumptions:**
+```bash
+# ❌ Assumes pid_max ≤ 99999 — fails silently in containers with large pid_max
+local stale_tmp="runs.jsonl.tmp.99999"
+
+# ✅ Use a process that is provably dead
+bash -c 'exit 0' &
+local dead_pid=$!
+wait "$dead_pid" 2>/dev/null || true
+local stale_tmp="runs.jsonl.tmp.${dead_pid}"
+```
+
+**Heredoc scope:**
+Functions defined outside a heredoc are not available inside it when the generated
+script executes. Calling them silently no-ops (if guarded with `|| true`) or crashes.
+```bash
+# ❌ _some_helper is not defined in the generated script's scope
+cat > script.sh <<'EOF'
+_some_helper 2>/dev/null || true   # silent no-op — always "succeeds"
+EOF
+
+# ✅ Either inline the logic, or define the function inside the heredoc
+```
+
+**Must check:**
+- [ ] Does the test use a hardcoded PID, port, or path that may conflict on the host?
+- [ ] Does the test call a function that is only defined in the outer shell, not inside a heredoc?
+- [ ] Is the "dead process" check using an actual exited process, not a guessed PID?
+
+---
+
+## Automated Safeguards
+
+### CI Gate Rules
+```yaml
+# .github/roll-checks.yml — enforced as a CI gate on every PR
+checks:
+  idempotency:
+    - pattern: "ingest|import|sync"
+      require_test: "idempotency"
+
+  cross_module_contract:
+    - files: ["src/*/index.ts"]
+      check: "shared_id_generation"
+
+  data_flow:
+    - require_integration_test: true
+```
+
+These run as CI gates on every PR. Slower-burn issues — dead code, doc staleness,
+structural drift — are caught by `roll-.dream`, the nightly code-health scan that
+files `REFACTOR-XXX` entries back into the backlog.
+
+### Pre-Commit Hook
+```bash
+#!/bin/bash
+# .git/hooks/pre-commit
+echo "🔍 Checking engineering common sense..."
+
+# Check idempotency tests
+if git diff --cached --name-only | grep -q "ingest\|import\|sync"; then
+  if ! grep -r "idempotency\|repeated run\|multiple times" tests/ 2>/dev/null; then
+    echo "❌ Missing idempotency tests!"
+    exit 1
+  fi
+fi
+
+echo "✅ Basic checks passed"
+```

package/guide/en/pricing.md

@@ -0,0 +1,116 @@
+# Pricing — Cost Visibility & Price Snapshots
+
+Roll computes per-cycle costs at **public per-token pricing** — not your
+subscription rate, but a comparable number you can track across projects and
+agents. This doc covers the `roll config prices` command, snapshot mechanism, and
+how historical costs survive price changes.
+
+## Where Costs Appear
+
+The dashboard (`roll loop status`) shows two cost-related columns per cycle:
+
+| Column | What it means |
+|--------|---------------|
+| **model** | Agent + version used (e.g. `deepseek-v4-pro`, `claude-sonnet-4-6`). Drives which price rates apply. |
+| **cost** | Cost computed at public per-token rates × token usage. Currency follows the vendor: `$` for Anthropic (USD), `¥` for DeepSeek and Kimi (CNY). |
+
+Roll uses **price snapshots** (not hardcoded constants) to look up rates.
+Snapshots live in `lib/prices/` under your Roll install directory.
+
+## Supported Vendors
+
+| Vendor | Currency | Source |
+|--------|----------|--------|
+| Anthropic (Claude) | USD | `platform.claude.com/docs/en/about-claude/pricing` |
+| DeepSeek | CNY | `api-docs.deepseek.com/zh-cn/quick_start/pricing/` |
+| Kimi (Moonshot) | CNY | `platform.kimi.com/docs/pricing/chat` |
+
+## `roll config prices` Command
+
+```bash
+roll config prices show        # Print the current price table (all vendors)
+roll config prices refresh     # Fetch official pricing docs for all vendors, diff, write if changed
+```
+
+### `roll config prices show`
+
+Prints the active snapshots' metadata and a table of all known models with
+their per-million-token rates:
+
+```
+in       Base input tokens
+out      Output tokens
+cw       Cache write tokens (billed at a premium over input)
+cr       Cache read tokens (billed at a deep discount)
+```
+
+Rates are **per million tokens**, in the vendor's native currency.
+
+### `roll config prices refresh`
+
+Fetches the official pricing page from each vendor, parses the rate table,
+and diffs it against the latest local snapshot. Supports per-vendor refresh:
+`roll config prices refresh anthropic|deepseek|kimi`.
+
+- **Rates changed** → writes a new snapshot file (`snapshot-YYYY-MM-DD.json`),
+  prints a diff (red = removed, green = added), and the dashboard picks up the
+  new rates on the next render.
+- **No change** → prints `up to date` and exits.
+
+If the network is down or the page can't be parsed, the command exits with an
+error message — **existing snapshots are never overwritten by a failed fetch**.
+
+## Price Snapshots
+
+Each snapshot is a JSON file in `lib/prices/` named by date:
+
+```
+lib/prices/
+  snapshot-2026-05-22.json
+  snapshot-2026-06-01.json   ← written by refresh when rates changed
+```
+
+A snapshot contains:
+
+| Field | Description |
+|-------|-------------|
+| `version` | ISO 8601 timestamp of when the snapshot was created |
+| `effective_at` | Date the vendor started charging these rates |
+| `source_url` | URL of the official pricing page used |
+| `prices` | Dict of `model → {in, out, cache_create, cache_read}` rates per million tokens |
+
+Snapshots are **never deleted** — every version is kept so you can audit which
+rate table applied at any point in time.
+
+## History Cost Solidification
+
+When a loop cycle finishes, Roll writes **two extra fields** into the usage
+event:
+
+| Field | Purpose |
+|-------|---------|
+| `cost_list_usd` | The cost at that moment's prices — frozen forever |
+| `prices_version` | Which snapshot version was used to compute it |
+
+The dashboard reads `cost_list_usd` first when rendering historical cycles.
+If the field is missing (cycles from before this feature shipped), it falls
+back to computing with the *current* snapshot and appends a dim `[legacy]`
+marker.
+
+**Net effect:** vendor price changes, `roll config prices refresh`, and Roll upgrades
+never rewrite historical cycle costs. "What you actually spent" is a fact —
+it stays put.
+
+## FAQ
+
+**Q: Does the cost column reflect my actual bill?**
+No. It uses public per-token rates. If you're on a subscription (Claude Pro,
+Team, etc.) your real cost is lower. Think of it as a comparable number.
+
+**Q: What happens when prices change?**
+Run `roll config prices refresh`. If rates changed, a new snapshot is written and new
+cycles use it. Old cycles keep their frozen `cost_list_usd`.
+
+**Q: Can I add prices for a different vendor?**
+Yes — `roll config prices refresh --vendor deepseek` (or `kimi`). The `--vendor` flag
+tells the fetcher which vendor's pricing page to scrape.

package/guide/en/project-setup.md

@@ -0,0 +1,126 @@
+# Roll — Project Setup
+
+## Initialize a Project
+
+From your project root:
+
+```bash
+roll init
+```
+
+`roll init` diagnoses the current directory state before any mutation:
+
+1. **Empty directory** — fresh start. Roll writes `AGENTS.md` plus an empty
+   `.roll/` scaffold (`backlog.md`, `features/`, `domain/`) ready for scoped
+   agent bindings in `.roll/agents.yaml`. No prompts. This is the **seed** adoption pattern — see
+   [patterns/seed-pattern.md](patterns/seed-pattern.md).
+2. **PRD/docs-only** — Roll finds requirements or product docs but no source or
+   manifests. It treats this as a new-project path and points you to design; it
+   does not route to existing-codebase onboarding.
+3. **Existing codebase without Roll** — Roll detects existing source files but no
+   `.roll/` directory. It does **not** silently scaffold; instead it surfaces
+   `$roll-onboard`, an interactive skill that scans the codebase, asks a short
+   set of cognition / scope / privacy questions, and produces
+   `.roll/init-diagnosis.yaml` and `.roll/onboard-plan.yaml` for review. After
+   reviewing the paired artifacts, run `roll init --apply`. It prints a checkpoint listing each planned file
+   operation, including action, target path, merge/create mode, and owner-content
+   handling, then waits for confirmation in an interactive terminal. In
+   non-interactive automation, use `roll init --apply --auto` after that review.
+   This is the **graft** pattern — see
+   [legacy-onboarding.md](legacy-onboarding.md) and
+   [patterns/graft-pattern.md](patterns/graft-pattern.md).
+4. **Already initialized** — `.roll/`, `AGENTS.md`, backlog, and features are
+   present. Roll prints `Already initialized` and `Next: roll status`.
+5. **Partial Roll** — some Roll markers are present but the setup is incomplete.
+   Roll prints the missing pieces and any stale pre-v2 markers. `roll init --repair`
+   previews the repair and asks in an interactive terminal; non-interactive
+   automation must use `roll init --repair --auto`. Repair only creates missing
+   Roll-owned files or merges Roll-owned sections, and records the changes in
+   `.roll/onboard-changeset.yaml` so `roll setup offboard` can reverse them later.
+
+After any of these paths, `roll next` is the continuation command. It reads the
+same Roll markers plus `.roll/brief.md`, `.roll/onboard-plan.yaml`, and
+`.roll/backlog.md`, then prints one next action: design from the brief, review
+and apply an onboard plan, repair partial markers, run the old-layout migration,
+start the loop on the next Todo card, or inspect status when nothing is
+actionable.
+
+Upgrading from a pre-2.0 layout (`BACKLOG.md` at root or `docs/features/`)?
+Run `npx @seanyao/roll@2 migrate` first — see
+[migration-2.0.md](migration-2.0.md). `roll init` will refuse to scaffold on top
+of a half-migrated project.
+
+## Sync After Updating Roll
+
+When roll ships a new version of its conventions or skills, sync them into your project:
+
+```bash
+roll sync
+```
+
+`sync` only overwrites files that roll manages (skills and global conventions). Your project-specific files (`.roll/backlog.md`, project source, etc.) are never touched.
+
+## Typical First-Use Sequence
+
+```bash
+curl -fsSL https://seanyao.github.io/roll/install | bash   # install roll
+roll setup                 # configure AI tools globally (one time per machine)
+cd my-project
+roll init                  # diagnose and route this project
+roll next                  # continue with design/apply/repair/migrate/loop/status
+roll loop on               # enable autonomous execution
+```
+
+`roll setup` syncs conventions for the AI tools installed on the machine.
+Agent semantics live in `~/.roll/agents.yaml` (Machine Scope) and
+`.roll/agents.yaml` (Project Scope). Run `roll agent migrate --dry-run` to
+convert older `primary_agent`, local-agent, pairing, or v3 route-slot data into
+the scoped model.
+
+## What Gets Created
+
+| File | Purpose |
+|------|---------|
+| `AGENTS.md` | Agent conventions: domain model, scope, coding standards (root-level — entry point for every AI client) |
+| `.roll/backlog.md` | Story tracking (Epic / Feature / Story / Fix / Refactor) |
+| `.roll/features/` | Per-feature deep docs (one file per Feature) |
+| `.roll/domain/` | DDD models, context map, architecture records |
+
+## Re-initializing
+
+`roll init` is safe to repeat. Complete projects get `Next: roll status`;
+partial projects get `roll init --repair` instead of another scaffold pass.
+
+## See Also
+
+- [installation.md](installation.md) — install and update roll
+- [conventions.md](conventions.md) — AGENTS.md structure and conventions
+- [patterns/](patterns/README.md) — three adoption patterns (seed / graft / replant)
+- [legacy-onboarding.md](legacy-onboarding.md) — graft Roll into an existing codebase
+- [migration-2.0.md](migration-2.0.md) — upgrade a pre-2.0 project to `.roll/` layout
+- [loop.md](loop.md) — enable autonomous execution
+
+## Git Hooks Auto-Setup (US-INFRA-008/009)
+
+Roll's TCR pre-commit gate lives in `hooks/pre-commit`. Git ignores this directory
+by default — `core.hooksPath` must point to it. Roll configures this automatically
+in three places so you never hit a "TCR gate bypassed" window:
+
+1. **`roll setup`** — configures `core.hooksPath=hooks` in the current repo.
+2. **Autonomous loop cycle preflight** — each cycle ensures the worktree it runs in
+   has the correct hooks path.
+3. **Claude Code SessionStart hook** (`.claude/settings.json`) — runs
+   `git config core.hooksPath hooks` every time a new Claude Code session opens in
+   this project.
+
+**Manual override:** If you've intentionally set `core.hooksPath` to something else,
+Roll will not overwrite it. The auto-setup only fires when the value is unset or
+equals the git default `.git/hooks`.
+
+**Troubleshooting:** If commits are going through without running tests:
+
+```bash
+git config core.hooksPath   # should print: hooks
+ls hooks/pre-commit          # should exist and be executable
+roll setup                   # re-runs the configuration step
+```