soturail 0.2.1

package/docs/assets/screenshots/run-expand.svg

@@ -0,0 +1,13 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 820 360" role="img" aria-labelledby="title desc">
+  <title id="title">SotuRail run and expand documentation example</title>
+  <desc id="desc">Static terminal-style documentation example showing raw_id recovery.</desc>
+  <rect width="820" height="360" rx="14" fill="#111827"/>
+  <text x="28" y="58" fill="#e5e7eb" font-family="Consolas, monospace" font-size="18">$ soturail run npm test</text>
+  <text x="28" y="98" fill="#fca5a5" font-family="Consolas, monospace" font-size="16">FAIL tests/app.test.ts</text>
+  <text x="28" y="128" fill="#fca5a5" font-family="Consolas, monospace" font-size="16">AssertionError: expected 1 to equal 2</text>
+  <text x="28" y="180" fill="#fb923c" font-family="Consolas, monospace" font-size="16">SotuRail run complete.</text>
+  <text x="28" y="210" fill="#e5e7eb" font-family="Consolas, monospace" font-size="16">Compressor: test-reducer</text>
+  <text x="28" y="240" fill="#e5e7eb" font-family="Consolas, monospace" font-size="16">raw_id: example123</text>
+  <text x="28" y="270" fill="#86efac" font-family="Consolas, monospace" font-size="16">Recovery: soturail expand example123</text>
+  <text x="28" y="320" fill="#94a3b8" font-family="Consolas, monospace" font-size="14">Documentation example; raw_id is illustrative.</text>
+</svg>

package/docs/assets/screenshots/stats.svg

@@ -0,0 +1,11 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 340" role="img" aria-labelledby="title desc">
+  <title id="title">SotuRail stats documentation example</title>
+  <desc id="desc">Static terminal-style documentation example for honest local stats.</desc>
+  <rect width="760" height="340" rx="14" fill="#0b1220"/>
+  <text x="28" y="58" fill="#e2e8f0" font-family="Consolas, monospace" font-size="18">$ soturail stats</text>
+  <text x="28" y="102" fill="#fb923c" font-family="Consolas, monospace" font-size="17">SotuRail local stats</text>
+  <text x="28" y="138" fill="#e2e8f0" font-family="Consolas, monospace" font-size="16">estimated_raw_tokens: from local manifests</text>
+  <text x="28" y="168" fill="#e2e8f0" font-family="Consolas, monospace" font-size="16">estimated_compressed_tokens: from reducers</text>
+  <text x="28" y="198" fill="#e2e8f0" font-family="Consolas, monospace" font-size="16">real_provider_cache_hits: not imported</text>
+  <text x="28" y="250" fill="#94a3b8" font-family="Consolas, monospace" font-size="14">Documentation example; no fake benchmark numbers shown.</text>
+</svg>

package/docs/assets/soturail-fox.svg

@@ -0,0 +1,17 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 220" role="img" aria-labelledby="title desc">
+  <title id="title">SotuRail fox logo</title>
+  <desc id="desc">A friendly minimal orange fox head with subtle circuit-like context rails underneath.</desc>
+  <g fill="none" stroke-linecap="round" stroke-linejoin="round">
+    <path d="M88 92 118 28l37 43h10l37-43 30 64-18 65c-9 32-32 49-54 49s-45-17-54-49L88 92Z" fill="#f97316" stroke="#7c2d12" stroke-width="8"/>
+    <path d="M118 28 129 89 88 92Z" fill="#fb923c" stroke="#7c2d12" stroke-width="7"/>
+    <path d="M202 28 191 89l41 3Z" fill="#fb923c" stroke="#7c2d12" stroke-width="7"/>
+    <path d="M117 104c18-16 68-16 86 0l-10 41c-6 23-21 36-33 36s-27-13-33-36l-10-41Z" fill="#fff7ed" stroke="#fed7aa" stroke-width="4"/>
+    <path d="M139 117c0 6-5 11-11 11s-11-5-11-11 5-11 11-11 11 5 11 11Zm64 0c0 6-5 11-11 11s-11-5-11-11 5-11 11-11 11 5 11 11Z" fill="#111827" stroke="none"/>
+    <path d="M151 138h18l-9 9-9-9Z" fill="#111827" stroke="none"/>
+    <path d="M143 160c11 8 23 8 34 0" stroke="#7c2d12" stroke-width="6"/>
+    <path d="M76 196h168M103 210h114" stroke="#38bdf8" stroke-width="6"/>
+    <path d="M116 186v31M160 184v33M204 186v31" stroke="#0f766e" stroke-width="4"/>
+    <circle cx="76" cy="196" r="5" fill="#38bdf8" stroke="none"/>
+    <circle cx="244" cy="196" r="5" fill="#38bdf8" stroke="none"/>
+  </g>
+</svg>

package/docs/assets/soturail-icon.svg

@@ -0,0 +1,10 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128" role="img" aria-labelledby="title desc">
+  <title id="title">SotuRail icon</title>
+  <desc id="desc">Compact orange fox icon with a small context rail mark.</desc>
+  <path d="M20 50 36 16l20 23h16l20-23 16 34-9 36c-6 22-22 33-35 33S35 108 29 86L20 50Z" fill="#f97316" stroke="#7c2d12" stroke-width="5" stroke-linejoin="round"/>
+  <path d="M39 59c11-9 39-9 50 0l-6 22c-4 13-12 20-19 20s-15-7-19-20l-6-22Z" fill="#fff7ed"/>
+  <circle cx="48" cy="67" r="5" fill="#111827"/>
+  <circle cx="80" cy="67" r="5" fill="#111827"/>
+  <path d="M59 79h10l-5 5-5-5Z" fill="#111827"/>
+  <path d="M27 116h74M42 124h44" stroke="#38bdf8" stroke-width="4" stroke-linecap="round"/>
+</svg>

package/docs/benchmarking.md

@@ -0,0 +1,34 @@
+# Benchmarking
+
+SotuRail benchmarks use deterministic local fixtures. They are intended to make reducer and context behavior measurable without claiming external superiority.
+
+```bash
+npm run build
+soturail bench prepare
+soturail bench run --engine ts
+soturail bench report
+```
+
+Outputs:
+
+- `benchmarks/results/latest.json`
+- `benchmarks/reports/latest.md`
+
+The suite groups results as:
+
+- terminal compression;
+- agent response compression;
+- JSON/tool payload compression;
+- knowledge structuring;
+- native performance when available.
+
+Knowledge-to-Rules is not judged as pure compression. It creates reusable structured rules, citations and validator metadata.
+
+Native comparison:
+
+```bash
+soturail bench run --engine native
+soturail bench compare-engines
+```
+
+Optional RTK or Squeez comparisons are only run when the user already has those tools on PATH.

package/docs/branding.md

@@ -0,0 +1,14 @@
+# Branding
+
+SotuRail means local context rails for AI coding agents. The name points to the idea of guiding repository, terminal and memory context along safe rails instead of flooding a model with unstable noise.
+
+The fox mascot is friendly and alert: a small signal that SotuRail is meant to help developers move quickly while still watching for safety and evidence.
+
+The rail/circuit line under the fox represents context flow: source files, command output, specs, memory and rules are routed into stable, recoverable payloads.
+
+Logo files:
+
+- `docs/assets/soturail-fox.svg` for the README and project pages.
+- `docs/assets/soturail-icon.svg` for compact usage.
+
+Both assets are original project identity artwork. They make no trademark or copyright claim beyond SotuRail project branding.

package/docs/hooks/claude.md

@@ -0,0 +1,24 @@
+# Claude Hooks
+
+SotuRail v0.2.1 includes a conservative Claude Code hook template.
+
+```bash
+soturail hooks install claude --dry-run
+soturail hooks install claude
+```
+
+Generated files:
+
+- `.claude/settings.json`
+- `.claude/hooks/soturail-pre-tool-use.js`
+- `.claude/hooks/soturail-post-tool-use.js`
+
+Existing files are backed up with `.soturail.bak` before being changed.
+
+The pre-tool hook inspects incoming tool payload text when Claude Code provides it. It blocks obvious destructive shell commands, including recursive deletion, `sudo`, `git push`, downloaded script piping and raw disk copy patterns. It also suggests `soturail run` for tests, builds and logs.
+
+The hook never routes `git push` through `soturail run`.
+
+## Limitation
+
+Claude Code hook schemas may vary by installed version. SotuRail writes a conservative documented template; if your Claude Code release expects a different schema, copy the generated command into the supported hook slot manually.

package/docs/hooks/codex.md

@@ -0,0 +1,5 @@
+# Codex Hooks
+
+Codex prompt-only integration uses `AGENTS.md`.
+
+SotuRail does not assume private Codex host hook APIs. The fallback rules describe when to index, read progressively, run through SotuRail and avoid `git push`.

package/docs/hooks/cursor.md

@@ -0,0 +1,5 @@
+# Cursor Hooks
+
+Cursor prompt-only integration writes `.cursor/rules/soturail.mdc` when installed.
+
+Existing files are backed up before SotuRail adds its rules.

package/docs/hooks/gemini.md

@@ -0,0 +1,5 @@
+# Gemini Hooks
+
+Gemini prompt-only integration uses `GEMINI.md`.
+
+The generated rules keep repository scans, progressive file reads and raw log recovery visible to Gemini CLI sessions.

package/docs/hooks/host-capabilities.json

@@ -0,0 +1,22 @@
+{
+  "claude": {
+    "mode": "adapter-template",
+    "target": "CLAUDE.md",
+    "notes": "Pre-tool hooks vary by host version; prompt-only rules are always safe."
+  },
+  "codex": {
+    "mode": "prompt-only",
+    "target": "AGENTS.md",
+    "notes": "Uses repository guidance rather than unstable host internals."
+  },
+  "gemini": {
+    "mode": "prompt-only",
+    "target": "GEMINI.md",
+    "notes": "Uses local project guidance."
+  },
+  "cursor": {
+    "mode": "prompt-only",
+    "target": ".cursor/rules/soturail.mdc",
+    "notes": "Cursor rules file fallback."
+  }
+}

package/docs/hooks.md

@@ -0,0 +1,15 @@
+# Agent Hooks
+
+SotuRail hook support is cautious. Claude gets a conservative hook template first; Codex, Gemini and Cursor remain prompt-only fallbacks because host APIs are not all stable.
+
+```bash
+soturail hooks list
+soturail hooks doctor
+soturail hooks install claude --dry-run
+soturail hooks install all --dry-run
+soturail hooks prompt-only codex
+```
+
+Installers create backups before modifying existing files. If a host config location is uncertain, SotuRail generates prompt-only guidance instead of guessing.
+
+Claude install writes `.claude/settings.json` and hook scripts under `.claude/hooks/`. Dry-run prints every file that would change.

package/docs/knowledge-to-rules.md

@@ -0,0 +1,23 @@
+# Knowledge-to-Rules Engine
+
+SotuRail can ingest heavy documents into reusable structured rules:
+
+```bash
+soturail ingest docs/requirements.md --type requirements
+soturail rules list
+soturail rules check
+soturail rules export --format yaml
+```
+
+Supported v0.2.0 inputs:
+
+- Markdown
+- TXT
+- JSON
+- YAML
+
+PDF extraction is documented as experimental and disabled unless a safe text extraction path is added later.
+
+Rules are deterministic and cite source file, source section and content hash. SotuRail should not invent rules that do not appear in the source.
+
+Benchmarks report this feature as knowledge structuring, not pure compression. A structured rule set may be larger than a short source document because it adds ids, citations, severities and validator metadata that can be reused across agent sessions.

package/docs/metrics.md

@@ -0,0 +1,35 @@
+# Metrics
+
+SotuRail metrics are local, append-only and transparent.
+
+## Sources
+
+- `.soturail/raw/index.jsonl`
+- `.soturail/metrics/events.jsonl`
+- `.soturail/cache/blocks.jsonl`
+- `.soturail/dedupe/index.jsonl`
+- benchmark JSON reports under `benchmarks/results/`
+
+## Reported Values
+
+- estimated raw tokens;
+- estimated compressed tokens;
+- compression ratio;
+- command count;
+- expansion count;
+- manual omission/failure count when present;
+- estimated cache stability score;
+- real provider cache hits only if imported metadata exists.
+- response compression reduction and preservation counts;
+- rules ingestion and validation counts;
+- benchmark fixture measurements.
+
+## Token Estimation
+
+v0.1.0 uses:
+
+```text
+Math.ceil(text.length / 4)
+```
+
+This is deterministic and useful for local comparisons. It is not a provider tokenizer.

package/docs/mvp.md

@@ -0,0 +1,21 @@
+# MVP Scope
+
+SotuRail v0.1.0 validates the core foundation:
+
+- local workspace creation;
+- cross-platform file scanning with deny rules;
+- progressive file reading;
+- safe tee-stream command execution;
+- raw log recovery;
+- SDD artifact generation;
+- Git-linked memory;
+- prompt-cache-friendly block ordering;
+- honest local metrics.
+
+## Non-goals
+
+- No telemetry.
+- No remote service.
+- No native compiled dependencies.
+- No automatic `git push`.
+- No provider cache-hit claims unless metadata is manually imported.

package/docs/native-runner.md

@@ -0,0 +1,52 @@
+# Native Runner
+
+SotuRail v0.2.1 keeps TypeScript as the public CLI, policy orchestration, docs and npm distribution layer. Rust is used only for optional hot paths where streaming, lower overhead and binary execution are useful.
+
+## Responsibility Split
+
+- TypeScript: CLI commands, config validation, safety policy, workspace manifests, docs, metrics and npm package distribution.
+- Rust: optional reducers and native tee-stream command execution.
+
+## Native Commands
+
+- `soturail-native --version`
+- `soturail-native reduce-generic`
+- `soturail-native reduce-git`
+- `soturail-native reduce-test`
+- `soturail-native reduce-json`
+- `soturail-native run --raw-log <path> --summary-json <path> -- <command...>`
+
+The native runner spawns a subprocess, tees stdout/stderr to the terminal, writes the same bytes to a raw log and records a compact JSON summary. TypeScript safety checks remain the primary gate, and the Rust binary also refuses obvious destructive command strings when called directly.
+
+## Detection and Fallback
+
+The TypeScript CLI checks:
+
+1. `native/soturail-native/target/release/soturail-native`
+2. `native/soturail-native/target/debug/soturail-native`
+3. `dist/native/soturail-native`
+4. `soturail-native` on PATH
+
+`--engine auto` falls back to TypeScript when native is missing. `--engine native` fails clearly when native is unavailable.
+
+## Build
+
+```bash
+npm run build          # TypeScript only
+npm run build:native   # Rust release binary
+npm run build:all      # TypeScript + Rust
+```
+
+## Native Benchmarks
+
+```bash
+npm run build:native
+soturail bench run --engine native
+soturail bench compare-engines
+```
+
+## Current Limitations
+
+- Native binaries are not prebuilt for npm yet.
+- Interactive TTY behavior is best-effort.
+- TypeScript remains the source of truth for workspace manifests and safety prompts.

package/docs/prompt-caching.md

@@ -0,0 +1,27 @@
+# Prompt Caching
+
+SotuRail formats context so stable blocks appear before dynamic session data:
+
+1. `static_header`
+2. `governance`
+3. `config`
+4. `repo_map`
+5. `approved_specs`
+6. `approved_memory`
+7. `dynamic_footer`
+
+The cache normalizer writes `.soturail/cache/blocks.jsonl` with block ids, source paths, hashes, stable order and token estimates.
+
+## Dynamic Data
+
+Dynamic data must stay after stable blocks:
+
+- timestamps;
+- raw ids;
+- command status;
+- recent logs;
+- ephemeral user questions.
+
+## Honesty
+
+`estimated_cache_stability_score` is a local estimate. v0.2.0 never claims real provider cache hits unless imported metadata exists.

package/docs/pt-BR/visao-geral.md

@@ -0,0 +1,26 @@
+# Visao geral do SotuRail
+
+SotuRail e um Context OS local-first para agentes de IA que trabalham com codigo.
+
+Ele ajuda a:
+
+- indexar o repositorio sem lotar o contexto do modelo;
+- ler arquivos grandes de forma progressiva;
+- executar comandos com uma politica de seguranca;
+- salvar logs brutos recuperaveis;
+- comprimir saidas longas do terminal;
+- separar blocos estaveis de dados dinamicos para prompt caching;
+- manter memoria local ligada ao commit Git atual;
+- comprimir respostas de agentes em modos profissionais;
+- transformar documentos em regras estruturadas e validadores;
+- instalar regras de hooks ou fallback prompt-only para agentes;
+- gerar benchmarks locais reproduziveis;
+- relatar metricas honestas.
+
+O comando principal e:
+
+```bash
+soturail
+```
+
+No v0.2.1, os numeros de tokens, reducao e cache sao estimativas locais quando nao ha metadados reais importados. Knowledge-to-Rules e tratado como estruturacao reutilizavel, nao apenas compressao. SotuRail nao inventa acertos reais de cache de provedores.

package/docs/public-roadmap.md

@@ -0,0 +1,20 @@
+# Public Roadmap
+
+SotuRail is early but functional. The public roadmap prioritizes local-first evidence, safety and reproducible measurement.
+
+## Near Term
+
+- Polish v0.2.x native hot paths.
+- Improve reducer quality checks.
+- Harden Claude hook templates.
+- Publish clearer benchmark interpretation docs.
+
+## v0.3.0 Direction
+
+- MCP server.
+- Tree-sitter repository map.
+- Hardened PDF extraction.
+- Semantic memory with embeddings.
+- Native runner packaging for npm prebuilds.
+- External benchmark comparison docs.
+- Real provider cache metadata import.

package/docs/reducers.md

@@ -0,0 +1,19 @@
+# Reducers
+
+Reducers compress terminal output while preserving the recovery path to raw logs.
+
+## Generic
+
+Keeps first and last lines, error-looking lines, warnings, permission failures, timeouts, file paths and repeated-line summaries.
+
+## Git
+
+Handles status, diff, log, show and branch output heuristically. It preserves changed files, rename/delete markers, conflict markers and hunk headers.
+
+## Test
+
+Handles Vitest, Jest, Mocha, Pytest, Maven Surefire and Gradle test output at a heuristic level. It preserves failing test names, assertions, expected/received summaries, stack traces and file paths.
+
+## JSON TOON Lite
+
+Emits a compressed representation with relevant `path: value` lines. It preserves error messages, statuses, ids, file paths, booleans and nulls that affect debugging, while collapsing repetitive arrays and object structure. The reduced representation is clearly labeled when it is not valid JSON.

package/docs/release-checklist.md

@@ -0,0 +1,13 @@
+# Release Checklist
+
+Before publishing:
+
+- [ ] `npm install`
+- [ ] `npm run build`
+- [ ] `npm test`
+- [ ] Verify `soturail --help`
+- [ ] Verify `soturail init` in a clean temp repo
+- [ ] Confirm docs mention limitations honestly
+- [ ] Confirm no telemetry exists
+- [ ] Confirm no `git push` was run automatically
+- [ ] Tag release manually after review

package/docs/response-compression.md

@@ -0,0 +1,15 @@
+# Agent Response Compression
+
+`soturail format` compresses verbose AI or documentation output deterministically. It does not call an external LLM.
+
+Modes:
+
+- `normal` - light cleanup.
+- `concise` - direct technical instructions.
+- `ultra` - maximum command-focused compression.
+- `review` - code-review style grouped by severity.
+- `commit` - Conventional Commit suggestion.
+- `debug` - symptom, cause, fix, verification and next command.
+- `docs` - README/docs-oriented language.
+
+The reducer preserves fenced code blocks, shell commands, file paths, line numbers, warnings, security notes and failure information.

package/docs/rules.md

@@ -0,0 +1,23 @@
+# Rules
+
+Extracted rules are written to:
+
+- `.soturail/rules/rules.yml`
+- `.soturail/rules/checklist.md`
+- `.soturail/rules/citations.json`
+- `.soturail/rules/validators/`
+
+Each rule includes:
+
+- id
+- title
+- description
+- source_file
+- source_section
+- severity
+- validation_type
+- suggested_validator_name
+- created_at
+- content_hash
+
+`soturail rules check` runs deterministic validators for package metadata, required files, forbidden files, README sections, CI workflow presence and documentation presence when the rule text supports it.

package/docs/security-model.md

@@ -0,0 +1,32 @@
+# Security Model
+
+SotuRail reduces accidental damage from AI-assisted terminal use. It is not a sandbox and does not make untrusted commands safe.
+
+## Blocked by Default
+
+- `rm -rf`
+- `sudo`
+- `format`
+- `dd if=`
+- `curl | sh`
+- `wget | sh`
+- `del /s`
+- `git push`
+
+## Unsafe Confirmation
+
+Dangerous commands can only run when the exact phrase is supplied:
+
+```text
+I_UNDERSTAND_THIS_CAN_DESTROY_DATA
+```
+
+This is intentionally verbose so accidental bypasses are unlikely.
+
+## Raw Logs
+
+Raw command logs remain on disk so compressed summaries can always be audited.
+
+## Limitations
+
+SotuRail does not isolate processes, prevent all shell tricks or replace OS permissions. Treat it as a policy and evidence layer.

package/docs/spec-driven-workflow.md

@@ -0,0 +1,22 @@
+# Spec-Driven Workflow
+
+SotuRail specs live under `.soturail/specs/`.
+
+```bash
+soturail spec new "feature idea"
+soturail spec status
+soturail spec validate
+soturail spec task add .soturail/specs/001-feature "Write tests"
+soturail spec task list .soturail/specs/001-feature
+soturail spec task done .soturail/specs/001-feature 1
+```
+
+v0.2.0 specs include:
+
+- `constitution.md`
+- `spec.md`
+- `plan.md`
+- `tasks.md`
+- `verification.md`
+- `context-budget.md`
+- `security-impact.md`

package/docs/usage.md

@@ -0,0 +1,64 @@
+# Usage
+
+## Initialize
+
+```bash
+soturail init
+```
+
+Creates `.soturail/` and starter docs without overwriting existing files.
+
+## Index
+
+```bash
+soturail index
+```
+
+Writes `.soturail/indexes/repo-map.json` and `.soturail/indexes/tree.txt`.
+
+## Read
+
+```bash
+soturail read src/core/file-scanner.ts --query "ignore rules"
+soturail read src/core/file-scanner.ts --full
+```
+
+Files under 150 lines are printed fully. Larger files include the first 15 lines, query matches with margin, the last 10 lines and reversible collapsed markers.
+
+## Run and Expand
+
+```bash
+soturail run npm test
+soturail expand <raw_id>
+```
+
+The runner streams output live, writes the same output to a raw log and prints a compressed summary.
+
+## Specs and Memory
+
+```bash
+soturail spec new "safe command profiles"
+soturail memory add "We block git push by default"
+soturail memory search "git push"
+```
+
+## Doctor and Stats
+
+```bash
+soturail doctor
+soturail doctor cache
+soturail stats
+```
+
+## v0.2.x Workflows
+
+```bash
+soturail bench prepare
+soturail bench run --engine ts
+soturail hooks install codex --dry-run
+soturail format README.md --mode concise
+soturail ingest README.md --type docs
+soturail rules check
+soturail native doctor
+soturail bench compare-engines
+```