threadwell 0.0.0 → 0.0.1

package/docs/memory.md

@@ -0,0 +1,484 @@
+# Threadwell Memory
+
+Threadwell Memory is the native project-memory layer for the Threadwell harness. It is designed to make continuity durable, searchable, explainable, and safe without adopting an external plugin or forcing an aggressive workflow.
+
+## Goals
+
+- Preserve operational continuity across turns, compactions, and sessions.
+- Store searchable event history without dumping raw outputs into prompt context.
+- Maintain project language in `CONTEXT.md`.
+- Preserve durable decisions in ADRs under `docs/adr/`.
+- Retrieve only the memory relevant to the current task.
+- Keep memory local-first, inspectable, redacted, and easy to disable.
+
+## Non-goals
+
+- Do not copy or depend on context-mode internals.
+- Do not make `packages/agent` aware of Threadwell memory, markdown files, `.threadwell`, `.threadwell`, or ADRs.
+- Do not block normal tools by default.
+- Do not make Feature Packets/spec workflows mandatory for small changes.
+- Do not store secrets, raw `.env` data, private keys, or unnecessary large outputs.
+
+## Layer model
+
+Threadwell Memory has four layers with separate responsibilities.
+
+```text
+Threadwell
+├─ operational continuity
+│  ├─ .threadwell/continuity/*
+│  ├─ .threadwell/continuity/* compatibility fallback
+│  ├─ AUTONOMOUS_EXECUTION.md
+│  ├─ ACTIVE_QUEUE.md
+│  └─ STATE.md
+├─ event memory
+│  └─ .threadwell/memory.sqlite
+├─ semantic memory
+│  └─ CONTEXT.md
+└─ decision memory
+   └─ docs/adr/*.md
+```
+
+### Operational continuity
+
+Continuity answers: what are we doing now, what is next, what is blocked, and how should the next agent resume?
+
+Sources:
+
+- `.threadwell/continuity/*` for primary native runtime state after path migration.
+- `.threadwell/continuity/*` as a compatibility fallback for existing projects.
+- `AUTONOMOUS_EXECUTION.md`, `ACTIVE_QUEUE.md`, and `STATE.md` for human-readable continuity sessions.
+
+Continuity stores transient operational state:
+
+- current goal
+- active queue item
+- next action
+- blockers
+- open questions
+- modified files
+- latest validation
+- handoff summary
+
+It should not become a long-term event database or a terminology glossary.
+
+### Event memory
+
+Event memory answers: what happened before, and what parts are relevant now?
+
+Planned storage:
+
+```text
+.threadwell/
+  memory.sqlite
+  memory.lock
+  memory-config.json
+  artifacts/
+  exports/
+```
+
+Event memory records compact, redacted, searchable events:
+
+- user intent
+- assistant summary
+- tool call/result summary
+- file modification
+- validation command/result
+- blocker
+- open question
+- decision candidate
+- context candidate
+- ADR candidate
+- handoff
+
+Large outputs should be summarized and stored as artifacts or content references, not injected into prompt context.
+
+### Semantic memory: `CONTEXT.md`
+
+Semantic memory answers: what language does this project use?
+
+`CONTEXT.md` stores stable project vocabulary and mental models:
+
+- domain terms
+- preferred names
+- relationships between concepts
+- conventions
+- ambiguities resolved by the user
+- words to avoid
+
+It must not store task history, raw logs, or ephemeral progress.
+
+### Decision memory: ADRs
+
+Decision memory answers: what have we decided, why, and what does that imply?
+
+ADRs live under `docs/adr/` and store durable decisions with tradeoffs:
+
+- status
+- context
+- decision
+- consequences
+- links to files, tasks, events, or previous ADRs
+
+ADRs are for decisions that should not be re-litigated in future sessions.
+
+## Package boundaries
+
+### `packages/agent`
+
+Allowed:
+
+- generic lifecycle hooks
+- generic context-provider interfaces
+- generic message/context types
+
+Forbidden:
+
+- Threadwell-specific memory semantics
+- `.threadwell`, `.threadwell`, `CONTEXT.md`, or ADR paths
+- SQLite or filesystem persistence for memory
+- continuity settings or markdown rendering
+
+### `packages/coding-agent`
+
+Owns all Threadwell-specific memory behavior:
+
+- MemoryService
+- MemoryStore implementations
+- continuity integration
+- context and ADR services
+- retrieval and capsule rendering
+- settings
+- commands
+- TUI surfaces
+- privacy/redaction
+
+## Identity and storage model
+
+Threadwell Memory needs stable identity at three levels.
+
+### Project identity
+
+`project_id` identifies the working project. It should be derived in this order:
+
+1. Git repository root plus remote URL when available.
+2. Git repository root path when no remote is available.
+3. Normalized current working directory for non-git projects.
+
+The stored id should be a stable hash of the canonical identity source, not the raw path. Raw paths may appear in metadata for user inspection, but retrieval and storage should use the stable id.
+
+Project metadata should include:
+
+- `project_id`
+- `root_path`
+- `git_remote` when available
+- `current_branch` when available
+- `created_at`
+- `last_seen_at`
+
+### Session identity
+
+`session_id` identifies one Threadwell runtime session. It should be derived from the existing session file/id when available and otherwise generated at startup.
+
+Session metadata should include:
+
+- `session_id`
+- `project_id`
+- `continuity_session_id` when a continuity session is active
+- `cwd`
+- `started_at`
+- `last_seen_at`
+- `ended_at` when known
+- `model/provider` when available
+
+### Continuity identity
+
+`continuity_session_id` links memory events to operational continuity files. It is optional because memory can run without active continuity, but when present it should be attached to relevant events, capsules, handoffs, queue items, and ADR/context candidates.
+
+### Path layout
+
+New Threadwell runtime data should live under `.threadwell/` in the project root:
+
+```text
+.threadwell/
+  settings.json
+  memory.sqlite
+  memory.lock
+  memory-config.json
+  artifacts/
+  exports/
+  features/
+  continuity/
+    state.json
+    context.md
+    plan.md
+    queue.md
+    decision-log.md
+    next-action.md
+```
+
+Compatibility fallback paths remain readable during migration:
+
+```text
+.threadwell/settings.json
+.threadwell/continuity/
+  state.json
+  context.md
+  plan.md
+  queue.md
+  decision-log.md
+  next-action.md
+```
+
+Migration rules:
+
+- Prefer `.threadwell/settings.json` for project settings when it exists.
+- Read `.threadwell/settings.json` only when `.threadwell/settings.json` is absent.
+- Write new native continuity state to `.threadwell/continuity/`.
+- Read `.threadwell/continuity/` when `.threadwell/continuity/` is absent.
+- Never delete or rewrite `.threadwell` compatibility data without an explicit user action.
+
+Repo-level human memory remains in normal source-controlled files:
+
+```text
+CONTEXT.md
+docs/adr/*.md
+```
+
+### Path normalization
+
+- Store canonical absolute paths internally for runtime operations.
+- Store display paths relative to project root when possible.
+- Normalize Windows separators for matching and retrieval, but preserve original display paths where useful.
+- Treat case-insensitive path comparisons as platform-dependent: case-insensitive on Windows, case-sensitive elsewhere.
+- Never use raw absolute paths as the only stable project id.
+
+### Repo-less behavior
+
+When no git repository exists, Threadwell should still work using the current working directory as the project root. Moving that directory may create a new project identity unless the user later configures an explicit project alias.
+
+## Startup read order
+
+Threadwell startup context is assembled in this order:
+
+1. settings: read `continuity` and `memory.*` before installing stores/providers;
+2. continuity: load operational state first because it is the source of truth for current task, blockers, and next action;
+3. event memory: open/search `.threadwell/memory.sqlite` only when memory is enabled;
+4. `CONTEXT.md`: read stable semantic vocabulary without overwriting manual edits;
+5. ADRs: read accepted ADRs as current durable decisions, excluding superseded/rejected/proposed records from active context;
+6. retrieval capsule: render bounded cited context from the layers above.
+
+## Retrieval model
+
+Threadwell should inject a bounded memory capsule before model calls when useful.
+
+The capsule should be structured, cited, and small:
+
+```xml
+<threadwell_memory>
+  <current_task>...</current_task>
+  <relevant_decisions>...</relevant_decisions>
+  <context_terms>...</context_terms>
+  <open_questions>...</open_questions>
+  <modified_files>...</modified_files>
+  <validation>...</validation>
+</threadwell_memory>
+```
+
+Ranking should combine:
+
+- full-text relevance
+- recency
+- priority
+- current task
+- modified files
+- branch/session locality
+- ADR links
+- `CONTEXT.md` term links
+- explicit queue anchors
+
+The model should never receive a raw database dump.
+
+## Context pressure policy
+
+Threadwell should adapt to context pressure instead of enforcing hard routing rules by default.
+
+- Low pressure: normal tool behavior.
+- Medium pressure: summarize large outputs and index artifacts.
+- High pressure: prefer bounded retrieval, artifacts, and compact capsules.
+- Critical pressure: prioritize handoff, blockers, decisions, and next action.
+
+Hard blocking of tools can exist later as an opt-in safety mode, not as the default personality.
+
+## Storage backend decision
+
+Threadwell Memory should use SQLite with FTS5 as the durable local event index.
+
+Preferred implementation path:
+
+- Use `better-sqlite3` from `packages/coding-agent` for the first durable backend.
+- Keep `MemoryStore` pluggable so tests can use an in-memory store and future runtimes can switch backend if needed.
+- Use SQLite FTS5 for local full-text search and BM25-style ranking.
+- Store raw large outputs as artifacts/content refs, not as FTS rows by default.
+
+Why SQLite:
+
+- Local-first and project-local.
+- Good enough for thousands to hundreds of thousands of compact events.
+- FTS5 provides fast local search without external services.
+- Easy to inspect, export, purge, compact, and back up.
+
+Why not `node:sqlite` initially:
+
+- Threadwell still targets Node >=20, while built-in SQLite support is tied to newer Node versions and may be experimental depending on runtime.
+- Relying on it would make the memory layer less portable across installed Threadwell environments.
+
+Why not JSON-only storage:
+
+- JSON is acceptable for continuity state, but not for searchable event history, retention, ranking, and concurrent writes.
+- JSON can remain a fallback/no-op or export format, not the primary store.
+
+Dependency plan:
+
+1. Add the dependency only when implementing the durable store task.
+2. Keep no-op and in-memory stores available so tests and disabled mode do not require SQLite.
+3. Add DB doctor/recovery before considering memory default-on complete.
+4. Validate on Windows before enabling broad capture by default.
+
+Open storage risks:
+
+- Native package install/prebuild behavior on Windows.
+- FTS5 availability in bundled SQLite builds.
+- Concurrent agents writing to the same project DB.
+- Recovery behavior after interrupted writes.
+
+These risks are handled by later tasks for migrations, WAL/locks/recovery, doctor commands, concurrency tests, and rollout gates.
+
+## Settings and rollout policy
+
+Threadwell Memory should be configured as a nested `memory` setting so individual layers can be disabled without turning off the whole harness.
+
+Planned shape:
+
+```json
+{
+  "memory": {
+    "enabled": true,
+    "inject": true,
+    "semanticContext": true,
+    "adrSuggestions": true,
+    "candidateReview": true,
+    "hud": false,
+    "retentionDays": 90,
+    "maxDbMb": 256,
+    "indexToolOutputs": "summaries",
+    "artifactLargeOutputs": true,
+    "aggressiveRouting": false
+  }
+}
+```
+
+Default policy:
+
+- `memory.enabled`: default `false` during rollout, then default `true` only after the privacy, recovery, and opt-out gates pass.
+- `memory.inject`: default `false` during rollout, then default `true` only after the same gates pass; injection is always bounded by the capsule budget.
+- `memory.semanticContext`: default `true`; accepted `CONTEXT.md` terms can be injected when relevant.
+- `memory.adrSuggestions`: default `true`; ADR candidates are suggested, not auto-accepted.
+- `memory.candidateReview`: default `true`; low-confidence context/ADR candidates wait for review.
+- `memory.hud`: default `false`; memory status can appear in startup/footer without an always-on HUD.
+- `memory.indexToolOutputs`: default `"summaries"`; full raw outputs are artifacted only when needed.
+- `memory.artifactLargeOutputs`: default `true`; large outputs remain inspectable through file/content refs.
+- `memory.aggressiveRouting`: default `false`; Threadwell does not block normal tool usage by default.
+
+Relationship with `continuity`:
+
+- `continuity: false` disables operational continuity injection and `.threadwell/continuity/` writes.
+- `memory.enabled: false` disables event memory, retrieval capsules, `CONTEXT.md` suggestions, ADR suggestions, and memory commands that require a store.
+- If continuity is disabled but memory is enabled, Threadwell may still record searchable events, but it must not claim an active continuity plan.
+- If memory is disabled but continuity is enabled, `.threadwell/continuity/` remains functional without searchable event history.
+
+Rollout gates before default-on memory is considered complete:
+
+1. Redaction and ignore rules are tested.
+2. DB recovery and lock behavior are tested on Windows.
+3. `/memory doctor`, export, forget, and purge exist.
+4. User-facing opt-out is documented and surfaced in settings.
+5. Manual TUI validation confirms memory surfaces are useful and not noisy.
+
+## Privacy and safety
+
+Before storage or export, memory must redact:
+
+- API keys
+- bearer tokens
+- auth headers
+- `.env` values
+- private keys
+- obvious credentials
+
+Memory should respect:
+
+- `.gitignore`
+- `.threadwellignore`
+- explicit memory settings
+
+Destructive commands such as purge or forget require explicit confirmation.
+
+## Commands
+
+Implemented memory commands:
+
+```text
+/memory status
+/memory enable
+/memory disable
+/memory search <query>
+/memory recent [limit]
+/memory why [memory:<id>...|id]
+/memory doctor
+/memory export
+/memory forget <event-id> --confirm
+/memory purge --confirm
+```
+
+`/memory export` writes a JSONL file under `.threadwell/exports/`. `forget` and `purge` are destructive and require explicit `--confirm`.
+
+Implemented semantic context commands:
+
+```text
+/context review
+/context accept
+/context ignore
+/context edit
+/context doctor
+```
+
+Implemented ADR commands:
+
+```text
+/adr review
+/adr propose <decision text>
+/adr accept [id|index]
+/adr list
+/adr show <id>
+/adr supersede <old> <new>
+```
+
+Review and Feature Packet commands:
+
+```text
+/review
+/feature init <title>
+/feature plan
+/feature tasks
+/queue
+/queue long
+```
+
+## Open decisions
+
+- Storage backend: SQLite with FTS5 is preferred, but the exact Node dependency must be chosen after Windows install/runtime risk is reviewed.
+- Default rollout: event capture and bounded injection should be default-on only after privacy, recovery, and opt-out behavior are validated.
+- Large-output handling: define when an output becomes an artifact versus a compact event summary.
+- Candidate automation: decide which `CONTEXT.md` and ADR candidates can be auto-accepted, if any.
+- TUI surface: decide how much memory state belongs in startup/footer versus an optional HUD.