memorix 1.1.13 → 1.2.0

package/docs/1.2.0-IMPLEMENTATION-PLAN.md

@@ -0,0 +1,554 @@
+# Memorix 1.2 Implementation Plan
+
+Status: execution contract
+Branch: codex/1.2.0-multidimensional-memory
+Baseline: memorix 1.1.13
+Start: 2026-07-17
+Companion documents: [Product Story](1.2.0-PRODUCT-STORY.md), [Development Charter](1.2.0-DEVELOPMENT-CHARTER.md), and [Knowledge and Workflow Research](1.2.0-KNOWLEDGE-WORKFLOW-RESEARCH.md)
+
+## Mission
+
+Deliver Memorix 1.2 as a local-first, multi-agent project working-context
+system. It must turn durable project evidence into a small task Workset without
+forcing users to remember commands, trust stale summaries, or install a
+third-party graph product.
+
+The release is complete only when all phases in this document pass their
+implementation and verification gates. This is one continuous development goal,
+not a collection of disconnected feature experiments.
+
+## Product Contract
+
+For a normal request such as "continue the migration failure," a supported
+agent can obtain a compact Workset that answers:
+
+1. where to start in current code;
+2. what source-backed knowledge and workflow apply;
+3. what changed since that knowledge was formed;
+4. what is stale, incomplete, conflicted, or unknown;
+5. what verification proves the next change is safe.
+
+Memorix does not replace code reading, Git, tests, or agent judgment. It
+chooses and qualifies the evidence that makes those activities start faster.
+
+## Delivery Rules
+
+- Complete all phases in this single goal, but do not advance a phase until its
+  acceptance tests pass.
+- Keep the default MCP profile small. Project Context remains the primary
+  agent-facing entry point.
+- Keep raw evidence, claims, wiki pages, workflow templates, and task runs
+  separate. A summary is never allowed to overwrite its evidence.
+- Preserve local-first operation. No remote provider, cloud graph database, or
+  external CodeGraph installation is required for a useful result.
+- Make every automatic write reviewable. Project-versioned wiki and workflow
+  changes default to proposed patches, not silent edits.
+- Do not represent the existing memory visualization as a semantic graph or
+  maintained wiki until the corresponding contracts exist.
+- Maintain Windows-safe paths, bounded background work, source secrecy, and
+  existing agent integration behavior.
+
+## Existing Foundations to Reuse
+
+| Existing foundation | 1.2 use |
+| --- | --- |
+| Shared SQLite database and maintenance queue | New schema, non-blocking compilation, lint, and refresh jobs |
+| CodeGraphStore and CodeGraph Lite | Current structural evidence, snapshots, and bounded fallback provider |
+| Observation and Git Memory stores | Durable narrative and commit evidence |
+| Observation code references and freshness | Claim-to-code validity and stale detection |
+| Mini-skills and SkillsEngine | Legacy knowledge sources and native agent adapter format |
+| WorkspaceSyncEngine and WorkflowSyncer | Adapter-writing path, but no longer the canonical workflow format |
+| project_context and context_pack | Workset retrieval surface and compatibility path |
+| Dashboard and TUI Knowledge views | Knowledge, workflow, provenance, and health inspection surfaces |
+| Doctor, background service, hook lifecycle | Diagnostics, scheduling, and integration health reporting |
+
+## Target Architecture
+
+~~~mermaid
+flowchart LR
+    A["Evidence: Git, code snapshots, tests, documents, observations"] --> B["Claim Ledger"]
+    B --> C["Knowledge Workspace: Markdown wiki, index, lint"]
+    B --> D["Workflow Library: versioned playbooks"]
+    A --> E["Code State and provider health"]
+    C --> F["Workset Builder"]
+    D --> F
+    E --> F
+    F --> G["memorix_project_context"]
+    G --> H["Agent action and verification"]
+    H --> A
+~~~
+
+## Canonical Storage and Data Ownership
+
+### SQLite is the operational index
+
+SQLite remains the local operational store. It records metadata, relationships,
+validity, source references, job state, search metadata, and audit history. It
+does not become the only human-readable copy of long-lived project knowledge.
+
+Add a small migration runner backed by a schema_migrations table. New 1.2
+migrations are ordered and transactional. Existing idempotent initialization
+remains compatible; no existing user data is rewritten or discarded.
+
+### Knowledge artifacts have two modes
+
+1. **Local mode** stores generated Markdown artifacts under the Memorix data
+   directory, scoped by project identity. This is private by default.
+2. **Versioned mode** stores the same artifacts in a user-selected,
+   non-ignored project directory. The CLI must require an explicit path or
+   explicit init choice before creating project files. It must not assume that
+   the repository's ignored .memorix directory is versioned.
+
+Both modes use the same Markdown format. SQLite stores page metadata and index
+state, not an opaque replacement for the page.
+
+### Raw sources stay raw
+
+Documents, Git facts, tests, observations, and captured execution evidence are
+referenced by stable identifiers and hashes. The default policy does not copy
+all project files or all conversations into the wiki. Large, binary, secret, or
+generated paths remain excluded using existing CodeGraph and secret filters.
+
+## Data Contracts
+
+### CodeStateSnapshot
+
+~~~text
+CodeStateSnapshot
+  id
+  projectId
+  provider
+  baseRevision
+  worktreeFingerprint
+  worktreeState             clean, dirty, unavailable
+  changedPathCount
+  indexedAt
+  sourceEpoch
+  completenessJson
+  previousSnapshotId
+~~~
+
+A snapshot is created by a bounded refresh or a fast project-state check. The
+fingerprint derives from Git metadata and path status, not from storing a second
+copy of source code. Completeness records file limits, skipped oversized files,
+ignored paths, parser coverage, and scan errors.
+
+Add snapshotId and sourceEpoch columns to code files, symbols, edges, and new
+claim/code references where needed. Existing code reference hashes remain the
+fine-grained freshness signal.
+
+### Knowledge claims
+
+~~~text
+KnowledgeClaim
+  id
+  projectId
+  subject
+  predicate
+  objectValue
+  scope                    project, workspace, team, workflow, task
+  status                   active, superseded, disputed, unknown
+  confidence
+  observedAt
+  validFrom
+  validTo
+  supersededBy
+  reviewState
+  createdAt
+  updatedAt
+~~~
+
+A claim has evidence rows rather than a hidden free-form relationship:
+
+~~~text
+ClaimEvidenceRef
+  claimId
+  evidenceKind             observation, git, code, test, document, workflow, run
+  evidenceId
+  relation                 supports, contradicts, derives, verifies
+  snapshotId
+  locator
+  capturedHash
+  createdAt
+~~~
+
+Multiple claims with the same subject, predicate, and scope may coexist. A
+conflict is visible; it is not overwritten by the newest model output. A claim
+becomes superseded only through explicit evidence-aware resolution.
+
+### Wiki pages
+
+Each generated page is Markdown with frontmatter that includes a stable id,
+title, kind, status, claim ids, evidence references, code snapshot reference,
+tags, review state, and updated time. Supporting tables track page metadata,
+claim links, and page links for fast retrieval and linting.
+
+Required artifacts:
+
+- index.md: content-oriented page catalog;
+- log.md: append-only knowledge evolution log;
+- pages/: interlinked topic, subsystem, decision, and risk pages;
+- workflows/: canonical project playbooks;
+- schema.md: the local maintenance rules used by agents and linting.
+
+### Workflow inheritance
+
+A canonical workflow is Markdown with YAML frontmatter. It is not generated
+from an agent-specific source format.
+
+~~~text
+WorkflowSpec
+  id, title, description, status, version
+  triggers and task lenses
+  assumptions and required context
+  ordered phases and branches
+  guardrails and allowed tools
+  expected outputs and verification gates
+  claim, evidence, and code references
+  adapter compatibility
+  updatedAt
+
+WorkflowRun
+  id, workflowId, projectId
+  task summary and starting snapshot
+  phase state
+  selected evidence
+  outcome and verification verdict
+  failure reason
+  startedAt, completedAt
+~~~
+
+Workflow source files are canonical. WorkspaceSyncEngine renders native Skill,
+rule, or instruction adapters only through its existing preview/apply safety
+path. Existing Windsurf workflow conversion remains supported as an import path,
+not the source of truth.
+
+## Public Surface
+
+### Primary agent path
+
+Extend memorix_project_context. It remains a single compact tool and returns:
+
+- start files and current code facts;
+- selected current claims and relevant wiki pages;
+- at most two matching workflow starts;
+- stale, incomplete, conflict, and provider cautions;
+- focused verification gates;
+- provenance references in JSON and diagnostics output.
+
+When no knowledge or workflow advantage exists, return less context rather than
+a generic summary.
+
+### Advanced operator path
+
+Add a single advanced knowledge management surface in full/team profiles and a
+matching CLI namespace:
+
+~~~text
+memorix knowledge init
+memorix knowledge status
+memorix knowledge ingest
+memorix knowledge query
+memorix knowledge lint
+memorix knowledge workflow ...
+memorix knowledge review ...
+~~~
+
+The advanced MCP equivalent uses one grouped knowledge tool with action-based
+operations. It is absent from the lite profile by default. Existing memory,
+graph, and context tools remain backward compatible.
+
+### Dashboard, TUI, and Doctor
+
+- Rename the existing generated view to an honest Memory Overview until the
+  maintained wiki is available.
+- Add Knowledge Workspace, Claims, Workflows, and Evidence Drill-down views.
+- Add health badges for snapshot freshness, incomplete scans, pending review,
+  conflicts, broken links, and failed maintenance jobs.
+- Extend doctor with project knowledge workspace path, workflow adapter status,
+  page lint state, and provider/snapshot diagnostics.
+- Keep all dashboard reads project-scoped.
+
+## Phase 0: Evaluation Harness and Truthful Labels
+
+### Implementation
+
+- Add fixture repositories for TypeScript, Python, Go, docs-only, dirty
+  worktree, deleted-symbol, and incomplete-scan cases.
+- Add seeded observations, Git evidence, test evidence, mini-skills, explicit
+  graph relations, and expected task Worksets.
+- Rename or annotate current dashboard/TUI graph and wiki views so their scope
+  is accurate.
+- Add an evaluation runner that compares memory-only context, current 1.1
+  context, and candidate 1.2 Worksets using deterministic expected evidence and
+  token budgets.
+- The reusable fixture contract and run instructions live in
+  [the Evaluation Harness](1.2.0-EVALUATION-HARNESS.md).
+
+### Exit gate
+
+- Tests prove that the current memory map, structural code index, and
+  source-backed claim are distinguishable concepts.
+- Evaluation fixtures run without network access or model credentials.
+- No existing 1.1 context output regresses.
+
+## Phase 1: Versioned Code State
+
+### Implementation
+
+- Introduce schema_migrations and CodeStateSnapshot tables.
+- Implement Git revision and worktree fingerprint collection with bounded
+  Windows-safe commands.
+- Extend CodeGraphStore refresh transactions to persist a snapshot and source
+  epoch with every changed fact.
+- Extend status, Project Context JSON, and maintenance payloads with snapshot
+  completeness and freshness.
+- Mark old observation code references suspect when their qualifying snapshot is
+  no longer current or complete enough.
+
+### Exit gate
+
+- A new commit creates a new snapshot.
+- A dirty worktree differs from a clean one without reading all source files.
+- Incomplete scans are explicit and cannot be formatted as complete truth.
+- Existing incremental refresh, deletion handling, max file size, and no-block
+  MCP initialization tests still pass.
+
+## Phase 2: Claim Ledger
+
+### Implementation
+
+- Add claim, evidence-ref, and conflict/supersession schema and stores.
+- Implement claim validation, normalized identity, provenance checks, and
+  lifecycle transitions.
+- Derive low-risk claims from explicit observations, Git facts, and code-state
+  evidence. Keep model-generated claims in draft or review state until policy
+  permits publication.
+- Link claims to existing observation code references and snapshots.
+- Implement claim lookup and a compact claim selector for task lenses.
+
+### Exit gate
+
+- Claims always expose at least one evidence reference.
+- Contradictory active claims are shown as conflicted, not merged.
+- Deleting a file, changing a bound symbol, or changing snapshot completeness
+  affects claim confidence or status deterministically.
+- Secret filtering applies to claim content, source locators, and diagnostics.
+
+## Phase 3: Wiki Compiler
+
+### Implementation
+
+- Add local and versioned workspace initialization with an explicit project
+  path choice and write policy.
+- Implement Markdown page reader/writer, frontmatter validation, content hash,
+  index.md, log.md, link graph, and atomic write behavior.
+- Compile approved claims into pages and produce proposed patches under review
+  policy. Preserve raw source references and do not silently overwrite manual
+  edits.
+- Implement wiki lint for broken links, orphan pages, missing evidence,
+  superseded primary claims, unresolved conflicts, stale snapshot references,
+  and malformed frontmatter.
+- Index page metadata and approved content into existing retrieval without
+  making raw pages mandatory prompt context.
+
+### Exit gate
+
+- A fresh workspace initializes predictably in local mode and only writes to a
+  project path after explicit opt-in.
+- Ingesting evidence produces a reviewable page patch with source drill-down.
+- Lint catches a deliberately broken link and a deliberately superseded claim.
+- Manual edits survive a subsequent compiler run unless an explicit reviewed
+  patch replaces them.
+
+## Phase 4: Workflow Inheritance
+
+### Implementation
+
+- Add WorkflowSpec and WorkflowRun stores, parser, validator, selector, and
+  source-backed workflow pages.
+- Import current Windsurf workflows into canonical workflow files without
+  changing their original source.
+- Extend WorkspaceSyncEngine to render canonical workflows to supported agents
+  through preview/apply, preserving non-Memorix user files.
+- Add task-lens workflow selection and progressive delivery. The first phase is
+  compact; full instructions, scripts, and references are loaded only when
+  relevant.
+- Record workflow runs and verification outcomes. Repeated outcomes may produce
+  a workflow improvement proposal, never a silent source edit.
+
+### Exit gate
+
+- A release, bugfix, migration, and review fixture each select the correct
+  canonical workflow or no workflow when none is applicable.
+- Generated adapters pass existing workspace safety tests and do not rewrite
+  global agent settings.
+- A failed verification gate remains visible in the next Workset.
+- Existing mini-skills remain usable and are not mass-injected without budget
+  control.
+
+## Phase 5: Workset Retrieval
+
+### Implementation
+
+- Replace the current context assembly policy with a bounded evidence ranker
+  that combines code state, claims, wiki pages, workflows, Git facts,
+  observations, and verification evidence.
+- Define per-section caps, confidence thresholds, conflict policy, and token
+  budget accounting.
+- Keep prompt, summary, and JSON outputs. JSON carries stable provenance ids and
+  reasons for selection.
+- Keep context_pack compatible, but route it through the same ranking core.
+
+### Exit gate
+
+- A task returns source-backed start points, current claims, a workflow start
+  when applicable, cautions, and verification in a stable order.
+- A task with no useful Memorix advantage receives a small response.
+- No raw source dump, all-mini-skill dump, or unbounded page content enters the
+  default prompt.
+- Existing lite MCP tool profile count does not grow.
+
+## Phase 6: Dynamic Lifecycle and Maintenance
+
+### Implementation
+
+- Add maintenance job kinds for snapshot refresh, claim requalification, wiki
+  compilation, wiki lint, workflow analysis, and stale artifact review.
+- Reuse SQLite leases, dedupe, retry, cancellation, and bounded worker behavior.
+- Schedule from Git/hook/worktree signals, never from every prompt.
+- Add recovery semantics for failed jobs, incomplete provider output, workspace
+  file conflicts, and interrupted writes.
+
+### Exit gate
+
+- Foreground Project Context returns without waiting for corpus scans or model
+  synthesis.
+- Duplicate lifecycle events create one deduped job.
+- Restarted workers recover eligible jobs without duplicate wiki/log writes.
+- Failed background jobs surface through doctor and dashboard with sanitized
+  errors.
+
+## Phase 7: Parser and Provider Quality
+
+### Implementation
+
+- Define a provider contract for symbols, relations, locations, coverage,
+  confidence, and snapshot metadata.
+- Benchmark current Lite extraction against parser-backed candidates on Windows
+  install reliability, packaged size, supported fixtures, refresh time, and
+  relation accuracy.
+- Adopt a parser-backed provider only when it wins the agreed gate. Tree-sitter
+  is a candidate, not a pre-decided dependency.
+- Add optional external provider import only when normalized evidence can be
+  validated, versioned, diagnosed, and safely replaced by internal fallback.
+- The checked-in provider contract, privacy boundary, and reproducible quality
+  gate live in [Provider Quality](1.2.0-PROVIDER-QUALITY.md). A pre-existing
+  local CodeGraph index may provide task-scoped semantic outlines; Memorix
+  never creates, syncs, or silently persists external facts.
+
+### Exit gate
+
+- Provider selection is explicit and reported in status/doctor.
+- Lite remains a tested fallback for every supported installation.
+- Parser or external provider failure degrades evidence confidence without
+  breaking Project Context.
+- Benchmark results are checked into docs and reproducible locally.
+
+## Phase 8: Product Validation and Release
+
+### Implementation
+
+- Run deterministic evaluation fixtures across coding, debugging, onboarding,
+  refactor, release, and non-code tasks.
+- Run agent-facing smoke tests through packaged MCP and at least the installed
+  Codex integration without modifying the user's real project configuration.
+- Verify local workspace, versioned workspace, workflow adapter, dashboard, and
+  doctor paths.
+- Update README, Chinese README, setup, API reference, configuration, operator
+  playbook, changelog, and release notes.
+- Perform package inspection, clean npm installation, full test suite, build,
+  lint, and GitHub Actions checks before publishing 1.2.0.
+
+### Exit gate
+
+- Evaluation evidence shows that 1.2 reduces broad exploration or catches
+  stale/conflicting context compared with the 1.1 baseline.
+- All public product claims are backed by tests or documented measurement.
+- No secret, raw private source, or user global config mutation appears in
+  default behavior.
+- npm package, GitHub release, tag, changelog, docs, and supported integration
+  matrix are synchronized.
+
+## Module Map
+
+| Area | Primary new or changed modules |
+| --- | --- |
+| Schema and migrations | src/store/sqlite-db.ts, new migration helper, store tests |
+| Code State | src/codegraph/types.ts, store.ts, lite-provider.ts, freshness.ts, auto-context.ts |
+| Claims | new src/knowledge/claims.ts, claim-store.ts, evidence.ts, resolver.ts |
+| Wiki | new src/knowledge/workspace.ts, pages.ts, compiler.ts, lint.ts, markdown.ts |
+| Workflows | new src/knowledge/workflows.ts, workflow-store.ts, selector.ts, adapters.ts |
+| Ranking | new src/knowledge/workset.ts and integration in codegraph auto-context/context-pack |
+| Runtime | src/runtime/maintenance-jobs.ts, background handler registration |
+| CLI and MCP | cli command registry, commands/context.ts, new commands/knowledge.ts, server.ts |
+| Dashboard and TUI | dashboard server/static views, TUI data/views/theme |
+| Config and docs | resolved config, TOML types/loaders, setup/API/playbook/README |
+
+Private helper layout can change during implementation, but these boundaries and
+acceptance tests are the contract.
+
+## Test and Smoke Matrix
+
+| Layer | Required verification |
+| --- | --- |
+| Unit | IDs, migrations, snapshot fingerprints, claim lifecycle, page parser, workflow parser, ranking, token caps |
+| Store | SQLite reopen, migrations from 1.1 data, transactions, concurrent maintenance leases |
+| Integration | code change to claim caution, wiki compile/lint, workflow selection/adapters, Project Context JSON |
+| MCP | stdio initialize/list remains fast, default tool profile remains compact, Workset call succeeds in clean installation |
+| CLI | knowledge init/status/ingest/lint/workflow, doctor diagnostics, no project write without opt-in |
+| Dashboard/TUI | project isolation, provenance drill-down, visual empty/loading/error states |
+| Agent smoke | isolated Codex setup and real MCP call; other adapters tested through generated config fixtures |
+| Release | lint, build, serial full tests, pack inspection, isolated npm install, CI, tag, publish, release notes |
+
+## Performance, Privacy, and Safety Budgets
+
+- MCP initialize and tools/list cannot wait for indexing, wiki compilation, or
+  provider calls.
+- Project Context schedules expensive work and returns the best qualified
+  current answer.
+- Default prompt output has strict section caps and never includes full raw
+  documents or source files.
+- Source locators and error strings pass existing credential sanitization.
+- External providers are opt-in and receive only the data required by their
+  adapter. No automatic project export is allowed.
+- Project file writes use atomic replacement, workspace path validation, and a
+  review policy. No broad user configuration writes are part of workflow setup.
+- Every generated claim, page, workflow selection, and warning records its
+  provenance and reason.
+
+## Compatibility and Migration Policy
+
+- Existing observations, graph entities, graph relations, mini-skills,
+  CodeGraph tables, context_pack, and project_context continue to work.
+- Existing Knowledge Base and Knowledge Graph endpoints remain available during
+  migration, but the UI describes them accurately as overview and memory map
+  until maintained artifacts are enabled.
+- Existing Windsurf workflows remain readable and importable. No source agent
+  file is overwritten by canonical workflow creation.
+- New schema is additive. Rollback means disabling knowledge features, not
+  deleting claims, pages, or existing 1.1 data.
+- Versioned workspace files are never deleted by automatic cleanup.
+
+## Definition of Done
+
+1. Every Phase 0 through 8 exit gate passes in automated tests or reproducible
+   smoke evidence.
+2. A clean project can use Memorix without a graph database, cloud service, or
+   manual command ritual.
+3. A project can opt into a readable wiki and versioned workflows with reviewed,
+   source-backed changes.
+4. A new agent receives a compact, qualified Workset through existing natural
+   language task flow.
+5. The release is documented, packaged, published, tagged, and verified as
+   memorix 1.2.0.