memorix 1.1.13 → 1.2.0

package/docs/1.2.0-KNOWLEDGE-WORKFLOW-RESEARCH.md

@@ -0,0 +1,205 @@
+# Memorix 1.2 Knowledge and Workflow Research
+
+Status: adopted product direction
+Companion: [1.2 Product Story](1.2.0-PRODUCT-STORY.md) and [1.2 Development Charter](1.2.0-DEVELOPMENT-CHARTER.md)
+Date: 2026-07-17
+
+## Research Question
+
+Can Memorix become more than a store of narrative memories? The useful answer is
+not "add a graph." A coding project needs an evidence-backed knowledge base that
+agents can maintain, inspect, and use, plus a way to inherit the project's
+working methods without asking every new agent to rediscover them.
+
+The resulting 1.2 direction is a **Knowledge Workspace** and **Workflow
+Inheritance** layer on top of Memorix's existing local, multi-agent memory
+system.
+
+## What the Reference Systems Actually Teach
+
+| System | Verified design contribution | What Memorix should learn | What Memorix should not copy |
+| --- | --- | --- | --- |
+| [Cognee](https://docs.cognee.ai/) | Ingests varied data, derives chunks, entities, concepts, and ontologies, then combines graph and vector retrieval for agent memory. | Treat knowledge as an evolving substrate, not a flat pile of notes. | Rebuilding a generic ingestion platform, graph database matrix, and cloud product inside a coding-memory release. |
+| [Karpathy's LLM Wiki](https://gist.githubusercontent.com/karpathy/442a6bf555914893e9891c11519de94f/raw/) | Keeps immutable raw sources, a generated interlinked Markdown wiki, and a schema that teaches the agent how to ingest, query, and lint it. | Make knowledge a durable, human-readable artifact that compounds, cites its sources, and can be linted. | Pretending every retrieved chunk is a wiki page or hiding synthesized claims in a database only. |
+| [Graphiti / Zep](https://help.getzep.com/graphiti/getting-started/overview) | Builds temporal Context Graphs from episodes, tracks fact invalidation, and combines temporal, keyword, semantic, and graph retrieval. | A fact must have provenance and a lifecycle: current, superseded, disputed, or unknown. | Adding a graph database just to claim a temporal graph before we have trustworthy claims and invalidation rules. |
+| [Mem0](https://github.com/mem0ai/mem0) | Uses compact memory APIs, entity linking, temporal retrieval, and multi-signal ranking. | Evaluate retrieval quality and fuse independent signals rather than trusting one similarity score. | Treating vendor benchmark scores as directly comparable to a local coding-agent workflow. |
+| [Letta](https://docs.letta.com/guides/core-concepts/memory/context-hierarchy/) | Separates always-visible blocks, files, archival memory, and external retrieval by size and mutability. | Give every knowledge item an access mode: core protocol, on-demand page, archival evidence, or external source. | Injecting every permanent mini-skill or wiki page into every session. |
+| [LangMem](https://langchain-ai.github.io/langmem/) | Separates agent-managed hot-path memory from background extraction and consolidation. | Keep foreground task work fast; larger synthesis and maintenance must be bounded background work. | Coupling Memorix to a particular agent runtime or framework store. |
+| [TencentDB Agent Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory) | Uses layered memory, readable Markdown upper layers, traceable drill-down, background extraction, and skill/SOP generation from execution history. | Store high-level guidance as readable artifacts that drill down to evidence; let repeated successful work propose a workflow. | Making Mermaid the canonical truth or accepting self-reported benchmark gains without independent evaluation. |
+| [Supermemory](https://supermemory.ai/docs) | Treats memory, extraction, connectors, and retrieval as one context stack scoped to an entity. | Make scope explicit: project, workspace, team, workflow, and task are different containers. | Becoming an all-in-one hosted content connector platform in 1.2. |
+| [Memobase](https://github.com/memodb-io/memobase) | Separates durable user profile from event timeline and processes buffers outside the hot path. | Separate stable project protocol from episodic task evidence; batch maintenance rather than blocking the user. | Optimizing Memorix around consumer user-profile personalization instead of project work. |
+
+The social-media ranking that prompted this research is useful as a discovery
+list, but it is not a common benchmark or a product-fit analysis. Each project
+optimizes for a different unit of memory: user profile, agent runtime, generic
+documents, enterprise data, or project work.
+
+## Honest Audit of the Current Memorix Baseline
+
+Memorix already has real foundations. It is not empty branding. It has SQLite
+storage, observations, Git evidence, mini-skills, a code index, code-bound
+freshness, a dashboard Knowledge Base view, and a graph view.
+
+It is nevertheless not yet an industry-grade semantic knowledge graph or a
+Karpathy-style LLM Wiki:
+
+1. **CodeGraph Lite is a structural index, not a semantic knowledge graph.**
+   It records files, symbols, and primarily import-level edges. Its built-in
+   provider uses conservative language-specific regular expressions, which is a
+   valid fallback but not parser-grade code intelligence.
+2. **The current Knowledge Base is a generated view, not a maintained wiki.**
+   It groups active observations and mini-skills into sections at render time.
+   It does not persist interlinked Markdown pages that are updated when a new
+   source changes the project's synthesis.
+3. **The current Knowledge Graph is a deterministic memory map.** It creates
+   nodes from observations and mini-skills, then infers many edges from shared
+   entity name, category, source skill, or optional explicit relations. It has
+   useful provenance refs, but it has no claim identity, temporal validity,
+   confidence policy, contradiction state, or source-level truth-maintenance
+   loop.
+4. **Workflow support is format conversion, not workflow inheritance.** Existing
+   workspace sync can convert Windsurf workflow files to selected agent formats.
+   Mini-skills preserve small instructions, but they are not versioned playbooks
+   with inputs, steps, gates, outcomes, or learning from runs.
+
+The right response is not to delete these foundations. They become the evidence
+and compatibility layers for the next system.
+
+## Product Decision: Knowledge Workspace
+
+Memorix 1.2 will build a local, project-scoped Knowledge Workspace with four
+separate layers:
+
+~~~mermaid
+flowchart LR
+    A["Evidence: Git, code, tests, documents, observations"] --> B["Claims: provenance and validity"]
+    B --> C["Wiki: interlinked, human-readable knowledge"]
+    B --> D["Workflows: reusable project playbooks"]
+    C --> E["Task Workset: minimal context for the active task"]
+    D --> E
+    E --> F["Agent action and verification"]
+    F --> A
+~~~
+
+### Evidence
+
+Evidence includes Git commits, code-state snapshots, test or command results,
+approved documents, and high-signal observations. It carries source references
+and capture time. A synthesized page never replaces its evidence.
+
+### Claims
+
+A claim is a small, inspectable assertion, not merely a graph edge. It records:
+
+- subject, predicate, and object or value;
+- project, workspace, team, workflow, or task scope;
+- source and optional code references;
+- observed time and validity window;
+- active, superseded, disputed, or unknown status;
+- confidence.
+
+Claims give Memorix the temporal honesty borrowed from Graphiti without
+pretending that a visual graph alone is a knowledge system.
+
+### Wiki
+
+The wiki consists of maintained Markdown pages with structured frontmatter and
+links. A page can cover an architecture concept, a subsystem, a decision, a
+release procedure, or a known risk. Each page identifies its claims, evidence,
+code-state snapshot, status, and last review.
+
+The project can choose a versioned workspace for these pages so humans can read
+them, agents can update them through reviewed patches, and Git can show how the
+knowledge changed. SQLite and search indexes accelerate retrieval but are not
+the only copy of the knowledge.
+
+### Workflows
+
+A workflow is not an oversized AGENTS.md paragraph. It is a canonical,
+versioned playbook containing:
+
+- trigger, scope, assumptions, and required context;
+- ordered phases with optional branches;
+- allowed tools, guardrails, expected outputs, and verification gates;
+- source, claim, and code references;
+- target-agent adapters;
+- run evidence, outcome, and failure reason.
+
+For example, a project can own a bug-fix workflow, release workflow, migration
+workflow, or incident workflow. A new agent receives only a small trigger and
+the relevant first phase. It loads the full playbook only when the task matches.
+Repeated successful runs can propose improvements, but never silently rewrite a
+project protocol.
+
+## How Workflow Inheritance Feels to a User
+
+The user says, "Prepare the next release." Memorix should recognize a relevant
+project release playbook, return the current release facts and first gate, and
+let the agent continue using the normal tools for that environment. The user
+does not remember a slash command or manually paste a checklist.
+
+This is progressive disclosure:
+
+1. agent integration sees a short workflow trigger and description;
+2. task selection chooses at most one or two relevant workflows;
+3. the active Workset contains only the immediate phase, linked knowledge, and
+   verification gate;
+4. detailed procedures, source pages, and raw evidence are loaded on demand.
+
+Agents that support Skills receive a native skill adapter. Other MCP clients
+receive the same canonical instructions as prompt-ready context. Memorix must
+not rewrite a user's global agent configuration or claim a workflow is executed
+when the host does not expose the required hooks or tools.
+
+## 1.2 Scope and Non-Goals
+
+The 1.2 goal is **project knowledge that makes agents better at continuing
+work**, not a general company data lake.
+
+In scope:
+
+- local evidence, claims, wiki pages, and workflows with source drill-down;
+- code- and Git-aware freshness for project knowledge;
+- versioned project workflows with progressive agent delivery;
+- linting for missing sources, stale claims, broken links, conflicts, orphan
+  pages, and unverified workflow gates;
+- a dashboard that explains why a page or workflow was selected.
+
+Out of scope for 1.2:
+
+- cloning Cognee's multi-format enterprise ingestion breadth;
+- requiring Neo4j, a cloud API, or an external CodeGraph installation;
+- capturing every conversation as knowledge;
+- automatic writes to source-controlled knowledge without a clear project
+  policy and audit trail;
+- running a workflow engine inside every supported agent.
+
+## Revised Delivery Order
+
+1. **Truth baseline:** make the existing graph and wiki labels honest; add
+   fixtures that distinguish a memory map, a structural code index, and a
+   source-backed claim.
+2. **Evidence and Code State:** ship CodeStateSnapshot and completeness
+   semantics before letting generated knowledge claim current truth.
+3. **Claim ledger:** add provenance, validity, conflict, and source drill-down
+   for knowledge assertions.
+4. **Wiki compiler:** ingest approved sources into reviewable, interlinked
+   Markdown pages; add index, log, and lint rather than only search.
+5. **Workflow inheritance:** introduce canonical playbooks, task selection,
+   agent adapters, and run evidence.
+6. **Workset integration:** rank code state, claims, wiki pages, workflows, Git
+   evidence, and verification hints into project context.
+7. **Lifecycle and proof:** keep maintenance non-blocking, expose diagnostics,
+   and measure whether new agents start useful work faster with less broad
+   repository exploration.
+
+## Acceptance Test
+
+A newcomer can enter a project, say "continue the database migration issue,"
+and receive a compact, evidence-backed answer that names the right starting
+code, the current state of the migration, relevant decisions, a warning about
+superseded advice, the matching migration workflow if one exists, and the
+verification gate. Every recommendation can be drilled back to its source.
+
+If Memorix cannot establish that chain of evidence, it must give a smaller
+answer and say what it does not know.

package/docs/1.2.0-KNOWLEDGE-WORKSPACE.md

@@ -0,0 +1,68 @@
+# Memorix 1.2 Knowledge Workspace
+
+The Knowledge Workspace is the human-readable layer above the Claim Ledger.
+It turns approved, source-backed claims into reviewable Markdown rather than
+silently treating a generated summary as project truth.
+
+## Storage modes
+
+Local mode is the default. Its Markdown lives below the Memorix data directory,
+scoped by a stable project hash. It stays private to the local user.
+
+Versioned mode is explicit. The caller must supply an absolute workspace path
+inside the Git project. Memorix rejects paths under Git internals, common build
+directories, and paths Git reports as ignored. It never assumes that a hidden
+project directory is versioned.
+
+Both modes use the same layout:
+
+    schema.md
+    index.md
+    log.md
+    pages/
+    proposals/
+    workflows/
+
+SQLite stores workspace and page metadata for fast retrieval and diagnostics.
+Markdown remains the readable artifact. Memorix never copies all source files
+or all conversations into either one.
+
+## Review model
+
+Compilation is proposal-first:
+
+1. approved active claims are grouped into a topic page candidate;
+2. the candidate is written below proposals;
+3. a proposal records source claim ids, evidence ids, and code snapshot ids;
+4. an explicit apply operation writes the corresponding pages entry.
+
+An existing page whose content differs from the last applied content hash is
+treated as manually edited. Compilation leaves it untouched and writes a new
+proposal instead. Applying a proposal rechecks that base hash before the atomic
+replacement, so a concurrent human edit cannot be overwritten.
+
+The schema, index, and append-only log are Memorix-owned workspace artifacts.
+Topic pages are user-reviewable knowledge artifacts.
+
+## Page contract
+
+Each page uses YAML frontmatter with a stable id, title, kind, status, review
+state, claim ids, evidence references, code snapshot reference, tags, source
+hash, and timestamps. The body is compact: current claims, evidence drill-down,
+and internal links. It does not embed raw source or long conversation history.
+
+## Linting
+
+Workspace lint reports:
+
+- malformed frontmatter;
+- broken internal Markdown links;
+- pages that are not referenced by the index;
+- claim ids with no evidence;
+- superseded primary claims;
+- unresolved claim conflicts;
+- stale snapshot references;
+- missing proposal targets or manual-edit protection failures.
+
+Lint is advisory but visible. A broken or stale page cannot be silently treated
+as current by the future Workset builder.

package/docs/1.2.0-PRODUCT-STORY.md

@@ -0,0 +1,234 @@
+# Memorix 1.2 Product Story
+
+Status: active product narrative
+Companion: [1.2 Development Charter](1.2.0-DEVELOPMENT-CHARTER.md)
+Start: 2026-07-17
+
+## The Promise
+
+Memorix 1.2 makes a project easier to resume than to rediscover.
+
+When an agent joins a task, it should not begin with a blank chat, a giant
+transcript, or a demand that the user remember the right command. It should
+receive a small, trustworthy starting map: what this task touches, what changed,
+what the last people learned, what may no longer be true, and how to prove the
+next change works.
+
+Memorix does not replace reading code or making engineering judgments. It makes
+the first useful read and the first useful action arrive sooner.
+
+It also helps a project preserve the way it works: the release gates, migration
+procedures, debugging habits, and hard-won project protocols that should not
+disappear when a new agent joins.
+
+## The User Story
+
+Someone opens an existing project and writes a normal request:
+
+> Continue the timeout fix after the embedding-provider switch.
+
+The useful experience is not a list of memory commands. Before the agent starts
+wandering through the repository, it can ask Memorix for project context in the
+same natural-language task. Memorix returns a compact **Workset**:
+
+1. **Start here**: the few files or symbols most likely to matter, with a reason.
+2. **What is true now**: the current revision, relevant local changes, and code
+   facts that were actually indexed.
+3. **What the project learned**: only durable decisions, previous failures, or
+   handoff facts relevant to this task.
+4. **What to distrust**: old memories whose files changed, incomplete scans, or
+   conflicting evidence.
+5. **How to finish safely**: focused tests, commands, or checks tied to the
+   affected area.
+
+The agent still reads the source and decides how to fix the problem. The user
+does not need to learn a graph query, a retrieval mode, or a storage model. If
+Memorix has no useful advantage, it stays quiet instead of padding the context.
+
+## What Feels Automatic, and What Is Actually Happening
+
+The user experience should be simple: state the task normally. Existing agent
+guidance, plugins, or supported lifecycle hooks tell an agent that a
+non-trivial task is a good time to request `memorix_project_context`, passing
+that same task text. The agent receives a Workset and continues normally.
+
+Memorix does not secretly read every prompt, force a tool call for trivial
+work, or claim that all clients expose the same lifecycle events. An integration
+without hooks still relies on MCP guidance and the agent's tool choice. 1.2
+improves the relevance, freshness, and restraint of the returned Workset; it
+does not pretend to provide universal invisible automation.
+
+## A Project Is Also a Knowledge Base
+
+The product cannot stop at task handoff. Over time, a useful project develops
+its own vocabulary, architecture rationale, operational procedures, test
+conventions, release gates, and recurring failure patterns. Those things are
+not just old chat text. They are knowledge that agents should be able to
+inspect, update, and reuse.
+
+1.2 makes that knowledge a first-class **Knowledge Workspace**:
+
+- immutable or traceable evidence from code, Git, tests, documents, and
+  high-signal observations;
+- small claims that say what is believed, why, and whether it is still current;
+- interlinked Markdown wiki pages that people can read and agents can maintain;
+- reusable project workflows that describe how work should be performed and
+  verified.
+
+The wiki is not a chat export and the workflow is not a giant global prompt.
+Both are compact, source-backed project artifacts that load only when relevant.
+
+## Workflow Inheritance
+
+Every serious project has a way of working: how to investigate a bug, migrate a
+database, cut a release, recover from an incident, or review a risky change.
+1.2 treats those as versioned project playbooks rather than accidental habits
+buried in sessions or agent-specific configuration files.
+
+When a task matches a workflow, the agent receives its immediate first phase,
+relevant knowledge pages, and verification gate. It can drill into the full
+playbook only when needed. A successful run produces evidence; repeated,
+reviewed evidence can propose a better workflow. It never silently rewrites the
+project's process.
+
+## The Change From 1.1
+
+1.1 preserves project knowledge well: session notes, decisions, Git facts,
+skills, hooks, search, and a lightweight code-memory layer. That remains useful,
+but a stored note alone cannot prove that it still matches the code in front of
+an agent.
+
+1.2 changes the center of gravity from **remembering old text** to **assembling
+current evidence for one active task**.
+
+| Question | 1.1 foundation | 1.2 destination |
+| --- | --- | --- |
+| What did we learn? | Searchable observations and Git facts | Keep only task-relevant durable knowledge |
+| Where should work start? | Code Memory can suggest files | A ranked, source-backed start map |
+| Is an old note still safe? | Freshness labels on code references | Revision- and worktree-aware evidence quality |
+| How does a user use it? | Commands and MCP tools are available | A normal task produces a compact Workset |
+| What happens after code changes? | Refresh and hooks exist | Evidence is invalidated or refreshed without blocking work |
+
+## The Four Evidence Planes
+
+Memorix does not store a second copy of the repository. It keeps four different
+kinds of evidence separate, then combines only what the task needs.
+
+| Plane | Plain-language job | Example |
+| --- | --- | --- |
+| Narrative memory | Preserve the reasoning humans and agents would otherwise lose | "The timeout must cover both embedding retries and fallback." |
+| Code state | Say where that reasoning connects to the code that exists now | `src/config/embedding.ts`, `resolveEmbeddingLane()` |
+| Change evidence | Explain what moved since a fact was recorded | a new commit, a dirty config file, a deleted symbol |
+| Verification evidence | Show how the next agent can know its work is correct | the focused timeout test and its last result |
+
+The resulting Workset is not a transcript and not a code dump. It is closer to a
+good handoff written by someone who can see the current repository.
+
+The Knowledge Workspace sits above these evidence planes. It turns source-backed
+claims into maintained wiki pages and reusable workflows. The Workset reads from
+it; it does not blindly copy its contents into every task.
+
+## What "Dynamic" Means
+
+Dynamic does not mean constantly re-indexing the whole repository or silently
+inventing conclusions. It means a fact carries enough identity to be checked
+against the project that exists now.
+
+A **CodeStateSnapshot** is the receipt for that check. It records the Git
+revision when available, a compact fingerprint of local uncommitted changes,
+when indexing happened, which provider produced the facts, and whether paths or
+languages were skipped. It does not store all source code again.
+
+That lets Memorix make honest distinctions:
+
+- "This decision still points at unchanged code."
+- "The referenced file changed; use this note as a lead, not as truth."
+- "The scan was incomplete, so this start map may have blind spots."
+- "A local edit happened after the last snapshot; refresh only the affected
+  evidence when possible."
+
+The important behavior is graceful: foreground work gets the best current
+answer quickly, while larger refresh work stays bounded and non-blocking.
+
+## The Product Loop
+
+```mermaid
+flowchart LR
+    A["Agent receives a normal task"] --> B["Memorix builds a compact Workset"]
+    B --> C["Agent reads the right code and acts"]
+    C --> D["Git, tests, hooks, and explicit notes create evidence"]
+    D --> E["Code state and memory freshness are updated"]
+    E --> A
+```
+
+The loop is valuable only when it reduces repeated exploration. It must never
+turn every prompt, file read, or chat line into permanent memory.
+
+## Delivery Story
+
+### Act 1: Establish Trust
+
+Create an evaluation harness and versioned code-state snapshots. Before richer
+parsers or external graph integrations, Memorix must prove whether a fact came
+from the current project, a changed project, or an incomplete scan.
+
+### Act 2: Build the Smallest Useful Starting Map
+
+Teach `memorix_project_context` to rank current code, Git evidence, durable
+knowledge, and verification hints into a bounded Workset. The default should be
+short enough to help an agent begin, not long enough to replace its context
+window.
+
+### Act 3: Let Project Knowledge Compound
+
+Build a source-backed claim ledger and a reviewable Markdown wiki. New evidence
+updates the project's synthesis, flags contradictions, and preserves a path back
+to source code, Git facts, tests, or approved documents.
+
+### Act 4: Let Agents Inherit Useful Workflows
+
+Build canonical project playbooks for repeatable work. Adapt them to supported
+agent skill and rule formats, but keep one project-owned source of truth and
+load only the relevant phase for a task.
+
+### Act 5: Keep the Map Honest While Work Continues
+
+Use lightweight lifecycle signals and Git/worktree changes to mark evidence
+current, suspect, stale, or incomplete. A slow scan or an unavailable optional
+provider must not block an agent from working.
+
+### Act 6: Improve Structural Quality Without Lock-in
+
+Benchmark parser-backed and external CodeGraph-class providers on Windows,
+package size, fidelity, and recovery behavior. Adopt a provider only when it
+can be normalized, versioned, diagnosed, and safely replaced by the built-in
+path.
+
+### Act 7: Prove the Product Helps
+
+Test coding, debugging, onboarding, refactor, and non-code tasks with and
+without a Workset. Measure faster first useful action, fewer broad scans,
+correct stale-context warnings, and no unnecessary context cost. A feature that
+looks sophisticated but does not improve those outcomes does not belong in 1.2.
+
+## What 1.2 Will Not Become
+
+- A full IDE, language server, or code search replacement.
+- A mandatory third-party CodeGraph installation.
+- A generic company data lake or a clone of Cognee's broad ingestion platform.
+- A second chat-history archive that captures everything by default.
+- A huge MCP toolbox that makes every agent pay for graph features it will not
+  use.
+- A black box that hides whether a recommendation came from current code,
+  older memory, or an incomplete scan.
+
+## The Release Test
+
+1.2 is ready when a capable new agent can enter an unfamiliar project, receive
+a small and truthful workset from a normal task, locate the right starting code,
+understand relevant project history, notice stale information before trusting
+it, apply the relevant project workflow when one exists, and run the right
+verification without the user teaching it a Memorix ritual.
+
+If the project offers no reliable advantage over an ordinary file read,
+Memorix should say less, not more.