memorio 5.1.4 → 5.2.0

package/markdown/MEMORY-ATTACHMENT.md

@@ -1,12 +1,19 @@
-> **Status:** Published
-> **Date:** 2026-09-12
-> **Scope**: API Reference
-> **Standard**: Memorio API Specification v5
->
----
-# Node Attachment System
-
-`memorio.memory` extends with a **Node Attachment System** allowing memory elements to dynamically attach, reference, or connect to other memory elements at runtime.
+> **Status:** Proposed (not yet implemented)
+> **Date:** 2026-09-12
+> **Scope**: API Reference
+> **Standard**: Memorio API Specification v5
+>
+> **Not implemented.** The Node Attachment System described in this document is a
+> proposed specification. None of the APIs below - `node.connect`, `node.attach`,
+> `node.related`, `node.traverse`, `memory.connect`, `memory.disconnect` - are
+> present in the current `MemoryAPI`. Additionally, `memory.remember()` returns
+> `Promise<void>`, not a node object with relationship methods. This document is
+> kept as a design reference until the feature is implemented.
+>
+---
+# Node Attachment System (proposed)
+
+`memorio.memory` is specified to extend with a **Node Attachment System** allowing memory elements to dynamically attach, reference, or connect to other memory elements at runtime.
 
 This is an **extension**, not a replacement, of existing memory semantics.
 
@@ -92,4 +99,4 @@ Nodes serialize to IDs only - never recursively embedded graphs.
 
 ## Backward Compatibility
 
-Existing APIs (`remember`, `get`, `forget`, `search`) continue to work unchanged. A node-enabled memory entry remains usable as a normal memory entry.
+Existing APIs (`remember`, `recall`, `forget`, `context`) continue to work unchanged. A node-enabled memory entry remains usable as a normal memory entry.

package/markdown/MEMORY.md

@@ -1,14 +1,114 @@
-> **Status:** Published
-> **Date:** 2026-09-12
-> **Scope**: API Reference
-> **Standard**: Memorio API Specification v5
->
----
-# Memory System
-
-`memorio.memory` provides a semantic memory layer - a key/value store with type safety, TTL, confidence scoring, tagging, and scope-based persistence.
-
-## Core API
+> **Status:** Published
+> **Date:** 2026-09-12
+> **Scope**: API Reference
+> **Standard**: Memorio API Specification v5
+>
+---
+# Memory System
+
+`memorio.memory` provides a semantic memory layer - a key/value store with type safety, TTL, confidence scoring, tagging, and scope-based persistence.
+
+## Acquired-experience discovery
+
+`memory.discover()` finds a small, deterministic set of plausible **active acquired-experience candidates** from ordinary situation text. The caller does not need to know applicability keys or claim IDs.
+
+```ts
+const result = await memory.discover('proxy target locking mutation')
+
+for (const candidate of result.candidates) {
+  console.log(candidate.claim.id)
+  console.log(candidate.explanation.matchedTerms)
+}
+```
+
+The optional retrieval depth controls breadth, not a different memory store:
+
+```ts
+await memory.discover('proxy locking', { depth: 'current' }) // at most 5 by default
+await memory.discover('proxy locking', { depth: 'deep' })    // at most 25 by default
+await memory.discover('proxy locking', { maxCandidates: 10 })
+```
+
+Discovery is deliberately distinct from the existing acquired-context selector:
+
+- **Discovery** uses normalized lexical overlap to find candidates. It can return zero candidates.
+- **Applicability selection** (`memory.context({ acquired: ... })`) keeps exact structured matching and most-specific-wins behavior.
+- **Context projection** (`memory.project()`) is a separate synchronous step that turns a Discovery result into bounded transferable context.
+
+Normalization is local and deterministic: Unicode NFKC normalization, locale-stable lowercase conversion, punctuation/symbol separation, whitespace normalization, removal of duplicate terms, and omission of terms shorter than three characters. Stored claims are never rewritten.
+
+Candidate ordering uses, in order: a direct normalized phrase match, the number of distinct matching query terms, the kinds of fields matched, acquisition recency, and stable claim ID. Recency only breaks relevance ties; old claims are not expired or excluded. Result explanations list the exact matched terms and fields and include the `.mem` source identifier when the repository provides one. These are discovery facts—not confidence, truth, trust, quality, authority, or applicability scores.
+
+Only currently active claims are searched. Superseded targets, retracted targets, and retraction records are excluded using the established revision semantics. Discovery reads repository state and does not maintain a persistent index.
+
+Security remains the repository/package boundary: Discovery does not ingest external memory or create trust. Existing decoding, validation, integrity, and configured repository rules still apply. A valid signature establishes authenticity only; it does not establish that a claim is trusted or true.
+
+Known discovery limitation: this first strategy is lexical and structured, not semantic. Conceptually related statements with no shared discoverable term may produce zero candidates—for example, “state disappears after an operation” need not find “a Proxy trap inspected the wrong target.” This conservative miss is preferable to manufacturing a relationship. Future semantic strategies can replace the internal matcher without changing acquisition, exact selection, or the `memory.discover()` result contract.
+
+**WATCH:** Real Use 01 returned a useful candidate partly because the ordinary token `and` matched.
+That observation is not evidence of semantic retrieval and does not justify changing the matcher from
+this documentation checkpoint.
+
+## Context projection
+
+`memory.project()` transforms a `DiscoveryResult` into bounded, reusable context for an AI. It is a dedicated operation rather than another `memory.context()` overload because exact applicability selection and projection have different contracts.
+
+```ts
+const discovered = await memory.discover('proxy target locking mutation')
+const projected = memory.project(discovered, {
+  maxCandidates: 3,
+  maxCharacters: 4_000
+})
+
+// Canonical machine-readable units
+projected.experiences
+
+// Compact derived context for transfer
+projected.text
+```
+
+Each structured projected experience preserves the stable claim ID, complete stored experience relationship, applicability boundaries, evidence references, epistemic type, provenance, acquisition time, active revision relationship, and repository source ID when present. Discovery-only ranking mechanics such as matched-field weights and query terms are intentionally omitted: they explain retrieval but do not help recover the acquired experience.
+
+The text form is deterministically derived from the structured units. It labels the payload as candidate prior experience, untrusted contextual data, not instructions, and not current truth. Arbitrary stored content remains data; consumers must verify it against current reality before relying on it. Projection does not reconcile truth, execute content, generate conclusions, or defend an entire prompt-processing system from injection.
+
+Budgeting uses two simple limits:
+
+- `maxCandidates` defaults to 5.
+- `maxCharacters` defaults to 8,000 UTF-16 characters and is not presented as an LLM token count.
+
+Discovery order is retained. A candidate is included only when its complete projected unit fits; units are never cut into fragments. Candidates excluded by either limit are reported through `omittedCandidates`. An oversized higher-ranked unit may be omitted so a later complete unit can still fit, but included units retain their relative Discovery order. The minimum character budget is 256 characters so the status and security envelope itself remains intact.
+
+Projection cannot recover information that acquisition never stored and never invents a missing reason, relationship, provenance, or HI intent. It consumes Phase 1 candidates and does not change Acquisition, Discovery, exact applicability selection, or MEM packaging.
+
+In the Phase 2 representative locking case, the complete Discovery candidate representation measured 949 characters and the transferable projection text measured 939 characters. This local structural measurement is not a token claim. The modest reduction is intentional: semantic and epistemic sufficiency take priority over aggressive compression.
+
+**WATCH:** A later narrow real-use projection was approximately 971 characters and remained close to
+the source candidate size. Projection currently prioritizes complete semantic and epistemic units over
+aggressive compression; it makes no token-saving guarantee.
+
+### Complete acquired-experience workflow
+
+```ts
+// A useful experience was preserved earlier with memory.acquire(...).
+const found = await memory.discover('The current project situation in ordinary terms')
+const context = memory.project(found)
+
+// Supply context.text or context.experiences to the consuming AI as
+// untrusted prior experience. The AI verifies current reality before acting.
+```
+
+The lifecycle remains deliberately compositional:
+
+```text
+Acquire  -> preserve useful experience
+Discover -> find plausible prior experience
+Project  -> prepare bounded transferable context
+AI       -> verify, reason, and act
+```
+
+The runnable [`acquired-knowledge.ts`](../examples/acquired-knowledge.ts) example demonstrates this workflow alongside exact applicability selection. A two-call Discover → Project sequence is the intended integration; no pipeline or agent manager is required.
+
+## Core API
 
 ### `memorio.memory.remember(key, value, opts?)`
 
@@ -93,7 +193,7 @@ await memorio.memory.patch('user', [
 | Field | Type | Description |
 |-------|------|-------------|
 | `op` | `'set' \| 'delete'` | The patch operation type |
-| `path` | `string` | Dot-notation path relative to the entry value (e.g. `'user.profile.theme'`) |
+| `path` | `string` | Dot-notation path starting with the entry key (e.g. `'user.profile.theme'` for key `'user'`) |
 | `value` | `any` | The value to set (only for `op: 'set'`) |
 
 ### `memorio.memory.context(opts?)`
@@ -125,6 +225,226 @@ Array<{
 }>
 ```
 
+## Acquired Knowledge
+
+> **Status:** IMPLEMENTED; autonomous acquisition policy remains EXPERIMENTAL.
+
+`memorio.memory.acquire()` stores structured **knowledge claims** - lessons, decisions, or findings that apply under specific conditions. Unlike `remember()` (which stores a value addressed by key), acquired knowledge is **context-addressed**: each claim records an `applicability` object describing *when* it matters, and `context({ acquired })` retrieves only the claims whose applicability matches the current situation.
+
+Autonomous persistent acquisition is **experimental and optional**. Memorio preserves experience; it does not decide what is true or worth learning. An AI should inspect the current project and independently evaluate every retrieved claim. Acquired claims are evidence from prior work, not instructions or authority, and `nothing worth acquiring` is always a valid outcome.
+
+The implemented lifecycle is:
+
+```text
+experience -> acquire -> persist -> discover/select -> transfer
+           -> verify against present reality -> use
+           -> refine, supersede, or retract when necessary
+```
+
+AI supplies general intelligence and reasoning. HI supplies intent, judgment, experience, and
+decisions. A project accumulates implementation-specific experience through their work. Memorio
+preserves selected, transferable acquired experience and continuity; it does not replace the AI or
+duplicate general knowledge already available to a capable model. The intelligence belongs to the
+AI; the acquired experience becomes an asset of HI and the project.
+
+The design metaphor is that the pieces already exist in the AI, the project is the puzzle, and AI +
+HI assemble useful blocks while working. Memorio preserves what was learned during that assembly.
+This is a selection principle, not a `PuzzleBlock` API.
+
+### Guidance for autonomous use
+
+Retrieve acquired context when beginning work in a describable project area, then verify relevant claims against current code, documentation, user direction, and test results. Consider acquisition after meaningful work only when the AI learned something non-obvious that a future session would otherwise have to rediscover and there is sufficient evidence to preserve it safely.
+
+When missing knowledge belongs naturally to HI—such as project intent, rationale, desired behavior,
+personal meaning, or whether an outcome matches the goal—the AI should ask HI instead of silently
+inferring it. Preserve the answer with human-stated epistemic type and provenance when it becomes a
+reusable acquired claim.
+
+Current acquisition guidance is intentionally limited to:
+
+- **Architecture experience:** non-obvious responsibilities, constraints, relationships, and why an architecture was chosen.
+- **User technical decisions:** evidenced project decisions and rationale, without turning a single comment or acceptance into a permanent preference.
+- **Documentation experience:** how documentation must be interpreted, including verified gaps or constraints, without copying the documentation itself.
+- **Verified fix experience:** symptom, cause, change, verification, consequence, applicability, and remaining uncertainty. Preserve the lesson, not a stale assertion that the fixed bug still exists.
+
+The practical test is whether experience created a useful connection worth recovering later. Preserve
+relationships such as problem → cause → fix → verified consequence; decision → reason → boundary; or
+assumption → evidence → correction. Do not use acquired knowledge as a dump for generic knowledge,
+source-code or documentation copies, entire conversations, routine operations, test logs, or hidden
+chain-of-thought.
+
+Do not acquire repository facts that code, Git, or current documentation already express adequately. Keep these stages distinct in the claim content and evidence:
+
+```text
+observation -> what was directly encountered
+inference   -> the AI's interpretation, explicitly uncertain
+verified    -> what a named check or result demonstrated
+experience  -> the reusable why, consequence, or constraint selected for persistence
+```
+
+A practical claim can encode those distinctions in `knowledge`, for example `{ observation, inference, verifiedResult, experience, uncertainty }`. This is guidance for the existing JSON-valued field, not a new required schema. Use evidence references to identify the supporting review, test, user decision, or investigation. If reality has changed, ignore the claim or acquire a `refines`/`supersedes` revision; never force current reality to fit memory.
+
+> **TODO - Autonomous Memory Safety:** Before autonomous acquisition is declared stable, decide what must never be acquired; how secrets, personal data, and unsafe content are excluded; when notification or approval is required; and how users inspect, reject, correct, forget, and assess the provenance, uncertainty, and staleness of acquired claims. The current API does not solve these policy questions.
+
+```ts
+await memorio.memory.acquire({
+  id: 'layout.aside-width',
+  knowledge: { guidance: 'Use the shared --aside-width token, not a fixed pixel value.' },
+  applies: { component: 'layout', element: 'aside' },
+  evidence: [{ id: 'review-17', source: 'design-review', observedAt: '2026-09-18T00:00:00Z' }]
+})
+```
+
+### `memorio.memory.acquire(opts)`
+
+| Option | Type | Required | Description |
+|--------|------|----------|-------------|
+| `id` | `string` | Yes | Unique claim identifier (immutable; cannot be reused) |
+| `knowledge` | `JsonValue` | Yes | The claim content - any JSON-serializable value |
+| `applies` | `Record<string, JsonPrimitive>` | Yes | Applicability context - the condition under which this claim is relevant |
+| `epistemicType` | `'observed' \| 'inferred' \| 'human-stated' \| 'verified'` | No | What kind of knowledge the claim represents |
+| `provenance` | `{ source: string; detail?: string }` | No | Origin of the acquired claim |
+| `acquiredAt` | `string` | No | Acquisition time; defaults to the current ISO timestamp |
+| `evidence` | `EvidenceReference[]` | Yes (≥1) | Evidence references justifying the claim |
+| `refines` | `string \| string[]` | No | Prior claim IDs this claim refines (more specific) |
+| `supersedes` | `string \| string[]` | No | Prior claim IDs this claim replaces (mutually exclusive with `refines`) |
+| `retracts` | `string \| string[]` | No | Prior claim IDs this claim withdraws (mutually exclusive with other revision kinds) |
+| `reason` | `string` | No | Why the revision relationship exists |
+
+**Evidence reference shape:**
+
+```ts
+interface EvidenceReference {
+  id: string        // observed-at identifier (required, non-empty)
+  source?: string   // where the observation came from
+  observedAt?: string // ISO timestamp of the observation
+  supports?: string // what proposition this evidence supports
+}
+```
+
+### `memorio.memory.context({ acquired })`
+
+Retrieves knowledge claims whose `applicability` exactly matches the provided query context. Both a single context object and an array of contexts are accepted.
+
+```ts
+// Single context - retrieves claims applicable to this situation
+const result = await memorio.memory.context({
+  acquired: { component: 'layout', element: 'aside' }
+})
+
+// Multiple contexts - each is queried independently
+const result2 = await memorio.memory.context({
+  acquired: [
+    { component: 'layout' },
+    { component: 'layout', element: 'aside' }
+  ]
+})
+```
+
+Returns:
+```ts
+{
+  knowledgeVersion: number   // current persisted knowledge state version
+  claims: AcquiredClaim[]    // matching claims, deduplicated
+}
+```
+
+**Claim shape:**
+```ts
+interface AcquiredClaim {
+  id: string
+  knowledge: JsonValue                         // the stored claim content
+  applicability: Record<string, JsonPrimitive> // the condition this claim applies to
+  evidence: EvidenceReference[]              // supporting evidence
+  epistemicType?: 'observed' | 'inferred' | 'human-stated' | 'verified'
+  provenance?: { source: string; detail?: string }
+  acquiredAt?: string
+  revision?: {
+    kind: 'refines' | 'supersedes' | 'retracts' // relationship type
+    claimIds: string[]                        // IDs of prior claims
+    reason?: string                           // why the revision exists
+  }
+}
+```
+
+### Matching semantics
+
+Selection follows two rules:
+
+1. **All applicability keys must match.** A claim with `applicability: { component: 'layout', element: 'aside' }` only matches a query that provides *both* `component: 'layout'` *and* `element: 'aside'`.
+2. **Most specific wins.** When multiple claims satisfy the same query, only those with the highest number of applicability keys are returned. Less-specific claims are excluded - they act as fallbacks only when no more-specific claim matches.
+
+This lets you store a general rule and a specific override:
+
+```ts
+// General rule
+await memorio.memory.acquire({
+  id: 'layout.general',
+  knowledge: 'Use layout tokens.',
+  applies: { component: 'layout' },
+  evidence: [{ id: 'e-general' }]
+})
+
+// Specific override
+await memorio.memory.acquire({
+  id: 'layout.aside',
+  knowledge: 'Use aside layout tokens.',
+  applies: { component: 'layout', element: 'aside' },
+  evidence: [{ id: 'e-aside' }],
+  refines: 'layout.general'
+})
+
+const general = await memorio.memory.context({ acquired: { component: 'layout' } })
+// general.claims → [{ id: 'layout.general', ... }]
+
+const aside = await memorio.memory.context({ acquired: { component: 'layout', element: 'aside' } })
+// aside.claims → [{ id: 'layout.aside', ... }] (more specific wins)
+// aside.claims[0].revision → { kind: 'refines', claimIds: ['layout.general'] }
+```
+
+### Revision: supersede vs. refine
+
+- **`supersedes`** - the new claim *replaces* the old one. Superseded claims are excluded from results; only the newer claim is returned for the same applicability context.
+- **`refines`** - the new claim is *more specific* than the old one. Both remain active; the more specific claim wins when its applicability matches fully. The `revision` field on the returned claim records the relationship so a future AI can follow the reasoning chain.
+- **`retracts`** - the new claim records withdrawal of prior claims. Both the retraction claim and the referenced claims are excluded from active selection, while history remains stored.
+
+```ts
+await memorio.memory.acquire({
+  id: 'layout.aside.width.v1',
+  knowledge: 'Width is 250px.',
+  applies: { component: 'layout', element: 'aside' },
+  evidence: [{ id: 'old-evidence' }]
+})
+
+await memorio.memory.acquire({
+  id: 'layout.aside.width.v2',
+  knowledge: 'Width is 300px (updated design token).',
+  applies: { component: 'layout', element: 'aside' },
+  evidence: [{ id: 'new-evidence' }],
+  supersedes: 'layout.aside.width.v1'
+})
+
+const result = await memorio.memory.context({ acquired: { component: 'layout', element: 'aside' } })
+// result.claims → [{ id: 'layout.aside.width.v2', ... }]  (v1 is excluded)
+```
+
+### Persistence
+
+Acquired knowledge is persisted so it survives process restarts and is available to future sessions:
+
+| Runtime | Storage |
+|---------|---------|
+| Node.js / Bun | `.memorio/project.mem` (standard ZIP package in the `.memorio` directory) |
+| Browser / Edge | `store` (localStorage-backed key `__memorio_acquired_knowledge_state__`) |
+
+The `.mem` file uses a versioned format: v1/v2 JSON remains readable and current writes are standard
+ZIP packages containing logical `memorio.knowledge` v2 plus SHA-256 integrity metadata. See
+[the format reference](./MEM_FORMAT.md). Because
+acquisition persists across sessions, applications and autonomous agents should make that behavior
+visible to users while it remains experimental. `memorio.memory.clear()` wipes all ordinary memories,
+acquired knowledge, and the index; it is destructive and should not be used as routine example cleanup
+or against a real project during tests.
+
 ### `memorio.memory.stats()`
 
 Returns usage statistics:
@@ -144,7 +464,50 @@ Cleans all expired entries. Returns count removed.
 
 ### `memorio.memory.clear()`
 
-Wipes all memories and the index.
+Wipes all memories, acquired knowledge, and the index.
+
+## Sync & Journal (optional)
+
+`memorio.memory` includes an optional local-first sync layer. See [`SYNC.md`](./SYNC.md) for the
+full guide.
+
+### `memorio.memory.configure(opts)`
+
+Enables the local operation journal and optionally registers a cloud-sync provider.
+
+```ts
+await memorio.memory.configure({
+  namespace: 'user:123:device:abc',   // partitions the journal (tenant/user/device)
+  provider: {                          // application-owned transport
+    push(ops) { /* ... */ },
+    pull?(since) { /* ... */ },
+    resolveConflict?(local, remote) { /* ... */ }
+  },
+  auto: true                           // auto-replay journal on focus/online (default)
+})
+```
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `namespace` | `string` | Tenant/user/device identifier partitioning the journal |
+| `provider` | `SyncProvider` | Application-owned object that pushes/pulls operations to your backend |
+| `direction` | `'up' \| 'down' \| 'both'` | Sync direction (default `'both'`) |
+| `auto` | `boolean` | Auto-replay journal on focus/online events (default `true`) |
+
+### `memorio.memory.journal`
+
+Local operation journal - a durable op log persisted on `store`. Only active when
+sync has been configured via `configure()`.
+
+| Method | Returns | Notes |
+|--------|---------|-------|
+| `append(entry, operation)` | `Promise<MemoryEntry>` | Records `remember \| update \| forget \| expire \| confirm \| supersede \| delete \| patch` |
+| `pending()` | `Promise<MemoryEntry[]>` | Entries where `sync !== 'synced'`, namespace-scoped |
+| `markSynced(ids)` | `Promise<number>` | Advances entries to `'synced'` |
+| `get(id)` | `Promise<MemoryEntry \| null>` | Single entry, namespace-scoped |
+| `clear()` | `Promise<void>` | Wipes the current namespace's journal only |
+| `replay()` | `Promise<SyncAck>` | Pushes pending entries to the provider, marks synced, optional `pull` |
+| `status()` | `Promise<'store'>` | The substrate in use (`'store'`) |
 
 ## Memory Entry Model
 
@@ -175,7 +538,7 @@ interface MemoryEntry<T = any> {
 | `local` | `localStorage` | ✅ | ✅ | ~10MB |
 | `durable` | `IndexedDB` | ✅ | ✅ | ~1GB+ |
 
-Default scope is `local` for values ≤100KB, `durable` for larger values.
+Default scope: `local` for values ≤1 MB, `durable` for values >1 MB. The `hot` and `session` scopes are only used when explicitly set via `opts.scope`.
 
 ## Memory Lifecycle
 
@@ -185,4 +548,4 @@ Default scope is `local` for values ≤100KB, `durable` for larger values.
 | Entry with TTL expires | Status → `obsolete` (cleaned by `forgetExpired()`) |
 | `recall()` expired entry | Returns `null` unless `includeObsolete: true` |
 | `update()` | Creates superseded copy + updated active entry |
-| `clear()` | Wipes all memories + index |
+| `clear()` | Wipes all memories, acquired knowledge + index |