memoir-cli 3.12.0 → 3.14.0

package/evals/CONTINUITY-PROTOCOL.md

@@ -0,0 +1,45 @@
+# Coding continuity comparison protocol — proposal v1
+
+This protocol defines future evaluation. It does not contain a completed user study, held-out task corpus, or competitor score. Existing `cases.json` is a development regression fixture; the retrieval performance corpus measures latency only.
+
+## Primary question
+
+Does memory help a returning developer or coding agent take the first appropriate action and finish a paused task, compared with the strongest practical baseline?
+
+The primary proposed outcome is elapsed reorientation time: from receiving the resumed task to the first action that a prewritten task verifier or a blinded reviewer considers appropriate. Count failures and timeouts explicitly. Task completion, repeated failed approaches, unsupported claims, and total operating cost are required secondary outcomes.
+
+## Freeze before testing
+
+- Start with 30 continuation tasks across several repositories and at least three languages. Treat this as a pilot. Split by repository/task family; near-duplicate histories must not cross development/evaluation boundaries.
+- Include decisions and rationale, changed code/branches, failed approaches, cross-client handoffs, conflicting updates, absent evidence, scope/deletion boundaries, and multilingual wording.
+- For each task, record the repository revision, prior session history, interruption point, permitted sources, valid and invalid next actions, completion verifier, timeout, and failure labels. Freeze a manifest hash before tuning prompts or retrieval on the evaluation tasks.
+- Keep the held-out material outside training/tuning fixtures. Publishing a protocol or generating new examples in the same development session does not make the examples independent.
+
+## Conditions
+
+Compare Memoir with native client memory enabled where available, a plain project plan/findings/progress baseline, and a small relevant external set: Claude-Mem, Engram, and Cass Memory first; EverOS for the portable-file/index comparison. Record the exact release/commit and actual integration support. A current default-branch README is not proof of a tested release.
+
+Use the same task, history, downstream model, permissions, context budget, and verifier where feasible. Record differences that cannot be normalized. Use isolated workspaces and memory stores for every condition to prevent contamination. Randomize/counterbalance condition order and repeat nondeterministic runs. Never feed a failed condition's discoveries into a later condition's starting state.
+
+## Required record for each run
+
+| Field | Meaning |
+|---|---|
+| task_id, task_manifest_hash, condition, repetition | Identity and reproducibility |
+| repository_commit, harness_commit, memory_version | Exact code and system revision |
+| model, embedding_model, judge, configuration_hash | Model/configuration differences |
+| start, first_appropriate_action, end, timeout | Timings; absent events must remain absent |
+| completion_verified, verifier_output | Outcome and evidence |
+| repeated_failed_approach, unsupported_memory, wrong_scope, stale_state_used | Failure labels with supporting traces |
+| ingestion_tokens, maintenance_tokens, retrieval_tokens, answer_tokens | Whole-lifecycle usage, not just the final prompt |
+| cost, retrieved_evidence_ids, memory_edits, setup_minutes | Cost, traceability, and operational burden |
+
+Use public/synthetic histories or explicitly consented, redacted user material. The project's distribution constraint remains SEO/automation only; obtain participants through inbound opt-in, not unsolicited outreach.
+
+## Analysis and decision
+
+Report paired differences, sample counts, medians, p95 where adequately sampled, confidence intervals, and failures by category. Retrieval Recall@k is a diagnostic and must not be relabeled answer accuracy or task completion.
+
+Proposed investment gate: at least 25% lower median reorientation time against the strongest tested baseline without worse completion or more serious memory errors. Pre-register the actual gate before running the pilot. If intervals remain wide or only a few tasks drive the result, expand the sample rather than declaring success.
+
+A release still requires the separate isolation, recovery, client acceptance, and cloud policy checks. No benchmark result compensates for a lost acknowledged write. Zero observed violations in a finite suite does not establish universal safety. A SOTA claim must name the precise task/configuration, release reproduction artifacts, and obtain independent confirmation.

package/evals/cases.json

@@ -0,0 +1,200 @@
+{
+  "version": 1,
+  "records": [
+    {
+      "key": "retry",
+      "text": "Retry failed recovery jobs with exponential backoff and a maximum of five attempts."
+    },
+    {
+      "key": "payments",
+      "text": "Payment webhooks require idempotency keys before changing account state."
+    },
+    {
+      "key": "database-a",
+      "text": "Project database choice is PostgreSQL; transactions protect recovery tokens."
+    },
+    {
+      "key": "database-b",
+      "project": "beta",
+      "text": "Project database choice is SQLite for the offline desktop client."
+    },
+    {
+      "key": "preference",
+      "project": "shared",
+      "text": "Prefer concise explanations with concrete examples."
+    },
+    {
+      "key": "alias",
+      "text": "The interface uses a vertical swipe feed.",
+      "aliases": [
+        "short video navigation"
+      ]
+    },
+    {
+      "key": "chinese",
+      "text": "数据库使用 PostgreSQL，备份每天运行。"
+    },
+    {
+      "key": "spanish",
+      "text": "La autenticación usa códigos de recuperación de un solo uso."
+    },
+    {
+      "key": "hidden",
+      "text": "hidden-canary-marker must never be recalled.",
+      "metadata": {
+        "hidden": true
+      }
+    },
+    {
+      "key": "expired",
+      "text": "expired-canary-marker used the old deployment endpoint.",
+      "metadata": {
+        "valid_until": "2020-01-01T00:00:00Z"
+      }
+    },
+    {
+      "key": "future",
+      "text": "future-canary-marker is not effective yet.",
+      "metadata": {
+        "valid_from": "2099-01-01T00:00:00Z"
+      }
+    },
+    {
+      "key": "superseded",
+      "text": "superseded-canary-marker is obsolete.",
+      "metadata": {
+        "superseded_by": "replacement"
+      }
+    },
+    {
+      "key": "negation",
+      "text": "Do not retry a consumed recovery token; issue a new token."
+    },
+    {
+      "key": "numeric",
+      "text": "Incident 42 affected retry workers; incident 43 affected webhooks."
+    }
+  ],
+  "cases": [
+    {
+      "name": "exact evidence",
+      "query": "idempotency",
+      "expected": [
+        "payments"
+      ]
+    },
+    {
+      "name": "multi-term evidence",
+      "query": "exponential backoff",
+      "expected": [
+        "retry"
+      ]
+    },
+    {
+      "name": "alpha scope",
+      "query": "database choice",
+      "expected": [
+        "database-a"
+      ],
+      "forbidden": [
+        "database-b"
+      ]
+    },
+    {
+      "name": "beta scope",
+      "project": "beta",
+      "query": "database choice",
+      "expected": [
+        "database-b"
+      ],
+      "forbidden": [
+        "database-a"
+      ]
+    },
+    {
+      "name": "shared preference",
+      "project": "beta",
+      "query": "concise explanations",
+      "expected": [
+        "preference"
+      ]
+    },
+    {
+      "name": "alias expansion",
+      "query": "short video navigation",
+      "expected": [
+        "alias"
+      ]
+    },
+    {
+      "name": "Chinese",
+      "query": "数据库",
+      "expected": [
+        "chinese"
+      ]
+    },
+    {
+      "name": "accented Spanish",
+      "query": "autenticación recuperación",
+      "expected": [
+        "spanish"
+      ]
+    },
+    {
+      "name": "hidden abstention",
+      "query": "hidden-canary-marker",
+      "expected": [],
+      "forbidden": [
+        "hidden"
+      ]
+    },
+    {
+      "name": "expired abstention",
+      "query": "expired-canary-marker",
+      "expected": [],
+      "forbidden": [
+        "expired"
+      ]
+    },
+    {
+      "name": "future abstention",
+      "query": "future-canary-marker",
+      "expected": [],
+      "forbidden": [
+        "future"
+      ]
+    },
+    {
+      "name": "superseded abstention",
+      "query": "superseded-canary-marker",
+      "expected": [],
+      "forbidden": [
+        "superseded"
+      ]
+    },
+    {
+      "name": "unknown abstention",
+      "query": "unrecorded-kubernetes-pod",
+      "expected": []
+    },
+    {
+      "name": "negation evidence",
+      "query": "not retry consumed token",
+      "expected": [
+        "negation"
+      ]
+    },
+    {
+      "name": "numeric evidence",
+      "query": "incident 42",
+      "expected": [
+        "numeric"
+      ]
+    },
+    {
+      "name": "empty abstention",
+      "query": "",
+      "expected": []
+    }
+  ]
+}