@simbimbo/memory-ocmemog 0.1.11 → 0.1.13

package/CHANGELOG.md

@@ -1,5 +1,35 @@
 # Changelog
 
+## 0.1.13 — 2026-03-22
+
+Final hardening release before a possible 1.0 cut.
+
+### Highlights
+- tested the full shipped surface aggressively: full pytest suite, release gate, live sidecar contract smoke, packaging dry-run, installer surfaces, and governance summary responsiveness checks all passed together
+- fixed a supersession-governance regression that could suppress `supersession_recommendation` generation and break governance queue/review/summary and auto-resolve flows
+- moved dashboard supersession plain-English rewriting out of the render path so live dashboard loads stay fast while recommendations still carry human-readable text
+- added a lightweight cached governance review summary path for the dashboard, reducing review load time from multi-second scans to sub-second first load and near-instant cached refresh
+- simplified Governance Review UI output into more concise, single-row plain-English review items
+- hardened supersession summary generation against polluted transcript/log content with tighter preview normalization, aggressive noise stripping, bounded local-model rewriting, and safe heuristic fallbacks
+- fixed dashboard cursor handling so the dashboard route tolerates minimal/mock DB cursor implementations used by tests
+- hardened the integrated proof token/session identifiers to avoid fresh-state collisions during repeated release validation
+
+## 0.1.12 — 2026-03-21
+
+Release hardening, integrated proof validation, and native-ownership cleanup.
+
+### Highlights
+- fixed conversation-state self-healing so polluted continuity cleanup preserves valid checkpoints instead of deleting the entire checkpoint history for a thread/session/conversation
+- aligned FastAPI sidecar version reporting with the package version and added regression coverage for version drift
+- moved runtime defaults toward native `ocmemog` ownership for report-log and SQLite DB naming while preserving legacy `brain_*` file fallback for existing installs
+- made embedding configuration native-first (`OCMEMOG_*`) while keeping `BRAIN_*` aliases for compatibility
+- hardened the integrated memory contract proof and release gate so fresh-state proof, live sidecar smoke, and route/regression validation run together as the canonical pre-release bar
+- fixed release-gate/proof path bugs around HTTP method mismatches, async ingest/postprocess timing assumptions, and proof output capture
+- restored live sidecar request-path verification for `/memory/ingest`, `/memory/search`, `/memory/get`, and `/conversation/hydrate`
+- collapsed the legacy `brain/runtime/*` implementation tree into thin compatibility shims and removed orphan legacy side-modules that were no longer part of the shipped product contract
+- cleaned release docs, compat wording, and helper scripts so new deployments follow native `ocmemog` behavior by default and stale side-DB architecture references are removed
+- removed stray invalid transcript-watcher drift assertions from the test suite
+
 ## 0.1.11 — 2026-03-20
 
 Watcher reliability and release-quality follow-up.

package/README.md

@@ -12,7 +12,7 @@ It is designed to go beyond simple memory search by providing:
 Architecture at a glance:
 - **OpenClaw plugin (`index.ts`)** handles tools and hook integration
 - **FastAPI sidecar (`ocmemog/sidecar/`)** exposes memory and continuity APIs
-- **SQLite-backed runtime (`brain/runtime/memory/`)** powers storage, hydration, checkpoints, salience ranking, and pondering
+- **SQLite-backed runtime (`ocmemog/runtime/memory/`)** powers storage, hydration, checkpoints, salience ranking, and pondering
 
 Current local runtime architecture note:
 - `docs/architecture/local-runtime-2026-03-19.md`
@@ -21,8 +21,9 @@ Current local runtime architecture note:
 
 - `openclaw.plugin.json`, `index.ts`, `package.json`: OpenClaw plugin package and manifest.
 - `ocmemog/sidecar/`: FastAPI sidecar with `/memory/search` and `/memory/get`.
-- `brain/runtime/memory/`: copied brAIn memory package.
-- `brain/runtime/`: compatibility shims for state store, instrumentation, redaction, storage paths, and a few placeholder runtime modules needed for importability.
+- `ocmemog/runtime/`: native runtime surfaces used by the sidecar and memory engine.
+- `ocmemog/runtime/memory/`: local memory/runtime package used by the sidecar.
+- `brain/`: internal compatibility residue retained for transitional shim paths; not the primary runtime surface.
 - `scripts/ocmemog-sidecar.sh`: convenience launcher.
 
 ## Run the sidecar
@@ -38,11 +39,44 @@ pip install -r requirements.txt
 # http://127.0.0.1:17891/dashboard
 ```
 
+For local development and CI-style test runs, install test dependencies as well:
+
+```bash
+cd /path/to/ocmemog
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install -r requirements-test.txt
+```
+
+### Run the doctor check
+
+```bash
+./.venv/bin/python3 scripts/ocmemog-doctor.py
+./.venv/bin/python3 scripts/ocmemog-doctor.py --json
+./.venv/bin/python3 scripts/ocmemog-doctor.py --fix create-missing-paths --fix repair-queue
+./.venv/bin/python3 scripts/ocmemog-doctor.py --strict --check runtime/imports --check sqlite/schema-access
+```
+
+The doctor command currently checks:
+- runtime/imports
+- state/path-writable
+- sqlite/schema-access
+- sidecar/http-auth
+- queue/health
+- sidecar/transcript-watcher
+- sidecar/app-import
+- sidecar/transcript-roots
+- sidecar/env-toggles
+- vector/runtime-probe
+
 ## Optional: transcript watcher (auto-ingest)
 
 ```bash
-# defaults to ~/.openclaw/workspace/memory/transcripts if not set
+# defaults:
+# - transcript mode: ~/.openclaw/workspace/memory/transcripts
+# - session mode: ~/.openclaw/agents/main/sessions (used when OCMEMOG_TRANSCRIPT_DIR is unset)
 export OCMEMOG_TRANSCRIPT_DIR="$HOME/.openclaw/workspace/memory/transcripts"
+export OCMEMOG_SESSION_DIR="$HOME/.openclaw/agents/main/sessions"
 ./scripts/ocmemog-transcript-watcher.sh
 ```
 
@@ -80,11 +114,26 @@ Optional environment variables:
 - `OCMEMOG_OPENAI_API_KEY` (required for model-backed distill)
 - `OCMEMOG_OPENAI_API_BASE` (default: `https://api.openai.com/v1`)
 - `OCMEMOG_OPENAI_EMBED_MODEL` (default: `text-embedding-3-small`)
-- `BRAIN_EMBED_MODEL_LOCAL` (`simple` by default)
-- `BRAIN_EMBED_MODEL_PROVIDER` (`local-openai` to use the local llama.cpp embedding endpoint; `openai` remains available for hosted embeddings)
+- `OCMEMOG_EMBED_MODEL_LOCAL` (`simple` by default; legacy alias: `BRAIN_EMBED_MODEL_LOCAL`)
+- `OCMEMOG_EMBED_MODEL_PROVIDER` (`local-openai` to use the local llama.cpp embedding endpoint; `openai` remains available for hosted embeddings; legacy alias: `BRAIN_EMBED_MODEL_PROVIDER`)
 - `OCMEMOG_TRANSCRIPT_WATCHER` (`true` to auto-start transcript watcher inside the sidecar)
 - `OCMEMOG_TRANSCRIPT_ROOTS` (comma-separated allowed roots for transcript context retrieval; default: `~/.openclaw/workspace/memory`)
-- `OCMEMOG_API_TOKEN` (optional; if set, requests must include `x-ocmemog-token` or `Authorization: Bearer ...`)
+- `OCMEMOG_TRANSCRIPT_DIR` (default: `~/.openclaw/workspace/memory/transcripts`)
+- `OCMEMOG_SESSION_DIR` (default: `~/.openclaw/agents/main/sessions`)
+- `OCMEMOG_TRANSCRIPT_POLL_SECONDS` (poll interval for file/session watcher; default: `30`, or `120` in battery mode)
+- `OCMEMOG_INGEST_BATCH_SECONDS` (max lines per watcher batch; default: `30`, or `120` in battery mode)
+- `OCMEMOG_INGEST_BATCH_MAX` (max watcher batches before yield; default: `25`, or `10` in battery mode)
+- `OCMEMOG_SESSION_GLOB` (default file glob for session sources: `*.jsonl`)
+- `OCMEMOG_TRANSCRIPT_GLOB` (default file glob for transcripts: `*.log`)
+- `OCMEMOG_INGEST_ASYNC_WORKER` (`true` to keep async ingest queue processing enabled; defaults to `true`)
+- `OCMEMOG_INGEST_ASYNC_POLL_SECONDS` (`5` by default)
+- `OCMEMOG_INGEST_ASYNC_BATCH_MAX` (`25` by default)
+- `OCMEMOG_INGEST_ENDPOINT` (default: `http://127.0.0.1:17891/memory/ingest_async`)
+- `OCMEMOG_SHUTDOWN_DRAIN_QUEUE` (`true` to drain remaining queue entries during shutdown; defaults to `false`)
+- `OCMEMOG_WORKER_SHUTDOWN_TIMEOUT_SECONDS` (`0.35` by default)
+- `OCMEMOG_SHUTDOWN_DUMP_THREADS` (`true` to include worker thread dump output during shutdown joins; defaults to `false`)
+- `OCMEMOG_SHUTDOWN_TIMING` (`true` enables shutdown timing logs; defaults to `true`)
+- `OCMEMOG_API_TOKEN` (optional; if set, requests must include `x-ocmemog-token` or `Authorization: Bearer ...`; OpenClaw plugin users should also set the plugin `config.token` field)
 - `OCMEMOG_AUTO_HYDRATION` (`true` to re-enable prompt-time continuity prepending; defaults to `false` as a safety guard until the host runtime is verified not to persist prepended context into session history)
 - `OCMEMOG_LAPTOP_MODE` (`auto` by default; on macOS battery power this slows watcher polling, reduces ingest batch size, and disables sentiment reinforcement unless explicitly overridden)
 - `OCMEMOG_LOCAL_LLM_BASE_URL` (default: `http://127.0.0.1:18080/v1`; local OpenAI-compatible text endpoint, e.g. llama.cpp)
@@ -101,6 +150,8 @@ Optional environment variables:
 - `OCMEMOG_PONDER_MODEL` (default via launcher: `local-openai:qwen2.5-7b-instruct`; recommended for structured local memory refinement)
 - `OCMEMOG_LESSON_MINING_ENABLED` (default: `true`)
 
+Boolean env values are parsed case-insensitively and support `1/0`, `true/false`, `yes/no`, `on/off`, `y/n`, and `t/f`.
+
 ## Security
 
 - Sidecar binds to **127.0.0.1** by default. Keep it local unless you add auth + firewall rules.
@@ -162,19 +213,32 @@ launchctl bootstrap gui/$UID scripts/launchagents/com.openclaw.ocmemog.guard.pli
 
 ## Recent changes
 
-### 0.1.11 (current main)
+### 0.1.12 (current main)
 
-Current main includes the recent memory-quality, governance, and watcher hardening work, including:
-- transcript watcher duplicate-ingest prevention
-- watcher auth propagation for `OCMEMOG_API_TOKEN`
-- persisted queue stats restored on startup
-- durable watcher error logging
-- multi-part text preservation for session content arrays
-- retry-safe session/transcript handling with transcript provenance preserved
-- pytest declared as a repo test extra and available in the local project venv
+Current main now includes:
+- integrated release-gate validation with a fresh-state memory contract proof
+- live sidecar smoke checks for `/memory/ingest`, `/memory/search`, `/memory/get`, and `/conversation/hydrate`
+- hardened async ingest/postprocess handling and queue-health validation
+- native-first `ocmemog` config, DB, and report-log naming with legacy fallback support
+- sidecar/package version alignment and regression coverage for checkpoint self-heal behavior
+- major collapse of the legacy `brain/runtime/*` implementation surface into thin compatibility shims
+- removal of orphan legacy memory side-modules and stale architecture references to retired side DBs
 
 ## Release prep / publish
 
+Run the release gate first:
+
+```bash
+./scripts/ocmemog-release-check.sh
+```
+
+This command is the canonical pre-release and CI validation path. A pass now means:
+- strict doctor checks succeed on a clean temporary state
+- the broad regression subset plus sidecar route tests pass
+- live `/healthz`, `/memory/ingest`, `/memory/search`, `/memory/get`, and `/conversation/hydrate` checks pass on `OCMEMOG_RELEASE_LIVE_ENDPOINT`
+- the integrated memory contract proof runs in fresh state and emits `reports/release-gate-proof.json`
+- syntax checks, integration assertions, and non-blocking packing validation are attempted
+
 Example ClawHub publish command (update version + changelog first; do not reuse stale release text blindly):
 
 ```bash
@@ -205,6 +269,8 @@ plugins:
       config:
         endpoint: http://127.0.0.1:17891
         timeoutMs: 30000
+        # Optional if the sidecar is protected by OCMEMOG_API_TOKEN
+        token: your-ocmemog-token
 ```
 
 Development install:
@@ -229,7 +295,6 @@ What is working now:
 Current limitations before broader public rollout:
 - Some advanced inference- and embedding-dependent paths still depend on environment configuration and may degrade to simpler local behavior if provider access is unavailable
 - Packaging and install UX are aimed primarily at power users and local developers today
-- Public release/distribution metadata is still being tightened up
+- Distribution and release metadata are now tracked in `package.json`, `CHANGELOG.md`, and the release check workflow.
 
 When a richer path is unavailable, the sidecar is designed to fail soft with explicit warnings rather than crash.
- soft with explicit warnings rather than crash.

package/brain/runtime/__init__.py

@@ -1,13 +1,3 @@
-"""Minimal runtime shims required by the copied brAIn memory package."""
+from __future__ import annotations
 
-from . import config, inference, instrumentation, model_roles, model_router, state_store, storage_paths
-
-__all__ = [
-    "config",
-    "inference",
-    "instrumentation",
-    "model_roles",
-    "model_router",
-    "state_store",
-    "storage_paths",
-]
+from ocmemog.runtime import *  # noqa: F401,F403

package/brain/runtime/config.py

@@ -1,26 +1,3 @@
 from __future__ import annotations
 
-import os
-
-BRAIN_EMBED_MODEL_LOCAL = os.environ.get("BRAIN_EMBED_MODEL_LOCAL", "simple")
-BRAIN_EMBED_MODEL_PROVIDER = os.environ.get("BRAIN_EMBED_MODEL_PROVIDER", "")
-
-OCMEMOG_MEMORY_MODEL = os.environ.get("OCMEMOG_MEMORY_MODEL", "gpt-4o-mini")
-OCMEMOG_OPENAI_API_BASE = os.environ.get("OCMEMOG_OPENAI_API_BASE", "https://api.openai.com/v1")
-OCMEMOG_OPENAI_EMBED_MODEL = os.environ.get("OCMEMOG_OPENAI_EMBED_MODEL", "text-embedding-3-small")
-
-OCMEMOG_LOCAL_LLM_BASE_URL = os.environ.get("OCMEMOG_LOCAL_LLM_BASE_URL", "http://127.0.0.1:18080/v1")
-OCMEMOG_LOCAL_LLM_MODEL = os.environ.get("OCMEMOG_LOCAL_LLM_MODEL", "qwen2.5-7b-instruct")
-OCMEMOG_LOCAL_EMBED_BASE_URL = os.environ.get("OCMEMOG_LOCAL_EMBED_BASE_URL", "http://127.0.0.1:18081/v1")
-OCMEMOG_LOCAL_EMBED_MODEL = os.environ.get("OCMEMOG_LOCAL_EMBED_MODEL", "nomic-embed-text-v1.5")
-
-OCMEMOG_OLLAMA_HOST = os.environ.get("OCMEMOG_OLLAMA_HOST", "http://127.0.0.1:11434")
-OCMEMOG_OLLAMA_MODEL = os.environ.get("OCMEMOG_OLLAMA_MODEL", "qwen2.5:7b")
-OCMEMOG_OLLAMA_EMBED_MODEL = os.environ.get("OCMEMOG_OLLAMA_EMBED_MODEL", "nomic-embed-text:latest")
-
-OCMEMOG_PROMOTION_THRESHOLD = float(os.environ.get("OCMEMOG_PROMOTION_THRESHOLD", "0.5"))
-OCMEMOG_DEMOTION_THRESHOLD = float(os.environ.get("OCMEMOG_DEMOTION_THRESHOLD", "0.2"))
-
-OCMEMOG_PONDER_ENABLED = os.environ.get("OCMEMOG_PONDER_ENABLED", "true")
-OCMEMOG_PONDER_MODEL = os.environ.get("OCMEMOG_PONDER_MODEL", OCMEMOG_MEMORY_MODEL)
-OCMEMOG_LESSON_MINING_ENABLED = os.environ.get("OCMEMOG_LESSON_MINING_ENABLED", "true")
+from ocmemog.runtime.config import *  # noqa: F401,F403

package/brain/runtime/inference.py

@@ -1,153 +1,3 @@
 from __future__ import annotations
 
-import json
-import os
-import re
-import urllib.request
-
-from brain.runtime import config, state_store
-from brain.runtime.instrumentation import emit_event
-
-LOGFILE = state_store.reports_dir() / "brain_memory.log.jsonl"
-
-
-def _infer_openai_compatible(prompt: str, *, base_url: str, model: str, api_key: str | None = None, provider_label: str = "openai-compatible") -> dict[str, str]:
-    url = f"{base_url.rstrip('/')}/chat/completions"
-    payload = {
-        "model": model,
-        "messages": [{"role": "user", "content": prompt}],
-        "temperature": 0.2,
-    }
-    data = json.dumps(payload).encode("utf-8")
-    req = urllib.request.Request(url, data=data, method="POST")
-    if api_key:
-        req.add_header("Authorization", f"Bearer {api_key}")
-    req.add_header("Content-Type", "application/json")
-
-    try:
-        with urllib.request.urlopen(req, timeout=30) as resp:
-            response = json.loads(resp.read().decode("utf-8"))
-    except Exception as exc:
-        emit_event(LOGFILE, "brain_infer_error", status="error", provider=provider_label, error=str(exc))
-        return {"status": "error", "error": f"request_failed:{exc}"}
-
-    try:
-        output = response["choices"][0]["message"]["content"]
-    except Exception as exc:
-        emit_event(LOGFILE, "brain_infer_error", status="error", provider=provider_label, error=str(exc))
-        return {"status": "error", "error": "invalid_response"}
-
-    return {"status": "ok", "output": str(output).strip()}
-
-
-def _infer_ollama(prompt: str, model: str | None = None) -> dict[str, str]:
-    payload = {
-        "model": model or config.OCMEMOG_OLLAMA_MODEL,
-        "prompt": prompt,
-        "stream": False,
-    }
-    data = json.dumps(payload).encode("utf-8")
-    req = urllib.request.Request(f"{config.OCMEMOG_OLLAMA_HOST.rstrip('/')}/api/generate", data=data, method="POST")
-    req.add_header("Content-Type", "application/json")
-    try:
-        with urllib.request.urlopen(req, timeout=60) as resp:
-            response = json.loads(resp.read().decode("utf-8"))
-    except Exception as exc:
-        emit_event(LOGFILE, "brain_infer_error", status="error", provider="ollama", error=str(exc))
-        return {"status": "error", "error": f"ollama_failed:{exc}"}
-    output = response.get("response")
-    if not output:
-        emit_event(LOGFILE, "brain_infer_error", status="error", provider="ollama", error="invalid_response")
-        return {"status": "error", "error": "invalid_response"}
-    return {"status": "ok", "output": str(output).strip()}
-
-
-def _looks_like_local_openai_model(name: str) -> bool:
-    if not name:
-        return False
-    lowered = name.strip().lower()
-    return lowered.startswith("local-openai:") or lowered.startswith("local_openai:") or lowered.startswith("llamacpp:")
-
-
-def _normalize_local_model_name(name: str) -> str:
-    lowered = (name or "").strip()
-    for prefix in ("local-openai:", "local_openai:", "llamacpp:"):
-        if lowered.lower().startswith(prefix):
-            return lowered[len(prefix):]
-    return lowered
-
-
-def _looks_like_ollama_model(name: str) -> bool:
-    if not name:
-        return False
-    lowered = name.strip().lower()
-    if lowered.startswith("ollama:"):
-        return True
-    if "/" in lowered:
-        return False
-    return ":" in lowered
-
-
-def stats() -> dict[str, object]:
-    materialized_local = int(_LOCAL_INFER_STATS.get("local_success", 0)) + int(_LOCAL_INFER_STATS.get("cache_hits", 0))
-    est_prompt_tokens_saved = materialized_local * _AVG_PROMPT_TOKENS_SAVED
-    est_completion_tokens_saved = materialized_local * _AVG_COMPLETION_TOKENS_SAVED
-    est_cost_saved = (
-        (est_prompt_tokens_saved / 1000.0) * _EST_FRONTIER_INPUT_COST_PER_1K
-        + (est_completion_tokens_saved / 1000.0) * _EST_FRONTIER_OUTPUT_COST_PER_1K
-    )
-    return {
-        "cache_entries": len(_LOCAL_INFER_CACHE),
-        "warm_models": sorted(_MODEL_WARM_STATE.keys()),
-        "frontier_calls_avoided_est": materialized_local,
-        "prompt_tokens_saved_est": est_prompt_tokens_saved,
-        "completion_tokens_saved_est": est_completion_tokens_saved,
-        "cost_saved_usd_est": round(est_cost_saved, 4),
-        **{k: int(v) for k, v in _LOCAL_INFER_STATS.items()},
-    }
-
-
-def infer(prompt: str, provider_name: str | None = None) -> dict[str, str]:
-    if not isinstance(prompt, str) or not prompt.strip():
-        return {"status": "error", "error": "empty_prompt"}
-
-    use_ollama = os.environ.get("OCMEMOG_USE_OLLAMA", "").lower() in {"1", "true", "yes"}
-    model_override = provider_name or config.OCMEMOG_MEMORY_MODEL
-    if _looks_like_local_openai_model(model_override):
-        model = _normalize_local_model_name(model_override) or config.OCMEMOG_LOCAL_LLM_MODEL
-        return _infer_openai_compatible(
-            prompt,
-            base_url=config.OCMEMOG_LOCAL_LLM_BASE_URL,
-            model=model,
-            api_key=os.environ.get("OCMEMOG_LOCAL_LLM_API_KEY") or os.environ.get("LOCAL_LLM_API_KEY"),
-            provider_label="local-openai",
-        )
-    if use_ollama or _looks_like_ollama_model(model_override):
-        model = model_override.split(":", 1)[-1] if model_override.startswith("ollama:") else model_override
-        return _infer_ollama(prompt, model)
-
-    api_key = os.environ.get("OCMEMOG_OPENAI_API_KEY") or os.environ.get("OPENAI_API_KEY")
-    if not api_key:
-        return _infer_openai_compatible(
-            prompt,
-            base_url=config.OCMEMOG_LOCAL_LLM_BASE_URL,
-            model=config.OCMEMOG_LOCAL_LLM_MODEL,
-            api_key=os.environ.get("OCMEMOG_LOCAL_LLM_API_KEY") or os.environ.get("LOCAL_LLM_API_KEY"),
-            provider_label="local-openai",
-        )
-
-    model = model_override
-    return _infer_openai_compatible(
-        prompt,
-        base_url=config.OCMEMOG_OPENAI_API_BASE,
-        model=model,
-        api_key=api_key,
-        provider_label="openai",
-    )
-
-
-def parse_operator_name(text: str) -> dict[str, str] | None:
-    match = re.search(r"\bmy name is ([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\b", text, flags=re.IGNORECASE)
-    if not match:
-        return None
-    return {"name": match.group(1)}
+from ocmemog.runtime.inference import *  # noqa: F401,F403

package/brain/runtime/instrumentation.py

@@ -1,17 +1,3 @@
 from __future__ import annotations
 
-import json
-from datetime import datetime, timezone
-from pathlib import Path
-from typing import Any
-
-
-def emit_event(path: Path, event: str, **fields: Any) -> None:
-    path.parent.mkdir(parents=True, exist_ok=True)
-    payload = {
-        "timestamp": datetime.now(timezone.utc).isoformat(),
-        "event": event,
-        **fields,
-    }
-    with path.open("a", encoding="utf-8") as handle:
-        handle.write(json.dumps(payload, sort_keys=True) + "\n")
+from ocmemog.runtime.instrumentation import *  # noqa: F401,F403

package/brain/runtime/memory/__init__.py

@@ -1,13 +1,3 @@
-__all__ = [
-    "store",
-    "distill",
-    "candidate",
-    "promote",
-    "retrieval",
-    "reinforcement",
-    "integrity",
-    "vector_index",
-    "context_builder",
-    "memory_taxonomy",
-    "health",
-]
+from __future__ import annotations
+
+from ocmemog.runtime.memory import *  # noqa: F401,F403