@simbimbo/memory-ocmemog 0.1.4

package/docs/architecture/memory.md

@@ -0,0 +1,118 @@
+# ocmemog Memory Architecture
+
+## What this repo actually ships
+
+ocmemog vendors a subset of brAIn's memory package and wraps it with a small FastAPI sidecar. The authoritative local implementation lives in:
+
+- `brain/runtime/memory/store.py` for the main SQLite schema
+- `brain/runtime/memory/retrieval.py` for keyword-first retrieval
+- `brain/runtime/memory/vector_index.py` for embeddings and fallback semantic lookup
+- `ocmemog/sidecar/app.py` for the plugin-facing HTTP API
+
+Unlike brAIn, this repo does not ship the full cognition/runtime stack. Several modules under `brain/runtime/*` are compatibility shims so the copied memory package can import cleanly.
+
+## Storage layout
+
+By default, ocmemog stores state under `.ocmemog-state/` at the repo root unless `OCMEMOG_STATE_DIR` or `BRAIN_STATE_DIR` overrides it.
+
+Primary files:
+
+- `.ocmemog-state/memory/brain_memory.sqlite3`
+- `.ocmemog-state/reports/brain_memory.log.jsonl`
+- `.ocmemog-state/data/person_memory.db`
+- `.ocmemog-state/data/interaction_memory.db`
+- `.ocmemog-state/data/sentiment_memory.db`
+- `.ocmemog-state/data/unresolved_state.db`
+- `.ocmemog-state/data/memory_graph.db`
+
+The main SQLite database owns these tables:
+
+- Core memory: `knowledge`, `reflections`, `directives`, `tasks`
+- Promotion pipeline: `experiences`, `candidates`, `promotions`
+- Derived indexes: `memory_index`, `vector_embeddings`
+- Supporting records: `memory_events`, `environment_cognition`, `artifacts`, `runbooks`, `lessons`
+
+## Retrieval flow
+
+The current sidecar behavior is simpler than brAIn's full memory architecture:
+
+1. `/memory/search` calls `retrieval.retrieve_for_queries()`.
+2. Retrieval scans `knowledge`, `reflections`, `directives`, and `tasks` for substring matches.
+3. Result scoring combines:
+   - keyword hit: `1.0` on substring match
+   - reinforcement bonus: `reward_score * 0.5`
+   - confidence bonus: `promotion confidence * 0.3`
+4. If `knowledge` has no keyword hit, retrieval falls back to `vector_index.search_memory()`.
+5. The sidecar flattens the bucketed results into a plugin-friendly response.
+
+Operational limits:
+
+- Semantic fallback now rehydrates any embedded bucket (`knowledge`, `runbooks`, `lessons`) when there are no keyword hits.
+- Default embeddings are local hash vectors (`BRAIN_EMBED_MODEL_LOCAL=simple`), which are deterministic but weak.
+- `runbooks`, `lessons`, `directives`, `reflections`, and `tasks` are now included in the default searchable categories and embedding index.
+
+## Write paths
+
+The main repo-local write paths are:
+
+- `reinforcement.log_experience()` writes to `experiences`
+- `candidate.create_candidate()` writes to `candidates` and `memory_events`
+- `promote.promote_candidate()` writes to `promotions` plus one of `knowledge`, `runbooks`, or `lessons`
+- `vector_index.insert_memory()` writes to `memory_index` and `vector_embeddings`
+- `memory_links.add_memory_link()` writes link metadata inside the main memory DB
+- `person_memory`, `interaction_memory`, `sentiment_memory`, `unresolved_state`, and `memory_graph` each write to separate SQLite files under `.ocmemog-state/data`
+
+## Distillation and promotion
+
+The brAIn docs describe a richer distill/promote pipeline. In ocmemog today:
+
+- Distillation exists in `brain/runtime/memory/distill.py`
+- Model-backed distillation is not available because `brain/runtime/inference.py` is still a shim
+- The practical fallback is a first-line heuristic summary plus generated verification prompts
+- Promotion is available locally and writes promoted summaries into `knowledge`, `runbooks`, or `lessons`
+- Successful promotion also logs a reinforcement event and attempts vector indexing
+
+This means the pipeline is present in code, but only part of it is production-ready.
+
+## Integrity and health
+
+Available support paths:
+
+- `integrity.run_integrity_check()` checks for missing tables, orphan candidates, duplicate promotions, missing memory references, and index mismatches
+- `health.get_memory_health()` reports counts and a coarse integrity summary
+- `brain_memory.log.jsonl` captures retrieval, embedding, integrity, and promotion events
+
+Known caveat:
+
+- health/integrity currently treat `memory_index` as the "vector" coverage source, even though actual embeddings live in `vector_embeddings`
+
+## Sidecar contract
+
+The sidecar exposes:
+
+- `GET /healthz`
+- `POST /memory/search`
+- `POST /memory/get`
+- `POST /memory/ingest`
+- `POST /memory/distill`
+
+The sidecar also reports runtime readiness through `mode`, `missingDeps`, `todo`, and `warnings`. That status is important because several copied brAIn modules are shimmed and should be treated as degraded.
+
+## Runtime adapters
+
+ocmemog ships lightweight runtime adapters for inference + embeddings. They require environment configuration:
+
+- `brain/runtime/inference.py` → OpenAI chat completions (requires `OCMEMOG_OPENAI_API_KEY`)
+- `brain/runtime/providers.py` → OpenAI embeddings (requires `BRAIN_EMBED_MODEL_PROVIDER=openai` + API key)
+- `brain/runtime/model_roles.py` + `model_router.py` → role-to-model and provider routing
+
+Effect on behavior:
+
+- Distillation uses OpenAI when API key is set, otherwise falls back to heuristics
+- Embeddings use OpenAI when configured, otherwise fall back to local hash or sentence-transformers
+- Role-aware context selection is still partially stubbed because `brain.runtime.roles` is not present
+
+## TODO: Missing runtime dependencies
+
+- TODO: add a repo-local `brain.runtime.roles` implementation or remove role-priority logic from `context_builder`
+- TODO: decide whether to add additional provider backends beyond OpenAI

package/docs/release-checklist.md

@@ -0,0 +1,34 @@
+# Release Checklist
+
+Use this checklist before publishing an ocmemog release.
+
+## Versioning
+- [ ] Update `package.json` version
+- [ ] Ensure release tag matches package version
+- [ ] Update `CHANGELOG.md`
+- [ ] Confirm README examples reference the current version where applicable
+
+## Validation
+- [ ] `bash -n scripts/install-ocmemog.sh`
+- [ ] `bash -n scripts/ocmemog-install.sh`
+- [ ] `./scripts/install-ocmemog.sh --help`
+- [ ] `./scripts/install-ocmemog.sh --dry-run`
+- [ ] `python -m unittest tests.test_regressions`
+- [ ] `npm pack --dry-run`
+
+## Install flow
+- [ ] Verify default installer path still works: `./scripts/install-ocmemog.sh`
+- [ ] Verify optional prereq install path is documented correctly
+- [ ] Verify LaunchAgent load path still matches repo scripts
+- [ ] Verify sidecar health check passes after install
+
+## Public artifacts
+- [ ] Push `main`
+- [ ] Create/update GitHub release notes
+- [ ] Publish/update ClawHub wrapper skill if installer/docs changed
+- [ ] Publish npm package if auth is available
+
+## Post-release sanity
+- [ ] Confirm repo default branch contains the release commits
+- [ ] Confirm GitHub release URL works
+- [ ] Confirm package/install instructions are still accurate

package/docs/reports/ocmemog-code-audit-2026-03-14.md

@@ -0,0 +1,155 @@
+# ocmemog deep audit (2026-03-14)
+
+**Scope:** full plugin repo, sidecar, memory runtime, scripts. Goal: public‑facing quality with minimal surprises.
+
+---
+
+## 1) TypeScript plugin layer
+
+### `openclaw.plugin.json`
+- ✅ `id: memory-ocmemog`, `kind: memory`.
+- ✅ Config schema includes `endpoint`, `timeoutMs`.
+- **Callout risk:** none.
+
+### `index.ts`
+- ✅ Uses OpenClaw plugin scaffold; forwards memory tools to sidecar.
+- **Callout risk:** if sidecar is down, tool calls should return clear error (verify by manual test).
+
+### `package.json`
+- ✅ Public package metadata, MIT license, `files` list, `publishConfig.access=public`.
+- ✅ `openclaw.extensions` points to `index.ts` (JITI loads TS at runtime).
+- **Callout risk:** repository URL must exist when publishing.
+
+---
+
+## 2) Sidecar API (FastAPI)
+
+### `ocmemog/sidecar/app.py`
+
+**Endpoints:**
+- `/memory/search`, `/memory/get`, `/memory/ingest`, `/memory/distill`, `/memory/context`, `/memory/ponder`, `/metrics`, `/events`, `/dashboard`, `/healthz`.
+
+**Security / robustness:**
+- ✅ Added optional auth: `OCMEMOG_API_TOKEN` (header `x-ocmemog-token` or `Authorization: Bearer`).
+- ✅ Transcript context retrieval now **allow‑listed** by `OCMEMOG_TRANSCRIPT_ROOTS`.
+- ✅ Transcript snippet reading **streams** only needed lines.
+- ✅ Loopback bind still default in sidecar launcher.
+
+**Potential callouts:**
+- If user binds to `0.0.0.0` without token, sidecar is exposed.
+- `/events` is an open SSE stream; if exposed, may leak memory logs.
+
+---
+
+## 3) Compatibility / runtime shims
+
+### `ocmemog/sidecar/compat.py`
+- ✅ Detects missing deps; reports warnings instead of crashing.
+- **Callout risk:** warnings are expected in “degraded” mode.
+
+---
+
+## 4) Memory store & schema
+
+### `brain/runtime/memory/store.py`
+- ✅ SQLite schema created on demand.
+- ✅ Added `demotions` and `cold_storage` tables (archive demoted memories).
+
+**Callout risk:**
+- Cold storage grows unbounded (acceptable for “disk cheap” stance).
+
+---
+
+## 5) Memory pipelines
+
+### `brain/runtime/memory/retrieval.py`
+- ✅ Simple keyword match + semantic fallback.
+- **Callout risk:** keyword matching is naive; might be criticized for low recall. (Mitigate by embeddings + vector index.)
+
+### `brain/runtime/memory/vector_index.py`
+- ✅ Embedding index stored in SQLite; semantic search loads all embeddings in memory.
+- **Callout risk:** O(N) scan can be slow at scale. Acceptable for local use; note in docs.
+
+### `brain/runtime/memory/distill.py`
+- ✅ Uses Ollama inference if configured; heuristic fallback.
+- **Callout risk:** distill quality depends on local model; acceptable for “best effort”.
+
+### `brain/runtime/memory/promote.py`
+- ✅ Promotion threshold configurable (`OCMEMOG_PROMOTION_THRESHOLD`).
+- ✅ Demotion archives into `cold_storage` and removes from hot tables.
+- **Callout risk:** Demotion is destructive to hot table (archived safe).
+
+### `brain/runtime/memory/context_builder.py`
+- ✅ Uses retrieval for context blocks.
+- **Callout risk:** role registry missing in this repo (already noted in compat warnings).
+
+### `brain/runtime/memory/pondering_engine.py`
+- ✅ Pondering now exposed via API and writes summaries into `reflections`.
+
+---
+
+## 6) Embeddings & inference
+
+### `brain/runtime/embedding_engine.py`
+- ✅ Local embeddings supported (simple/hash or sentence-transformers).
+- ✅ Provider embeddings via OpenAI or Ollama.
+- **Callout risk:** if `BRAIN_EMBED_MODEL_LOCAL` is set, it overrides provider embeddings.
+
+### `brain/runtime/providers.py`
+- ✅ OpenAI and Ollama embedding calls.
+- ✅ Added structured error logging to `brain_memory.log.jsonl`.
+
+### `brain/runtime/inference.py`
+- ✅ Ollama inference supported; OpenAI fallback.
+- ✅ Added structured error logging.
+
+---
+
+## 7) Sidecar scripts
+
+### `scripts/ocmemog-sidecar.sh`
+- ✅ Defaults to Ollama (phi3 + nomic‑embed‑text).
+- ✅ Configurable thresholds + token.
+
+### `scripts/ocmemog-test-rig.py`
+- ✅ Stress rig: ingest, distill, promote, demote, ponder, research.
+- ✅ Supports shadow promotion mode (avoid rejecting useful data).
+
+### `scripts/ocmemog-demo.py`
+- ✅ Presentation‑friendly demo with `--pretty`.
+
+---
+
+## 8) Known TODOs / warnings
+- Sidecar reports missing role registry for role‑based prioritization.
+- Optional dependency: `sentence-transformers` (only if you want local non‑Ollama embeddings).
+- Vector index is naive (O(N)).
+
+---
+
+## 9) Dependencies
+
+**Python**
+- `fastapi`
+- `uvicorn[standard]`
+
+**System**
+- `ollama`
+
+---
+
+## 10) Recommendations before public launch
+
+1) **Add a short SECURITY note** to README:
+   - sidecar is local by default
+   - if bound to 0.0.0.0, require `OCMEMOG_API_TOKEN`
+
+2) **Add scale warning** for vector search (O(N))
+
+3) **Tag release** `v0.1.0` and publish npm package.
+
+---
+
+## 11) Summary
+
+The codebase is publish‑ready after the security hardening and doc updates above. Remaining risks are acceptable given local‑first design. No critical missing dependencies or crashes detected.

package/docs/usage.md

@@ -0,0 +1,223 @@
+# ocmemog Memory Usage
+
+## Current operating model
+
+ocmemog is a repo-local OpenClaw memory sidecar backed by SQLite. It is not a full brAIn runtime clone. The safe assumption is:
+
+- search/get over local memory are supported
+- heuristic embeddings are supported by default
+- several advanced brAIn memory flows are copied in but still degraded by missing runtime dependencies
+
+## Running the sidecar
+
+Use the provided launcher:
+
+```bash
+scripts/ocmemog-sidecar.sh
+```
+
+## Transcript watcher (auto-ingest)
+
+Manual watcher:
+
+```bash
+# defaults to ~/.openclaw/workspace/memory/transcripts if not set
+export OCMEMOG_TRANSCRIPT_DIR="$HOME/.openclaw/workspace/memory/transcripts"
+export OCMEMOG_INGEST_ENDPOINT="http://127.0.0.1:17890/memory/ingest"
+./scripts/ocmemog-transcript-watcher.sh
+```
+
+Auto-start inside sidecar:
+
+```bash
+export OCMEMOG_TRANSCRIPT_WATCHER=true
+./scripts/ocmemog-sidecar.sh
+```
+
+Useful environment variables:
+
+```bash
+export OCMEMOG_HOST=127.0.0.1
+export OCMEMOG_PORT=17890
+export OCMEMOG_STATE_DIR=/path/to/state
+export OCMEMOG_DB_PATH=/path/to/brain_memory.sqlite3
+export OCMEMOG_MEMORY_MODEL=gpt-4o-mini
+export OCMEMOG_OPENAI_API_KEY=sk-...
+export OCMEMOG_OPENAI_API_BASE=https://api.openai.com/v1
+export OCMEMOG_OPENAI_EMBED_MODEL=text-embedding-3-small
+export BRAIN_EMBED_MODEL_LOCAL=simple
+export BRAIN_EMBED_MODEL_PROVIDER=openai
+export OCMEMOG_TRANSCRIPT_DIR=$HOME/.openclaw/workspace/memory/transcripts
+export OCMEMOG_TRANSCRIPT_GLOB=*.log
+export OCMEMOG_TRANSCRIPT_POLL_SECONDS=1
+export OCMEMOG_INGEST_KIND=memory
+export OCMEMOG_INGEST_SOURCE=transcript
+export OCMEMOG_TRANSCRIPT_WATCHER=true
+```
+
+Default state location in this repo is `.ocmemog-state/`.
+
+## Plugin API
+
+Health:
+
+```bash
+curl http://127.0.0.1:17890/healthz
+```
+
+Realtime metrics + events:
+
+```bash
+curl http://127.0.0.1:17890/metrics
+curl http://127.0.0.1:17890/events
+```
+
+Dashboard:
+
+```bash
+open http://127.0.0.1:17890/dashboard
+```
+
+Search:
+
+```bash
+curl -s http://127.0.0.1:17890/memory/search \
+  -H 'content-type: application/json' \
+  -d '{"query":"deploy risk","limit":5,"categories":["knowledge","tasks"]}'
+```
+
+If `OCMEMOG_API_TOKEN` is set, include the header:
+
+```bash
+-H 'x-ocmemog-token: YOUR_TOKEN'
+```
+
+Get by reference:
+
+```bash
+curl -s http://127.0.0.1:17890/memory/get \
+  -H 'content-type: application/json' \
+  -d '{"reference":"knowledge:12"}'
+```
+
+Fetch linked context (transcript snippet):
+
+```bash
+curl -s http://127.0.0.1:17890/memory/context \
+  -H 'content-type: application/json' \
+  -d '{"reference":"knowledge:12","radius":10}'
+```
+
+Helper script:
+
+```bash
+./scripts/ocmemog-context.sh knowledge:12 10
+```
+
+Run pondering (writes summaries into reflections):
+
+```bash
+curl -s http://127.0.0.1:17890/memory/ponder \
+  -H 'content-type: application/json' \
+  -d '{"max_items":5}'
+```
+
+Fetch latest ponder recommendations:
+
+```bash
+curl -s http://127.0.0.1:17890/memory/ponder/latest?limit=5
+```
+
+Ingest content:
+
+```bash
+curl -s http://127.0.0.1:17890/memory/ingest \
+  -H 'content-type: application/json' \
+  -d '{"content":"remember this","kind":"memory","memory_type":"knowledge"}'
+```
+
+Ingest with context anchors (links to chat/transcript):
+
+```bash
+curl -s http://127.0.0.1:17890/memory/ingest \
+  -H 'content-type: application/json' \
+  -d '{
+        "content":"remember this",
+        "kind":"memory",
+        "memory_type":"knowledge",
+        "session_id":"session-123",
+        "thread_id":"thread-abc",
+        "message_id":"msg-987",
+        "transcript_path":"/path/to/transcript.log",
+        "transcript_offset":420,
+        "timestamp":"2026-02-22 16:04:52"
+      }'
+```
+
+Distill recent experiences:
+
+```bash
+curl -s http://127.0.0.1:17890/memory/distill \
+  -H 'content-type: application/json' \
+  -d '{"limit":10}'
+```
+
+Notes:
+
+- Valid sidecar categories today are `knowledge`, `reflections`, `directives`, `tasks`, `runbooks`, and `lessons`.
+- `/memory/get` currently expects a `table:id` reference.
+- Runtime degradation is reported in every sidecar response.
+
+## What is safe to rely on
+
+- `store.init_db()` creates the local schema automatically
+- `retrieval.retrieve_for_queries()` is the main sidecar search path
+- `vector_index.search_memory()` provides a semantic fallback over `knowledge`, `runbooks`, `lessons`, `directives`, `reflections`, and `tasks` when keyword retrieval misses
+- `probe_runtime()` exposes missing shim replacements and optional embedding warnings
+
+## What is not safe to rely on yet
+
+- `brain/runtime/memory/api.py`
+  - It targets missing/legacy tables and columns.
+- Provider-backed embeddings
+  - Available when `BRAIN_EMBED_MODEL_PROVIDER=openai` and `OCMEMOG_OPENAI_API_KEY` is set.
+  - Falls back to local embeddings when missing.
+- Model-backed distillation
+  - Available when `OCMEMOG_OPENAI_API_KEY` is set; otherwise falls back to heuristic distill.
+- Role-prioritized context building
+  - `brain.runtime.roles.role_registry` is not bundled here.
+- Full brAIn memory parity
+  - The repo ships only a subset of the original runtime architecture.
+
+## Suggested local workflows
+
+To inspect memory state quickly:
+
+```bash
+python3 - <<'PY'
+from brain.runtime.memory import store
+store.init_db()
+conn = store.connect()
+for table in ("knowledge", "reflections", "directives", "tasks", "candidates", "promotions"):
+    count = conn.execute(f"SELECT COUNT(*) FROM {table}").fetchone()[0]
+    print(table, count)
+conn.close()
+PY
+```
+
+To rebuild embeddings for recent knowledge rows:
+
+```bash
+python3 - <<'PY'
+from brain.runtime.memory import vector_index
+print(vector_index.index_memory())
+PY
+```
+
+## TODO: Missing runtime dependencies
+
+- TODO: wire a real inference backend before enabling distill/promote as an operator-facing workflow
+- TODO: wire real provider execution if `BRAIN_EMBED_MODEL_PROVIDER` is meant to do anything
+- TODO: add or remove role-based context selection; the current import path is absent
+- TODO: harden `/memory/get` with a table allow-list before exposing the sidecar outside trusted local use
+- TODO: decide whether to expose `runbooks` and `lessons` in the plugin API