contextos-memory-runtime 1.0.0rc2__tar.gz

contextos_memory_runtime-1.0.0rc2/.gitignore

@@ -0,0 +1,22 @@
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.coverage
+htmlcov/
+
+.venv/
+venv/
+
+.env
+.env.*
+
+*.db
+*.sqlite
+*.sqlite3
+
+dist/
+build/
+*.egg-info/
+
+.idea/
+.vscode/

contextos_memory_runtime-1.0.0rc2/CHANGELOG.md

@@ -0,0 +1,44 @@
+# ContextOS Changelog
+
+All notable changes to ContextOS will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0-rc2] - 2026-10-03
+
+- Distribution renamed to `contextos-memory-runtime`; imports and commands remain `contextos`.
+- Bounded startup serialization using cross-platform OS lifecycle locks (`msvcrt` on Windows, `flock` on POSIX) with configured timeouts (30s readiness, 45s lock).
+- OS TCP listener ownership verification: verifies that the configured loopback listener belongs to the verified ContextOS daemon process tree, rejecting spoofed endpoint PID claims.
+- Recoverable startup ownership: daemon publishes durable identity prior to binding, ensuring that readiness implies discoverable lifecycle registration even if the starter process exits prematurely.
+- Compare-before-delete lifecycle cleanup preserving survivor state during concurrent start or failed attempts.
+
+## [1.0.0-rc1] - 2026-10-02
+
+### Added
+- **Core Architecture & Persistence:** Local SQLite storage engine (Schema v7) with append-only event logging, memory lifecycle tracking, and deterministic local defaults. The runtime has Python package dependencies but needs no cloud account or model/tokenizer download with those defaults.
+- **Privacy & Security Boundary:** `PatternSecretScanner` detects supported credential shapes across CLI, API, MCP, and connectors; policy rejects, quarantines, or redacts findings. Selected graph/provenance outputs suppress path-shaped labels and terminal sanitization removes control sequences; universal private-path redaction is not a scanner capability. Attack matrix verified against ANSI/OSC escape injection, zero-width chars, malformed JSONL, SQLite contention, and credential leakage.
+- **Temporal Memory:** Slot-key state tracking supporting supersession, contradiction, and coexistence relations with optimized candidate peer lookup beyond 500 records.
+- **Hybrid RAG & Retrieval Engine:** BM25 lexical indexing + dense vector search combined via Reciprocal Rank Fusion (RRF), with batched candidate hydration.
+- **Token-aware Optimizer & Compiler:** Knapsack-based context selection under token budgets with fact/provenance evidence formatting and contextual framing.
+- **Deterministic Memory Graph:** Graph projection engine supporting bounded path traversal, entity/relation links, dirty-state tracking, and opt-in graph-assisted retrieval.
+- **Model Router & Telemetry:** Provider-agnostic router (`FakeProvider`, `Ollama`, `OpenAICompatible`) with per-invocation token telemetry and measurement basis provenance.
+- **Model Context Protocol (MCP):** STDIO protocol implementation (`contextos-mcp`) with permission checks and bounded output schemas.
+- **Connectors System:** Local file and JSON/JSONL connectors with incremental sync, content hashing, and skip invariants for unchanged files.
+- **Terminal UX & Dashboard:** Interactive CLI (`contextos stats`, `monitor`, `desktop`), rich terminal formatting, and diagnostic health checks (`contextos doctor`).
+- **Explainability & RAG Inspector:** Non-intrusive trace generation and context inspection CLI (`contextos inspect`) for single-pass and side-by-side retrieval mode comparison.
+- **Canonical Benchmarking & Evaluation Suite:** Synthetic benchmarking framework (`contextos.benchmarks.final`) evaluating 100 and 1,000 memory corpora across Vector, Hybrid, Hybrid+Graph, and ContextOS strategies.
+- **Packaging & Modular Distribution:** Core `contextos` package decoupled from heavyweight ML frameworks, making `sentence-transformers` an optional extra (`pip install "contextos-memory-runtime[embeddings]"`).
+
+### Release blocker repairs
+- Core defaults use hashed deterministic embeddings and word-token accounting labeled `APPROXIMATED`. Optional SentenceTransformer embeddings and exact tiktoken counting require explicit configuration.
+- First-run demo needs no model/tokenizer assets or external network. Startup failure closes the initialized database before propagating the error.
+- Wheel/sdist rules exclude local state and nested checkouts; `python tools/validate_release_artifacts.py` validates rebuilt archives.
+- The canonical synthetic benchmark retains explicit `cl100k_base` counting; token reductions are not provider-billed savings.
+
+### Known limitations
+- Answer quality is **NOT VALIDATED**. Deterministic embeddings are an offline baseline, not a SentenceTransformer-quality claim.
+- Graph retrieval stays opt-in because the current synthetic benchmark shows a ranking/latency tradeoff.
+- Live providers have not been comprehensively validated.
+- Latest independent cold CLI startup: approximately **2.27 seconds median** on Windows.
+- Python 3.12 is declared but not validated; release validation uses Windows/Python 3.13.5.

contextos_memory_runtime-1.0.0rc2/Makefile

@@ -0,0 +1,44 @@
+.PHONY: test lint typecheck check format install dev clean benchmark
+
+install:
+	pip install -e .
+
+dev:
+	pip install -e ".[dev]"
+
+test:
+	pytest tests/ -v --tb=short -m "not slow and not benchmark"
+
+test-all:
+	pytest tests/ -v --tb=short
+
+test-unit:
+	pytest tests/unit/ -v --tb=short
+
+test-integration:
+	pytest tests/integration/ -v --tb=short
+
+test-adversarial:
+	pytest tests/adversarial/ -v --tb=short
+
+test-cov:
+	pytest tests/ -v --tb=short --cov=contextos --cov-report=term-missing --cov-report=html -m "not slow and not benchmark"
+
+benchmark:
+	pytest tests/benchmarks/ -v --tb=short -m benchmark
+
+lint:
+	ruff check src/ tests/
+
+format:
+	ruff format src/ tests/
+	ruff check --fix src/ tests/
+
+typecheck:
+	mypy src/contextos/
+
+check: lint typecheck test
+
+clean:
+	rm -rf build/ dist/ *.egg-info .pytest_cache .mypy_cache .ruff_cache htmlcov/
+	find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true

contextos_memory_runtime-1.0.0rc2/PKG-INFO

@@ -0,0 +1,143 @@
+Metadata-Version: 2.5
+Name: contextos-memory-runtime
+Version: 1.0.0rc2
+Summary: Local-first personal AI memory runtime
+Author: ContextOS Contributors
+License-Expression: MIT
+Keywords: ai,context,local-first,memory,rag
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.12
+Requires-Dist: aiosqlite>=0.20.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: mcp<3,>=2
+Requires-Dist: numpy>=1.26.0
+Requires-Dist: psutil>=6.0.0
+Requires-Dist: pydantic-settings>=2.5.0
+Requires-Dist: pydantic>=2.9.0
+Requires-Dist: rank-bm25>=0.2.2
+Requires-Dist: rich>=13.8.0
+Requires-Dist: sqlite-vec>=0.1.0
+Requires-Dist: tiktoken>=0.7.0
+Requires-Dist: typer>=0.12.0
+Requires-Dist: uvicorn[standard]>=0.30.0
+Provides-Extra: dev
+Requires-Dist: mypy>=1.11.0; extra == 'dev'
+Requires-Dist: pre-commit>=3.8.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.3.0; extra == 'dev'
+Requires-Dist: ruff>=0.6.0; extra == 'dev'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=3.0.0; extra == 'embeddings'
+Provides-Extra: lancedb
+Requires-Dist: lancedb>=0.12.0; extra == 'lancedb'
+Provides-Extra: tui
+Requires-Dist: textual>=0.79.0; extra == 'tui'
+Description-Content-Type: text/markdown
+
+# ContextOS
+
+ContextOS is a local-first, model-independent AI memory runtime. It is not a chatbot. It accepts activity through a privacy boundary, extracts candidate facts, resolves temporal state, persists memories in SQLite, and supplies bounded context to a chosen model. A local daemon exposes a CLI and API; optional STDIO MCP and explicitly configured connectors use the same memory pipeline.
+
+## Why it exists
+
+Long conversation histories are expensive and can carry stale or contradictory facts. ContextOS combines lexical and dense retrieval, temporal eligibility, a token-aware selector, and a context compiler. It records what it selected and what it could prove, without claiming that fewer context tokens automatically improve answers or provider billing.
+
+## Quick start
+
+Python 3.12+ is declared; this release has been verified on Windows with Python 3.13. Python 3.12 is not yet validated. Core defaults and the offline demo require no commercial API key, model download, or tokenizer cache.
+
+### Installation
+
+Core lightweight installation (uses deterministic/local capabilities):
+```powershell
+pip install contextos-memory-runtime
+```
+
+Core defaults use hashed deterministic embeddings and word-token accounting labeled `APPROXIMATED`. These embeddings are an offline baseline; SentenceTransformer retrieval quality is not implied. FakeProvider is a local simulation.
+
+Rebuilt RC2 archives are approximately **220 KB for the wheel** and **300 KB for the sdist** (decimal units). These sizes exclude installed dependencies. Run `python tools/validate_release_artifacts.py` after a rebuild for exact byte counts.
+
+With optional local sentence-transformers embeddings extra:
+```powershell
+pip install "contextos-memory-runtime[embeddings]"
+```
+
+Installing the extra does not change defaults. Explicitly set `[embedding]` with `model = "all-MiniLM-L6-v2"` in your ContextOS `config.toml` to select it; first use may download that model unless provisioned locally. Exact context counting is also opt-in: set `[token_counter]` with `encoding = "cl100k_base"` or `"o200k_base"`, and provision its tokenizer cache before offline use. Default `encoding = "deterministic"` needs no tokenizer assets.
+
+For development:
+```powershell
+pip install -e ".[dev]"
+```
+
+### Usage
+
+```powershell
+contextos demo
+contextos start
+contextos doctor
+contextos stats
+"I prefer concise technical explanations." | contextos memories remember
+contextos inspect "What explanation style do I prefer?"
+contextos stop
+```
+
+The daemon listens on loopback by default. Review your local configuration before changing its host, connector roots, MCP permissions, or provider settings. Prefer `memories remember` for private text: a command-line argument may be visible in process lists and shell history.
+
+## Architecture and memory lifecycle
+
+Input flows through privacy scanning, extraction, temporal acceptance, and SQLite persistence. Retrieval reads the memory store through ephemeral BM25 and dense indexes. The graph is a separate deterministic projection. The optimizer selects within a token budget; the compiler turns selected memories into context with fact/provenance evidence. The router sends the prepared request to a configured local or optional remote provider. Graph-assisted retrieval remains opt-in because its ranking performance is mixed in the local benchmark.
+
+See [architecture](docs/architecture.md), [explainability](docs/explainability.md), and [RAG inspector](docs/rag-inspector.md) for exact boundaries. Explainability and inspection describe ContextOS preparation, not an unseen provider wire payload. Provider dispatch is `NOT_ATTEMPTED` during inspection.
+
+## Privacy and integrations
+
+Secret scanning does not universally redact private filesystem paths. Selected graph/provenance output surfaces suppress path-shaped labels; terminal sanitization removes control sequences. These are separate output protections.
+
+Secrets are scanned before accepted memory is stored; the legacy `--skip-secret-scan` request field does not disable that boundary. Metadata dashboards omit memory bodies, prompts, source paths, and credentials by default. Explicit content-view commands can reveal private text on your terminal; use them deliberately. Local storage is not a substitute for OS disk encryption or trusted local-user access.
+
+Credential-free local file and JSON/JSONL connectors must be configured explicitly; none are registered by default. See [connectors](docs/connectors.md). MCP is disabled by default, uses STDIO, and separates read, write, and telemetry permissions; see [MCP](docs/mcp.md). Ollama and compatible local endpoints are first-class options; external providers are optional. FakeProvider supports deterministic tests/demo and is identified as simulated.
+
+## Terminal product
+
+| Command | Purpose |
+| --- | --- |
+| `contextos status`, `health`, `doctor` | Daemon state and non-destructive diagnostics |
+| `contextos stats [--model ID] [--provider ID] [--compare] [--today/--week]` | Measured activity by provider/model and token basis |
+| `contextos monitor [--model ID] [--provider ID]` | Poll the same bounded local dashboard |
+| `contextos inspect "query" [--mode hybrid] [--graph] [--memory UUID] [--compare] [--json]` | Retrieval-to-compiler decision evidence |
+| `contextos explain "query"`, `preview "query"` | Explain or preview context without provider dispatch |
+| `contextos graph stats/search/show` | Projection statistics and bounded graph paths |
+| `contextos memories current/conflicts/history` | Temporal metadata; content is opt-in |
+| `contextos connectors list/status/sync` | Registered connector state and explicit sync |
+| `contextos models list`, `contextos telemetry` | Provider inventory and measured invocation records |
+| `contextos benchmark [--extended]`, `contextos demo` | Isolated synthetic evaluation and offline walkthrough |
+
+The dashboard groups provider and model together and keeps context-token counts separate from provider-reported usage. A reduction bar appears only when a single known context tokenizer basis can be compared. Older telemetry with unknown provenance is not silently combined. `--today` is the current UTC day; `--week` is a rolling seven-day window. Session-wide history is not yet persisted as a distinct aggregate.
+Connector status includes currently tracked source-item count; the existing schema does not retain last-sync accepted/unchanged/failed totals for historical display.
+
+## Evidence and limits
+
+The [benchmark](docs/benchmarking.md) compares full history, vector, hybrid, hybrid+graph, and compiled ContextOS context on ten fixed questions at 100 and 1,000 synthetic memories. It reports relevance ranking and context tokens, not answer quality. The [security](docs/security.md), [performance](docs/performance.md), [demo](docs/demo.md), and [release checklist](docs/release-checklist.md) documents separate measured, simulated, and unverified claims. A 5,000-memory run is opt-in. No cloud deployment or commercial-provider proof is implied by the local test suite.
+
+Answer quality is **NOT VALIDATED**. Live providers have not been comprehensively validated. Graph retrieval stays opt-in because the current synthetic benchmark shows a recall/latency tradeoff. Latest independent cold CLI startup was approximately **2.27 seconds median** on Windows; this is a local measurement, not a guarantee.
+
+## Development
+
+```powershell
+python -m pytest tests -q -ra
+python -m compileall -q src tests
+python -m contextos.benchmarks.final
+python -m pip check
+git diff --check
+```
+
+`make test`, `make lint`, `make typecheck`, and `make check` are available when `make` is installed. The package is currently version 1.0.0-rc2. License: MIT.
+
+Background `contextos start` polls the initialized daemon for up to 10 seconds before reporting success. Existing-daemon readiness requires the health PID to match the verified recorded PID at the configured host and port. Startup and stop share a Windows/POSIX lifecycle lock with a 15-second acquisition timeout; concurrent starters wait, then report the verified daemon as already running. The persistent `contextos.lock` file is reusable and OS lock ownership is released on process exit. PID publication is atomic, and failed startup cleans up only its own process tree and matching PID state. `contextos doctor` can be run immediately after a successful start.

contextos_memory_runtime-1.0.0rc2/README.md

@@ -0,0 +1,100 @@
+# ContextOS
+
+ContextOS is a local-first, model-independent AI memory runtime. It is not a chatbot. It accepts activity through a privacy boundary, extracts candidate facts, resolves temporal state, persists memories in SQLite, and supplies bounded context to a chosen model. A local daemon exposes a CLI and API; optional STDIO MCP and explicitly configured connectors use the same memory pipeline.
+
+## Why it exists
+
+Long conversation histories are expensive and can carry stale or contradictory facts. ContextOS combines lexical and dense retrieval, temporal eligibility, a token-aware selector, and a context compiler. It records what it selected and what it could prove, without claiming that fewer context tokens automatically improve answers or provider billing.
+
+## Quick start
+
+Python 3.12+ is declared; this release has been verified on Windows with Python 3.13. Python 3.12 is not yet validated. Core defaults and the offline demo require no commercial API key, model download, or tokenizer cache.
+
+### Installation
+
+Core lightweight installation (uses deterministic/local capabilities):
+```powershell
+pip install contextos-memory-runtime
+```
+
+Core defaults use hashed deterministic embeddings and word-token accounting labeled `APPROXIMATED`. These embeddings are an offline baseline; SentenceTransformer retrieval quality is not implied. FakeProvider is a local simulation.
+
+Rebuilt RC2 archives are approximately **220 KB for the wheel** and **300 KB for the sdist** (decimal units). These sizes exclude installed dependencies. Run `python tools/validate_release_artifacts.py` after a rebuild for exact byte counts.
+
+With optional local sentence-transformers embeddings extra:
+```powershell
+pip install "contextos-memory-runtime[embeddings]"
+```
+
+Installing the extra does not change defaults. Explicitly set `[embedding]` with `model = "all-MiniLM-L6-v2"` in your ContextOS `config.toml` to select it; first use may download that model unless provisioned locally. Exact context counting is also opt-in: set `[token_counter]` with `encoding = "cl100k_base"` or `"o200k_base"`, and provision its tokenizer cache before offline use. Default `encoding = "deterministic"` needs no tokenizer assets.
+
+For development:
+```powershell
+pip install -e ".[dev]"
+```
+
+### Usage
+
+```powershell
+contextos demo
+contextos start
+contextos doctor
+contextos stats
+"I prefer concise technical explanations." | contextos memories remember
+contextos inspect "What explanation style do I prefer?"
+contextos stop
+```
+
+The daemon listens on loopback by default. Review your local configuration before changing its host, connector roots, MCP permissions, or provider settings. Prefer `memories remember` for private text: a command-line argument may be visible in process lists and shell history.
+
+## Architecture and memory lifecycle
+
+Input flows through privacy scanning, extraction, temporal acceptance, and SQLite persistence. Retrieval reads the memory store through ephemeral BM25 and dense indexes. The graph is a separate deterministic projection. The optimizer selects within a token budget; the compiler turns selected memories into context with fact/provenance evidence. The router sends the prepared request to a configured local or optional remote provider. Graph-assisted retrieval remains opt-in because its ranking performance is mixed in the local benchmark.
+
+See [architecture](docs/architecture.md), [explainability](docs/explainability.md), and [RAG inspector](docs/rag-inspector.md) for exact boundaries. Explainability and inspection describe ContextOS preparation, not an unseen provider wire payload. Provider dispatch is `NOT_ATTEMPTED` during inspection.
+
+## Privacy and integrations
+
+Secret scanning does not universally redact private filesystem paths. Selected graph/provenance output surfaces suppress path-shaped labels; terminal sanitization removes control sequences. These are separate output protections.
+
+Secrets are scanned before accepted memory is stored; the legacy `--skip-secret-scan` request field does not disable that boundary. Metadata dashboards omit memory bodies, prompts, source paths, and credentials by default. Explicit content-view commands can reveal private text on your terminal; use them deliberately. Local storage is not a substitute for OS disk encryption or trusted local-user access.
+
+Credential-free local file and JSON/JSONL connectors must be configured explicitly; none are registered by default. See [connectors](docs/connectors.md). MCP is disabled by default, uses STDIO, and separates read, write, and telemetry permissions; see [MCP](docs/mcp.md). Ollama and compatible local endpoints are first-class options; external providers are optional. FakeProvider supports deterministic tests/demo and is identified as simulated.
+
+## Terminal product
+
+| Command | Purpose |
+| --- | --- |
+| `contextos status`, `health`, `doctor` | Daemon state and non-destructive diagnostics |
+| `contextos stats [--model ID] [--provider ID] [--compare] [--today/--week]` | Measured activity by provider/model and token basis |
+| `contextos monitor [--model ID] [--provider ID]` | Poll the same bounded local dashboard |
+| `contextos inspect "query" [--mode hybrid] [--graph] [--memory UUID] [--compare] [--json]` | Retrieval-to-compiler decision evidence |
+| `contextos explain "query"`, `preview "query"` | Explain or preview context without provider dispatch |
+| `contextos graph stats/search/show` | Projection statistics and bounded graph paths |
+| `contextos memories current/conflicts/history` | Temporal metadata; content is opt-in |
+| `contextos connectors list/status/sync` | Registered connector state and explicit sync |
+| `contextos models list`, `contextos telemetry` | Provider inventory and measured invocation records |
+| `contextos benchmark [--extended]`, `contextos demo` | Isolated synthetic evaluation and offline walkthrough |
+
+The dashboard groups provider and model together and keeps context-token counts separate from provider-reported usage. A reduction bar appears only when a single known context tokenizer basis can be compared. Older telemetry with unknown provenance is not silently combined. `--today` is the current UTC day; `--week` is a rolling seven-day window. Session-wide history is not yet persisted as a distinct aggregate.
+Connector status includes currently tracked source-item count; the existing schema does not retain last-sync accepted/unchanged/failed totals for historical display.
+
+## Evidence and limits
+
+The [benchmark](docs/benchmarking.md) compares full history, vector, hybrid, hybrid+graph, and compiled ContextOS context on ten fixed questions at 100 and 1,000 synthetic memories. It reports relevance ranking and context tokens, not answer quality. The [security](docs/security.md), [performance](docs/performance.md), [demo](docs/demo.md), and [release checklist](docs/release-checklist.md) documents separate measured, simulated, and unverified claims. A 5,000-memory run is opt-in. No cloud deployment or commercial-provider proof is implied by the local test suite.
+
+Answer quality is **NOT VALIDATED**. Live providers have not been comprehensively validated. Graph retrieval stays opt-in because the current synthetic benchmark shows a recall/latency tradeoff. Latest independent cold CLI startup was approximately **2.27 seconds median** on Windows; this is a local measurement, not a guarantee.
+
+## Development
+
+```powershell
+python -m pytest tests -q -ra
+python -m compileall -q src tests
+python -m contextos.benchmarks.final
+python -m pip check
+git diff --check
+```
+
+`make test`, `make lint`, `make typecheck`, and `make check` are available when `make` is installed. The package is currently version 1.0.0-rc2. License: MIT.
+
+Background `contextos start` polls the initialized daemon for up to 10 seconds before reporting success. Existing-daemon readiness requires the health PID to match the verified recorded PID at the configured host and port. Startup and stop share a Windows/POSIX lifecycle lock with a 15-second acquisition timeout; concurrent starters wait, then report the verified daemon as already running. The persistent `contextos.lock` file is reusable and OS lock ownership is released on process exit. PID publication is atomic, and failed startup cleans up only its own process tree and matching PID state. `contextos doctor` can be run immediately after a successful start.

contextos_memory_runtime-1.0.0rc2/docs/architecture.md

@@ -0,0 +1,37 @@
+# Architecture
+
+ContextOS is a single-machine daemon, not a chatbot or hosted account service. The CLI talks to the FastAPI daemon on the configured loopback address. `wire_services()` composes SQLite repositories, privacy/extraction/temporal services, retrieval indexes, graph projection, optimizer, compiler, model routing, telemetry, connectors, explainability, and inspection.
+
+## Write and read paths
+
+```text
+CLI / API / opt-in MCP / configured connector
+  -> IngestionPipeline (privacy scan, rule-based candidate extraction, provenance event)
+  -> TemporalMemoryService.accept() (per-candidate durable transaction)
+  -> SQLite memories and relations
+  -> retrieval/graph projection invalidation
+
+query -> index synchronization -> BM25 + local dense embeddings -> eligibility
+      -> rank fusion -> optional graph augmentation -> token-aware selection
+      -> fact compiler -> optional provider routing -> invocation telemetry
+```
+
+The rule-based extractor and deterministic graph entity/relation extractor are deliberately narrower than an LLM. Ambiguous or unsupported facts may not be extracted. Multiple candidate writes are individually atomic, not one transaction for the entire request. Connector cursor advancement follows its own failure semantics; see [connectors](connectors.md).
+
+## Storage, retrieval, and time
+
+SQLite schema version 7 stores memories, events, temporal relations, graph projection/supports, connector state, and model-invocation telemetry. The database uses WAL. BM25 and in-memory dense indexes are rebuildable from SQLite. The retrieval index synchronizer checks a persisted-memory fingerprint, then rebuilds as needed. Candidate hydration uses bounded batches on the SQLite repository, with the repository-protocol fallback still supported. This avoids per-candidate SQLite round trips but index synchronization still scales with corpus size.
+
+The current temporal scope excludes stale states by default. Explicit historical/all scopes can retrieve them. The temporal peer lookup checks the latest active same-subject/property peer, without the old 500-row truncation. The graph is a persisted deterministic projection of memories and relations with a dirty bit and bounded traversal (maximum three hops, explicit node/edge limits). Graph-assisted ranking is opt-in, not a global default.
+
+## Context and evidence
+
+The optimizer chooses candidate memories under a budget; the compiler emits bounded facts/context. These are distinct operations: selected memories need not contribute a compiled fact. Phase 13 explainability uses recorded retrieval, temporal, graph, optimizer, and compiler evidence. Phase 14 inspection adds a structured, bounded read-only view over one such execution. Its optional comparison runs additional retrieval strategies explicitly. Neither mechanism sends a provider request. A logical request fingerprint is not a hash of provider wire bytes; downstream adapter transformations cannot be inferred from it. Unknown candidate or graph membership remains `NOT_AVAILABLE`.
+
+Telemetry records model/provider identifiers, counts, timing, measurement source/tokenizer, status, and errors, not raw prompts/responses. A provider-reported input count and a local preflight/context count are different measurement bases and must not be added together. The dashboard groups by provider, model, and context measurement basis. See [RAG inspector](rag-inspector.md) and [explainability](explainability.md).
+
+## Integrations and trust
+
+MCP is optional STDIO with separately configured permissions. Local-file and JSON import connectors are explicit and credential-free; no connector is registered by default. FakeProvider is a simulation for tests/demo. Ollama and OpenAI-compatible local endpoints can be configured; external providers remain optional. The daemon is not a multi-user authorization boundary. Loopback binding, trusted local-user access, filesystem permissions, and disk encryption remain operational responsibilities.
+
+No schema migration was added for Phases 14-18. The inspector, dashboards, benchmark, and demo use existing v7 data and temporary benchmark/demo databases.

contextos_memory_runtime-1.0.0rc2/docs/benchmarking.md

@@ -0,0 +1,13 @@
+# Benchmarking and claim boundaries
+
+Run `python -m contextos.benchmarks.final` or `contextos benchmark --json`. The default measures isolated 100- and 1,000-memory SQLite corpora, ten fixed questions, and two iterations. `--extended` adds 5,000 records and is deliberately optional. Direct benchmark seeding creates synthetic fixture rows only in a disposable database; it is not a production ingestion path. All index sizes are checked against the intended corpus to prevent a false scale claim.
+
+The five comparisons are full-history corpus order, dense/vector retrieval, hybrid retrieval, hybrid+graph, and ContextOS hybrid retrieval followed by optimizer/compiler (graph off by default). Fixed relevant memory IDs are declared before retrieval. Recall/Precision/Hit at 1/3/5/10, MRR, and NDCG@10 use those IDs. Duplicate relevant records are distinct IDs. Full history has no measured retrieval latency because it is represented as the corpus-order context baseline rather than an executed retrieval engine. Its token total includes all records, including a stale one; other strategies use current scope.
+
+For compiled ContextOS context, `required_source_coverage` counts relevant labeled memory IDs that actually appear as source IDs of emitted facts, divided by all relevant IDs across queries. It is an evidence-coverage proxy, not semantic answer correctness; duplicate relevant memories are counted separately. `stale_source_facts` counts emitted facts attributed to explicitly stale fixture IDs. The separate temporal suite evaluates relation classification and current-state handling.
+
+The run records candidate/optimized/compiled context token totals with one tokenizer basis, weighted reduction, selected-memory and emitted/excluded/duplicate-fact counts, provenance coverage, budget utilization, stale inclusion query count, mean/median/p95 local latency (including separate final retrieval/optimizer/compiler), top-10 hybrid/graph overlap, graph rebuild time, inspector time, index counts, SQLite bytes, process RSS, and graph counts. The fixture directly seeds memory rows without provenance events, so its zero provenance coverage is a property of the fixture, not of normal ingestion. Temporal relation classification is evaluated in the separate existing 17-case synthetic temporal harness included in final JSON. No generated answers are graded, so answer quality is `NOT AVAILABLE`. The benchmark does not prove realistic production quality, provider billing reduction, or universal graph value. Fixed data/IDs reduce variance, but timing and tied ranking can still vary by run and platform.
+
+See [final measured run](final-benchmark.md) and its [condensed machine-readable snapshot](final-benchmark.json) for actual Windows/Python results, and [performance](performance.md) for before/after optimization scope. The benchmark command itself emits the full JSON schema. Existing canonical harnesses remain `python -m contextos.benchmarks.connectors`, `.terminal`, and `.explainability`.
+
+The canonical final benchmark explicitly selects exact `cl100k_base` counting to preserve the published snapshot basis. Provision its tokenizer cache before offline benchmarking; first use otherwise requires an asset download. Core startup and the offline demo use the deterministic word approximation labeled `APPROXIMATED`, with no tokenizer assets.

contextos_memory_runtime-1.0.0rc2/docs/connectors.md

@@ -0,0 +1,25 @@
+# ContextOS connectors
+
+Phase 11 supports credential-free local activity sources: explicitly configured
+text/Markdown files, JSON/JSONL imports, and a deterministic fake connector for
+tests. A connector returns bounded `ConnectorItem` values; it cannot write a
+memory row. The manager sends every item through privacy, extraction, temporal
+acceptance, persistence, and normal graph/index invalidation.
+
+Source identity is `(connector_id, external_id)` plus revision/content hash.
+Unchanged items are skipped before ingestion. Cursor and source identity state
+are persisted without raw source content. Failed items stop cursor advancement;
+the next sync retries from the prior safe cursor.
+
+Local files are restricted to configured resolved roots, an extension allowlist
+(`.txt`, `.md`, `.json`, `.jsonl`), deterministic ordering, UTF-8 decoding, and
+a file-size limit. Symlink targets outside an allowed root are excluded.
+
+Source deletion records the source item as deleted. The default policy keeps
+derived memories: deleting a source is not a user request to purge memory.
+Disabling a connector stops sync and retains state and memories. Future policy
+may explicitly expire source-bound memory; automatic purge is never performed.
+
+Connector input, metadata, Markdown, HTML-like text, and URLs are untrusted
+data. Privacy scanner findings, raw content, credentials, and secret values are
+not stored in cursor state or connector telemetry.

contextos_memory_runtime-1.0.0rc2/docs/demo.md

@@ -0,0 +1,7 @@
+# Offline golden demo
+
+Run `contextos demo --json` or `python -m contextos.demo` without starting a daemon, cloud account, API key, or local model server. It creates a temporary SQLite database, wires the normal deterministic local services, and removes the directory when finished. No real user data is read or modified.
+
+The demo sends a preference and a later changed preference through ingestion and temporal acceptance, syncs a fake connector twice to show accepted then unchanged behavior, builds and traverses the graph, runs explainability and inspection without provider dispatch, invokes FakeProvider once, queries its telemetry, and confirms a secret-shaped input is rejected. The output reports measured IDs/counts/statuses and inspector token diff. FakeProvider response is simulated, not a production-provider proof. The synthetic demo is a walkthrough, not a benchmark or answer-quality assessment.
+
+The first run needs no network, model weights, or existing tokenizer cache. It explicitly selects deterministic embeddings and word-token accounting labeled `APPROXIMATED` with tokenizer `deterministic-word-approximation`. FakeProvider usage is simulated; approximate context counts are not exact BPE counts or provider-billed savings.

contextos_memory_runtime-1.0.0rc2/docs/explainability.md

@@ -0,0 +1,73 @@
+# ContextOS explainability
+
+Phase 13 exposes an ephemeral, deterministic account of one retrieval, optimizer, and compiler execution. The final context is what ContextOS prepared for a downstream model. Standalone explanations and previews mark provider dispatch as `NOT_ATTEMPTED`; actual provider dispatch evidence is captured only when a real model invocation executes through `ModelService.ask(..., explain=True)`. The explanation trace has a server-generated UUID; timings and trace IDs can vary while ranking and decision fields remain deterministic for unchanged inputs and configuration.
+
+## What the trace can establish
+
+### 1. Retrieval & Channels
+For memories returned by the retrieval stage, the trace reports lexical BM25 rank and raw score, dense rank and raw score, fused score, reciprocal rank contributions, metadata adjustments, rank, and source strategies. BM25 and dense scores use different scales; compare ranks/fused scores rather than interpreting one as a calibrated percentage.
+
+Candidate absence is categorized truthfully without guessing:
+- `SELECTED`: Memory was retrieved, selected by the optimizer, and included in the compiled context.
+- `COMPILER_EXCLUDED`: Memory was selected by the optimizer but the compiler's persisted-in-memory `included_memory_ids` evidence does not include it.
+- `RETRIEVED_BUT_OPTIMIZER_EXCLUDED`: Memory was retrieved, but optimizer excluded it due to token budget or redundancy constraints.
+- `RESULT_LIMIT`: Memory was present in the fused/reranked candidate set before the final top-K truncation cut it.
+- `CHANNEL_NOT_RETRIEVED`: Memory is indexed and eligible, but was not surfaced by bounded candidate channels for this query.
+- `INDEX_NOT_PRESENT`: Memory is directly proven absent from both lexical BM25 and dense vector indices, and graph retrieval was not active for this explanation. Graph-channel membership is not inferred from those two stores.
+- `TEMPORALLY_INELIGIBLE`: Stored memory state or temporal scope policy excluded the candidate.
+- `NOT_AVAILABLE`: Evidence is insufficient to establish why the memory was absent.
+
+Channel candidate IDs and pre-limit IDs are retained with explicit truncation flags. If a requested memory is outside a truncated snapshot, the explanation returns `NOT_AVAILABLE`; it does not guess between channel absence and result-limit exclusion.
+
+### 2. Graph Evidence
+Graph traces capture structured path nodes and edges collected during graph traversal without second-pass querying:
+- Traversal bounds: strictly bounded to `max_hops <= 3` (up to 3 edges and up to 4 nodes per path), `max_nodes <= 100`, and `max_edges <= 250`.
+- `GraphPathNode`: `node_id`, `node_type`, safe sanitized label, and `project_scope` (if stored). Missing labels are reported as `null` or excluded; labels are never invented.
+- `GraphPathEdge`: `edge_type`, `confidence`, `supporting_memory_ids`, and `project_scope`.
+- `GraphCandidateEvidence`: `seed_memory_id`, `candidate_memory_id`, `hop_count`, `graph_score`, `path_nodes`, `path_edges`, and `scope_match` evidence.
+
+### 3. Temporal Evidence
+The structured `TemporalEvidenceResolver` reports provable persisted state and relations:
+- Lifecycle status (`active`, `superseded`, `expired`, `deleted`, `purged`).
+- Stored relations (`CORRECTS`, `COEXISTS_WITH`, `CONTRADICTS`, `SUPERSEDES`).
+- Related endpoint state (`present`, `deleted`, or `missing`) resolved for either direction of a persisted relation.
+- Replacement memory ID (`replacement_memory_id`) when superseded.
+- Observed and effective timestamps.
+- Acceptance rationale is strictly `null` (no fictional historical reasoning or motives are inferred).
+
+### 4. Compiler transformations
+Compiler actions are recorded from emitted and excluded compiler facts: `RAW_INCLUDED`, `DEDUPLICATED`, `MERGED`, `COMPRESSED`, `RESCUED_FROM_OVERSIZED_MEMORY`, and `DROPPED`. Exact raw inclusion, merge source count, rescue input kind, and exclusion reason are evidenced. When output differs from the original memory but the compiler does not record whether it was clause extraction or compression, the trace reports `UNKNOWN` rather than claiming `COMPRESSED`. The trace does not invent a transformation for a memory with no compiler fact; optimizer selection and compiler inclusion are reported separately.
+
+### 5. Provider Dispatch Proof & Receipt
+For standalone explanations (`contextos explain`, MCP `contextos_explain_context`, `POST /api/v1/explain`), the dispatch state is strictly:
+- `ProviderDispatchState.NOT_ATTEMPTED`: "prepared by ContextOS; no provider dispatch attempted"
+
+For real model invocations (`ModelService.ask`), evidence tracks:
+- `state`:
+  - `REQUEST_CONSTRUCTED`: Context and ModelRequest constructed and fingerprinted.
+  - `DISPATCH_ATTEMPTED`: Provider invocation `generate()` was attempted at the adapter boundary (does not claim verified wire delivery).
+  - `RESPONSE_RECEIVED`: Downstream provider returned a successful response.
+  - `DISPATCH_FAILED`: Provider invocation raised an error or timed out. Evidence is attached to the raised Python exception and returned in the structured `/ask` API error response when available. SQLite telemetry records only generic `status="error"`, not the receipt.
+- `compiled_context_sha256`: SHA-256 fingerprint of the exact compiled context text.
+- `logical_request_sha256`: SHA-256 fingerprint of the actual `ModelRequest` fields observed immediately before provider invocation, represented as system prompt, compiled context, and user prompt. Provider adapters subsequently transform this into provider-specific payloads; this is not a wire-payload hash.
+- `context_match`: Boolean proof confirming the actual downstream `ModelRequest` object carried the exact compiled ContextOS context string before hashing.
+- `preflight_input_tokens`: Preflight token count calculated before dispatch.
+- `provider_input_tokens`: Token count returned directly by the downstream provider (kept separate from ContextOS context tokens).
+
+## Privacy
+
+Responses omit raw prompt text, private credentials, source URIs, connector metadata, and filesystem paths by default. Provenance includes only whitelisted source categories and event UUIDs; unknown source values are replaced with `unknown`. Terminal output is sanitized against ANSI CSI, OSC titles, OSC hyperlinks, carriage returns, backspaces, nulls, and bidi overrides. Traces are ephemeral (never persisted), and no schema v8 migration is required.
+
+## Use
+
+```powershell
+contextos explain "What are my Python preferences?" --budget 1000 --mode hybrid --limit 25
+contextos explain "What are my Python preferences?" --json
+contextos explain "What are my Python preferences?" --show-content
+contextos explain "What are my Python preferences?" --memory-id 12345678-1234-4234-8234-123456789012
+contextos explain "What did I previously prefer?" --temporal-scope historical
+contextos preview "What are my Python preferences?" --explain
+```
+
+The API accepts `POST /api/v1/explain` with `query`, `mode`, `budget`, `limit`, `graph`, `temporal_scope`, `include_content`, and optional `target_memory_id`. Query length is capped at 10,000 characters, budget at 8,000 tokens, and candidates at 100.
+

contextos_memory_runtime-1.0.0rc2/docs/final-benchmark.json

@@ -0,0 +1,46 @@
+{
+  "label": "LOCAL SYNTHETIC DETERMINISTIC BENCHMARK",
+  "command": "python -m contextos.benchmarks.final",
+  "environment": {"python": "3.13.5", "platform": "Windows-11-10.0.26200-SP0"},
+  "queries": 10,
+  "iterations": 2,
+  "tokenizer": "cl100k_base",
+  "scope": "One measured local run; condensed snapshot; full JSON emitted by command",
+  "corpora": {
+    "100": {
+      "lexical_index_count": 100,
+      "dense_index_count": 100,
+      "graph_nodes": 48,
+      "graph_edges": 55,
+      "graph_rebuild_ms": 76.887,
+      "sqlite_bytes": 278528,
+      "process_rss_bytes": 108990464,
+      "strategies": {
+        "full_history": {"recall_at_5": 0.3, "mrr": 0.2482, "ndcg_at_10": 0.2312, "candidate_tokens": 34040, "compiled_tokens": 34040, "mean_ms": null},
+        "vector": {"recall_at_5": 0.8, "mrr": 0.6867, "ndcg_at_10": 0.6932, "candidate_tokens": 3102, "compiled_tokens": 3102, "mean_ms": 20.369},
+        "hybrid": {"recall_at_5": 0.95, "mrr": 0.7833, "ndcg_at_10": 0.8263, "candidate_tokens": 2854, "compiled_tokens": 2854, "mean_ms": 20.852},
+        "hybrid_graph": {"recall_at_5": 0.6, "mrr": 0.4472, "ndcg_at_10": 0.5388, "candidate_tokens": 2956, "compiled_tokens": 2956, "mean_ms": 72.693},
+        "contextos": {"recall_at_5": 0.9, "mrr": 0.8, "ndcg_at_10": 0.7806, "candidate_tokens": 2854, "optimized_tokens": 1082, "compiled_tokens": 868, "weighted_token_reduction": 0.6959, "mean_ms": 24.419, "facts_emitted": 82, "required_source_coverage": 0.8571, "stale_source_facts": 0, "provenance_coverage": 0}
+      }
+    },
+    "1000": {
+      "lexical_index_count": 1000,
+      "dense_index_count": 1000,
+      "graph_nodes": 228,
+      "graph_edges": 235,
+      "graph_rebuild_ms": 544.252,
+      "sqlite_bytes": 1069056,
+      "process_rss_bytes": 122015744,
+      "strategies": {
+        "full_history": {"recall_at_5": 0.3, "mrr": 0.2482, "ndcg_at_10": 0.2312, "candidate_tokens": 343640, "compiled_tokens": 343640, "mean_ms": null},
+        "vector": {"recall_at_5": 0.6, "mrr": 0.6461, "ndcg_at_10": 0.6378, "candidate_tokens": 3098, "compiled_tokens": 3098, "mean_ms": 193.776},
+        "hybrid": {"recall_at_5": 0.9, "mrr": 0.7833, "ndcg_at_10": 0.7971, "candidate_tokens": 2884, "compiled_tokens": 2884, "mean_ms": 196.635},
+        "hybrid_graph": {"recall_at_5": 0.5, "mrr": 0.3483, "ndcg_at_10": 0.4069, "candidate_tokens": 3022, "compiled_tokens": 3022, "mean_ms": 435.317},
+        "contextos": {"recall_at_5": 0.85, "mrr": 0.8, "ndcg_at_10": 0.7568, "candidate_tokens": 2884, "optimized_tokens": 762, "compiled_tokens": 650, "weighted_token_reduction": 0.7746, "mean_ms": 207.143, "facts_emitted": 58, "required_source_coverage": 0.7857, "stale_source_facts": 0, "provenance_coverage": 0}
+      }
+    }
+  },
+  "temporal_fixture": {"cases": 17, "contextos_relation_accuracy": 1.0, "naive_relation_accuracy": 0.4117647058823529},
+  "answer_quality": null,
+  "answer_quality_reason": "No generated answers or independent grading"
+}

contextos_memory_runtime-1.0.0rc2/docs/final-benchmark.md

@@ -0,0 +1,22 @@
+# Final local benchmark snapshot
+
+Command: `python -m contextos.benchmarks.final` (default: 100 and 1,000 records; two iterations of ten fixed queries). Environment: Windows 11 `10.0.26200`, Python 3.13.5, deterministic local embedding, SQLite, `cl100k_base` context counter. The temporary synthetic corpora include duplicate, stale, scoped preference, project/tool, negated deployment, and long-memory records. Ground-truth relevant IDs were specified before retrieval. BM25 and dense index counts were asserted equal to each corpus size. This is one local measured run, not production or model-answer evidence.
+
+| Memories | Strategy | Recall@5 | MRR | NDCG@10 | Candidate tokens | Compiled tokens | Mean ms |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| 100 | Full history | 0.300 | 0.248 | 0.231 | 34,040 | 34,040 | not measured |
+| 100 | Vector | 0.800 | 0.687 | 0.693 | 3,102 | 3,102 | 20.369 |
+| 100 | Hybrid | 0.950 | 0.783 | 0.826 | 2,854 | 2,854 | 20.852 |
+| 100 | Hybrid + graph | 0.600 | 0.447 | 0.539 | 2,956 | 2,956 | 72.693 |
+| 100 | ContextOS final | 0.900 | 0.800 | 0.781 | 2,854 | 868 | 24.419 |
+| 1,000 | Full history | 0.300 | 0.248 | 0.231 | 343,640 | 343,640 | not measured |
+| 1,000 | Vector | 0.600 | 0.646 | 0.638 | 3,098 | 3,098 | 193.776 |
+| 1,000 | Hybrid | 0.900 | 0.783 | 0.797 | 2,884 | 2,884 | 196.635 |
+| 1,000 | Hybrid + graph | 0.500 | 0.348 | 0.407 | 3,022 | 3,022 | 435.317 |
+| 1,000 | ContextOS final | 0.850 | 0.800 | 0.757 | 2,884 | 650 | 207.143 |
+
+ContextOS's weighted reduction of supplied context bodies was 69.59% at 100 and 77.46% at 1,000; compiled-body totals were 868 and 650 tokens. This is not provider-billed-token reduction. Over twenty query executions, the compiler emitted 82/58 facts and selected memories; ID-based required-source coverage was 85.71%/78.57%, stale-source facts were zero, and provenance coverage was 0 because direct fixture rows have no provenance event IDs. This does not describe normal ingestion. Context budget utilization was 21.7%/16.25% under a 200-token limit. The latest 1,000-record graph had 228 nodes/235 edges; SQLite size was 1,069,056 bytes and whole-process RSS was 122,015,744 bytes. These are synthetic evidence proxies, not answer correctness.
+
+All twenty full-history query iterations included stale records, whereas the current-scope strategies did not. The separate seventeen-case temporal fixture yielded 1.0 ContextOS relation-classification/current-state accuracy versus 0.412/0.471 for two naive baselines; this is fixture-specific, not a real-world accuracy estimate.
+
+Supported: ContextOS reduced measured supplied context tokens on this corpus; hybrid Recall@5 exceeded vector. Mixed: final compilation lost 0.05 Recall@5 versus raw hybrid at both sizes but had slightly higher MRR (0.800 versus 0.783). Opt-in graph augmentation degraded ranking and added latency here, so no graph-default change is justified. Answer quality was not measured. See [methodology](benchmarking.md) and the [condensed JSON snapshot](final-benchmark.json) before generalizing.