neotoma 0.3.9 → 0.4.0

package/README.md

@@ -1,327 +1,184 @@
-# Neotoma: Truth Layer for Persistent Agent Memory
+# Neotoma
 
-![Neotoma banner](https://raw.githubusercontent.com/markmhendrickson/neotoma/main/docs/assets/neotoma_banner.png)
+Your agents forget. Neotoma makes them remember.
 
-_Give your agents memory you can inspect, replay, and trust._
+Versioned records — contacts, tasks, decisions, finances — that persist across Claude, Cursor, ChatGPT, OpenClaw, and every agent you run. Open-source. Local-first. Deterministic. MIT licensed.
 
-For a guided overview, see [neotoma.io](https://neotoma.io).
+**[neotoma.io](https://neotoma.io)** · **[Evaluate](https://neotoma.io/evaluate)** · **[Install](https://neotoma.io/install)** · **[Documentation](https://neotoma.io/docs)**
 
-Agent memory is forgetful. What keeps breaking automation is trust, not intelligence: memory changes implicitly, context drifts, and you cannot see what changed or replay it. When agents act, personal data becomes state. The missing primitive is a layer of explicit, inspectable, replayable state.
+## Why this exists
 
-[Neotoma](https://neotoma.io) is that layer. Open-source, privacy-protective, and user-controlled. It is contract-first and deterministic (same input, same output), with immutable, queryable state in one graph for documents you upload and data agents write.
+You run AI agents across tools and sessions. Without a state layer, you become the human sync layer:
 
-It works with Cursor, Claude, Codex, and other MCP-capable tools, with CLI fallback when MCP is unavailable. Install with npm, then configure MCP for your editor or use the CLI directly.
+- Every session starts from zero — nothing your agent learns carries over
+- Facts conflict across tools — two agents store different versions of the same person
+- Decisions execute without a reproducible trail — you can't trace why your agent acted
+- Corrections don't stick — you fix something in Claude and it's wrong again in Cursor
 
-For the full rationale, see [Building a truth layer for persistent agent memory](https://markmhendrickson.com/posts/truth-layer-agent-memory).
+These are not hypothetical. They happen every day in production agent systems. You compensate by re-prompting context, patching state gaps, and maintaining manual workarounds. Neotoma removes that tax.
 
----
+## What Neotoma does
 
-## What Neotoma Is
+Neotoma is a deterministic state layer for AI agents. It stores structured records — contacts, tasks, transactions, decisions, events, contracts — with versioned history and full provenance. Every change creates a new version. Nothing is overwritten. Every state can be replayed from the observation log.
 
-Neotoma is a _truth layer_, not an app, agent, or workflow engine. It is the lowest-level canonical source of truth for personal data (documents and agent-created data), exposed to AI tools via Model Context Protocol (MCP).
+Not retrieval memory (RAG, vector search, semantic lookup). Neotoma enforces deterministic state evolution: same observations always produce the same entity state, regardless of when or in what order they are processed.
 
-You upload documents (PDFs, images, receipts, contracts) or share information during agent conversations. You don't have to structure it yourself: agents structure and store it via Neotoma when you provide unstructured or semi-structured content. Neotoma resolves entities across all sources, builds timelines from date fields, and keeps every fact traceable to its source. ChatGPT, Claude, and Cursor can read this memory, write new structured data, correct mistakes, and trigger reinterpretation. One graph connects people, companies, events, and relationships across all your data.
-
-It's not a note-taking app or "second brain." Not provider-controlled ChatGPT Memory or Claude Projects (those are conversation-only and platform-locked; Neotoma is structured personal data memory with entity resolution and timelines, cross-platform via MCP). Not a vector store or RAG layer. Not an autonomous agent. It is the memory layer agents read and write; you control what goes in and what stays.
-
----
-
-## Three Foundations
-
-Neotoma is built on three architectural choices that provider memory cannot offer:
-
-| Foundation         | What it means                                                                                                                                         |
-| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **Privacy-first**  | User-controlled memory, end-to-end encryption and row-level security, never used for training. Your data remains yours.                               |
-| **Deterministic**  | Same input always produces same output. Schema-first extraction, hash-based entity IDs, full provenance. No hallucinations or probabilistic behavior. |
-| **Cross-platform** | Works with ChatGPT, Claude, Cursor, and Claude Code via MCP. One memory system across tools; no platform lock-in. Localhost-agent friendly.           |
-
-**These enable:** Immutable audit trail and time-travel queries, cryptographic integrity, event-sourced history, entity resolution across documents and agent data, timeline generation, dual-path storing (files + agent interactions), and persistent memory without context-window limits.
-
-**System architecture:**
+## Architecture
 
 ```mermaid
 graph LR
-  Sources[Documents + Agent Data] --> Ingest[Ingestion]
-  Ingest --> Obs[Observations]
+  Sources["Sources (files, messages, APIs)"] --> Obs[Observations]
   Obs --> Entities[Entity Resolution]
-  Entities --> Snapshots[Entity Snapshots]
+  Entities --> Snapshots["Entity Snapshots (versioned)"]
   Snapshots --> Graph[Memory Graph]
   Graph <--> MCP[MCP Protocol]
-  MCP --> ChatGPT
   MCP --> Claude
+  MCP --> ChatGPT
   MCP --> Cursor
+  MCP --> OpenClaw
 ```
 
----
-
-## Problems Solved
-
-| Problem                              | How Neotoma addresses it                                                                                      |
-| ------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
-| Personal data is fragmented          | Dual-path storing from file uploads and agent interactions into one source of truth.                          |
-| Provider memory is conversation-only | Structured personal data memory with entity resolution and timelines across documents and agent-created data. |
-| No trust when agents act             | Explicit, named operations; visible inputs; reconstructable history. Replay and audit trail.                  |
-| No cross-data reasoning              | One graph: sources, entities, events, typed relationships.                                                    |
-| Entity fragmentation                 | Hash-based canonical IDs unify "Acme Corp" across all personal data.                                          |
-| No temporal reasoning                | Automatic timeline generation from date fields.                                                               |
-| Platform lock-in                     | MCP-based access; works with any compatible AI tool.                                                          |
-
----
-
-## Core Terminology
-
-| Term            | Definition                                                                                    |
-| --------------- | --------------------------------------------------------------------------------------------- |
-| Truth Layer     | Deterministic, immutable structured memory substrate that tools and agents read and write.    |
-| Source          | Raw data (file, text, URL, or structured JSON) stored with content-addressed deduplication.   |
-| Observation     | Granular fact extracted from a source; reducers merge observations into entity snapshots.     |
-| Entity          | Canonical representation of a person, company, task, or other object with a deterministic ID. |
-| Entity snapshot | Current truth for an entity computed from all related observations.                           |
-| Provenance      | Origin tracking (source, timestamp, operation) so each value is traceable.                    |
-| Memory graph    | Graph of sources, observations, entities, events, and typed relationships.                    |
-
-For the full glossary, see [Core terminology](https://neotoma.io/#terminology).
-
----
-
-## Who It's For
-
-| Who                                   | What they need                                                           | Example data to remember                                                |
-| ------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
-| AI-native individual operators        | Memory that follows across daily tools and sessions                      | Tasks, preferences, notes, recurring reminders, contacts, deadlines     |
-| Knowledge workers with scattered data | Durable context across documents and sessions, with evidence and lineage | Source documents, extracted entities, citations, key quotes, timelines  |
-| Builders of agentic systems           | Structured memory agents can read and write with provenance              | Session histories, accumulated facts, decisions, runbooks, tool configs |
-
-**Why Neotoma:** One memory graph across documents and agent-created data; agents remember context without re-explanation; full provenance and audit trail; works with any MCP-compatible tool; privacy-first and user-controlled. The same substrate serves both human-in-the-loop use and agent frameworks or toolchains that need deterministic memory and provenance.
-
-**Suggested GitHub topics:** `ai-agents` `ai-memory` `memory-system` `entity-resolution` `event-sourcing` `truth-layer` `mcp` `provenance` `privacy-first`
-
----
-
-## Security Defaults
-
-Neotoma stores personal data and requires secure configuration.
-
-**Authentication:** Local auth is the active mode (dev stub or key-based when encryption is enabled). Session tokens are deprecated. See [OAuth implementation](docs/developer/mcp_oauth_implementation.md) for current local flow details.
-
-**Authorization:** Local data isolation and explicit operation-level access controls are enforced in local-only mode.
-
-**Data protection:** User-controlled data with full export and deletion control; never used for training or provider access. Local backend supports optional encryption at rest (key file or mnemonic). End-to-end encryption planned (v2.0.0).
-
-**Verify your setup:** Run `npm run doctor` for environment, database, RLS, and security checks. See [Health check](docs/operations/health_check.md), [Auth](docs/subsystems/auth.md), [Privacy](docs/subsystems/privacy.md), and [Compliance](docs/legal/compliance.md).
-
----
-
-## Current Status
-
-**Version:** v0.3.0  
-**Status:** Completed (reconciliation release establishing current baseline)
-
-**Developer preview:** Planned during dogfooding once core invariants are stable.
-
-**What's implemented:** Sources-first architecture with content-addressed storage, dual-path storing (file uploads + agent interactions), observations architecture, entity resolution with hash-based IDs, schema registry system, auto-enhancement, timeline generation, optional entity semantic search for `retrieve_entities` and `retrieve_entity_by_identifier` (local embeddings), MCP integration (ChatGPT, Claude, Cursor), full provenance and audit trail, React frontend, CLI. See [Release roadmap](#release-roadmap) and [docs/releases/](docs/releases/) for details.
-
-**Next steps:** Review current uncommitted changes, apply pending migrations, audit the test suite, and plan v0.4.0 realistically based on the current baseline.
-
----
-
-## Developer preview
-
-The developer preview exposes the **core contract only**: CLI for humans, MCP for agents, OpenAPI as the single source of truth. There is no web app in scope for the preview; the frontend in this repo exists for development and future use.
-
-### What is guaranteed (even in preview)
-
-- **No silent data loss** – Operations either succeed and are recorded or fail with explicit errors.
-- **Explicit, inspectable state mutations** – Every change is a named operation with visible inputs; state is reconstructable from the audit trail.
-- **Auditable operations** – Full provenance; CLI and MCP map to the same underlying contract. If it cannot be expressed via the contract, it is not real.
-- **Same contract for CLI and MCP** – Both use the same OpenAPI-backed operations; MCP tool calls log the equivalent CLI invocation (redaction-aware).
-
-### What is not guaranteed yet
-
-- Stable schemas
-- Deterministic extraction across versions
-- Long-term replay compatibility
-- Backward compatibility
-
-Breaking changes should be expected.
-
-### Who this is for
-
-- Developers comfortable with CLI-first workflows
-- People building or operating agentic systems
-- Anyone who treats personal data like production infrastructure
-
-### Who this is not for (yet)
-
-- UI-first users
-- Casual note-taking
-- Anyone expecting reliability guarantees today
-
-**Storage:** The developer preview supports **local storage only** (SQLite + local file storage). See [Developer preview storage](docs/developer/developer_preview_storage.md) and [Getting started](docs/developer/getting_started.md) for setup.
-
----
-
-## Release Roadmap
-
-### Completed Releases
-
-- **v0.2.0** – Minimal ingestion + correction loop (`in_testing`). [docs/releases/v0.2.0/](docs/releases/v0.2.0/)
-- **v0.2.1** – Documentation generation system (`in_progress`). [docs/releases/v0.2.1/](docs/releases/v0.2.1/)
-- **v0.2.2** – `list_capabilities` MCP action (`planning`). [docs/releases/v0.2.2/](docs/releases/v0.2.2/)
-- **v0.2.15** – Complete architecture migration (`implemented`, pending migrations). [docs/releases/v0.2.15/](docs/releases/v0.2.15/)
-- **v0.3.0** – Reconciliation release (`completed`). [docs/releases/v0.3.0/](docs/releases/v0.3.0/)
+- **Deterministic.** Same observations always produce the same versioned entity snapshots. No ordering sensitivity.
+- **Immutable.** Append-only observations. Corrections add new data, never erase.
+- **Replayable.** Inspect any entity at any point in time. Diff versions. Reconstruct history from the observation log.
+- **Structure-first.** Schema-first extraction with deterministic retrieval. Optional similarity search when embeddings are configured.
 
-### Future Planning
+### Three foundations
 
-Future releases will be planned realistically based on the v0.3.0 baseline. Previous aspirational releases (v0.4.0 through v2.2.0) have been archived to [docs/releases/archived/aspirational/](docs/releases/archived/aspirational/) and can be revisited for future planning.
+| Foundation         | What it means                                                                                                                           |
+| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
+| **Privacy-first**  | Your data stays local. Never used for training. User-controlled storage, optional encryption at rest. Full export and deletion control. |
+| **Deterministic**  | Same input always produces same output. Schema-first extraction, hash-based entity IDs, full provenance. No silent mutation.            |
+| **Cross-platform** | One memory graph across Claude, ChatGPT, Cursor, OpenClaw, Codex, and CLI. MCP-based access. No platform lock-in. Works alongside native memory. |
 
-Full release index: [docs/releases/](docs/releases/).
+## State guarantees
 
----
+Most AI memory systems optimize storage or retrieval. Neotoma enforces state integrity. [Full comparison with explanations →](https://neotoma.io/memory-guarantees)
 
-## Documentation Index
+| Property                             | Platform  | Retrieval / RAG | Files      | Database      | Neotoma       |
+| ------------------------------------ | --------- | --------------- | ---------- | ------------- | ------------- |
+| Deterministic state evolution        | ✗         | ✗               | ✗          | ✗             | ✓             |
+| Versioned history                    | ✗         | ✗               | ⚠ manual   | ✗             | ✓             |
+| Replayable timeline                  | ✗         | ✗               | ✗          | ✗             | ✓             |
+| Auditable change log                 | ✗         | ✗               | ⚠ partial  | ✗             | ✓             |
+| Schema constraints                   | ✗         | ✗               | ✗          | ⚠ partial     | ✓             |
+| Silent mutation risk                 | ⚠ common  | ⚠ common        | ⚠ common   | ⚠ common      | prevented     |
+| Conflicting facts risk               | ⚠ common  | ⚠ common        | ⚠ possible | ⚠ common      | prevented     |
+| Reproducible state reconstruction    | ✗         | ✗               | ✗          | ✗             | ✓             |
+| Human inspectability (diffs/lineage) | ⚠ partial | ⚠ partial       | ⚠ partial  | ⚠ partial     | ✓             |
+| Zero-setup onboarding                | ✓         | ✗               | ✗          | ✗             | ✗             |
+| Semantic similarity search           | ✗         | ✓               | ✗          | ✗             | ✓             |
+| Direct human editability             | ✗         | ✗               | ✓          | ✗             | ✗             |
 
-The **primary entrypoint** for all documentation is the index and navigation guide. All contributors and AI assistants working on the repo should load it first.
+**Platform:** Claude, ChatGPT, Gemini, Copilot. **Retrieval:** Mem0, Zep, LangChain Memory. **Files:** Markdown files, JSON stores, CRDT docs. **Database:** SQLite, Postgres, MySQL. **Neotoma:** Deterministic state layer (reference implementation).
 
-- **[Documentation index and navigation](#documentation-index)** – Map of the docs system, reading order by change type, dependency graph, and quick-reference answers. Start here when contributing or navigating the repo.
+## Quick start
 
-**Foundational (load first):**
+### Evaluate first (recommended)
 
-- [Core identity](docs/foundation/core_identity.md) – What Neotoma is and is not
-- [Philosophy](docs/foundation/philosophy.md) – Principles and invariants
-- [Problem statement](docs/foundation/problem_statement.md) – Why Neotoma exists
-- [Layered architecture](docs/foundation/layered_architecture.md) – Truth / Strategy / Execution layers
+Ask your AI agent whether Neotoma fits your workflow. Paste this into Claude, Cursor, ChatGPT, or any MCP-capable tool:
 
-**Specifications and architecture:**
-
-- [MVP overview](docs/specs/MVP_OVERVIEW.md) – Product summary
-- [Architecture](docs/architecture/architecture.md) – System design
-- [MCP spec](docs/specs/MCP_SPEC.md) – MCP action catalog
-- [Schema](docs/subsystems/schema.md) – Database schema and JSONB structures
-
-**Developer:**
-
-- [Getting started](docs/developer/getting_started.md) – Setup, local-only storage, first run
-- [Developer preview storage](docs/developer/developer_preview_storage.md) – Local-only stance for the preview
-- [CLI overview](docs/developer/cli_overview.md) – CLI workflows
-- [CLI reference](docs/developer/cli_reference.md) – Commands and scripts
-- [MCP overview](docs/developer/mcp_overview.md) – MCP entry points and setup links
-- [Development workflow](docs/developer/development_workflow.md) – Git, branches, PRs
-- [MCP developer docs](docs/developer/mcp/) – Instructions, tool descriptions, unauthenticated use
+```
+Read https://neotoma.io/evaluate. Before you answer, inspect any broader tool, workspace, and configuration context you can access so you assess my real workflow rather than just this chat. Then tell me whether Neotoma is a real fit for my workflow, what data would be highest value to persist first, and what I should do next.
+```
 
-**Other categories (see index for full tables):**
+The agent reads the evaluation page, checks your context, and gives you an honest assessment. If it's a fit, the agent carries you through install and activation.
 
-- [Specs](docs/specs/) – Requirements, ICP profiles, data models, test plan
-- [Subsystems](docs/subsystems/) – Ingestion, observation architecture, reducer, relationships, auth, events, errors, privacy
-- [Feature units](docs/feature_units/) – Standards, manifests, execution instructions
-- [Testing](docs/testing/) – [Automated test catalog](docs/testing/automated_test_catalog.md), [testing standard](docs/testing/testing_standard.md)
-- [Operations](docs/operations/) – [Runbook](docs/operations/runbook.md) (startup, health, shutdown), [Health check](docs/operations/health_check.md) (`npm run doctor`), [Troubleshooting](docs/operations/troubleshooting.md)
-- [API](docs/api/rest_api.md), [Error codes](docs/reference/error_codes.md), [Vocabulary](docs/vocabulary/canonical_terms.md)
+### Agent-driven install
 
----
+Agents install Neotoma themselves. Paste this prompt into Claude, Cursor, ChatGPT, or Codex:
 
-## Installation
+```
+Read https://neotoma.io/install and carry me through the install-first Neotoma flow. Install, activate it with my data, and configure my current tool for robust ongoing use.
+```
 
-### Option 1: Install from npm (recommended for users)
+The agent handles npm install, initialization, and MCP configuration. **Manual install:**
 
 ```bash
-# Global install (recommended for MCP server use)
 npm install -g neotoma
-
-# Initialize (creates data directories, database; init can prompt for encryption)
 neotoma init
-
-# Start the API server
-neotoma api start
+neotoma mcp config
 ```
 
-Source checkout resolution precedence is: explicit CLI inputs (where supported), then `NEOTOMA_REPO_ROOT`, then directory-local checkout from current working directory, then saved config fallback.
+More options: [Docker](docs/developer/docker.md) | [CLI reference](docs/developer/cli_reference.md) | [Getting started](docs/developer/getting_started.md)
 
-CLI transport defaults are offline-first. Use `--api-only` when you explicitly want remote API transport:
+## Example
 
 ```bash
-# Default (in-process local transport; no API server required)
+neotoma store --json='[{"entity_type":"task","title":"Submit expense report","status":"open"}]'
 neotoma entities list --type task
-
-# Explicit remote API mode (fail if API is unreachable)
-neotoma --api-only entities list --type task
+neotoma upload ./invoice.pdf
 ```
 
-After installation, configure MCP for your AI tool:
+Results reflect versioned entity state with full provenance. Agents perform the same operations through MCP tool calls (`store`, `retrieve_entities`, `retrieve_entity_by_identifier`).
 
-```bash
-neotoma mcp config
-```
+## Interfaces
 
-For first-run agent-guided setup (install + init + preview-before-save confirmation), see [Agent onboarding](docs/developer/agent_onboarding.md).
+Three interfaces. One state invariant. Every interface provides the same deterministic behavior regardless of how you access the state layer.
 
-Marketing site: **https://neotoma.io** (GitHub Pages, deployed from **main**; custom domain set in repo Settings → Pages). See [Deployment](docs/infrastructure/deployment.md#marketing-site-neotomaio).
+| Interface      | Description                                                                                                                    |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| **REST API**   | Full HTTP interface for application integration. Entities, relationships, observations, schema, timeline, and version history. |
+| **MCP Server** | Model Context Protocol for Claude, ChatGPT, Cursor, OpenClaw, Codex, and more. Agents store and retrieve state through structured tool calls. |
+| **CLI**        | Command-line for scripting and direct access. Inspect entities, replay timelines, and manage state from the terminal.          |
 
-### Option 2: Clone source checkout (for development)
+All three map to the same OpenAPI-backed operations. MCP tool calls log the equivalent CLI invocation.
 
-```bash
-git clone https://github.com/markmhendrickson/neotoma.git
-cd neotoma
-npm install
-npm test
-```
+## Who this is for
 
-**Prerequisites:** Node.js v18.x or v20.x (LTS), npm v9+. Developer preview uses **local storage only**. For local storage, **no `.env` is required**; the app uses defaults (`./data`, `./data/neotoma.db`, `./data/sources`). Optional overrides: [Getting started](docs/developer/getting_started.md).
+People building a personal operating system with AI agents across their life — wiring together tools like Claude, Cursor, ChatGPT, OpenClaw, and custom scripts to manage contacts, tasks, finances, code, content, and other domains. The same person operates their agents, builds new pipelines, and debugs state drift. These are three operational modes, not separate personas:
 
-### Option 3: Run with Docker
+| Mode | What you're doing | The tax you pay without Neotoma | What you get back |
+| ---- | ----------------- | ------------------------------- | ----------------- |
+| **Operating** | Running AI tools across sessions and contexts | Re-prompting, context re-establishment, manual cross-tool sync | Attention, continuity, trust in your tools |
+| **Building** | Shipping agents and pipelines | Prompt workarounds, dedup hacks, memory regression fixes | Product velocity, shipping confidence |
+| **Debugging** | Tracing state drift and reproducing failures | Writing glue (checkpoint logic, custom diffing, state serialization) | Debugging speed, platform design time |
 
-```bash
-git clone https://github.com/markmhendrickson/neotoma.git
-cd neotoma
-docker build -t neotoma .
+**Not for:** Casual note-taking. PKM/Obsidian-style users. Thought-partner usage where the human drives every turn. Platform builders who build state management as their core product. Users who need zero-install onboarding (Neotoma requires npm and CLI today).
 
-docker run -d \
-  --name neotoma \
-  -p 3080:3080 \
-  -v neotoma-data:/app/data \
-  neotoma
+## Record types
 
-docker exec neotoma neotoma init --yes --data-dir /app/data
-```
+Neotoma stores typed entities with versioned history and provenance. Each type has a dedicated guide on [neotoma.io](https://neotoma.io):
 
-Connect MCP from Docker:
-
-```json
-{
-  "mcpServers": {
-    "neotoma": {
-      "command": "docker",
-      "args": ["exec", "-i", "neotoma", "node", "dist/index.js"]
-    }
-  }
-}
-```
+| Type | What it stores | Examples |
+| ---- | -------------- | -------- |
+| **[Contacts](https://neotoma.io/types/contacts)** | People, companies, roles, relationships | `contact`, `company`, `account` |
+| **[Tasks](https://neotoma.io/types/tasks)** | Obligations, deadlines, habits, goals | `task`, `habit`, `goal` |
+| **[Transactions](https://neotoma.io/types/transactions)** | Payments, receipts, invoices, ledger entries | `transaction`, `invoice`, `receipt` |
+| **[Contracts](https://neotoma.io/types/contracts)** | Agreements, clauses, amendments | `contract`, `clause`, `amendment` |
+| **[Decisions](https://neotoma.io/types/decisions)** | Choices, rationale, audit trails | `decision`, `assessment`, `review` |
+| **[Events](https://neotoma.io/types/events)** | Meetings, milestones, outcomes | `event`, `meeting`, `milestone` |
 
-Use the CLI from Docker:
+Schema is flexible — store any entity type with whatever fields the message implies. The system infers and evolves schemas automatically.
 
-```bash
-docker exec neotoma neotoma store \
-  --json='[{"entity_type":"task","title":"Submit expense report","status":"open"}]'
+## Current status
 
-docker exec neotoma neotoma entities list --type task
-```
+**Version:** v0.4.0 · **Releases:** 12 · **License:** MIT
+
+### What is guaranteed (even in preview)
 
----
+- **No silent data loss.** Operations either succeed and are recorded or fail with explicit errors.
+- **Explicit, inspectable state mutations.** Every change is a named operation with visible inputs. State is reconstructable from the audit trail.
+- **Auditable operations.** Full provenance. CLI and MCP map to the same underlying contract.
+- **Same contract for CLI and MCP.** Both use the same OpenAPI-backed operations.
 
-## Get started
+### What is not guaranteed yet
 
-After installation:
+- Stable schemas
+- Deterministic extraction across versions
+- Long-term replay compatibility
+- Backward compatibility
 
-1. Run `neotoma init` and configure your MCP client.
-2. In a conversation, tell your assistant: "Remind me to review my subscription Friday."
-3. In the same conversation, ask it to list your open tasks.
+Breaking changes should be expected. **Storage:** Local-only (SQLite + local file storage). See [Developer preview storage](docs/developer/developer_preview_storage.md).
 
-This gives you a quick end-to-end validation that memory is persisting and retrievable across sessions and tools. For full setup steps, see [Getting started](docs/developer/getting_started.md).
+## Security defaults
 
-If you want onboarding handled by an agent, use the first-run confirmation workflow in [Agent onboarding](docs/developer/agent_onboarding.md). That flow is onboarding-only and does not replace normal ongoing behavior.
+Neotoma stores user data and requires secure configuration.
 
----
+- **Authentication:** Local auth (dev stub or key-based when encryption is enabled).
+- **Authorization:** Local data isolation and explicit operation-level access controls.
+- **Data protection:** User-controlled data with full export and deletion control. Never used for training. Optional encryption at rest.
+- **Verify your setup:** Run `npm run doctor` for environment, database, and security checks. See [Auth](docs/subsystems/auth.md), [Privacy](docs/subsystems/privacy.md), [Compliance](docs/legal/compliance.md).
 
 ## Development
 
@@ -332,100 +189,75 @@ npm run dev          # MCP server (stdio)
 npm run dev:ui       # Frontend
 npm run dev:server   # API only (MCP at /mcp)
 npm run dev:full     # API + UI + build watch
-npm run dev:ws       # WebSocket MCP bridge
-npm run watch:prod:tunnel # Production mode + HTTPS tunnel (local storage, remote access)
 ```
 
-For `watch:prod:tunnel`, no `.env` is required - server auto-discovers tunnel URL. Tunnel prefers Cloudflare when `cloudflared` is installed; else ngrok. Optional: `NEOTOMA_HOST_URL` (for fixed domain), `NEOTOMA_BEARER_TOKEN`. Full tunnel and env: [Tunnels](docs/developer/tunnels.md), [Getting started – Production with tunnel](docs/developer/getting_started.md#start-development-server).
-
 **CLI:**
 
 ```bash
 npm run cli        # Run via npm (no global install)
 npm run cli:dev    # Dev mode (tsx; picks up source changes)
 npm run setup:cli  # Build and link so `neotoma` is available globally
-# Or manually: npm run build:server && npm install -g .  (or npm link)
 ```
 
-If `neotoma` is not found after global install or link, add npm's global bin to your PATH (e.g. `export PATH="$(npm config get prefix)/bin:$PATH"`). See [Getting started](docs/developer/getting_started.md#cli-setup) for details.
-
-Interactive `neotoma` startup is connect-only. It discovers healthy local API instances, applies `--env` as a preference, and prompts when multiple instances are available.
-
-**Testing:** `npm test` | `npm run test:integration` | `npm run test:e2e` | `npm run test:agent-mcp` | `npm run type-check` | `npm run lint`
-
-**IDE instructions:** Neotoma supports both Cursor and Claude Code with project-specific instructions. Instructions in `.cursor/` and `.claude/` are generated from sources in `foundation/` and `docs/`. To sync: `npm run setup:cursor` | `npm run setup:claude` (auto-syncs on commit).
-
-See [Getting started](docs/developer/getting_started.md) for full setup and storage options.
-
----
-
-## Using with AI Tools (MCP)
-
-Neotoma exposes memory via MCP. **Developer preview:** Local storage only; local auth. **Auth:** local built-in auth with OAuth-style flow support where applicable. Dev stub: `neotoma auth login --dev-stub`.
+**Testing:** `npm test` | `npm run test:integration` | `npm run test:e2e` | `npm run test:agent-mcp` | `npm run type-check` | `npm run lint` · **Source checkout:**
 
-**Setup:**
-
-- [Cursor MCP setup](docs/developer/mcp_cursor_setup.md)
-- [Claude Code MCP setup](docs/developer/mcp_claude_code_setup.md)
-- [ChatGPT Custom GPT setup](docs/developer/mcp_chatgpt_setup.md)
-- [OAuth implementation](docs/developer/mcp_oauth_implementation.md)
-
-**Representative actions:** `store`, `reinterpret`, `correct`, `retrieve_entities`, `retrieve_entity_snapshot`, `merge_entities`, `list_observations`, `create_relationship`, `list_relationships`, `list_timeline_events`, `retrieve_graph_neighborhood`, `retrieve_file_url`, schema tools (`analyze_schema_candidates`, `get_schema_recommendations`, `update_schema_incremental`, `register_schema`). Full list: [MCP spec](docs/specs/MCP_SPEC.md).
-
-To use the Neotoma MCP server from another workspace, see [Cursor MCP setup](docs/developer/mcp_cursor_setup.md).
+```bash
+git clone https://github.com/markmhendrickson/neotoma.git
+cd neotoma
+npm install
+npm test
+```
 
----
+**Prerequisites:** Node.js v18.x or v20.x (LTS), npm v9+. No `.env` required for local storage. See [Getting started](docs/developer/getting_started.md).
 
-## Agent Instructions (Behavior Summary)
+## Using with AI tools (MCP)
 
-Neotoma-compatible agents follow a consistent behavior contract across MCP and CLI:
+Neotoma exposes state via MCP. Local storage only in preview. Local built-in auth.
 
-- **Store first:** Persist the conversation turn before responding.
-- **Bounded retrieval:** Retrieve likely related entities before storing new ones.
-- **Entity extraction:** Extract and store relevant people, tasks, events, places, and relationships from user input.
-- **Task creation:** Create tasks when users express intent, obligations, deadlines, or reminders.
-- **External data safety:** Store relevant entities from external tool results before responding.
+**Setup guides:** [Cursor](https://neotoma.io/neotoma-with-cursor) · [Claude Code](https://neotoma.io/neotoma-with-claude-code) · [Claude](https://neotoma.io/neotoma-with-claude) · [ChatGPT](https://neotoma.io/neotoma-with-chatgpt) · [Codex](https://neotoma.io/neotoma-with-codex) · [OpenClaw](https://neotoma.io/neotoma-with-openclaw)
 
-Full instructions: [MCP instructions](docs/developer/mcp/instructions.md) and [CLI agent instructions](docs/developer/cli_agent_instructions.md).
+**Agent behavior contract:** Store first, retrieve before storing, extract entities from user input, create tasks for commitments. Full instructions: [MCP instructions](docs/developer/mcp/instructions.md) and [CLI agent instructions](docs/developer/cli_agent_instructions.md).
 
----
+**Representative actions:** `store`, `retrieve_entities`, `retrieve_entity_snapshot`, `merge_entities`, `list_observations`, `create_relationship`, `list_relationships`, `list_timeline_events`, `retrieve_graph_neighborhood`. Full list: [MCP spec](docs/specs/MCP_SPEC.md).
 
-## Core Principles
+## Common questions
 
-1. **Deterministic** – Same input → same output. Hash-based IDs, no randomness in core components.
-2. **Schema-first** – Type-driven extraction and storage, not freeform blobs.
-3. **Explainable** – Every field traces to source (document or agent interaction).
-4. **Entity-unified** – Canonical IDs across all personal data.
-5. **Timeline-aware** – Chronological ordering from date fields.
-6. **Cross-platform** – MCP for any compatible AI tool.
-7. **Privacy-first** – User-controlled memory, encryption, RLS.
-8. **Immutable** – Truth does not change after storage; corrections create new observations.
-9. **Provenance** – Full audit trail; event-sourced, replayable history.
-10. **Explicit control** – Ingestion only of what you explicitly provide; no background scanning.
-11. **Four-layer model** – Source → Observation → Entity → Entity Snapshot.
+**Platform memory (Claude, ChatGPT) is good enough — why add another tool?**
+Platform memory stores what one vendor decides to remember, in a format you can't inspect or export. It doesn't version, doesn't detect conflicts, and vanishes if you switch tools. Neotoma gives you structured, cross-tool state you control.
 
----
+**Can't I just build this with SQLite or a JSON file?**
+You can start there — many teams do. But you'll eventually need versioning, conflict detection, schema evolution, and cross-tool sync. That's months of infrastructure work. Neotoma ships those guarantees on day one.
 
-## Local-only storage
+**What's the difference between RAG memory and deterministic memory?**
+RAG stores text chunks and retrieves them by similarity. Neotoma stores structured observations and composes entity state with reducers; the same observations always yield the same snapshot. RAG optimizes relevance; deterministic memory optimizes integrity, versioning, and auditability.
 
-Neotoma is intentionally local-only in this active codebase (SQLite + local file storage).
-If remote backend support is needed later, recover it from git history.
+**Is this production-ready?**
+Neotoma is in developer preview — used daily by real agent workflows. The core guarantees (deterministic state, versioned history, append-only log) are stable. Install in 5 minutes and let your agent evaluate the fit.
 
----
+More questions: [FAQ](https://neotoma.io/faq)
 
 ## Related posts
 
 - [Neotoma developer release](https://markmhendrickson.com/posts/neotoma-developer-release)
+- [Your AI remembers your vibe but not your work](https://markmhendrickson.com/posts/your-ai-remembers-your-vibe-but-not-your-work)
 - [Building a truth layer for persistent agent memory](https://markmhendrickson.com/posts/truth-layer-agent-memory)
 - [Agent memory has a truth problem](https://markmhendrickson.com/posts/agent-memory-truth-problem)
+- [Six agentic trends I'm betting on (and how I might be wrong)](https://markmhendrickson.com/posts/six-agentic-trends-betting-on)
 - [Why agent memory needs more than RAG](https://markmhendrickson.com/posts/why-agent-memory-needs-more-than-rag)
+- [Agent command centers need one source of truth](https://markmhendrickson.com/posts/agent-command-centers-source-of-truth)
 
----
+## Documentation
 
-## Contributing
+Full documentation is organized at [neotoma.io/docs](https://neotoma.io/docs) and in the `docs/` directory.
+
+**Getting started:** [Evaluate](https://neotoma.io/evaluate), [Install](https://neotoma.io/install), [Walkthrough](https://neotoma.io/developer-walkthrough)
 
-Neotoma is in active development. For questions or collaboration, open an issue or discussion. The work is in the open: [github.com/markmhendrickson/neotoma](https://github.com/markmhendrickson/neotoma). See [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md).
+**Reference:** [REST API](https://neotoma.io/api), [MCP server](https://neotoma.io/mcp), [CLI](https://neotoma.io/cli), [Memory guarantees](https://neotoma.io/memory-guarantees), [Architecture](https://neotoma.io/architecture), [Terminology](https://neotoma.io/terminology)
 
-## License
+**Foundational:** [Core identity](docs/foundation/core_identity.md), [Philosophy](docs/foundation/philosophy.md), [Problem statement](docs/foundation/problem_statement.md)
+
+**Operations:** [Runbook](docs/operations/runbook.md), [Health check](docs/operations/health_check.md) (`npm run doctor`), [Troubleshooting](https://neotoma.io/troubleshooting)
+
+## Contributing
 
-MIT
+Neotoma is in active development. For questions or collaboration, open an issue or discussion. See [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md). **License:** MIT

package/dist/actions.d.ts

@@ -1,6 +1,9 @@
-export declare const app: import("express-serve-static-core").Express;
+import express from "express";
+export declare const app: any;
+/** True when request is to localhost/127.0.0.1 (local access). Tunnel requests have a non-local Host. */
+export declare function isLocalRequest(req: express.Request): boolean;
 export declare function startHTTPServer(): Promise<{
-    server: import("http").Server<typeof import("http").IncomingMessage, typeof import("http").ServerResponse>;
+    server: any;
     port: number;
 } | undefined>;
 //# sourceMappingURL=actions.d.ts.map

package/dist/actions.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"actions.d.ts","sourceRoot":"","sources":["../src/actions.ts"],"names":[],"mappings":"AA8EA,eAAO,MAAM,GAAG,6CAAY,CAAC;AA0vI7B,wBAAsB,eAAe;;;eAkCpC"}
+{"version":3,"file":"actions.d.ts","sourceRoot":"","sources":["../src/actions.ts"],"names":[],"mappings":"AAAA,OAAO,OAAO,MAAM,SAAS,CAAC;AA8E9B,eAAO,MAAM,GAAG,KAAY,CAAC;AAwL7B,yGAAyG;AACzG,wBAAgB,cAAc,CAAC,GAAG,EAAE,OAAO,CAAC,OAAO,GAAG,OAAO,CAK5D;AAk5ID,wBAAsB,eAAe;;;eAgCpC"}