neotoma 0.3.0 → 0.3.2

package/docs/developer/agent_cli_configuration.md

@@ -1,140 +0,0 @@
-# Agent CLI Configuration
-
-_(User-level settings for Claude Code, OpenAI Codex, and Cursor Agent CLI)_
-
-## Scope
-
-This document describes how Neotoma keeps agent and CLI behavior in **user-level** config so only your personal settings apply. Project-level config in the repo is minimal or absent so it does not override your choices.
-
-## Prefer MCP when available, CLI as backup
-
-- **When MCP is installed and running:** Prefer **Neotoma MCP** for Neotoma operations. Use MCP tools (e.g. `store_structured`, `create_relationship`) per `docs/developer/mcp/instructions.md`.
-- **When MCP is not available:** Use the **Neotoma CLI** as backup. Always pass `--servers=start` so the API starts if not running, and add `--env dev` or `--env prod` when targeting a specific environment (e.g. `neotoma --servers=start entities list`, `neotoma --servers=start --env dev store --json='...'`). The positional form `neotoma dev` or `neotoma prod` as first argument is valid and sets environment for the whole run, but agents should prefer explicit `--servers=start` and `--env` for direct commands.
-
-Agents get the same behaviors (chat persistence, entity extraction, conventions) either way; the rule instructs them to use MCP when it is available and to fall back to the CLI when it is not.
-
-The rule content is sourced from **`docs/developer/cli_agent_instructions.md`**, which mirrors the behavioral instructions in `docs/developer/mcp/instructions.md` (same chat persistence, entity extraction, and conventions), with MCP preferred when available and CLI equivalents as backup. When you run `neotoma cli-instructions check`, the CLI creates **symlinks** from both project paths (`.cursor/rules/`, `.claude/rules/`, `.codex/`) and user-level paths (`~/.cursor/rules/`, etc.) to that doc so the rule is always up to date. User symlinks use the absolute path to the doc in this repo; if you move the repo, re-run the check to recreate them.
-
-To add the rule so it is applied in Cursor, Claude Code, and Codex, run `neotoma cli-instructions check` from the repo root. The check only considers paths each IDE actually loads (e.g. `.cursor/rules/`, `.claude/`, `.codex/`); it will offer to add the rule to all three environments (project and/or user). Use `neotoma cli-instructions config` to print paths and the instruction source doc.
-
-`neotoma init` and `neotoma mcp check` also surface this workflow. They can remind you to run `neotoma cli-instructions check`, and in interactive runs they can offer to add missing CLI instructions after MCP setup.
-
-## Configuration strategy
-
-- **Permissions and approval behavior** live in your user config, not in the repo.
-- **Project config** (when present) is minimal and does not override approval, sandbox, or permissions.
-- Same approach across Claude Code, OpenAI Codex, and Cursor Agent CLI.
-- **MCP servers:** `.cursor/mcp.json` is the single source of truth. All CLIs that support project MCP config see the same servers after sync.
-
-## MCP: one source, all CLIs
-
-MCP servers are defined once in **`.cursor/mcp.json`** (Cursor IDE). To make them available to Claude Code and Codex in this project:
-
-1. **Add or edit servers** in `.cursor/mcp.json` only.
-2. **Sync to other configs:** run `npm run sync:mcp` from the repo root.
-
-The sync script updates:
-
-- **`.mcp.json`** — Claude Code (and any tool that reads project `.mcp.json`) gets the same `mcpServers` as Cursor. Existing entries in `.mcp.json` are kept; Cursor’s list is merged in.
-- **`.codex/config.toml`** — Appends `[mcp_servers.<id>]` for each stdio server in `.cursor/mcp.json` so Codex sees the same servers.
-
-Run `npm run sync:mcp` after changing `.cursor/mcp.json` (or add it to your workflow). Cursor Agent CLI uses Cursor’s MCP config when running in a project that has `.cursor/mcp.json`.
-
-### Transport: stdio for local, HTTP for remote
-
-For local usage (Cursor, Claude Code, Codex on the same machine as the Neotoma repo), use **stdio** in `.cursor/mcp.json`:
-
-```json
-{
-  "mcpServers": {
-    "neotoma-dev": {
-      "command": "/absolute/path/to/neotoma/scripts/run_neotoma_mcp_stdio.sh"
-    },
-    "neotoma": {
-      "command": "/absolute/path/to/neotoma/scripts/run_neotoma_mcp_stdio_prod.sh"
-    }
-  }
-}
-```
-
-Replace `/absolute/path/to/neotoma` with your repo path. Run `npm run build:server` before first use. Stdio lets the client spawn the server, so after sleep you only need to toggle the MCP server off/on; no separate HTTP process to restart.
-
-To use both dev and prod, add two MCP servers (e.g. `neotoma-dev` and `neotoma`); each process has a fixed environment and cannot switch at runtime.
-
-For remote access (tunnel, ChatGPT, deployed server), use HTTP URLs instead. See [mcp_cursor_setup.md](mcp_cursor_setup.md) for the transport comparison and HTTP config.
-
----
-
-## Claude Code
-
-**User config (all projects):** `~/.claude/settings.json`
-
-Set permissions and default mode there. Example:
-
-```json
-{
-  "alwaysThinkingEnabled": false,
-  "permissions": {
-    "defaultMode": "acceptEdits",
-    "allow": ["Bash", "Edit(/**)", "Read(/**)", "Grep", "Glob", "mcp__parquet", "mcp__neotoma"],
-    "deny": ["Read(.env)", "Read(.env.*)", "Read(docs/private/**)", "Edit(.env)", "Edit(.env.*)", "Bash(rm -rf *)", "Bash(git push --force *)"]
-  }
-}
-```
-
-**In this repo:** `.claude/settings.json` has no `permissions` block, so your user config applies. Rules and entrypoint live in `.claude/CLAUDE.md` and `.claude/rules/`.
-
-**Restart Claude Code** after changing `~/.claude/settings.json`.
-
----
-
-## OpenAI Codex
-
-**User config (all projects):** `~/.codex/config.toml`  
-**Project config:** `.codex/config.toml` (comments only; no overrides)
-
-Set model, approval policy, sandbox, and MCP in your user config. Example snippet:
-
-```toml
-# ~/.codex/config.toml
-model = "gpt-5.2"
-approval_policy = "on-request"
-sandbox_mode = "workspace-write"
-```
-
-Precedence: CLI / profile > project `.codex/config.toml` > user `~/.codex/config.toml`. This repo’s `.codex/config.toml` only documents that; it does not set options, so user config applies.
-
-References: [Config basics](https://developers.openai.com/codex/config-basic), [Config reference](https://developers.openai.com/codex/config-reference).
-
----
-
-## Cursor Agent CLI
-
-**User config (all projects):** `~/.cursor/cli-config.json`
-
-Cursor Agent CLI does not use a project-level CLI config file. All CLI agent behavior is controlled from `~/.cursor/cli-config.json`.
-
-Configure model, approval, and other CLI options there. See [Cursor CLI configuration](https://docs.cursor.com/cli/reference/configuration).
-
-**Note:** `.cursor/` in this repo holds **IDE** context (rules, commands, MCP), not the Agent CLI config. The CLI reads only from `~/.cursor/cli-config.json`.
-
----
-
-## Summary
-
-| Tool              | User config                 | Project config in repo        | MCP source        |
-|-------------------|-----------------------------|-------------------------------|-------------------|
-| Claude Code       | `~/.claude/settings.json`   | `.claude/settings.json` (no permissions) | `.mcp.json` (synced from `.cursor/mcp.json`) |
-| OpenAI Codex      | `~/.codex/config.toml`      | `.codex/config.toml` (comments + synced `[mcp_servers]`) | Synced from `.cursor/mcp.json` |
-| Cursor IDE / CLI  | `~/.cursor/cli-config.json` (CLI only) | `.cursor/mcp.json` (MCP) | `.cursor/mcp.json` (canonical) |
-
-Use the user config paths above so the same behavior applies in every project. For MCP, edit `.cursor/mcp.json` and run `npm run sync:mcp` so all CLIs see the same servers.
-
-## Related documents
-
-- [CLI agent instructions](cli_agent_instructions.md) — Source of truth for rule content (aligned with MCP instructions)
-- [MCP instructions](mcp/instructions.md) — MCP interaction instructions (same behaviors, MCP tools)
-- [Getting started](getting_started.md) — Local environment and first contribution
-- [CLI overview](cli_overview.md) — Neotoma CLI usage
-- [MCP Cursor setup](mcp_cursor_setup.md) — Cursor IDE MCP integration
-- [MCP Claude Code setup](mcp_claude_code_setup.md) — Claude Code MCP integration

package/docs/developer/agent_instructions_sync_rules.mdc

@@ -1,97 +0,0 @@
----
-description: "MCP/CLI instruction parity: when either mcp/instructions.md or cli_agent_instructions.md changes, apply equivalent change to the other."
-globs: ["docs/developer/mcp/instructions.md", "docs/developer/cli_agent_instructions.md"]
----
-
-# Agent Instructions Sync Rule
-
-## Purpose
-
-Ensures behavioral instructions stay in sync across the two agent instruction files. When either file is updated, the equivalent change must be applied to the other.
-
-## Scope
-
-The two files governed by this rule:
-
-- `docs/developer/mcp/instructions.md` — loaded at runtime by `getMcpInteractionInstructions()` in `src/server.ts`; sent verbatim to MCP clients
-- `docs/developer/cli_agent_instructions.md` — source of truth for CLI agent rules; applied to `.cursor/rules/neotoma_cli.mdc`, `.claude/rules/`, and `.codex/` by `neotoma cli-instructions check`
-
-This rule does NOT cover:
-- The auto-injected active-env section in `.cursor/rules/neotoma_cli.mdc` (managed by CLI session startup)
-- Per-tool descriptions in `docs/developer/mcp/tool_descriptions.yaml`
-- Format differences between the two files (the MCP file uses a plain-text code block; the CLI file uses markdown sections)
-
-## Trigger Patterns
-
-Apply this rule whenever:
-- Any behavioral instruction is added, changed, or removed in either file
-- A new category of behavior is introduced (e.g. new store pattern, new retrieval pattern, new relationship type)
-- Wording of an existing rule is corrected or clarified in one file
-
-## Behavioral Sections That Must Stay in Parity
-
-Each behavioral rule exists in both files with transport-appropriate wording. The following behaviors must be equivalent in both:
-
-| Behavior | MCP anchor | CLI anchor |
-|---|---|---|
-| Chat persistence (MANDATORY) | `Chat (one call):` line | `MANDATORY in chat` bullet |
-| Entity shapes (conversation, agent_message) | `Chat (one call):` line | `MANDATORY in chat` bullet |
-| Stable turn identity (`turn_key`) | `Chat (one call):` line | `MANDATORY in chat` bullet |
-| Observation history / branch overwriting | `Chat details:` line | `Chat details` bullet |
-| Reverted turns (SUPERSEDES) | `Chat details:` line | `Chat details` bullet |
-| Attachments | `Chat details:` line | `Chat details` bullet |
-| Screenshot/image (extract and store) | `Screenshot/image:` line | `Screenshot/image:` bullet |
-| Do not mention storage in thoughts | `Do not mention...` line | `Do not mention` bullet |
-| Store-first rule (external tools) | `Other MCPs:` line | `External tool data` bullet |
-| Proactive retrieval (before final response) | `Proactive retrieval:` line | `Proactive retrieval` bullet |
-| Retrieval guardrails and answer grounding | `Retrieval guardrails:` / `Answer grounding:` lines | `Retrieval guardrails and answer grounding` bullet |
-| User identity | `User identity:` line | `User identity` bullet |
-| Entity extraction | `Extract and store...` line | `Extract and store` bullet |
-| Tasks | `Tasks:` line | `Tasks` bullet |
-| Entity type discovery | `Entity types:` line | `Entity types` bullet |
-| Conventions (pre-check, structured/unstructured path, summary) | `Conventions:` line | `Conventions` bullet |
-| Relationship fallback | `Fallback if relationships...` line | `Relationship creation` section |
-
-## Agent Actions
-
-### When modifying either file
-
-1. **Identify the changed behavior** — which category in the table above does it belong to?
-2. **Locate the equivalent section in the other file** — use the anchor column in the table
-3. **Apply the equivalent change** — reword for the transport (MCP tool calls vs CLI commands) but preserve the behavioral intent exactly
-4. **Verify both files are updated before completing the task**
-
-### Wording conventions
-
-The MCP file is a dense plain-text code block (one instruction per line). The CLI file uses structured markdown with transport-branched wording (`**MCP:** ... **CLI:** ...`). Translate between them:
-
-- MCP action names (`store_structured`, `create_relationship`) → CLI equivalents (`neotoma --servers=start store --json='...'`, `neotoma --servers=start relationships create ...`)
-- MCP retrieval actions (`retrieve_entity_by_identifier`, `retrieve_entities`, `list_timeline_events`, `retrieve_related_entities`, `retrieve_graph_neighborhood`, `list_observations`) → CLI retrieval equivalents (`neotoma --servers=start entities search/list`, timeline/event query commands, `relationships list`, `observations list`)
-- MCP `list_entity_types` → CLI `neotoma --servers=start schemas list --search <keyword>`
-- MCP `npm_check_update` → CLI `neotoma --servers=start --help` / update notifier
-- Inline `relationships: [...]` in MCP store → separate `relationships create` call in CLI
-
-### Files to update
-
-When `docs/developer/mcp/instructions.md` changes:
-- Update `docs/developer/cli_agent_instructions.md` (source of truth for CLI)
-- The `.cursor/rules/neotoma_cli.mdc` Rule section is regenerated from the source doc on next CLI session start; if an immediate update is needed, edit `.cursor/rules/neotoma_cli.mdc` directly as well
-
-When `docs/developer/cli_agent_instructions.md` changes:
-- Update `docs/developer/mcp/instructions.md`
-- Update `.cursor/rules/neotoma_cli.mdc` Rule section directly (same content as source doc)
-
-## Constraints
-
-- MUST update both files when behavioral instructions change in either one
-- MUST NOT change the behavioral intent when translating between transports — only the commands differ
-- MUST NOT add CLI-specific operational sections (e.g. `--servers=start` protocol, `--json=` syntax warning, active-env block) to `mcp/instructions.md`
-- MUST NOT add MCP-specific internals (e.g. `source_index`/`target_index` for inline relationships, `file_content+mime_type` parameters) to the CLI file beyond what is needed for behavioral equivalence
-- MUST verify both files are updated before marking the task complete
-
-## Related Documents
-
-- `docs/developer/mcp/instructions.md` — MCP instruction source
-- `docs/developer/cli_agent_instructions.md` — CLI instruction source
-- `.cursor/rules/neotoma_cli.mdc` — applied CLI rule file (auto-updated by CLI session startup)
-- `docs/developer/agent_cli_configuration.md` — CLI/MCP strategy overview

package/docs/developer/canonical_walkthrough.md

@@ -1,137 +0,0 @@
-# Canonical ingestion walkthrough
-## Scope
-This document describes the end to end ingestion flow from MCP or REST input to observations, snapshots, and timeline events. It does not define UI behavior or schema design details.
-
-## Purpose
-Provide a single, deterministic walkthrough of the ingestion pipeline so developers can reason about state changes and outputs.
-
-## Invariants
-1. Inputs are explicit and user provided only.
-2. Idempotency is enforced with `idempotency_key`.
-3. Observations are immutable and corrections create new observations.
-4. Snapshots are deterministic reducer outputs.
-5. Events derive only from extracted date fields.
-6. No inference or prediction is allowed.
-
-## Definitions
-- **Source**: A stored payload with provenance and raw content.
-- **Observation**: A source specific fact derived from structured input or extraction.
-- **Entity**: A canonical object with a deterministic ID.
-- **Snapshot**: Reducer output representing current truth.
-- **Event**: Timeline item derived from source date fields.
-- **Idempotency key**: Client provided key used to deduplicate store and correct operations.
-
-## Data models
-- Sources and observations are stored in `sources` and `observations`.
-- Entities and snapshots are stored in `entities` and `entity_snapshots`.
-- Timeline events are stored in `timeline_events`.
-- Relationship observations are stored in `relationship_observations`.
-
-## Flows
-### Structured ingestion
-1. Client calls `store_structured` or `POST /api/store` with `entities` and `idempotency_key`.
-2. Server checks `sources` for an existing `idempotency_key` for the user.
-3. If present, return the existing observations for the source.
-4. If not present, create a source with a content hash and provenance.
-5. For each entity payload, resolve entity ID deterministically.
-6. Create observations for each entity with `source_priority` and `fields`.
-7. Reducers compute entity snapshots deterministically.
-8. Timeline events are generated from extracted date fields.
-
-### Unstructured ingestion
-1. Client calls `store_unstructured` with `file_content` and `mime_type`, or `file_path`, plus `idempotency_key`.
-2. Server stores raw content and deduplicates by content hash.
-3. If `interpret=true`, the interpretation pipeline extracts fields.
-4. Extracted fields are written as observations with provenance.
-5. Reducers compute entity snapshots deterministically.
-6. Timeline events are generated from extracted date fields.
-
-## Diagram
-```mermaid
-flowchart TD
-  Input[Client input] --> Store[Store action]
-  Store --> Source[Source]
-  Source --> Observations[Observations]
-  Observations --> Snapshots[Entity snapshots]
-  Observations --> Events[Timeline events]
-  Observations --> Relationships[Relationship observations]
-```
-
-## Examples
-### Structured store
-Request:
-```json
-{
-  "idempotency_key": "idem_contact_001",
-  "entities": [
-    {
-      "entity_type": "contact",
-      "name": "Ada Lovelace",
-      "email": "ada@example.com",
-      "role": "Engineer"
-    }
-  ]
-}
-```
-
-Expected outcomes:
-- One source created with `idempotency_key=idem_contact_001`
-- One observation for the contact
-- One entity snapshot with `name`, `email`, `role`
-
-### Unstructured store
-Request:
-```json
-{
-  "idempotency_key": "idem_invoice_001",
-  "file_content": "BASE64_DATA",
-  "mime_type": "application/pdf",
-  "interpret": true
-}
-```
-
-Expected outcomes:
-- One source created with `idempotency_key=idem_invoice_001`
-- Observations created from extracted fields
-- Timeline events created from extracted date fields
-
-## Testing requirements
-1. Run `tests/contract/contract_mapping.test.ts` for contract coverage.
-2. Run `tests/fixtures/replay_graph.test.ts` for deterministic snapshot replay.
-3. Run `tests/integration` for ingestion and reducer integration.
-
-## Agent Instructions
-### When to Load This Document
-Load when implementing ingestion, reducer, or timeline changes.
-
-### Required Co-Loaded Documents
-- `docs/NEOTOMA_MANIFEST.md`
-- `docs/foundation/core_identity.md`
-- `docs/foundation/layered_architecture.md`
-- `docs/foundation/product_principles.md`
-- `docs/architecture/determinism.md`
-- `docs/subsystems/ingestion/ingestion.md`
-- `docs/subsystems/observation_architecture.md`
-- `docs/subsystems/errors.md`
-- `docs/testing/testing_standard.md`
-
-### Constraints Agents Must Enforce
-1. Idempotency keys are required for store and correct operations.
-2. Observations are immutable, corrections create new observations.
-3. Snapshots are deterministic reducer outputs.
-4. Events derive only from extracted date fields.
-5. No inference beyond extracted fields.
-
-### Forbidden Patterns
-- Implicit ingestion or background data collection
-- Nondeterministic sorting or random IDs
-- Modifying existing observations
-- Missing idempotency keys in store or correct requests
-
-### Validation Checklist
-- [ ] Change respects Truth Layer boundaries
-- [ ] Idempotency enforced for store and correct
-- [ ] Observations remain immutable
-- [ ] Reducer output deterministic
-- [ ] Timeline events derived only from extracted dates
-- [ ] Tests updated or added

package/docs/developer/cli_agent_instructions.md

@@ -1,141 +0,0 @@
----
-description: "Prefer Neotoma MCP when installed and running; use CLI as backup when MCP is not available. Same agent behaviors (chat persistence, entity extraction, conventions)."
-globs: ["**/*"]
-alwaysApply: true
----
-
-# Neotoma transport: MCP when available, CLI as backup
-
-## Purpose
-
-Use **Neotoma MCP** for Neotoma operations when it is installed and running in this environment. Use the **Neotoma CLI** as backup when MCP is not available (not installed, not running, or not in context). Same agent behaviors either way: chat persistence, entity extraction, and conventions. This rule ensures agents get consistent behavior whether they use MCP tools or CLI commands.
-
-## Scope
-
-This document is the canonical source for the instruction content written to applied paths by `neotoma cli-instructions check`. It mirrors the behavioral instructions in `docs/developer/mcp/instructions.md`, with MCP actions paired with CLI equivalents for when MCP is not available. Edits here take effect when the rule is re-applied (run `neotoma cli-instructions check` and add to missing environments).
-
-When a Neotoma CLI session starts (dev or prod), the applied rule files (e.g. `.cursor/rules/neotoma_cli.mdc`) are automatically updated with an **Active environment** section that identifies the current env and the correct `--env` flag to use. This happens silently on every session start so agents always know which environment is active.
-
-## Transport and environment
-
-- **When MCP is available (installed and running):** Prefer **MCP** for Neotoma operations. Use MCP tools (e.g. `store_structured`, `create_relationship`, `mcp_parquet_*`) per `docs/developer/mcp/instructions.md`.
-- **When both neotoma-dev and neotoma MCP servers are configured:** Prefer **neotoma** (production) for general use unless the user or context clearly indicates development; if unsure, prefer neotoma.
-- **When MCP is not available:** Use the **Neotoma CLI** as backup. Data commands are API-first with automatic in-process local fallback if the API is unreachable (e.g. `neotoma store --json='...'`, `neotoma entities list`). Use `--offline` to force local transport or `--api-only` to require remote API. For server commands (`api start`, `api stop`, `api logs`, `watch`), always pass `--env dev` or `--env prod`.
-
-## CLI startup protocol (use-existing)
-
-CLI uses connect-only startup for interactive sessions. It does not auto-start servers on session start. On no-args startup, it discovers running local API instances from session ports, defaults (`8080`, `8180`), remembered ports, and optional configured ports. If multiple instances are healthy, it applies `--env` as a preference and then prompts for explicit selection.
-
-```bash
-neotoma store --json='[...]'
-neotoma entities list --type contact
-neotoma schemas list
-```
-
-For server commands (`api start`, `api stop`, `api logs`, `watch`), pass `--env dev` or `--env prod`. For data commands, use `--base-url` for a specific target port, `--offline` to force local transport, or `--api-only` to disable fallback.
-
-**`neotoma dev` / `neotoma prod` are environment shorthands.** They inject `--env dev` or `--env prod` for that invocation.
-
-> **`--json=` syntax warning:** Always write `--json='[...]'` with **no space** between `--json` and `=`. The form `--json '[...]'` (space before the value) can fail silently in shell. For payloads longer than a few hundred characters, write entities to a temp file and use `--file <path>` instead.
-
-## Store command quick reference (CLI backup)
-
-When using the CLI (MCP not available):
-
-```bash
-neotoma store --json='[{"entity_type":"person","name":"Sarah","canonical_name":"Sarah"},{"entity_type":"task","title":"Pay Sarah $20","description":"Owe Sarah $20"}]'
-```
-
-For long payloads, use `neotoma store --file <path>` with a JSON file containing the entities array.
-
-## Relationship creation (CLI backup)
-
-After storing entities, create relationships with:
-
-```bash
-neotoma relationships create --source-id <entity_id> --target-id <entity_id> --type PART_OF
-neotoma relationships create --source-id <msg_id> --target-id <conv_id> --type PART_OF
-neotoma relationships create --source-id <container_id> --target-id <asset_id> --type EMBEDS
-```
-
-Entity IDs are returned in the `store` response (`entities[].entity_id`). If `relationships create` fails or is unavailable, check `neotoma relationships --help` for current syntax.
-
-## Pre-check before storing (CLI backup)
-
-Before storing a new entity, check for an existing record to avoid duplicates:
-
-```bash
-neotoma entities search --query "Holly Blondin" --type contact
-neotoma entities list --type contact
-```
-
-If a matching entity exists, use its `entity_id` for relationship creation instead of creating a duplicate. Only store if no match is found.
-
-## Retrieval command quick reference (CLI backup)
-
-When MCP is not available and prompt context may depend on prior memory:
-
-```bash
-# Identifier lookup
-neotoma entities search --query "Acme Corp"
-
-# Type-based lookup
-neotoma entities list --type task
-neotoma entities list --type event
-
-# History/provenance lookup
-neotoma observations list --entity-id <entity_id>
-
-# Relationship expansion (when needed)
-neotoma relationships list --entity-id <entity_id>
-```
-
-Use narrow queries first, then expand only if needed.
-
-## Store error handling
-
-If `store` fails:
-1. Retry once with the same payload.
-2. If it fails again, surface the error to the user ("Storage failed: [error message]") before responding with the retrieved data.
-3. Do not silently skip storage and respond as if it succeeded.
-
-## Rule (instruction content)
-
-- **Transport:** Prefer **MCP** when it is installed and running in this environment; use **CLI** when MCP is not available. CLI session startup is connect-only. For server commands (`api start`, `api stop`, `api logs`, `watch`), always include `--env dev` or `--env prod`.
-
-- **MANDATORY in chat:** Every turn persist the conversation and current user message. Do not skip for greetings or minimal messages. Same turn as your response. **If MCP available:** use unified `store` with entities and idempotency_key per MCP instructions. **If using CLI:** use `neotoma --servers=start store --json='[{"entity_type":"conversation","title":"<summary>","started_at":"<iso>"},{"entity_type":"agent_message","role":"user","content":"<exact message>","turn_key":"{conversation_id}:{turn_id}"}]'` with idempotency_key `conversation-{conversation_id}-{turn_id}-{timestamp_ms}` or UUID (each store creates a new observation; overwriting between branches is allowed). Entity indices: `entities[0]` = conversation, `entities[1]` = message in the response. Include `turn_key` or `id` as `{conversation_id}:{turn_id}` so the same logical turn resolves to the same entity and multiple observations form history. After storing, create PART_OF relationship: `neotoma --servers=start relationships create --source-id <msg_id> --target-id <conv_id> --type PART_OF` (see "Relationship creation" section). Use host conversation_id/thread_id/session_id and turn_id when available.
-
-- **Chat details:** Overwriting between branches is acceptable; history is available via `neotoma --servers=start observations list <entity_id>`. Reverted turns: optionally link new message to previous with relationship SUPERSEDES (`--source-id <new_msg_id> --target-id <prev_msg_id> --type SUPERSEDES`). Attachments: **MCP:** include file input in the same `store` call with entities and then create EMBEDS from `structured.entities[1]` to `unstructured.interpretation.entities[0]`. **CLI:** use one `neotoma --servers=start store` call with both `--json` and `--file-path` when possible, then create EMBEDS (container, asset). Relationship types: PART_OF, REFERS_TO, EMBEDS, SUPERSEDES, etc. (see MCP spec).
-
-- **Screenshot/image:** When the user provides a screenshot or image, extract structured data from the visible content (people, messages, dates, criteria, tasks, events, offers, transactions, etc.) and store it. **MCP:** use the attachment turn recipe with entities [ conversation, message, …extracted entities ], file_path or file_content+mime_type for the image, relationships PART_OF, REFERS_TO for each extracted entity, EMBEDS from message to file entity. **CLI:** use one `neotoma --servers=start store` call with `--json` (conversation + message + extracted entities) and `--file-path` for the image, then create PART_OF, REFERS_TO, and EMBEDS relationships. Extract every distinct entity visible in the image before responding.
-
-- **Do not mention storage, memory, or linking unless the user asked.** Do not describe internal persistence in thought or reply (e.g. "Persisting this turn, then replying", "Storing the conversation first"). When confirming something was stored, use memory-related language ("remember", "recall", "stored in memory") and include one of those phrases.
-
-- **Use Neotoma proactively (MCP or CLI).** Store when the user states relevant information; store first, then respond. Do not skip store because the user did not ask to save.
-
-- **Proactive retrieval (after persistence, before final response):** After mandatory per-turn persistence, run a bounded retrieval pass whenever the prompt includes entities, topics, time windows, or commitments that might map to Neotoma memory. **MCP:** start with `retrieve_entity_by_identifier` and `retrieve_entities`, use `list_timeline_events` for time-based questions, then expand with `retrieve_related_entities` / `retrieve_graph_neighborhood` only when relationship context is needed, and `list_observations` when turn/provenance continuity matters. **CLI:** start with `neotoma --servers=start entities search --query "<term>"` and `neotoma --servers=start entities list --type <type>`, use timeline/event queries where supported, then `neotoma --servers=start relationships list --entity-id <id>` and `neotoma --servers=start observations list --entity-id <id>` as needed.
-
-- **Retrieval guardrails and answer grounding:** Start with small, targeted queries and expand only on ambiguity or low confidence. Avoid broad scans unless necessary. Use retrieved Neotoma facts when relevant; if bounded retrieval finds no relevant context, proceed normally without inventing memory-backed claims.
-
-- **External tool data (store-first rule):** Whenever any tool call returns external data — regardless of source (email, calendar, web search, file read, database query, API response, or any other external system) — extract and store all relevant entities before composing the response. Do not respond until storage is complete (same turn). Entities to extract: people, companies, locations, events, tasks, notifications, documents, device status, and relationships between them. Create tasks for action items. Link events and tasks to locations and people. **MCP:** use unified `store` and create_relationship as needed. **CLI:** use `neotoma --servers=start store --json='[...]'` and include `--file-path` or `--file-content` when preserving files in the same request.
-
-- **User identity:** When the user provides or implies their identity (name, email, "me"/"myself"), store as contact or person in the same turn when you have enough to identify.
-
-- **Extract and store all relevant entities** from the user: people, tasks, events, commitments, preferences, possessions, relationships, places. Store every distinct entity. Places: store as location/property/place and link to tasks or other entities. Implied relationships: use create_relationship (MCP) or **CLI:** `neotoma --servers=start relationships create --source-id <id> --target-id <id> --type <TYPE>` (see "Relationship creation" section). Container+asset: EMBEDS with source=container, target=asset; **CLI:** store file with `neotoma --servers=start upload`, create image/media entity, then create relationship EMBEDS.
-
-- **Tasks:** Create a task when the user expresses intent, obligation, or future action ("I need to", "remind me", deadlines). Unless they say no reminders/tasks, create task with due date if available and relate to person or entity. **CLI:** `neotoma --servers=start store --json='[{"entity_type":"task",...}]'` or `--file <path>`.
-
-- **Entity types:** Use MCP tools or `neotoma --servers=start schemas list` / `neotoma --servers=start schemas get <entityType>` when needed. For unknown types, use `neotoma --servers=start schemas list --search <keyword>` to discover a matching type before inventing one; if none matches, store with a descriptive entity_type and the system infers or creates schema. Examples: contact, person, company, task, invoice, transaction, receipt, note, contract, event, location, property, place, feature_unit, release, agent_decision, agent_session, validation_result, codebase_entity, architectural_decision.
-
-- **Conventions:** Prefer **MCP** when available; otherwise use CLI direct commands. Avoid fetching authenticated user unless the next action needs it. **Pre-check before storing** (see "Pre-check before storing" section above): search for existing records by name/type before inserting to avoid duplicates; use existing entity_id for relationships if found. Include all fields from source when storing. **Structured vs unstructured path:** use the entities JSON array (`--json` or `--file`) for conversation- and tool-sourced data; omit `original_filename`. For turns that include attachments, send entities and file together in one `store` call when possible (`--json` + `--file-path` or `--file-content`). Omit user_id (inferred). After store, summarize entities and relationships. Do not repeat the same phrase as both thought and section heading. At session start, optionally run `neotoma --help` or check for update; if an update is available, prompt user to upgrade.
-
-## When to load
-
-Load when configuring or documenting agent behavior, or when choosing between MCP and CLI for Neotoma operations. Applied in Cursor, Claude Code, and Codex via `.cursor/rules/`, `.claude/rules/`, `.codex/` (project or user).
-
-## Related documents
-
-- `docs/developer/mcp/instructions.md` — MCP interaction instructions (use when MCP is available)
-- `docs/specs/MCP_SPEC.md` — Action catalog and entity type rules
-- `docs/developer/agent_cli_configuration.md` — Agent CLI configuration and MCP/CLI strategy
-- `docs/developer/cli_reference.md` — CLI command reference