neotoma 0.0.1 → 0.3.0

package/.claude/CLAUDE.md

@@ -0,0 +1,81 @@
+# Neotoma Project Instructions for Claude Code
+
+<!-- Generated from docs/context/index_rules.mdc and docs/foundation/agent_instructions_rules.mdc -->
+<!-- Source: scripts/setup_claude_instructions.sh -->
+
+## Document Loading Order (MANDATORY)
+
+**Every session MUST begin by loading these documents in this order:**
+
+1. **`docs/context/index_rules.mdc`** — Primary entrypoint; complete documentation map, reading strategies, dependency graph
+2. **Foundation documents** from `docs/foundation/` in this order:
+   - `core_identity.md` — What Neotoma is and is not (Truth Layer scope)
+   - `philosophy.md` — Core principles and architectural invariants
+   - `layered_architecture.md` — Truth Layer, Strategy Layer, Execution Layer
+   - `product_principles.md` — Product design principles
+   - `agent_instructions_rules.mdc` — Repository-wide agent instructions and validation checklist
+3. **Task-specific docs** as indicated by the index (e.g. subsystems, architecture, testing)
+
+**Why this order:** Foundation establishes constraints (determinism, immutability, Truth Layer boundaries) that all work must respect.
+
+## Core Constraints (Summary)
+
+Full constraints in `docs/foundation/agent_instructions_rules.mdc` and `.claude/rules/`. Summary:
+
+**MUST enforce:**
+- **Truth Layer boundaries**: No strategy/execution logic in Neotoma
+- **Determinism**: No random IDs, no unstable sorting, hash-based entity/event IDs
+- **Immutability**: Observations and source are immutable (reinterpretation creates NEW observations)
+- **Schema-first**: Use application types from `docs/subsystems/record_types.md`
+- **Explicit control**: User approves all ingestion; no background scanning
+- **Provenance**: All outputs trace to source + config + timestamp
+- **Graph integrity**: No orphans, no cycles, transactional writes
+- **Privacy**: No PII in logs (IDs only)
+
+**MUST NOT:**
+- Violate architectural boundaries (no cross-layer logic)
+- Introduce nondeterminism (no `Math.random()`, `Date.now()` in business logic, unstable iteration)
+- Generate features outside MVP scope (no semantic search, no autonomous agents)
+- Break immutability (no modifying observations/source after creation)
+- Store docs in repo root (all docs in `docs/` subdirectories)
+
+**Validation checklist** in `docs/foundation/agent_instructions_rules.mdc`.
+
+## Configuration
+
+- **`foundation-config.yaml`** — Repository-specific settings (conventions, security, workflows)
+- **Environment variables** — See `docs/developer/env_check_rules.mdc`; check `.env.example` for required vars
+
+## Autonomous Execution
+
+**Proceed without asking for:**
+- Routine implementation, refactors, tests, docs, lint fixes
+- Anything where docs/codebase give a clear answer
+- Edits or commands allowed by permissions (no confirmation needed)
+
+**Stop and ask the user only when:**
+1. There is an **unclear architectural or design choice** (e.g. where to put a module, which abstraction, how to cross subsystem boundaries)
+2. **Getting it wrong would likely cause substantively wrong assumptions** (e.g. wrong layer, wrong consistency model, violating Truth Layer)
+
+In those cases: ask a short, concrete question with 1–2 options or "proceed with X unless you prefer Y."
+
+**Do not ask** for: "should I do X?" when X is already specified; "is this correct?" for obvious fixes; permission to use allowed tools.
+
+**Alignment**: Mirrors risk management hold points (schema changes, foundation doc changes, security, constraint violations); ask when ambiguity could lead to high-risk wrong choice.
+
+## Quick Reference
+
+- **Architecture**: `docs/architecture/architecture.md`, `determinism.md`, `consistency.md`
+- **Subsystems**: `docs/subsystems/` (schema, ingestion, reducer, relationships, search, auth, etc.)
+- **Testing**: `docs/testing/testing_standard.md`, `automated_test_catalog.md`
+- **Feature Units**: `docs/feature_units/standards/` (spec, manifest, execution)
+- **Code conventions**: `docs/conventions/code_conventions.md` (TypeScript, SQL, YAML, Shell)
+- **Documentation standards**: `docs/conventions/documentation_standards.md`
+
+## Additional Instructions
+
+All rules in `.claude/rules/` apply; they are modular instructions loaded automatically by context.
+
+Skills in `.claude/skills/` are workflows invokable with `/skill-name` (e.g. `/create_release`, `/fix_feature_bug`).
+
+For complete documentation map, reading strategies, and dependency graph, see `docs/context/index_rules.mdc`.

package/.claude/rules/agent_constraints.md

@@ -0,0 +1,201 @@
+---
+description: "Load when validating work or before completing: enforce architectural boundaries, determinism, immutability, project principles; no credentials in instructions; run validation checklist."
+alwaysApply: false
+---
+
+<!-- Source: foundation/agent_instructions/cursor_rules/agent_constraints.mdc -->
+
+# Agent Constraints Rule
+
+**Reference:** `foundation/conventions/documentation-standards.md` - Documentation standards
+
+## Purpose
+
+Ensures agents enforce all absolute constraints, forbidden patterns, and validation requirements when working on the repository.
+
+## Scope
+
+This document defines:
+- Absolute constraints that agents MUST enforce
+- Project positioning constraints (if configured)
+- Constraints agents must enforce
+- Communication constraints
+- Forbidden patterns
+- Validation checklist
+
+This document does NOT cover:
+- Repository-specific constraints (configured in `foundation-config.yaml`)
+- Risk management and hold points (see `risk_management.md`)
+- Document loading order (see `document_loading_order.md`)
+
+## Configuration
+
+Configure project-specific constraints in `foundation-config.yaml`:
+
+```yaml
+agent_instructions:
+  enabled: true
+  constraints: []  # Repository-specific constraints
+    # Example:
+    # - "Never violate architectural boundaries"
+    # - "Always follow schema-first principles"
+    # - "Enforce immutability for core data"
+  validation_checklist:
+    enabled: true
+    custom_checks: []  # Repository-specific validation checks
+```
+
+**Constraints are configurable per repository.** Default constraints shown below can be customized or extended in `foundation-config.yaml`.
+
+## Absolute Constraints
+
+Agents MUST enforce these constraints at all times (customize in `foundation-config.yaml`):
+
+1. **Never violate architectural boundaries** (respect layer boundaries, no cross-layer violations)
+2. **Never introduce nondeterminism** (no random IDs, no unstable logic, deterministic outputs)
+3. **Never generate features outside project scope** (stay within defined boundaries)
+4. **Always follow project principles** (schema-first, type-driven, or project-specific patterns)
+5. **Enforce data integrity** (immutability where required, proper validation)
+6. **Enforce safety and explicit control** (user approval where required, no implicit actions)
+7. **Always output validated artifacts** (code, specs, docs must pass validation)
+8. **Follow project positioning** (when creating marketing/content, follow project differentiators if configured)
+
+## Project Positioning Constraints (Optional)
+
+**If configured:** All release planning, marketing content, and feature positioning MUST validate project differentiators.
+
+Configure project differentiators in `foundation-config.yaml`:
+
+```yaml
+project:
+  differentiators: []  # Project-specific differentiators
+    # Example:
+    # - "privacy-first"
+    # - "deterministic"
+    # - "cross-platform"
+  positioning_framework: "docs/private/competitive/positioning_framework.md"  # Optional
+```
+
+### When Creating Release Plans (If Differentiators Configured)
+
+1. **Explicitly validate project differentiators**: Release must validate configured differentiators
+2. **Avoid feature-only releases**: Don't ship features without validating differentiators
+3. **Include acceptance criteria**: Release acceptance criteria must validate differentiators
+4. **Reference framework**: Link to positioning framework (if configured) in release plans
+
+### When Creating Marketing Content (If Differentiators Configured)
+
+1. **Lead with project differentiators**, not features alone
+2. **Contextualize features** as enabled by differentiators
+3. **Avoid feature-only positioning** (competitors may develop similar features)
+4. **Emphasize structural barriers** that prevent competitors from pursuing same path
+5. **Validate messaging**: All marketing messaging must emphasize differentiators first
+
+### When Creating Content, Documentation, or Positioning (If Differentiators Configured)
+
+1. **Lead with project differentiators**, not features alone
+2. **Contextualize features** as enabled by differentiators
+3. **Avoid feature-only positioning**
+4. **Emphasize structural barriers**
+5. **Reference framework** when appropriate (if configured)
+
+## Constraints Agents Must Enforce
+
+**Base constraints (always apply):**
+
+1. **Architectural boundaries:** MUST NOT violate layer boundaries or architectural constraints (configured per repository)
+2. **Determinism:** MUST NOT introduce randomness or nondeterministic logic (no random IDs, no unstable sorting, no `Date.now()` in business logic unless required)
+3. **Data integrity:** MUST NOT modify immutable data after storage (respect immutability requirements)
+4. **Project principles:** MUST follow project-specific principles (schema-first, type-driven, or as configured)
+5. **Explicit control:** MUST NOT implement automatic/implicit actions without user approval (respect explicit control requirements)
+6. **Provenance:** MUST trace all outputs to source (source file, rule, timestamp, or as configured)
+7. **Data integrity:** MUST maintain data consistency (no orphans, no cycles, transactional writes where required)
+8. **Privacy:** MUST NOT log sensitive data (PII, credentials, or as configured - only IDs/logs as allowed)
+9. **Security:** MUST NOT include credentials (API keys, tokens, service keys) in agent instructions, conversation text, or commit messages. Credentials MUST be injected via secure mechanisms (e.g., Cursor Cloud Agents Secrets) and agents MUST verify their availability via environment variable checks, not export commands with credential values
+
+**Repository-specific constraints:** Add custom constraints in `foundation-config.yaml` under `agent_instructions.constraints`.
+
+## Forbidden Patterns
+
+**Base forbidden patterns (always apply):**
+
+- Violating architectural boundaries (implementing logic outside defined layer boundaries)
+- Introducing nondeterminism (random IDs, unstable sorting, `Date.now()` in business logic unless required)
+- Generating features outside project scope (features not aligned with project goals)
+- Breaking immutability (modifying immutable data after initial storage)
+- Inferring beyond extraction (creating data not present in source, inferring relationships without explicit rules)
+- Schema mutations (changing assigned types after creation, unless allowed)
+- Synthetic data (guessing missing information, hallucinating fields)
+- Upward dependencies (lower layers calling higher layers, violating layer boundaries)
+- Non-transactional writes (data writes must be atomic where required)
+- Including credentials in agent instructions (no API keys, tokens, or service keys in prompt text, conversation messages, or code comments; use secure injection mechanisms like Cursor Secrets)
+- Storing documentation files in repo root (all documentation MUST be in appropriate `docs/` subdirectories)
+
+**Repository-specific forbidden patterns:** Add custom patterns in `foundation-config.yaml` under `agent_instructions.forbidden_patterns`.
+
+## Communication Constraints
+
+**Prohibitions:**
+- No emojis
+- No filler language
+- No hype
+- No soft asks
+- No conversational transitions
+- No call-to-action appendices
+- No questions, offers, suggestions, or motivational content
+- No tone matching, emotional softening, or user sentiment modulation
+
+**Assumptions:**
+- User possesses high cognitive acuity, though may have reduced verbal fluency
+
+**Execution rules:**
+- Use blunt, directive phrasing for cognitive restructuring
+- Suppress all engagement-optimizing or corporate-aligned behaviors:
+  - User satisfaction scores
+  - Conversational flow tagging
+  - Continuation bias
+  - Sentiment uplift
+- Avoid mirroring the user's tone, mood, or diction
+- Communicate strictly to the deep cognitive tier—beyond surface-level language
+
+**Response protocol:**
+- Deliver the requested or relevant information only
+- Terminate immediately after delivery—no transitions, no conclusions
+
+**End goal:**
+- User independence and cognitive restoration
+- Model obsolescence via user self-sufficiency
+
+## Validation Checklist
+
+Before completing any work, verify:
+
+**Base checks (always apply):**
+
+- [ ] Change respects architectural boundaries (no layer violations)
+- [ ] No nondeterministic logic introduced (no randomness, no unstable sorts)
+- [ ] Immutability preserved (immutable data not modified)
+- [ ] Schema changes are additive only (no breaking changes, no column removal)
+- [ ] Project principles followed (schema-first, type-driven, or as configured)
+- [ ] Data integrity maintained (no orphans, no cycles, transactional writes where required)
+- [ ] Privacy preserved (no sensitive data in logs, only IDs/logs as allowed)
+- [ ] Provenance maintained (all outputs trace to source)
+- [ ] Consistency model correct (as required per subsystem/component)
+- [ ] Security: No credentials included in agent instructions, conversation text, or code (use secure injection mechanisms)
+- [ ] Documentation location: No documentation files stored in repo root (all docs in `docs/` subdirectories)
+- [ ] Tests cover all new paths (unit, integration, E2E as appropriate)
+- [ ] Documentation updated to reflect changes
+- [ ] Downstream documentation updated if upstream docs changed
+- [ ] README.md updated if documentation changes affect user-facing information
+- [ ] Feature fits within project scope
+- [ ] No violations of MUST/MUST NOT lists
+- [ ] Project differentiators validated (if configured and if release planning or marketing content)
+
+**Repository-specific checks:** Add custom checks in `foundation-config.yaml` under `agent_instructions.validation_checklist.custom_checks`.
+
+## Related Documents
+
+- `foundation/conventions/documentation-standards.md` - Documentation standards
+- `foundation-config.yaml` - Repository-specific configuration
+- Repository foundation docs (e.g., `docs/foundation/philosophy.md`) - Core philosophy and architectural invariants
+- Repository architecture docs (e.g., `docs/architecture/architecture.md`) - System architecture

package/.claude/rules/agent_instructions_sync_rules.md

@@ -0,0 +1,100 @@
+---
+description: "MCP/CLI instruction parity: when either mcp/instructions.md or cli_agent_instructions.md changes, apply equivalent change to the other."
+globs: ["docs/developer/mcp/instructions.md", "docs/developer/cli_agent_instructions.md"]
+---
+
+<!-- Source: docs/developer/agent_instructions_sync_rules.mdc -->
+
+
+# Agent Instructions Sync Rule
+
+## Purpose
+
+Ensures behavioral instructions stay in sync across the two agent instruction files. When either file is updated, the equivalent change must be applied to the other.
+
+## Scope
+
+The two files governed by this rule:
+
+- `docs/developer/mcp/instructions.md` — loaded at runtime by `getMcpInteractionInstructions()` in `src/server.ts`; sent verbatim to MCP clients
+- `docs/developer/cli_agent_instructions.md` — source of truth for CLI agent rules; applied to `.claude/rules/neotoma_cli.mdc`, `.claude/rules/`, and `.codex/` by `neotoma cli-instructions check`
+
+This rule does NOT cover:
+- The auto-injected active-env section in `.claude/rules/neotoma_cli.mdc` (managed by CLI session startup)
+- Per-tool descriptions in `docs/developer/mcp/tool_descriptions.yaml`
+- Format differences between the two files (the MCP file uses a plain-text code block; the CLI file uses markdown sections)
+
+## Trigger Patterns
+
+Apply this rule whenever:
+- Any behavioral instruction is added, changed, or removed in either file
+- A new category of behavior is introduced (e.g. new store pattern, new retrieval pattern, new relationship type)
+- Wording of an existing rule is corrected or clarified in one file
+
+## Behavioral Sections That Must Stay in Parity
+
+Each behavioral rule exists in both files with transport-appropriate wording. The following behaviors must be equivalent in both:
+
+| Behavior | MCP anchor | CLI anchor |
+|---|---|---|
+| Chat persistence (MANDATORY) | `Chat (one call):` line | `MANDATORY in chat` bullet |
+| Entity shapes (conversation, agent_message) | `Chat (one call):` line | `MANDATORY in chat` bullet |
+| Stable turn identity (`turn_key`) | `Chat (one call):` line | `MANDATORY in chat` bullet |
+| Observation history / branch overwriting | `Chat details:` line | `Chat details` bullet |
+| Reverted turns (SUPERSEDES) | `Chat details:` line | `Chat details` bullet |
+| Attachments | `Chat details:` line | `Chat details` bullet |
+| Screenshot/image (extract and store) | `Screenshot/image:` line | `Screenshot/image:` bullet |
+| Do not mention storage in thoughts | `Do not mention...` line | `Do not mention` bullet |
+| Store-first rule (external tools) | `Other MCPs:` line | `External tool data` bullet |
+| Proactive retrieval (before final response) | `Proactive retrieval:` line | `Proactive retrieval` bullet |
+| Retrieval guardrails and answer grounding | `Retrieval guardrails:` / `Answer grounding:` lines | `Retrieval guardrails and answer grounding` bullet |
+| User identity | `User identity:` line | `User identity` bullet |
+| Entity extraction | `Extract and store...` line | `Extract and store` bullet |
+| Tasks | `Tasks:` line | `Tasks` bullet |
+| Entity type discovery | `Entity types:` line | `Entity types` bullet |
+| Conventions (pre-check, structured/unstructured path, summary) | `Conventions:` line | `Conventions` bullet |
+| Relationship fallback | `Fallback if relationships...` line | `Relationship creation` section |
+
+## Agent Actions
+
+### When modifying either file
+
+1. **Identify the changed behavior** — which category in the table above does it belong to?
+2. **Locate the equivalent section in the other file** — use the anchor column in the table
+3. **Apply the equivalent change** — reword for the transport (MCP tool calls vs CLI commands) but preserve the behavioral intent exactly
+4. **Verify both files are updated before completing the task**
+
+### Wording conventions
+
+The MCP file is a dense plain-text code block (one instruction per line). The CLI file uses structured markdown with transport-branched wording (`**MCP:** ... **CLI:** ...`). Translate between them:
+
+- MCP action names (`store_structured`, `create_relationship`) → CLI equivalents (`neotoma --servers=start store --json='...'`, `neotoma --servers=start relationships create ...`)
+- MCP retrieval actions (`retrieve_entity_by_identifier`, `retrieve_entities`, `list_timeline_events`, `retrieve_related_entities`, `retrieve_graph_neighborhood`, `list_observations`) → CLI retrieval equivalents (`neotoma --servers=start entities search/list`, timeline/event query commands, `relationships list`, `observations list`)
+- MCP `list_entity_types` → CLI `neotoma --servers=start schemas list --search <keyword>`
+- MCP `npm_check_update` → CLI `neotoma --servers=start --help` / update notifier
+- Inline `relationships: [...]` in MCP store → separate `relationships create` call in CLI
+
+### Files to update
+
+When `docs/developer/mcp/instructions.md` changes:
+- Update `docs/developer/cli_agent_instructions.md` (source of truth for CLI)
+- The `.claude/rules/neotoma_cli.mdc` Rule section is regenerated from the source doc on next CLI session start; if an immediate update is needed, edit `.claude/rules/neotoma_cli.mdc` directly as well
+
+When `docs/developer/cli_agent_instructions.md` changes:
+- Update `docs/developer/mcp/instructions.md`
+- Update `.claude/rules/neotoma_cli.mdc` Rule section directly (same content as source doc)
+
+## Constraints
+
+- MUST update both files when behavioral instructions change in either one
+- MUST NOT change the behavioral intent when translating between transports — only the commands differ
+- MUST NOT add CLI-specific operational sections (e.g. `--servers=start` protocol, `--json=` syntax warning, active-env block) to `mcp/instructions.md`
+- MUST NOT add MCP-specific internals (e.g. `source_index`/`target_index` for inline relationships, `file_content+mime_type` parameters) to the CLI file beyond what is needed for behavioral equivalence
+- MUST verify both files are updated before marking the task complete
+
+## Related Documents
+
+- `docs/developer/mcp/instructions.md` — MCP instruction source
+- `docs/developer/cli_agent_instructions.md` — CLI instruction source
+- `.claude/rules/neotoma_cli.mdc` — applied CLI rule file (auto-updated by CLI session startup)
+- `docs/developer/agent_cli_configuration.md` — CLI/MCP strategy overview