neotoma 0.3.0 → 0.3.2

package/docs/foundation/mmry_comparison.md

@@ -1,176 +0,0 @@
-# Neotoma vs. mmry.io: Marketing Comparison
-## Competitive Dynamic Summary
-**Overall Assessment: Complementary, Not Competitive**
-mmry.io and Neotoma operate in adjacent but distinct market segments with minimal direct competition:
-- **Different Data Domains:** mmry indexes web content (social media, videos, bookmarks); Neotoma structures personal data (PDFs, images, receipts, agent-created data)
-- **Different User Problems:** mmry solves "I can't find that thing I saw online"; Neotoma solves "AI agents have no memory across sessions"
-- **Different Value Propositions:** mmry enables rediscovery of consumed web content; Neotoma enables structured memory for AI agents
-- **Different Integration Models:** mmry uses optional local inference (Ollama); Neotoma uses core MCP integration for agent memory
-**Competitive Risk: Low**
-- No overlap in primary use cases or target users
-- Different technical architectures (search engine vs. structured memory substrate)
-- Both can coexist in a user's toolset without conflict
-- Potential integration opportunity: mmry could index Neotoma's structured records
-**Strategic Positioning:**
-- Neotoma occupies a new category (Deterministic Personal Memory Engine) that mmry does not address
-- mmry focuses on web content recall; Neotoma focuses on document structure for AI
-- Clear differentiation enables coexistence without direct competition
-- Both products reinforce the broader trend toward privacy-first, user-owned data tools
-**Market Context:**
-Both products address fragmented personal data but through fundamentally different lenses:
-- mmry = horizontal search tool for web content consumption
-- Neotoma = vertical memory substrate for AI-native workflows
-## Executive Summary
-**mmry.io** is a privacy-first personal search engine that indexes content you've already seen online (Twitter, Reddit, YouTube, bookmarks) to enable search across your digital history.
-**Neotoma** is a deterministic truth layer that transforms fragmented personal data into structured, queryable knowledge for AI agents via MCP.
-Both products address fragmented personal data, but serve fundamentally different use cases: mmry focuses on **searching web content you've consumed**, while Neotoma focuses on **structuring documents and conversation context for AI agents**.
-## Core Value Propositions
-### mmry.io
-- **"Find the memories you've lost"** — Search engine for things you've already seen online
-- Privacy-first search across Twitter, Reddit, YouTube, bookmarks, Pocket, Raindrop
-- Local-first architecture with optional Ollama integration
-- Community-driven plugin ecosystem
-- No algorithms, no AI training on your data
-### Neotoma
-- **"Structured Personal Data Memory for AI Agents"** — Deterministic memory substrate for AI agents
-- Dual-path ingestion: file uploads + agent interactions via MCP
-- Structured extraction from PDFs, images, receipts, contracts, AND agent-created data
-- Entity resolution and timeline generation across all personal data
-- MCP-exposed memory for ChatGPT, Claude, Cursor (cross-platform)
-- Schema-first, deterministic, explainable architecture
-## Feature Comparison
-| Feature                 | mmry.io                                                | Neotoma                                                                 |
-| ----------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- |
-| **Primary Data Source** | Web content (Twitter, Reddit, YouTube, bookmarks)      | Personal data (PDFs, images, receipts, contracts, agent-created data)   |
-| **Ingestion Method**    | Plugin-based imports from web services                 | File uploads + MCP agent interactions                                   |
-| **Data Structure**      | Indexed web content (text, links, media)               | Structured records with schema types (invoice, receipt, contract, etc.) |
-| **Search Capability**   | Full-text search across indexed content                | Structured search + filters + full-text (bounded eventual consistency)  |
-| **Entity Resolution**   | Not applicable                                         | Hash-based canonical IDs unify entities across documents                |
-| **Timeline Generation** | Not applicable                                         | Automatic chronological ordering from date fields                       |
-| **AI Integration**      | Optional Ollama for local inference                    | MCP server for ChatGPT, Claude, Cursor (read/write memory)              |
-| **Privacy Model**       | Local-first, no third-party data sharing               | Encrypted storage, row-level security, full provenance                  |
-| **Extensibility**       | Community plugins (Twitter, Pocket, Raindrop, YouTube) | External providers (Gmail, Plaid) + MCP actions                         |
-| **Target Use Case**     | "I saw this article/tweet/video but can't find it"     | "AI agents need structured memory of my personal data"                  |
-| **Data Ownership**      | Your indexed web history                               | Your structured personal data memory (documents + agent-created data)   |
-## Use Case Differentiation
-### mmry.io Use Cases
-- **"I remember reading something about X on Twitter but can't find it"** — Search indexed tweets
-- **"What was that YouTube video I watched last month?"** — Search transcribed video content
-- **"I bookmarked this article but forgot where"** — Search across Pocket/Raindrop imports
-- **"Find that Reddit thread I commented on"** — Search indexed Reddit content
-**Core Problem:** Web content you've consumed is scattered across platforms and hard to rediscover.
-### Neotoma Use Cases
-- **"AI agent needs to remember my preferences and context across sessions"** — Persistent memory via MCP
-- **"Extract structured data from invoices and receipts"** — Schema-first extraction from documents
-- **"Store project context during AI conversation"** — Agent creates structured data via MCP `store_record`
-- **"Build timeline across documents and agent-created data"** — Automatic chronological ordering across all personal data
-- **"Unify 'Acme Corp' across all personal data regardless of source"** — Entity resolution with canonical IDs
-- **"AI agent should remember structured data from conversations"** — Structured data (not just conversation) stored via MCP
-**Core Problem:** Personal data is fragmented and provider memory is conversation-only, preventing AI agents from building persistent structured memory with entity resolution and timelines.
-## Technical Architecture Comparison
-### mmry.io
-- **Architecture:** Local-first search engine with plugin system
-- **Storage:** Local database (not specified)
-- **Processing:** Optional Ollama for local inference
-- **Extensibility:** Community plugins for data sources
-- **Security:** Sandboxed plugins, privacy-first design
-### Neotoma
-- **Architecture:** Five-layer architecture (External → Infrastructure → Domain → Application → Presentation)
-- **Storage:** PostgreSQL with row-level security
-- **Processing:** Deterministic extraction (regex/parsing), no LLM in MVP
-- **Extensibility:** MCP actions, external providers (Gmail, Plaid)
-- **Security:** Encryption, immutable truth layer, full provenance
-## Market Positioning
-### mmry.io
-**Category:** Personal Search Engine / Digital Memory Tool
-**Positioning:**
-- Privacy-first alternative to traditional search engines
-- Community-driven, non-commercial project
-- Focus on web content discovery and recall
-**Target Users:**
-- Heavy web content consumers (Twitter, Reddit, YouTube users)
-- Privacy-conscious individuals who want local-first search
-- Users frustrated by platform-specific search limitations
-### Neotoma
-**Category:** Deterministic Personal Memory Engine (DPME) / AI Memory Substrate
-**Positioning:**
-- Truth layer beneath AI tools
-- Structured memory for AI agents
-- Foundation for agent-native personal computing
-**Target Users:**
-- AI-native individuals (heavy ChatGPT, Claude, Cursor users)
-- High-context knowledge workers (researchers, analysts, consultants, lawyers)
-- AI-native founders and small teams (2–20 people)
-## Competitive Advantages
-### mmry.io Advantages
-- **Web Content Focus:** Specialized in indexing web content you've consumed
-- **Community Ecosystem:** Plugin marketplace for extensibility
-- **Privacy-First:** Local-first architecture, no third-party data sharing
-- **Non-Commercial:** Community-driven project without VC monetization
-### Neotoma Advantages
-**Defensible Architectural Choices:**
-- **Privacy-first:** User-controlled memory, no provider access, never used for training
-- **Deterministic:** Reproducible outputs, explainable, full provenance (vs. ML-based probabilistic)
-- **Cross-platform:** Works with ChatGPT, Claude, Cursor via MCP (not platform-locked)
-**Feature Capabilities (Enabled by Defensible Differentiators):**
-- **AI-Native:** Built specifically for AI agent memory via MCP
-- **Structured Data:** Schema-first extraction with entity resolution and timelines
-- **Cross-Document Reasoning:** Entity unification enables reasoning across documents
-- **Multi-Modal:** PDFs, images, OCR support for document ingestion
-- **Conversation Context:** Agents can read/write memory during conversations
-## Overlap and Distinction
-### Where They Overlap
-- Both address fragmented personal data
-- Both emphasize privacy and data ownership
-- Both enable search across previously inaccessible content
-- Both are local-first/privacy-first architectures
-### Where They Diverge
-| Dimension          | mmry.io                                        | Neotoma                                                                |
-| ------------------ | ---------------------------------------------- | ---------------------------------------------------------------------- |
-| **Data Type**      | Web content (tweets, posts, videos, bookmarks) | Documents (PDFs, images, receipts, contracts) + conversation context   |
-| **Primary Use**    | Rediscovering web content you've seen          | Structuring documents for AI agent memory                              |
-| **Structure**      | Indexed text/media                             | Schema-first structured records with entities and timelines            |
-| **AI Integration** | Optional local inference (Ollama)              | Core MCP integration for agent memory                                  |
-| **Target Problem** | "I can't find that thing I saw online"         | "AI agents have no memory across sessions"                             |
-| **User Workflow**  | Import web content → Search                    | Upload documents / Agent conversations → Structured memory → AI access |
-## Marketing Messaging Differentiation
-### mmry.io Messaging
-- "Find the memories you've lost"
-- "Privacy first search engine for all the things you have already seen"
-- "Your memories belong to you"
-- Focus on web content discovery and recall
-### Neotoma Messaging
-**Lead with Defensible Differentiators:**
-- "Privacy-first deterministic memory for AI tools"
-- "User-controlled, cross-platform via MCP"
-- "Truth Layer for AI Memory"
-- "Deterministic personal knowledge substrate"
-- "The truth engine behind your AI tools"
-- "The foundation for agent-native personal computing"
-- Focus on structured memory for AI agents
-## Strategic Implications
-### For Neotoma
-**Differentiation Strategy:**
-- Emphasize AI-native positioning (MCP integration, agent memory)
-- Highlight structured data extraction (schema-first, entity resolution, timelines)
-- Position as substrate beneath AI tools, not a search engine
-- Target AI-native users who need persistent agent memory
-**Competitive Response:**
-- mmry.io is not a direct competitor — different use cases and data types
-- Both can coexist: mmry for web content search, Neotoma for document memory
-- Potential integration opportunity: mmry could index Neotoma's structured records for search
-**Market Positioning:**
-- Neotoma occupies a new category (DPME) that mmry.io does not address
-- mmry.io focuses on web content recall; Neotoma focuses on document structure for AI
-- Clear differentiation enables coexistence without direct competition
-## Conclusion
-**mmry.io** and **Neotoma** serve complementary but distinct use cases:
-- **mmry.io** = Search engine for web content you've consumed (Twitter, Reddit, YouTube, bookmarks)
-- **Neotoma** = Structured memory substrate for AI agents (documents, conversation context)
-Both address fragmented personal data, but through different lenses:
-- mmry enables **rediscovery** of web content
-- Neotoma enables **structuring** of documents for AI agent memory
-**Key Distinction:** mmry helps you find things you've seen online; Neotoma helps AI agents remember and reason about your documents and conversations.
-## References
-- [mmry.io](https://mmry.io/) — Official website
-- [Neotoma README](../../README.md) — Neotoma overview

package/docs/foundation/philosophy.md

@@ -1,103 +0,0 @@
-# Neotoma Core Philosophy and Principles
-
-## 5.1 Idempotence Above All (System-Level Determinism)
-
-A given operation MUST always produce the same final state:
-- Same file uploaded twice → same record (deduplicated via content_hash)
-- Same source + same interpretation config → same observations (idempotent, no duplicates)
-- Same entity name → same entity_id (hash-based, deterministic)
-- Same observations + same merge rules → same snapshot (deterministic reducer)
-- Same query + same DB state → same search order
-
-**System-Level Guarantees:**
-- **Deterministic components**: Content hashing, deduplication, reducer computation, entity ID generation (must be byte-for-byte deterministic)
-- **Idempotent operations**: LLM interpretation (model is stochastic, but system enforces idempotence through canonicalization, hashing, and deduplication)
-
-**No randomness in deterministic components. No duplicate observations from idempotent operations.**
-
-## 5.2 Explicit User Control
-- Neotoma ingests **only what the user explicitly provides**
-- No automatic ingestion, no background scanning
-- No reading email bodies or cloud drives without user action
-- User controls all data entry
-
-## 5.3 Strong Boundaries
-The Truth Layer MUST contain **no**:
-- Inference or prediction
-- Heuristics beyond allowed extraction rules
-- Semantic-only retrieval (MVP: structural retrieval primary; entity semantic search allowed as optional path over structured state when text query provided)
-- Strategy or execution logic
-
-**Note:** Entity semantic search embeds structured output (entity snapshots), not raw chunks; structure drives what is searchable.
-
-**Note:** AI interpretation for unstructured files is permitted with:
-- Config logging (model, temperature, prompt_hash)
-- Canonicalization and hashing for idempotence
-- No claims of replay determinism
-
-## 5.4 Truth is Immutable
-Once stored, these MUST NOT change:
-- `raw_text` (original extracted text)
-- `schema_type` (assigned document type)
-- `extracted_fields` (deterministic output)
-- Entity IDs (hash-based, stable)
-- Event timestamps (from source fields)
-Metadata (`created_at`, `updated_at`) is mutable for auditing.
-
-## 5.5 Schema-First Processing
-All extraction, linking, and event creation derive from:
-**schema_type → extraction rules → extracted_fields → entities + events**
-
-## 5.6 No Synthetic Data
-Neotoma never:
-- Guesses missing information
-- Infers relationships beyond explicit extraction
-- Predicts future states
-- Hallucinates data
-
-**Processing approach:** AI interpretation is used for unstructured data extraction with audit trail and idempotence guarantees. The LLM extracts fields present in the source document but does not synthesize or infer data.
-
-## 5.7 Full Explainability
-Every output MUST trace to:
-- Source file (provenance)
-- Extraction rule (which regex/parser)
-- Source field (for events: which date field?)
-- Entity derivation (which text → which normalized value → which hash)
-**Nothing is hidden or magical.**
-
-## Architectural Invariants (MUST/MUST NOT)
-### MUST
-1. **Determinism:** Same input → same output, always
-2. **Immutability:** Once stored, truth never changes (except metadata)
-3. **Provenance:** All records trace to source file + timestamp + user
-4. **Schema-first:** All extraction derives from schema type
-5. **Entity IDs:** Hash-based, globally unique, deterministic
-6. **Event IDs:** Hash-based, deterministic from record + field + date
-7. **Graph integrity:** No orphan nodes, no cycles, typed edges only
-8. **Transactional writes:** All graph inserts in single transaction
-9. **User isolation:** RLS enforced (future: per-user data access)
-10. **MCP-only mutations:** All writes via validated MCP actions
-11. **Strong consistency:** Core records, entities, events immediately visible
-12. **Explainability:** Every output traces to input + rule
-13. **Privacy:** No PII in logs, RLS for data protection
-14. **Explicit control:** User approves all ingestion
-15. **Truth Layer boundaries:** No strategy or execution logic
-16. **Event-sourced updates:** All state changes via Domain Events → Reducers
-17. **Pure Strategy:** Strategy Layer has no side effects
-18. **Pure Execution:** Execution Layer emits Domain Events, never writes truth directly
-### MUST NOT
-1. **No semantic-only retrieval** (structural retrieval remains primary; entity semantic search permitted as supplemental path; set queries, relationship traversal, timelines remain structural)
-2. **No automatic ingestion** (explicit user action required)
-3. **No replay-deterministic claims for AI interpretation** (auditable but not replay-deterministic; see `docs/architecture/determinism.md`)
-4. **No nondeterminism in core components** (no random IDs for entities/events, no Date.now() in entity ID generation)
-5. **No schema mutation** (once assigned, schema_type never changes)
-6. **No observation modification** (immutable after creation; reinterpretation creates NEW observations)
-7. **No inferred entities beyond extraction** (only extract what's present in source)
-8. **No strategy logic** (that's Strategy Layer, e.g., Agentic Portfolio)
-9. **No execution logic** (that's Execution Layer, e.g., Agentic Wallet)
-10. **No agent behavior** (Neotoma is not an agent)
-11. **No predictive features** (no forecasting, no recommendations)
-12. **No synthetic data** (no guessing, no hallucination beyond interpretation)
-13. **No upward layer dependencies** (Domain never calls Application)
-14. **No PII in logs** (record IDs only, not extracted fields)
-15. **No breaking schema changes** (additive evolution only)

package/docs/foundation/problem_statement.md

@@ -1,41 +0,0 @@
-# Why Neotoma Exists: The Problem
-## 6.1 Personal Data is Fragmented
-Users store critical personal data across:
-- Email attachments (Gmail, Outlook)
-- Downloads folders (PDFs, images)
-- WhatsApp/iMessage screenshots
-- Cloud drives (Dropbox, Google Drive)
-- Desktop files, phone photos
-- App exports (bank statements, receipts)
-- Scanner outputs
-- AI conversations (ChatGPT, Claude, Cursor)
-**No system unifies this into durable, structured memory with entity resolution and timelines.**
-Provider memory (ChatGPT Memory, Claude Memory, Gemini Personal Context) offers conversation-only memory—it can't structure documents, resolve entities, or build timelines across all personal data.
-## 6.2 AI Has Limited Structured Memory
-LLMs have conversation-only memory (ChatGPT Memory, Claude Memory, Gemini Personal Context) but cannot reason across structured personal data:
-- Structured extraction (can't extract fields from documents)
-- Entity resolution (doesn't know "Acme Corp" = "ACME CORP" across documents and agent-created data)
-- Timeline generation (can't build chronological sequences across all personal data)
-- Cross-data relationships (can't link entities across documents and conversations)
-- Cross-platform memory (locked to specific provider or OS)
-**Provider memory is conversation-only. Neotoma provides structured personal data memory.**
-
-Agentic and multi-agent systems lack a shared, deterministic memory layer: agents and toolchains have no single source of truth for context, provenance, or cross-session state. Neotoma provides that substrate.
-## 6.3 Neotoma Provides the Missing Substrate
-Neotoma gives AI structured personal data memory built on three defensible architectural choices that provider memory cannot offer:
-**Defensible Architectural Choices (Competitors Cannot Pursue):**
-1. **Privacy-First Architecture** — User-controlled memory, no provider access, never used for training (vs. provider-controlled memory with provider access)
-   - **Why Defensible:** Providers/startups won't pursue due to business model conflicts (data collection, training use)
-2. **Deterministic Extraction** — Same input → same output, always; reproducible, explainable, no hallucinations (vs. ML-based probabilistic)
-   - **Why Defensible:** Providers/startups won't pursue due to ML-first organizational identity and speed-to-market constraints
-3. **Cross-Platform Access** — Works with ChatGPT, Claude, Cursor via MCP, not platform-locked (vs. platform-specific memory)
-   - **Why Defensible:** Providers won't pursue due to platform lock-in business models; startups won't pursue due to separate consumer app positioning
-**Feature Capabilities (Enabled by Defensible Differentiators):**
-- Dual-path ingestion (file uploads + agent interactions)
-- Entity resolution (deterministic hash-based canonical IDs across all personal data)
-- Timeline generation (deterministic chronological ordering across all personal data)
-- Structured extraction (schema-first from documents and agent-created data)
-- Stable record IDs (persistent references)
-- Explicit provenance (trust and auditability)
-**Strategic Positioning:** Provider memory (ChatGPT, Claude, Gemini) and startups (Supermemory.ai) are developing similar feature capabilities (structured memory, entity resolution, timelines), but cannot pursue Neotoma's defensible architectural choices due to structural constraints. See [`docs/private/competitive/defensible_differentiation_framework.md`](../private/competitive/defensible_differentiation_framework.md).
-**Neotoma is the "RAM + HDD" for AI-native personal computing.**

package/docs/foundation/product_positioning.md

@@ -1,69 +0,0 @@
-# Neotoma Product Positioning and Differentiation
-## 7.1 Positioning
-Neotoma is **not a PKM or app**. Neotoma is:
-- A **substrate** for AI tools
-- A **structured memory system** for personal/professional data (documents + agent-created data)
-- A **data foundation** beneath all AI interactions
-- The **Rosetta Stone** of a user's life data
-- The **one layer beneath all AI tooling**
-
-Neotoma serves both human-in-the-loop workflows (people using AI tools) and agentic systems that need a deterministic, queryable memory substrate (agent frameworks, orchestration pipelines, observability stacks).
-
-**Marketing positioning:**
-- "Structured personal data memory for AI agents"
-- "Memory infrastructure for agentic systems"
-- "Cross-platform memory that works with all your AI tools"
-- "The truth engine behind your AI tools"
-- "The foundation for agent-native personal computing"
-- "The private data substrate that LLM app layers depend on for context engineering"
-
-**Substrate Positioning in LLM Stack:**
-
-Neotoma positions as the substrate layer beneath the emerging "LLM app layer" (applications like Cursor that orchestrate LLM calls for specific verticals). As [Andrej Karpathy observes in his 2025 LLM Year in Review](https://x.com/karpathy/status/2002118205729562949), LLM labs produce generally capable models, but LLM apps "organize, finetune and actually animate teams of them into deployed professionals in specific verticals by supplying private data, sensors and actuators and feedback loops." Neotoma provides that critical "private data" component—the structured memory substrate that LLM app layers depend on for effective context engineering.
-
-**LLM Stack Architecture:**
-- **LLM Labs** (OpenAI, Anthropic, Google): Produce foundation models
-- **LLM App Layer** (Cursor, vertical-specific apps): Orchestrate LLM calls, provide context engineering, application-specific GUIs
-- **Neotoma (Truth Layer)**: Provides structured personal data substrate—the private data that enables LLM apps to be effective
-
-**vs. Provider Memory:** ChatGPT, Claude, and Gemini offer conversation-only memory (platform-locked, provider-controlled). Neotoma provides structured personal data memory with entity resolution, timelines, and cross-platform access via MCP.
-## 7.2 Differentiation
-Neotoma is **not competing** with Notion, Evernote, Google Docs, or PKM systems.
-**Those tools store files. Neotoma understands files.**
-**Provider memory is conversation-only. Neotoma structures personal data.**
-### Defensible Differentiators
-**Core Architectural Choices (Long-Term Defensible):**
-These differentiators are defensible because competitors find them structurally difficult to pursue:
-1. **Privacy-First Architecture (User-Controlled Memory)**
-   - User-controlled memory with no provider access, never used for training
-   - End-to-end encryption with row-level security
-   - User owns data with full export and deletion control
-   - **Why Defensible:** Providers cannot pursue this due to business model conflicts (data collection, training use). Startups cannot pursue this due to provider-controlled revenue models.
-2. **Deterministic Extraction (vs. ML-Based Probabilistic)**
-   - Schema-first field extraction with deterministic, explainable results
-   - Same input always produces same output (reproducible)
-   - No hallucinations or probabilistic behavior
-   - Hash-based entity IDs ensure deterministic, tamper-evident records
-   - **Creates a "Verifiable Domain" for Personal Data:** By making extraction deterministic and verifiable, Neotoma creates objective (non-gameable) results similar to verifiable domains in math and code. This enables reliable, consistent results that LLMs can depend on, compensating for the ["jagged intelligence" problem](https://x.com/karpathy/status/2002118205729562949) where LLMs excel in verifiable domains but struggle with consistency elsewhere.
-   - **Why Defensible:** Providers cannot pursue this due to ML-first organizational identity. Startups cannot pursue this due to speed-to-market constraints.
-3. **Cross-Platform Access (MCP Integration)**
-   - Works with ChatGPT, Claude, Cursor via MCP (not platform-locked)
-   - Memory persists across all AI tools, not just one platform
-   - **Localhost Agent Compatible:** Aligns with localhost agent architectures (Claude Code) that prioritize "private environment, data and context" on the user's computer, as [Karpathy describes](https://x.com/karpathy/status/2002118205729562949). Neotoma serves as the private data substrate that localhost agents depend on.
-   - **Why Defensible:** Providers cannot pursue this due to platform lock-in business models. Startups cannot pursue this due to separate consumer app positioning.
-**Feature Capabilities (Enabled by Defensible Differentiators):**
-These features are valuable but not defensible alone (competitors are developing similar capabilities):
-- **Dual-path ingestion:** File uploads + agent interactions via MCP (not conversation-only)
-- **Entity resolution:** Deterministic hash-based canonical IDs across all personal data with cryptographic integrity
-- **Timeline generation:** Deterministic automatic chronological ordering across all personal data
-- **Multi-modal ingestion:** PDFs, images, OCR, agent interactions
-- **Full provenance:** Every field traces to source (document or agent interaction)
-- **Immutability:** Truth never changes; immutable audit trail with cryptographic integrity
-- **Event-sourced history:** Complete event log enables historical replay and time-travel queries
-- **Cryptographic integrity:** Hash-based entity IDs and event chaining ensure deterministic, tamper-evident records
-- **Type-stable graph:** No orphans, no cycles, typed edges
-**Strategic Positioning:** Neotoma combines defensible differentiators (privacy-first, deterministic, cross-platform) with feature capabilities (entity resolution, timelines, dual-path ingestion). Competitors can replicate features but cannot pursue the same architectural choices due to structural constraints.
-
-**Compensating for Jagged Intelligence:** LLMs display ["jagged intelligence"](https://x.com/karpathy/status/2002118205729562949)—simultaneously genius and cognitively challenged, strong in verifiable domains but inconsistent elsewhere. Neotoma's deterministic substrate compensates for this by providing a reliable truth layer that agents can depend on. By creating a verifiable domain for personal data with objective (non-gameable) results, Neotoma ensures consistent, reproducible outcomes despite LLM inconsistency.
-
-**This is a new category: Deterministic Personal Memory Engine (DPME).**

package/docs/foundation/product_principles.md

@@ -1,18 +0,0 @@
-# Neotoma Product Principles
-## 10.1 Truth Before Experience
-Correctness > convenience. Never sacrifice accuracy for UX.
-## 10.2 Explicit Over Implicit
-Users control ingestion. No automatic background processing.
-## 10.3 Minimal Over Magical
-UI is an inspection window, not an AI showpiece. Show truth, don't embellish.
-## 10.4 Schema Over Semantics
-LLMs cannot determine truth; schemas do. Type-driven, not inference-driven.
-## 10.5 Structure Over Interpretation
-Store structured facts, not meaning. Extract fields, don't interpret intent.
-## 10.6 Determinism Over Heuristics
-If a rule can be misinterpreted, formalize it. No ambiguity.
-## 10.7 Privacy Over Automation
-- No background ingestion
-- No message body ingestion
-- No auto-scanning of cloud drives
-- User explicitly provides all data

package/docs/foundation/quality_requirements.md

@@ -1,68 +0,0 @@
-# Neotoma Quality Requirements
-## 20. Testing and Quality Requirements
-### 20.1 Test Types (All Required)
-**Unit Tests:**
-- Pure functions (extraction, ID generation)
-- Deterministic (run 100 times → 100 same results)
-- Fast (<10ms per test)
-**Integration Tests:**
-- Service interactions with test DB
-- Full ingestion pipeline
-- Graph insertion with transactions
-**E2E Tests (Playwright):**
-- Upload file → see record details
-- Search → click result → view detail
-- Timeline view → filter by date
-**Property-Based Tests:**
-- Invariant verification (e.g., entity ID always same length)
-- Determinism proofs
-### 20.2 Coverage Targets
-| Code Type         | Lines | Branches | Critical Paths |
-| ----------------- | ----- | -------- | -------------- |
-| Domain Logic      | >85%  | >85%     | 100%           |
-| Application Layer | >80%  | >80%     | 100%           |
-| UI Components     | >75%  | >75%     | N/A            |
-**Critical paths (100% required):**
-- Ingestion pipeline
-- Entity resolution
-- Graph insertion
-- Search ranking
-## 21. Observability Requirements
-All operations MUST emit:
-**Metrics:**
-- Counters (e.g., `neotoma_record_upload_total{status="success"}`)
-- Histograms (e.g., `neotoma_record_upload_duration_ms`)
-**Logs:**
-- Structured JSON logs
-- NO PII (record IDs only, not extracted fields)
-- Include `trace_id` for distributed tracing
-**Events:**
-- State changes (e.g., `record.created`, `ingestion.failed`)
-- Payload: metadata only, no PII
-**Traces:**
-- Distributed tracing spans (e.g., `ingestion.ingest_file`)
-- Propagate `trace_id` through all layers
-## 22. Privacy and Security Commitments
-### 22.1 Privacy (PII Handling)
-**MUST NOT:**
-- Log PII from `properties` (names, SSN, addresses, phone)
-- Log full `raw_text` (may contain PII)
-- Log auth tokens or credentials
-- Store PII unencrypted (use RLS, future: encryption)
-**MAY Log:**
-- Record IDs (UUIDs)
-- Schema types
-- Error codes
-- Performance metrics (file size, duration)
-### 22.2 Security
-**Authentication:**
-- Local auth (dev stub or key-based when encryption enabled)
-- OAuth for MCP client connections
-**Authorization:**
-- Row-Level Security (RLS) when applicable
-- MVP: All authenticated users see all records (single-user)
-- Future: Per-user isolation via `user_id` column + RLS policies
-**Data Protection:**
-- Database encryption at rest (optional, key file or mnemonic)
-- HTTPS for all API calls
-- WSS (WebSocket Secure) for MCP connections

package/docs/foundation/timeline_events.md

@@ -1,19 +0,0 @@
-# Timeline and Event Doctrine
-## 16.1 Event Generation Rules
-Events MUST:
-- Derive from extracted date fields (never inferred)
-- Reflect actual dates in documents
-- Be timestamp-normalized (ISO 8601)
-- Be source-field-linked (traceability)
-- Have deterministic IDs (hash-based)
-Events MUST NOT:
-- Be inferred or predicted
-- Be created without source date
-- Mutate after creation
-## 16.2 Timeline Requirements
-Timeline MUST be:
-- Chronological (sorted by `event_timestamp`)
-- Deterministic (same events → same order)
-- Stable (events never removed without user action)
-- Source-linked (every event → source record + field)
-**AI MUST rely on timeline for temporal reasoning, not guess.**

package/docs/foundation/user_workflows.md

@@ -1,38 +0,0 @@
-# Neotoma Key User Workflows (MVP)
-## Workflow 1: Upload Documents → See Structured Memory
-User uploads invoices, receipts, contracts, travel docs.
-They see:
-- Extracted fields (invoice_number, amount, vendor)
-- Entities (companies, people automatically identified)
-- Events (timeline of dates from documents)
-- Graph relationships (which entities in which records)
-## Workflow 2: Agent Creates Structured Data → See in Memory
-User provides structured data during AI conversation (ChatGPT, Claude, Cursor).
-Agent stores data via MCP `store_record`:
-- User preference (theme: dark)
-- Project context (project_name, deadline, collaborators)
-- Contact information (person, company, role)
-They see:
-- Structured data in memory (queryable via MCP)
-- Entities unified with documents ("Acme Corp" from conversation matches "Acme Corp" from invoice)
-- Events added to timeline
-- Cross-platform access (data available in ChatGPT, Claude, Cursor)
-## Workflow 3: Connect Gmail → Import Attachments
-User selects labels (Receipts, Travel, Finance).
-System ingests attachments only (never email bodies).
-## Workflow 4: Ask AI Questions
-- "Summarize this contract"
-- "Show me all data involving Acme Corp" (documents + agent-created data)
-- "What are all my travel events next month?"
-- "What expires soon?"
-**AI MUST use MCP and the graph, not guess.**
-## Workflow 5: Explore the Timeline
-Chronological view of events from all personal data:
-- Flights (departure/arrival) — from documents
-- Contract effective dates — from documents
-- Invoice issued/due dates — from documents
-- Passport expiry — from documents
-- Project deadlines — from agent-created data
-- Event dates — from agent-created data
-**AI and UI MUST show the same events (single source of truth).**
-Timeline works across documents AND agent-created data, differentiating from conversation-only provider memory.