neotoma 0.3.0 → 0.3.2

package/docs/reports/dr_cintas_pageindex_structure_over_embeddings_report.md

@@ -1,77 +0,0 @@
----
-title: "Dr. Cintas / PageIndex 'Structure Over Embeddings' — Neotoma Positioning Report"
-status: "report"
-created_date: "2026-02-06"
-sources:
-  - "https://x.com/dr_cintas/status/2019045152350756869 (Dr. Alvaro Cintas, Feb 4 2026)"
-  - "docs/private/insights/dr_cintas_tweet_2019045152350756869_relevance_analysis.md"
----
-
-# Dr. Cintas / PageIndex "Structure Over Embeddings" — Neotoma Positioning Report
-
-## Scope
-
-This report summarizes Dr. Alvaro Cintas's tweet promoting PageIndex (RAG without vector DBs, document trees instead of embeddings) and the application of that narrative to Neotoma's positioning. It does not change code or schema.
-
-## Purpose
-
-Provide a single reference for: (1) what the tweet claims and promotes, (2) how "structure over embeddings" aligns with Neotoma's architecture, and (3) how Neotoma should use this narrative in positioning and messaging.
-
-## Executive Summary
-
-| Question | Answer |
-|----------|--------|
-| Does the tweet support or contradict Neotoma? | **Supports.** Same direction: structure over embeddings, reason over structure, no embedding/chunking dependency. |
-| Positioning implication? | **Use "structure over embeddings" and "reason over structure" in messaging.** Neotoma = structure-first memory; PageIndex = structure-based RAG. Different layers; same narrative family. |
-| Competitive overlap? | **None.** PageIndex is RAG over documents; Neotoma is Truth Layer for personal data. Complementary narrative, not same product. |
-
----
-
-## 1. Source Summary
-
-**Source:** Dr. Alvaro Cintas (@dr_cintas), Twitter/X, February 4, 2026.  
-**URL:** https://x.com/dr_cintas/status/2019045152350756869
-
-**Tweet content (abbreviated):** "Vector databases just got disrupted. You can now build RAG without Vector DBs. PageIndex is a new open-source library that uses document trees instead of embeddings. It achieves 98.7% on FinanceBench by letting LLMs reason over structure rather than matching keywords. No Embeddings, No Chunking. 100% Open Source."
-
-**Author:** Professor, PhD Computer Science & Engineering; educates on AI, cybersecurity, technology. 122K+ followers. High engagement (1K+ likes, 131 retweets, 77.7K views).
-
-**What it promotes:** PageIndex as an alternative to vector-DB-based RAG: document trees, reasoning over structure, benchmark (FinanceBench 98.7%), no embeddings, no chunking, open source.
-
----
-
-## 2. Alignment with Neotoma
-
-**Conclusion: The tweet supports Neotoma's positioning; it does not contradict it.**
-
-| Tweet / PageIndex theme | Neotoma alignment |
-|-------------------------|-------------------|
-| Structure over embeddings | Neotoma is structure-first: schema, observations, entity graph, deterministic retrieval. No vector search in core. |
-| Reason over structure (vs. matching keywords) | Neotoma retrieves by schema, entities, timelines, MCP tools; not by embedding similarity. |
-| No embeddings, no chunking | Neotoma does not depend on chunking or embedding pipelines for core memory; schema-driven extraction and graph. |
-| Open, transparent approach | Neotoma's open/cross-platform and deterministic design fits the same family of "explainable, structure-first" narrative. |
-
-**Difference (additive, not contradictory):** PageIndex targets RAG over documents (retrieval for LLM context). Neotoma is a Truth Layer for personal data (memory, entities, timelines). Same narrative (structure over embeddings); different layer and use case. Neotoma should not position as "RAG" or "PageIndex competitor" but can use the same messaging themes.
-
----
-
-## 3. Recommendations
-
-1. **Messaging:** Add "structure over embeddings" and "reason over structure" as supporting themes in positioning (docs, README, talks). State that Neotoma's memory is structure-first and does not rely on vector search in the core.
-2. **Boundary:** Keep category clear: Neotoma = Truth Layer for personal data; PageIndex = RAG over documents. Reference the trend without conflating products.
-3. **Docs:** If not already present, add a short "why we don't use vector search" or "structure-first memory" section, with optional pointer to this narrative (e.g., this tweet or similar content).
-4. **Monitor:** Track PageIndex and similar structure-based RAG work for messaging and ecosystem alignment; do not treat as direct competitors.
-
----
-
-## 4. Conclusion
-
-Dr. Cintas's tweet reinforces that "structure over embeddings" and "reason over structure" are credible, high-engagement narratives. Neotoma's architecture (deterministic, schema-first, structured memory, no semantic search in MVP) is aligned with this direction. The tweet does not describe Neotoma but validates that structure-first, non-embedding approaches are marketable and technically credible. Use this narrative to support Neotoma's positioning while keeping Neotoma clearly in the Truth Layer / personal-data memory category.
-
----
-
-## Related Documents
-
-- `docs/private/insights/dr_cintas_tweet_2019045152350756869_relevance_analysis.md` — Full relevance analysis
-- `docs/foundation/core_identity.md`
-- `docs/foundation/product_positioning.md`

package/docs/reports/failure_gallery.md

@@ -1,97 +0,0 @@
-# Failure gallery
-## Scope
-This document catalogs deterministic failure modes for ingestion, MCP actions, and reducers. It does not define remediation playbooks or release processes.
-
-## Purpose
-Provide a single reference for common failure modes, expected error envelopes, and diagnostics.
-
-## Invariants
-1. Failure cases must map to canonical error codes.
-2. Errors must not include PII.
-3. Idempotent operations must not create duplicates.
-4. Observations remain immutable even after failures.
-
-## Definitions
-- **Failure case**: A deterministic scenario that produces a known error envelope.
-- **Error envelope**: Structured error response with `error_code`, `message`, and `timestamp`.
-- **Replay**: Re running the same input and verifying identical failure output.
-
-## Failure cases
-### Missing authentication
-**Scenario:** MCP or REST call without valid authentication.
-**Expected error_code:** `AUTH_REQUIRED`
-**Notes:** Rejects before any data access.
-
-### Idempotency key reuse with conflicting payload
-**Scenario:** Reuse `idempotency_key` with a different structured payload.
-**Expected error_code:** `DB_CONSTRAINT_VIOLATION` or `VALIDATION_INVALID_FORMAT`
-**Notes:** Must return existing source or reject with conflict.
-
-### Invalid schema update
-**Scenario:** `update_schema_incremental` includes invalid field definition.
-**Expected error_code:** `VALIDATION_INVALID_FORMAT`
-**Notes:** No schema version is created.
-
-### Source not found
-**Scenario:** Fetch source by ID that does not exist.
-**Expected error_code:** `RESOURCE_NOT_FOUND`
-**Notes:** 404 response, no side effects.
-
-### Relationship key mismatch
-**Scenario:** Get relationship snapshot for a non existent key.
-**Expected error_code:** `RESOURCE_NOT_FOUND`
-**Notes:** No relationship snapshot returned.
-
-### Timeline query with invalid date
-**Scenario:** Timeline filter uses invalid date format.
-**Expected error_code:** `VALIDATION_INVALID_FORMAT`
-**Notes:** No events returned.
-
-### Observation creation with user mismatch
-**Scenario:** Provided `user_id` does not match authenticated user.
-**Expected error_code:** `FORBIDDEN`
-**Notes:** No observation created.
-
-## Examples
-### Error envelope
-```json
-{
-  "error_code": "RESOURCE_NOT_FOUND",
-  "message": "Source not found",
-  "details": {
-    "resource": "source",
-    "id": "src_missing"
-  },
-  "timestamp": "2024-01-10T00:00:00.000Z"
-}
-```
-
-## Testing requirements
-1. Add regression tests when a new failure case is discovered.
-2. Validate error envelopes in integration tests.
-3. Use deterministic fixtures for replay tests.
-
-## Agent Instructions
-### When to Load This Document
-Load when adding new error codes, adjusting error responses, or writing failure regression tests.
-
-### Required Co-Loaded Documents
-- `docs/NEOTOMA_MANIFEST.md`
-- `docs/subsystems/errors.md`
-- `docs/reference/error_codes.md`
-- `docs/testing/testing_standard.md`
-
-### Constraints Agents Must Enforce
-1. Every failure case maps to a canonical error code.
-2. Error messages do not include PII.
-3. Failures do not mutate observations or sources.
-
-### Forbidden Patterns
-- Returning raw stack traces to clients
-- Using non canonical error codes
-- Recording PII in error messages
-
-### Validation Checklist
-- [ ] Failure case uses canonical error code
-- [ ] Error envelope is structured and deterministic
-- [ ] Tests added for new failure cases

package/docs/reports/fixture_expansion_summary.md

@@ -1,204 +0,0 @@
-# Fixture Expansion Implementation Summary
-
-Generated: 2026-01-22
-
-## Overview
-
-Comprehensive expansion of test fixtures to support all 1.0 schemas with multiple variants for thorough testing coverage.
-
-## Implementation Complete
-
-### Phase 1: Schema Consolidation ✅
-
-**Objective**: Prevent drift between schema definitions and type catalog
-
-**Changes**:
-- Extended `EntitySchema` interface with `EntitySchemaMetadata`
-- Added metadata to all 50 schemas (label, description, category, aliases)
-- Refactored `record_types.ts` to derive from `schema_definitions.ts`
-- Created validation script (`scripts/validate-schema-sync.ts`)
-
-**Result**: Single source of truth eliminates drift risk
-
-### Phase 2: DATA_DIR Analysis ✅
-
-**Objective**: Identify missing schemas from production data
-
-**Changes**:
-- Created analysis script (`scripts/analyze_data_dir_for_schemas.ts`)
-- Analyzed 72 DATA_DIR entity types
-- Added 16 priority schemas:
-  - **Productivity**: outcome, strategy, process, task_attachment, task_comment, task_dependency, task_story
-  - **Health**: habit, workout, emotion, habit_completion, habit_objective
-  - **Knowledge**: belief, domain, research, argument
-
-**Result**: Coverage improved from 38% to 65% (47/72 types)
-
-### Phase 3: JSON Fixture Expansion ✅
-
-**Objective**: 5 variants per schema for comprehensive testing
-
-**Before**: 34 files with 1-3 variants each (~80 total fixtures)  
-**After**: 34 files with 5 variants each (170 total fixtures)
-
-**Variant Coverage**:
-- **Minimal**: Required fields only
-- **Complete**: All fields populated
-- **Edge Cases**: Boundaries, nulls, special values
-- **Update Variants**: Temporal changes, same entity over time
-- **State Transitions**: Status changes (pending → completed, active → inactive)
-
-**Entity Type Coverage**:
-- **Finance** (17 types): All core financial types
-- **Productivity** (11 types): Tasks, outcomes, projects, processes
-- **Knowledge** (7 types): Contacts, research, beliefs, domains
-- **Health** (6 types): Habits, workouts, meals, exercises, emotions
-
-### Phase 4: PDF Fixture Expansion ✅
-
-**Objective**: Unstructured document fixtures for extraction testing
-
-**Before**: 6 PDFs (transaction_receipt, bank_statement, invoice, tax_form, holding_statement, contract)  
-**After**: 11 PDFs covering 11 entity types
-
-**New PDFs Added** (5):
-- `sample_receipt.pdf` - Purchase receipt
-- `sample_note.pdf` - Meeting notes
-- `sample_research.pdf` - Research paper
-- `sample_meal.pdf` - Meal log
-- `sample_exercise.pdf` - Exercise log
-
-**Documentation**: All 11 PDFs have corresponding `.md` files documenting expected extraction
-
-**Coverage**: 22% of schemas (11/50) - focused on document-heavy types
-
-## Final Statistics
-
-### Schemas
-- **Total**: 50 schemas (was 34)
-- **New**: 16 schemas from DATA_DIR analysis
-- **Metadata**: 100% have complete metadata
-- **Validation**: ✅ All pass validation
-
-### JSON Fixtures (Structured)
-- **Files**: 34
-- **Variants**: 170 (5 per file)
-- **Coverage**: 68% of schemas (34/50)
-- **Validation**: ✅ TypeScript compilation passes
-
-### PDF Fixtures (Unstructured)
-- **Files**: 11 PDFs + 11 docs
-- **Coverage**: 22% of schemas (11/50)
-- **Types**: Finance (4), Productivity/Knowledge (4), Health (2), Other (1)
-
-### Other Fixtures
-- **CSV**: 10 files (bulk data testing)
-- **Integration**: 1 multi-record scenario file
-- **Helpers**: 3 TypeScript utility files
-
-### Directory Structure
-```
-tests/fixtures/
-├── json/           # 34 files, 170 variants
-├── csv/            # 10 files
-├── pdf/            # 11 PDFs + 11 docs + 1 txt
-├── integration/    # 1 scenario file
-├── helpers.ts
-├── types.ts
-└── validation.ts
-```
-
-## Validation Results
-
-✅ **Schema Sync**: All 50 schemas validated  
-✅ **TypeScript**: Compilation passes  
-✅ **Coverage**: 65% DATA_DIR types covered  
-✅ **Organization**: Unified structure in `tests/fixtures/`  
-✅ **Documentation**: README and standard docs updated
-
-## Testing Capabilities
-
-### JSON Fixtures Support
-- **Unit testing**: Programmatic fixture generation with helpers
-- **Integration testing**: Multi-variant testing for temporal changes
-- **Edge case testing**: Minimal and boundary value fixtures
-- **State transition testing**: Status change sequences
-
-### PDF Fixtures Support
-- **Extraction testing**: Text extraction from documents
-- **Type detection**: Schema type identification
-- **Entity resolution**: Entity extraction from documents
-- **Multi-format testing**: Financial, productivity, health documents
-
-### CSV Fixtures Support
-- **Bulk import testing**: Multi-record ingestion
-- **CSV parsing**: Structured data extraction
-- **Entity resolution**: Cross-record entity deduplication
-
-## Files Changed
-
-### New Files
-- `scripts/validate-schema-sync.ts` - Schema validation script
-- `scripts/analyze_data_dir_for_schemas.ts` - DATA_DIR analysis script
-- `tests/fixtures/README.md` - Unified fixture documentation
-- `tests/fixtures/json/*.json` - 16 new JSON fixture files
-- `tests/fixtures/pdf/sample_*.pdf` - 5 new PDF fixtures
-- `tests/fixtures/pdf/sample_*.md` - 5 new PDF documentation files
-- `docs/reports/data_dir_schema_analysis.md` - Analysis report
-- `docs/reports/fixture_expansion_summary.md` - This document
-
-### Modified Files
-- `src/services/schema_definitions.ts` - Added metadata + 16 new schemas
-- `src/config/record_types.ts` - Now derives from schema_definitions
-- `tests/fixtures/json/*.json` - Expanded all 34 files to 5 variants
-- `scripts/generate_pdf_fixtures.ts` - Added 5 new PDF generators
-- `docs/testing/fixtures_standard.md` - Updated paths and coverage
-- 5 Playwright test files - Updated fixture paths
-
-### Deleted Directories
-- `src/__fixtures__/` - Migrated to `tests/fixtures/`
-- `playwright/tests/fixtures/` - Migrated to `tests/fixtures/`
-- `tests/integration/fixtures/` - Migrated to `tests/fixtures/`
-
-## Recommendations for Further Expansion
-
-### JSON Fixtures
-✅ **Complete**: All 34 types have 5 variants
-
-### PDF Fixtures
-Current: 11/50 types (22%)
-
-**High Priority** (10 more PDFs recommended):
-- **Finance**: liability_document, property_deed, income_statement, balance_sheet, purchase_receipt
-- **Productivity**: project_spec, goal_document, task_list
-- **Knowledge**: contact_card, document
-
-**Low Priority**: Task metadata types, system types (relationship, location, address)
-
-### CSV Fixtures
-Current: 10 files
-
-**Potential Additions** (5-10 more CSVs):
-- habits.csv, workouts.csv, emotions.csv
-- beliefs.csv, domains.csv, research.csv
-- outcomes.csv, strategies.csv, processes.csv
-
-## Next Steps
-
-1. **Schema Addition**: Add remaining 25 DATA_DIR types if valuable
-2. **PDF Expansion**: Add 10 high-priority PDFs (Phase 2 continuation)
-3. **CSV Expansion**: Add 5-10 CSVs for new entity types
-4. **Testing**: Run full test suite with expanded fixtures
-5. **Documentation**: Update any test documentation referencing fixture counts
-
-## Success Criteria Met
-
-✅ All schemas have complete metadata  
-✅ Schema consolidation prevents drift  
-✅ DATA_DIR analysis identifies gaps  
-✅ JSON fixtures: 5 variants per type (170 total)  
-✅ PDF fixtures expanded (6 → 11)  
-✅ Unified fixture location (`tests/fixtures/`)  
-✅ All validations pass  
-✅ Documentation updated  
-✅ Old directories removed

package/docs/reports/mcp_action_names_review.md

@@ -1,131 +0,0 @@
-# MCP Action Names Review Against Canonical Terms
-
-**Date:** 2026-01-08  
-**Reference:** [`docs/vocabulary/canonical_terms.md`](../vocabulary/canonical_terms.md)
-
-## Summary
-
-Review of all MCP action names against canonical terminology to ensure alignment.
-
-## Canonical Terms Reference
-
-### "Retrieving" (Lines 81-92)
-**Definition:** The process of querying and retrieving [entities](#entity), [observations](#observation), [entity snapshots](#entity-snapshot), and related data from the [memory graph](#memory-graph).
-
-**Forbidden Synonyms:**
-- ❌ "fetch" (too generic)
-- ❌ "load" (implies loading into memory, retrieving is querying)
-- ❌ "get" (too generic, use specific action names)
-
-**Example:** "Use `retrieve_entities` to query [entities](#entity) by type, or `get_entity_snapshot` to get current truth for an [entity](#entity)."
-
-**Note:** The example itself uses `get_entity_snapshot`, suggesting that while "get" is discouraged as a generic term, specific action names like `get_entity_snapshot` may be acceptable if they are sufficiently descriptive.
-
-### "Storing" (Lines 106-118)
-**Definition:** The process of uploading, [extracting](#extraction), and inserting [source](#source) into the [memory graph](#memory-graph).
-
-**Status:** ✅ Already renamed from `ingest` to `store`
-
-### "Interpretation" (Lines 120-131)
-**Definition:** Versioned interpretation attempt on unstructured [source](#source).
-
-**Action:** ✅ `reinterpret` aligns with this term
-
-## Current MCP Actions Analysis
-
-### ✅ Actions That Align with Canonical Terms
-
-| Action Name | Purpose | Canonical Term | Status |
-|------------|---------|----------------|--------|
-| `store` | Unified storing for all source | "Storing" | ✅ Aligned |
-| `retrieve_entities` | Query entities with filters | "Retrieving" | ✅ Aligned |
-| `list_observations` | Query observations for entity | "Retrieving" | ✅ Aligned |
-| `list_relationships` | Query entity relationships | "Retrieving" | ✅ Aligned |
-| `list_timeline_events` | Query timeline events | "Retrieving" | ✅ Aligned |
-| `list_entity_types` | List all available entity types | "Retrieving" | ✅ Aligned |
-| `reinterpret` | Re-run AI interpretation | "Interpretation" | ✅ Aligned |
-| `create_relationship` | Create typed relationship | "Relationship" | ✅ Aligned |
-| `merge_entities` | Merge duplicate entities | "Entity" | ✅ Aligned |
-| `correct` | Create correction observation | "Observation" | ✅ Aligned |
-
-### ⚠️ Actions with "get_" Prefix - Review Needed
-
-| Action Name | Purpose | Issue | Recommendation |
-|------------|---------|-------|----------------|
-| `get_file_url` | Get signed URL for file access | Uses "get" (forbidden synonym for "Retrieving") | Consider `retrieve_file_url` or keep if "get URL" is a standard pattern |
-| `get_entity_snapshot` | Get entity snapshot with provenance | Uses "get" (forbidden synonym) | However, canonical terms example uses this name. Could rename to `retrieve_entity_snapshot` but may break existing usage |
-| `get_field_provenance` | Get provenance chain for field | Uses "get" (forbidden synonym) | Could rename to `retrieve_field_provenance` |
-| `get_entity_by_identifier` | Find entity by identifier | Uses "get" (forbidden synonym) | Could rename to `retrieve_entity_by_identifier` |
-| `get_related_entities` | Get entities via relationships | Uses "get" (forbidden synonym) | Could rename to `retrieve_related_entities` |
-| `get_graph_neighborhood` | Get complete graph context | Uses "get" (forbidden synonym) | Could rename to `retrieve_graph_neighborhood` |
-
-## Key Finding
-
-The canonical terms document explicitly states that **"get" is a forbidden synonym** for "Retrieving" because it's "too generic." However, the same document's example uses `get_entity_snapshot`:
-
-> "Use `retrieve_entities` to query [entities](#entity) by type, or `get_entity_snapshot` to get current truth for an [entity](#entity)."
-
-This creates an inconsistency:
-- The rule says "get" is forbidden
-- The example uses `get_entity_snapshot`
-
-## Recommendation
-
-### Option 1: Rename All "get_" Actions to "retrieve_" (Consistent with Canonical Terms)
-
-**Breaking Changes:**
-- `get_file_url` → `retrieve_file_url`
-- `get_entity_snapshot` → `retrieve_entity_snapshot`
-- `get_field_provenance` → `retrieve_field_provenance`
-- `get_entity_by_identifier` → `retrieve_entity_by_identifier`
-- `get_related_entities` → `retrieve_related_entities`
-- `get_graph_neighborhood` → `retrieve_graph_neighborhood`
-
-**Pros:**
-- Fully aligned with canonical terms
-- Consistent naming pattern (`retrieve_*` for all retrieval operations)
-- Removes ambiguity about whether "get" is acceptable
-
-**Cons:**
-- Breaking change for all existing MCP clients
-- The canonical terms example uses `get_entity_snapshot`, so this contradicts that example
-
-### Option 2: Keep "get_" Prefix for Specific Actions (Acceptable Specific Names)
-
-**Rationale:**
-- While "get" is forbidden as a generic term, specific action names like `get_entity_snapshot` may be acceptable if they are sufficiently descriptive
-- The canonical terms example uses `get_entity_snapshot`, suggesting it's acceptable in practice
-- `get_file_url` follows HTTP/REST conventions (GET request pattern)
-
-**Status:** Keep as-is, but update canonical terms document to clarify that specific action names with "get_" prefix are acceptable when sufficiently descriptive.
-
-### Option 3: Hybrid Approach - Rename Some, Keep Others
-
-**Rename:**
-- `get_entity_by_identifier` → `retrieve_entity_by_identifier` (query operation)
-- `get_related_entities` → `retrieve_related_entities` (query operation)
-- `get_graph_neighborhood` → `retrieve_graph_neighborhood` (query operation)
-
-**Keep:**
-- `get_file_url` (follows REST/HTTP conventions)
-- `get_entity_snapshot` (used in canonical terms example)
-- `get_field_provenance` (specific retrieval operation, may be acceptable)
-
-## Decision Required
-
-The canonical terms document has an inconsistency:
-1. It states "get" is forbidden (line 92)
-2. It uses `get_entity_snapshot` in an example (line 87)
-
-**Questions to resolve:**
-1. Should we strictly follow the rule and rename all "get_" actions to "retrieve_"?
-2. Or should we clarify that specific action names with "get_" prefix are acceptable?
-3. Should we update the canonical terms document to resolve this inconsistency?
-
-## Recommended Action
-
-**Immediate:** Clarify the canonical terms document to specify:
-- "get" as a standalone generic term is forbidden
-- Specific action names with "get_" prefix are acceptable if they are sufficiently descriptive (e.g., `get_entity_snapshot`, `get_file_url`)
-
-**Future:** Consider standardizing on "retrieve_" prefix for all retrieval operations in a future major version release (v2.0.0) to fully align with canonical terms.