neotoma 0.3.0 → 0.3.2

package/docs/reports/TEST_FIXES_SUMMARY.md

@@ -1,96 +0,0 @@
-# Test Fixes Summary
-
-**Date**: 2026-01-15  
-**Status**: 8/58 tests still failing (down from 23/58)
-
-## Fixes Applied
-
-### 1. Method Name Mismatches (Fixed 15 tests)
-
-**Problem**: Tests were calling MCP actions with snake_case names (e.g., `retrieve_entity_snapshot`), but the actual server methods use camelCase (e.g., `retrieveEntitySnapshot`).
-
-**Solution**: Created a `callMCPAction` helper function that converts snake_case to camelCase and calls the correct method.
-
-**Files Modified**:
-- `tests/integration/mcp_actions_matrix.test.ts` - Added helper, fixed 18 method calls
-- `tests/integration/mcp_auto_enhancement.test.ts` - Added helper, fixed 3 method calls
-
-**Tests Fixed**: 15 tests now passing
-
-### 2. Relationship Snapshot Timing (Partially Fixed)
-
-**Problem**: `createRelationship` was trying to retrieve a relationship snapshot immediately after creating it, but snapshot creation is async and might not be complete yet.
-
-**Solution**: Added retry logic with exponential backoff (3 attempts with 100ms, 200ms delays) and changed from `.single()` to `.maybeSingle()`.
-
-**Files Modified**:
-- `src/server.ts` - Updated `createRelationship` method
-
-**Status**: Still 3 relationship tests failing, may need longer delays or different approach
-
-### 3. Auto-Enhancement Test Expectations (Partially Fixed)
-
-**Problem**: Tests were checking `unknown_fields_count` immediately, but raw fragments might not be created yet, or the count might be 0 if all fields match the schema.
-
-**Solution**: 
-- Removed strict `unknown_fields_count` check
-- Added wait for auto-enhancement processor
-- Focus on verifying raw_fragments count instead
-
-**Files Modified**:
-- `tests/integration/mcp_auto_enhancement.test.ts`
-
-**Status**: Still 2 auto-enhancement tests failing - raw fragments not being created
-
-## Remaining Failures (8 tests)
-
-### Relationship Operations (3 failures)
-1. `create_relationship` - should create relationship between entities
-2. `create_relationship` - should return CYCLE_DETECTED for circular relationships  
-3. `list_relationships` - should list relationships for entity
-
-**Error**: `Failed to retrieve relationship snapshot: Cannot coerce the result to a single JSON object`
-
-**Possible Causes**:
-- Snapshot creation takes longer than retry window
-- Snapshot might not be created if observation creation fails
-- Query might be returning 0 rows or multiple rows
-
-**Next Steps**:
-- Increase retry delay or add more attempts
-- Check if `createRelationshipObservations` is actually creating observations
-- Verify relationship_snapshots table structure
-
-### Auto-Enhancement (2 failures)
-1. `should create queue entries when storing unknown fields` - No raw_fragments found
-2. `should process queue and create schema recommendations` - No queue items processed
-
-**Possible Causes**:
-- Schema matching might be too permissive (treating all fields as known)
-- `id` field might be treated specially and not counted as unknown
-- Raw fragments query might be wrong (checking wrong fields)
-
-**Next Steps**:
-- Verify parquet file structure matches expectations
-- Check if schema matching logic excludes `id` field
-- Add debug logging to see what fields are being stored
-
-### Other Tests (3 failures)
-1. `retrieve_entity_snapshot` - should retrieve entity snapshot with provenance
-2. `merge_entities` - should merge two entities
-3. `list_observations` - should list observations for entity
-
-**Status**: Need to investigate error messages for these tests
-
-## Test Results
-
-**Before**: 23/58 failing (40% failure rate)  
-**After**: 8/58 failing (14% failure rate)  
-**Improvement**: 65% reduction in failures
-
-## Next Actions
-
-1. Investigate relationship snapshot creation timing
-2. Debug auto-enhancement raw fragments creation
-3. Check error messages for remaining 3 test failures
-4. Consider adding integration test helpers for async operations

package/docs/reports/VOCABULARY_CONFORMANCE_REVIEW.md

@@ -1,177 +0,0 @@
-# Vocabulary Conformance Review
-
-**Date:** 2025-01-27  
-**Scope:** Review of `docs/` and `src/services/schema_definitions.ts` against `docs/vocabulary/canonical_terms.md`
-
-## Summary
-
-This review identifies terminology inconsistencies across documentation and code that need to be updated to align with canonical vocabulary.
-
-## ✅ Fixed Issues
-
-### 1. `src/services/schema_definitions.ts`
-**Status:** ✅ **FIXED**
-
-**Changes Made:**
-- `RecordTypeSchema` → `EntitySchema`
-- `RECORD_TYPE_SCHEMAS` → `ENTITY_SCHEMAS`
-- `EXPANDED_RECORD_TYPE_SCHEMAS` → `EXPANDED_ENTITY_SCHEMAS`
-- Updated comments: "Record Types" → "Entity Types", "schema" → "entity schema" when referring to definitions
-- Updated function parameter names: `recordType` → `entityType`
-- Added note about default schemas and fallback behavior
-
-**Impact:** Low - function signatures unchanged, only internal names updated
-
-### 2. `docs/architecture/schema_handling.md`
-**Status:** ✅ **FIXED**
-
-**Changes Made:**
-- Updated all terminology to use canonical terms with markdown links
-- "record" → "[source](#source)" or "[entity](#entity)"
-- "record type" → "[entity type](#entity-type)"
-- "schema" → "[entity schema](#entity-schema)"
-- "ingestion" → "[storing](#storing)"
-- "parsing/analysis" → "[extraction](#extraction)"
-- Updated SQL examples to use `observations` and `raw_fragments` tables
-- Updated references to point to correct documentation
-
-### 3. `docs/subsystems/ingestion/ingestion.md`
-**Status:** ✅ **PARTIALLY FIXED**
-
-**Changes Made:**
-- Updated title: "Ingestion" → "[Storing](#storing)"
-- Updated "interpretation runs" → "[interpretations](#interpretation)"
-- Added canonical term markdown links throughout
-- Updated pipeline diagram labels
-- Updated scope and path descriptions
-
-**Remaining:** Some internal references may still use "ingestion" in less critical sections
-
-### 4. `docs/subsystems/schema.md`
-**Status:** ✅ **PARTIALLY FIXED**
-
-**Changes Made:**
-- Updated Section 3: "properties" → "fields" (observations)
-- Updated all SQL query examples to use `observations` table
-- Updated references from `records.type` to `entity_type`
-- Updated extraction metadata section to reference `raw_fragments`
-- Added notes about legacy architecture for historical reference
-- Updated index examples to use observations table
-
-**Remaining:** Some examples in Section 3 still reference legacy structure (may be intentional for examples)
-
-## ⚠️ Critical Issues Requiring Updates
-
-### 2. `docs/subsystems/record_types.md`
-**Status:** ⚠️ **NEEDS MAJOR UPDATE**
-
-**Issues:**
-- Entire document uses "record types" terminology (deprecated)
-- References `records.type` column (legacy table)
-- References `src/config/record_types.ts` (may not exist or need update)
-- Should be renamed/updated to use "entity types" terminology
-
-**Recommendation:**
-- Rename to `entity_types.md` or `docs/subsystems/entity_types.md`
-- Update all references from "record type" to "[entity type](#entity-type)"
-- Update references from `records.type` to entity type classification
-- Verify `src/config/record_types.ts` exists and update if needed
-
-### 3. `docs/subsystems/ingestion/ingestion.md`
-**Status:** ⚠️ **NEEDS UPDATE**
-
-**Issues:**
-- Uses "ingestion" terminology (should be "[storing](#storing)")
-- References "interpretation runs" (should be "[interpretation](#interpretation)")
-- May reference deprecated record-based architecture
-
-**Recommendation:**
-- Update "ingestion" → "[storing](#storing)" throughout
-- Update "interpretation run" → "[interpretation](#interpretation)"
-- Verify alignment with sources-first architecture
-
-### 4. `docs/subsystems/schema.md`
-**Status:** ⚠️ **NEEDS UPDATE**
-
-**Issues:**
-- Section 3.11 references `records` table and `properties` JSONB field
-- References `record_types.md` (legacy document)
-- Uses "record type" terminology in some places
-
-**Recommendation:**
-- Update references to use observations/sources architecture
-- Update `properties` references to observation `fields`
-- Update links to point to entity types documentation
-
-## 📋 Files Requiring Review (295 files found)
-
-The following files contain potential deprecated terms. Priority review needed for:
-
-### High Priority (Core Documentation)
-1. `docs/subsystems/record_types.md` - Entire document needs rewrite
-2. `docs/subsystems/ingestion/ingestion.md` - Core ingestion documentation
-3. `docs/subsystems/schema.md` - Core schema documentation
-4. `docs/architecture/source_material_model.md` - Should be canonical
-5. `docs/specs/DATA_MODELS.md` - Data model specifications
-
-### Medium Priority (Subsystem Docs)
-6. `docs/subsystems/sources.md` - Sources architecture
-7. `docs/subsystems/observation_architecture.md` - Observation architecture
-8. `docs/subsystems/relationships.md` - Relationships
-9. `docs/subsystems/reducer.md` - Reducer documentation
-10. `docs/subsystems/events.md` - Events documentation
-
-### Lower Priority (Release/Historical Docs)
-- Release documentation (v0.1.0, v0.2.0, etc.) - May be historical
-- Implementation logs - Historical records
-- Compliance reports - Historical records
-
-## 🔍 Common Issues Found
-
-### Deprecated Terms to Replace
-1. **"record"** → "[source](#source)" or "[entity](#entity)" (context-dependent)
-2. **"record type"** → "[entity type](#entity-type)"
-3. **"capability"** → "[entity schema](#entity-schema)"
-4. **"ingestion"** → "[storing](#storing)"
-5. **"interpretation run"** → "[interpretation](#interpretation)"
-6. **"parsing"** → "[extraction](#extraction)"
-7. **"analysis"** → "[extraction](#extraction)" (when referring to deterministic extraction)
-
-### Forbidden Terms to Remove
-- "smart" (marketing language)
-- "intelligent" (implies non-determinism)
-- "learn" (Neotoma doesn't learn)
-- "understand" (Neotoma doesn't understand)
-
-## 📝 Recommended Action Plan
-
-### Phase 1: Critical Core Documentation (Immediate)
-1. ✅ Fix `src/services/schema_definitions.ts` - **DONE**
-2. ✅ Update `docs/architecture/schema_handling.md` - **DONE**
-3. ✅ Update `docs/subsystems/ingestion/ingestion.md` terminology - **PARTIALLY DONE**
-4. ✅ Update `docs/subsystems/schema.md` references - **PARTIALLY DONE**
-5. ⚠️ Update `docs/subsystems/record_types.md` → `entity_types.md` - **PENDING**
-
-### Phase 2: Architecture Documentation (High Priority)
-5. Review and update `docs/architecture/` files
-6. Review and update `docs/specs/` files
-7. Review and update `docs/subsystems/` files
-
-### Phase 3: Historical Documentation (Lower Priority)
-8. Review release documentation (may leave as historical)
-9. Review implementation logs (may leave as historical)
-
-## 🎯 Success Criteria
-
-Documentation is conformant when:
-- ✅ No use of deprecated terms ("record", "record type", "capability") in active documentation
-- ✅ All references use canonical terms with proper markdown links
-- ✅ Code comments and variable names use canonical terminology
-- ✅ Examples demonstrate correct usage
-- ✅ Historical documents are clearly marked or updated
-
-## Notes
-
-- Historical release documentation (v0.1.0, v0.2.0) may intentionally preserve legacy terminology for historical accuracy
-- Implementation logs may preserve original terminology for audit trail
-- Focus updates on active, current documentation that guides development

package/docs/reports/WHY_TESTS_MISSED_BIGINT_ISSUE.md

@@ -1,144 +0,0 @@
-# Why Tests Didn't Catch BigInt Serialization Issue
-
-## Summary
-
-The automated tests did not catch the BigInt serialization issue because:
-
-1. **No parquet-specific integration tests exist** - There are no automated tests that test the MCP `store` action with parquet files
-2. **Test scripts are manual, not automated** - The parquet test scripts (`test_parquet_ingestion.ts`, `test_dsnp_parquet_reading.ts`) are manual scripts, not part of the test suite
-3. **Integration tests don't cover parquet file_path parameter** - Existing MCP action tests use JSON entities directly, not parquet files via `file_path`
-4. **No unit tests for parquet reader** - The `parquet_reader.ts` service has no unit tests
-5. **Test data may not contain BigInt values** - Even if tests existed, they might use files without Int64/BigInt fields
-
-## Detailed Analysis
-
-### 1. Missing Parquet Integration Tests
-
-**What exists:**
-- `tests/integration/release/v0.1.0/it_006_mcp_actions.test.ts` - Tests MCP actions but:
-  - Tests `store_record` with JSON entities directly
-  - Tests `upload_file` but only checks endpoint exists (accepts any status code)
-  - Does NOT test `store` action with `file_path` parameter for parquet files
-
-**What's missing:**
-- Integration test that calls MCP `store` action with `file_path` pointing to a parquet file
-- Test that verifies BigInt values are properly serialized
-- Test that exercises the full flow: parquet file → read → convert → store → retrieve
-
-### 2. Manual Test Scripts (Not Automated)
-
-**Scripts found:**
-- `scripts/test_parquet_ingestion.ts` - Tests reading parquet files only
-- `scripts/test_dsnp_parquet_reading.ts` - Tests library functionality only
-
-**Issues:**
-- These are manual scripts, not part of `npm test`
-- They test reading, not storing via MCP
-- They use `JSON.stringify` which would catch BigInt IF:
-  - The test data contains BigInt values
-  - The script actually calls the store action (it doesn't)
-
-### 3. No Unit Tests for Parquet Reader
-
-**Missing:**
-- Unit tests in `tests/` directory for `src/services/parquet_reader.ts`
-- Tests that verify BigInt conversion in `convertBigIntValues()`
-- Tests that verify entity conversion with various data types
-
-**Why this matters:**
-- Unit tests could have caught the BigInt issue early
-- Could test edge cases (nested objects, arrays with BigInt, etc.)
-
-### 4. Test Data May Not Contain BigInt
-
-**Observation:**
-- The test scripts use `accounts.parquet` and `transactions.parquet`
-- These files might not contain Int64/BigInt fields
-- The `tasks.parquet` file (which triggered the issue) might be the first to contain BigInt values
-
-**Evidence:**
-- Test script at line 39 uses `JSON.stringify(result.entities[i], null, 2)`
-- If test data had BigInt, this would have failed
-- Suggests test data doesn't contain BigInt values
-
-## Recommendations
-
-### 1. Add Parquet Integration Test
-
-Create `tests/integration/mcp_store_parquet.test.ts`:
-
-```typescript
-describe("MCP Store with Parquet Files", () => {
-  it("should store parquet file via MCP store action", async () => {
-    // Create test parquet file with BigInt values
-    // Call MCP store action with file_path
-    // Verify entities are created
-    // Verify BigInt values are converted to numbers
-  });
-});
-```
-
-### 2. Add Unit Tests for Parquet Reader
-
-Create `tests/unit/parquet_reader.test.ts`:
-
-```typescript
-describe("Parquet Reader", () => {
-  it("should convert BigInt values to numbers", () => {
-    // Test convertBigIntValues function
-  });
-  
-  it("should handle nested objects with BigInt", () => {
-    // Test recursive conversion
-  });
-  
-  it("should handle arrays with BigInt", () => {
-    // Test array conversion
-  });
-});
-```
-
-### 3. Create Test Parquet File with BigInt
-
-Create a test fixture parquet file that contains:
-- Int64 fields (BigInt values)
-- Nested objects with BigInt
-- Arrays with BigInt
-- Mixed data types
-
-### 4. Add JSON Serialization Test
-
-Test that all MCP response methods handle BigInt:
-
-```typescript
-it("should serialize responses with BigInt values", () => {
-  const data = { id: BigInt(12345678901234567890) };
-  expect(() => JSON.stringify(data)).toThrow(); // Before fix
-  // After fix: should convert to number
-});
-```
-
-### 5. Add to CI/CD Pipeline
-
-Ensure parquet tests run in CI:
-- Add parquet test files to test fixtures
-- Run integration tests with parquet files
-- Verify BigInt handling in all code paths
-
-## Root Cause
-
-The issue wasn't caught because:
-1. **Feature was tested manually** - Parquet support was tested via manual scripts, not automated tests
-2. **Test coverage gap** - No integration tests for the parquet → MCP store flow
-3. **Data-dependent bug** - Only manifests with files containing BigInt values
-4. **Missing edge case testing** - BigInt serialization is an edge case that wasn't considered
-
-## Action Items
-
-- [x] Create integration test for MCP store with parquet files (`tests/integration/mcp_store_parquet.test.ts`)
-- [x] Create unit tests for parquet_reader service (`tests/unit/parquet_reader.test.ts`)
-- [x] Create test helper to generate parquet files with BigInt values (`tests/helpers/create_test_parquet.ts`)
-- [x] Add BigInt serialization tests for all JSON.stringify calls (`tests/unit/bigint_serialization.test.ts`)
-- [x] Export `convertBigIntValues` function for testing (`src/services/parquet_reader.ts`)
-- [ ] Update CI/CD to run parquet tests (requires CI/CD configuration update)
-- [ ] Document test coverage requirements for new features

package/docs/reports/beyond_rag_agent_memory_neotoma_architecture_report.md

@@ -1,103 +0,0 @@
----
-title: "Beyond RAG for Agent Memory: Neotoma Architecture Report"
-status: "report"
-created_date: "2026-02-04"
-sources:
-  - "arxiv 2602.02007 (Hu et al., xMemory paper)"
-  - "DAIR.AI X / PDF (Beyond RAG for Agent Memory)"
-  - "docs/private/insights/xmemory_beyond_rag_agent_memory_paper_relevance_analysis.md"
-  - "docs/private/insights/dair_ai_beyond_rag_agent_memory_relevance_analysis.md"
----
-
-# Beyond RAG for Agent Memory: Neotoma Architecture Report
-
-## Scope
-
-This report consolidates analysis of the paper "Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation" (Hu et al., arxiv 2602.02007) and related DAIR.AI content, and states implications for Neotoma's architecture, defensibility, and robustness. It does not change code or schema.
-
-## Purpose
-
-Provide a single reference for: (1) whether the paper supports or contradicts Neotoma's architecture, (2) what the paper implies for defensibility of Neotoma's choices, and (3) whether Neotoma remains robust if inference is used only to suggest unbreaking schema changes.
-
-## Executive Summary
-
-| Question | Answer |
-|----------|--------|
-| Does the paper support or contradict Neotoma? | **Supports.** Same direction: beyond RAG, structure over similarity, intact units. No contradiction. |
-| Defensibility implication? | **Strengthens** determinism/schema-first (paper shows LLM-based structure is fragile). **Does not** make "beyond RAG" defensible by itself; defensibility rests on Neotoma's specific choices. |
-| Robust if inference suggests schema changes? | **Yes,** provided inference only suggests; application of schema changes and all extraction/reducer execution remain deterministic. |
-
----
-
-## 1. Paper Summary
-
-**Source:** Hu et al., "Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation," arxiv 2602.02007 (preprint Feb 2026). King's College London, Alan Turing Institute.
-
-**Problem:** Standard RAG (embed stored memories, retrieve fixed top-k by similarity, concatenate) is mismatched to agent memory. RAG assumes large, heterogeneous corpora with diverse passages; agent memory is a bounded, coherent stream with highly correlated (often near-duplicate) spans. Consequences: fixed top-k collapses to redundant context; post-hoc pruning fragments temporally linked evidence.
-
-**Proposed approach:** "Decoupling to aggregation." Disentangle memories into semantic components, organise into a hierarchy, use structure to drive retrieval (not similarity ranking over raw spans). Preserve intact units; do not prune inside evidence blocks.
-
-**xMemory:** Four-level hierarchy (messages → episodes → semantics → themes), sparsity-semantics objective for theme split/merge, top-down retrieval: select compact set of themes/semantics, then expand to episodes and raw messages only when it reduces reader uncertainty. Beats Naive RAG, A-Mem, MemoryOS, LightMem, Nemori on LoCoMo and PerLTQA (answer quality and token efficiency). Paper cites A-Mem and MemoryOS as fragile due to LLM-generated structured memory (formatting deviations, failed updates).
-
----
-
-## 2. Neotoma Alignment: Support vs Contradiction
-
-**Conclusion: The paper supports Neotoma's architecture; it does not contradict it.**
-
-| Paper principle | Neotoma alignment |
-|-----------------|-------------------|
-| RAG wrong for agent memory | Neotoma avoids RAG/semantic search in MVP. |
-| Structure over similarity | Neotoma retrieves by schema, entities, timelines, relationships (MCP tools), not by embedding similarity. |
-| Intact units; no pruning inside evidence | Observations and reducer outputs are intact; no ad-hoc pruning inside them. |
-| Control redundancy without fragmenting evidence | Deterministic merge and schema-typed structure; no prompt pruning. |
-
-**Difference (additive, not contradictory):** xMemory builds structure with embeddings and LLM-generated summaries. Neotoma builds structure with schema-first, deterministic extraction and reducers. Same high-level direction; different implementation. The paper does not require embeddings or LLMs for "beyond RAG."
-
----
-
-## 3. Defensibility Implications
-
-Neotoma's defensible differentiators: (1) privacy-first, (2) deterministic/verifiable domain, (3) cross-platform MCP.
-
-| Differentiator | Paper implication |
-|----------------|-------------------|
-| **Determinism / schema-first** | **Strengthens.** Paper shows LLM-based structure (A-Mem, MemoryOS) is brittle. Supports that deterministic, schema-first structure is a more reliable (and structurally harder for LLM-centric competitors to copy) path. |
-| **"Beyond RAG" as category** | **Does not support defensibility.** Paper shows the direction is valid but achievable without Neotoma's choices (e.g. xMemory uses embeddings/LLMs). Defensibility must rest on Neotoma's specific choices, not on the category. |
-| **Privacy / MCP** | **Neutral.** Paper's impact statement recommends consent and minimisation but does not implement. No implication for or against Neotoma's privacy-first or cross-platform defensibility. |
-
-**Net:** The paper supports defensibility of the deterministic, schema-first architecture by highlighting fragility of the alternative. It does not imply the overall "beyond RAG" position is defensible by itself.
-
----
-
-## 4. Robustness When Inference Suggests Unbreaking Schema Changes
-
-**Question:** Is Neotoma's architecture still relatively robust if inference (e.g. LLM) is used to *suggest* unbreaking schema changes?
-
-**Conclusion: Yes, provided inference is strictly in a suggestion role.**
-
-| Condition | Rationale |
-|-----------|-----------|
-| **Schema changes are applied deterministically** | Migrations or config updates are applied by tooling and/or human approval, not by the model. Same schema + same data → same observations and reducer output. |
-| **Extraction and reducer stay deterministic** | No LLM in the path from input to observations or entity snapshots. Intact units and same-input–same-output hold. |
-| **Schema remains source of truth** | Canonical schema is defined by config/migrations. Inference only proposes; it does not define or mutate authority. |
-
-**Risk to avoid:** Inference must not creep into execution (e.g. auto-applying every suggestion, or LLM-driven merge/extraction). If "suggest unbreaking schema changes" is implemented as "model proposes; human or deterministic pipeline approves and applies," robustness and defensibility of the deterministic core are preserved. The paper's critique of "LLM-generated structure" applies when the model *drives* structure; it does not apply when the model only *suggests* schema evolution that is then applied deterministically.
-
----
-
-## 5. Recommendations
-
-1. **Cite arxiv 2602.02007** in architecture and positioning docs when explaining why agent memory retrieval should move beyond RAG and why structure and intact units matter.
-2. **Preserve intact-unit invariant:** Do not prune or compress inside an observation boundary; document this as alignment with the paper.
-3. **Position determinism explicitly:** In comparisons with xMemory or similar work, state that Neotoma achieves "beyond RAG" with deterministic, schema-first structure rather than LLM/embedding-based hierarchy.
-4. **If inference is used for schema suggestions:** Keep it advisory only; require deterministic application and approval; document that the critical path (ingest → extract → reduce → store → retrieve) remains deterministic.
-
----
-
-## 6. Related Documents
-
-- `docs/private/insights/xmemory_beyond_rag_agent_memory_paper_relevance_analysis.md` — Full paper relevance analysis
-- `docs/private/insights/dair_ai_beyond_rag_agent_memory_relevance_analysis.md` — DAIR.AI tweet/PDF relevance analysis
-- `docs/foundation/core_identity.md` — Neotoma scope and non-RAG stance
-- `docs/foundation/product_positioning.md` — Defensible differentiators

package/docs/reports/dhh_clankers_with_claws_neotoma_report.md

@@ -1,102 +0,0 @@
----
-title: "DHH 'Clankers with claws' — Neotoma Need and Positioning Report"
-status: "report"
-created_date: "2026-02-06"
-sources:
-  - "https://world.hey.com/dhh/clankers-with-claws-9f86fa71 (David Heinemeier Hansson, Feb 5 2026)"
-  - "docs/private/insights/dhh_clankers_with_claws_relevance_analysis.md"
----
-
-# DHH "Clankers with claws" — Neotoma Need and Positioning Report
-
-## Scope
-
-This report summarizes David Heinemeier Hansson's Hey World post "Clankers with claws," his general thesis that agents can navigate human interfaces and do not need agent-specific ones, and the critical application of that thesis to the need for Neotoma. It does not change code or schema.
-
-## Purpose
-
-Provide a single reference for: (1) what DHH claimed and showed, (2) whether his thesis undermines or supports the need for Neotoma, and (3) how Neotoma should position itself (memory vs. execution, structured cross-source view vs. human UIs).
-
-## Executive Summary
-
-| Question | Answer |
-|----------|--------|
-| Does DHH's thesis undermine the need for Neotoma? | **No.** His point applies to **execution** interfaces (MCPs/APIs for doing things). Neotoma is a **memory/truth** layer. |
-| Do agents need Neotoma if they can "navigate human ones"? | **Yes, for cross-source structured memory.** No single human UI shows "all my entities" or "timeline from every source"; Neotoma provides that view. |
-| Positioning implication? | **Sharpen:** Neotoma = canonical, structured memory for the user's data. Separate "memory MCP" from "execution MCP" in all messaging. |
-
----
-
-## 1. Source Summary
-
-**Source:** David Heinemeier Hansson, "Clankers with claws," Hey World, February 5, 2026.  
-**URL:** https://world.hey.com/dhh/clankers-with-claws-9f86fa71
-
-**What he did:** Ran an AI agent ("Kef") on OpenClaw with **zero accommodations**: no skills, no MCPs, no APIs. The agent signed up for HEY, then Fizzy, created a Fizzy board with five idea cards and web-sourced images, joined Basecamp via invite, and completed all steps without corrections. He used Claude Opus 4.5 (later re-ran with Kimi K2.5 via OpenRouter). He ran the agent on a Proxmox VM isolated from his personal data and logins.
-
-**What he argued:** Agent accommodations (MCPs, CLIs, APIs) are a "temporary crutch." The future is agents interacting via "human affordances" (web, email, standard UIs), like self-driving cars that eventually do not need special equipment (e.g. LIDAR). Human interfaces will be "more than adequate."
-
-**General claim:** Agents do not need agent-specific interfaces, because they can navigate human ones sufficiently.
-
----
-
-## 2. Critical Application: Does This Undermine Neotoma?
-
-### 2.1 Execution vs. memory
-
-- DHH's thesis targets **execution** tooling: browser automation, API wrappers, MCPs that let the agent *do* things (click, submit, call services). Neotoma's MCP is for **memory**: read/write structured records, entities, timelines, ingestion. So "agents don't need agent-specific interfaces" applies to execution; it does **not** apply to an interface for **structured, aggregated memory** that no single human app provides.
-
-### 2.2 No single human interface for cross-source structured truth
-
-- Human interfaces are **per-source**: Gmail, Drive, Notion, local files each have their own UI. There is **no** human interface that shows:
-  - "All my entities across all my data"
-  - "Timeline of events from every source"
-  - "Resolved canonical list of people/companies"
-- Those are **aggregated, normalized views** that only a dedicated layer (Neotoma) can provide. The agent's options are: (a) navigate every source each time and infer structure itself (costly, inconsistent, no single source of truth), or (b) read from Neotoma. Neotoma's "interface" is not a substitute for a human UI; it is a substitute for the agent re-crawling and re-deriving structure from dozens of human UIs every time.
-
-### 2.3 What DHH's experiment did not test
-
-- **One agent, one platform.** His demo did not test cross-platform or user-owned memory.
-- **Sandbox, not real data.** The agent created new accounts; it did not reason over DHH's existing personal data (invoices, contacts, documents). For "agent reasons over my existing fragmented data," a user-controlled layer (Neotoma) with a bounded interface (MCP) remains the safe, structured option.
-- **Structured use cases.** The demo was signup and collaboration. It did not address "reason over my tax documents," "who are all my vendors across my files," or "timeline of my contracts." For those, structured + queryable + provenance (Neotoma) matters.
-
-### 2.4 Where his view strengthens Neotoma
-
-- **Privacy:** He isolated the agent from his data. That aligns with "give the agent a bounded memory layer you control" (Neotoma) instead of full machine access.
-- **Multi-tool users:** His experiment was one agent. Users who use multiple agents or tools (Cursor, ChatGPT, Claude, OpenClaw) need one truth layer so "what I know" is not siloed per platform.
-- **Structured, auditable memory:** As agents do more, the need for "what does the user know?" in a structured, auditable form grows. Neotoma's deterministic, provenance-backed layer is where that lives.
-
----
-
-## 3. Summary Table: DHH Claim vs. Effect on Neotoma
-
-| DHH claim / observation | Effect on need for Neotoma | Condition |
-|-------------------------|----------------------------|-----------|
-| Agents can act without MCP/skills/APIs | Neutral / strengthens | Only if we frame Neotoma as memory layer, not execution. |
-| MCPs are a temporary crutch | Weakens if misread | Own the "memory MCP" narrative; separate from execution MCP. |
-| Human affordances will be adequate | Weakens for execution; neutral/strengthens for memory | Adequate for *doing* things. For *structured cross-source memory*, a dedicated layer (Neotoma) still required. |
-| Isolation and caution | Strengthens | Aligns with bounded, user-controlled memory (Neotoma). |
-| One agent, one platform | Does not test Neotoma | Neotoma's need is clearest for multi-platform and structured personal data. |
-
----
-
-## 4. Recommendations
-
-1. **Positioning:** Add "Truth Layer vs. execution layer" and "memory MCP vs. execution MCP" to `docs/foundation/product_positioning.md` (or equivalent). State explicitly that Neotoma's MCP is the interface to **cross-source structured memory**, not to execution.
-2. **Messaging:** Use one clear line in content and talks: "Agents that use human affordances (web, email) still need structured, user-controlled memory for cross-source truth. Neotoma is that layer."
-3. **Reference architecture:** Document "agent in isolated environment (e.g. VM) + Neotoma MCP for approved personal data" as a recommended, privacy-aligned pattern.
-4. **Monitor narrative:** If "we don't need MCP" spreads, consistently separate memory MCP (Neotoma) from execution MCP so Neotoma is not lumped with "unnecessary tooling."
-
----
-
-## 5. Conclusion
-
-DHH's general point—agents can navigate human interfaces and may not need agent-specific execution interfaces—**does not undermine** the need for Neotoma when that need is defined as: a user-owned, cross-platform, **structured truth layer** for personal data that agents (and humans) can query and update via a bounded interface. It would undermine Neotoma only if Neotoma were positioned as **execution** infrastructure or as the only way agents can remember anything. Applied critically, his thesis **sharpens** positioning: Neotoma is the **canonical, structured memory** for the user's personal and professional data, regardless of how agents execute. There is no single human interface for that; Neotoma's MCP is the interface to a view only a dedicated truth layer can provide.
-
----
-
-## Related Documents
-
-- `docs/private/insights/dhh_clankers_with_claws_relevance_analysis.md` — Full relevance analysis and critical application
-- `docs/foundation/core_identity.md`
-- `docs/foundation/product_positioning.md`