neotoma 0.3.0 → 0.3.2

package/docs/releases/v0.2.0/status.md

@@ -1,167 +0,0 @@
-# Release v0.2.0 — Status
-**Release**: Minimal Ingestion + Correction Loop  
-**Release Type**: Not Marketed  
-**Deployment**: Production (neotoma.io)  
-**Status**: `in_testing`  
-**Last Updated**: 2025-12-31
-
-## Summary
-v0.2.0 implements the minimal sources-first ingestion architecture with content-addressed storage, versioned interpretation, corrections, and entity merging. All 13 feature units completed and tested.
-
-## Phase Status
-| Phase                          | Status      | Progress |
-| ------------------------------ | ----------- | -------- |
-| Phase 1: Schema + Storage      | ✅ Complete | 6/6 FUs  |
-| Phase 2: MCP Tools + Services  | ✅ Complete | 7/7 FUs  |
-| Phase 3: Query Updates         | ✅ Complete | 1/1 FU   |
-
-## Feature Unit Status
-### Phase 1: Schema + Storage Foundation
-| Feature Unit                      | Status      | Notes                                    |
-| --------------------------------- | ----------- | ---------------------------------------- |
-| FU-110: Sources Table Migration   | ✅ Complete | Content-addressed storage with RLS       |
-| FU-111: Interpretation Runs Table | ✅ Complete | Versioned interpretation tracking        |
-| FU-113: Entity Extensions         | ✅ Complete | user_id + merge tracking                 |
-| FU-114: Observation Extensions    | ✅ Complete | source_id linkage                        |
-| FU-115: Raw Fragments Extensions  | ✅ Complete | Unknown field storage                    |
-| FU-116: Entity Merges Table       | ✅ Complete | Audit log                                |
-
-### Phase 2: MCP Tools + Services
-| Feature Unit                         | Status      | Notes                                |
-| ------------------------------------ | ----------- | ------------------------------------ |
-| FU-120: Raw Storage Service          | ✅ Complete | SHA-256, synchronous storage         |
-| FU-121: Interpretation Service       | ✅ Complete | Schema validation, entity resolution |
-| FU-122: MCP ingest() Tool            | ✅ Complete | Core ingestion                       |
-| FU-123: MCP ingest_structured() Tool | ✅ Complete | Pre-structured data                  |
-| FU-124: MCP reinterpret() Tool       | ✅ Complete | Re-interpretation                    |
-| FU-125: MCP correct() Tool           | ✅ Complete | Corrections                          |
-| FU-126: MCP merge_entities() Tool    | ✅ Complete | Entity deduplication                 |
-
-### Phase 3: Query Updates
-| Feature Unit                | Status      | Notes                       |
-| --------------------------- | ----------- | --------------------------- |
-| FU-134: Query Updates       | ✅ Complete | Provenance, merge exclusion |
-
-## Integration Tests Status
-| Test ID | Name                             | Status     |
-| ------- | -------------------------------- | ---------- |
-| IT-001  | Raw File Ingestion Flow          | ✅ Passing |
-| IT-002  | Content Deduplication            | ✅ Passing |
-| IT-003  | Reinterpretation Immutability    | ✅ Passing |
-| IT-004  | Correction Override              | ✅ Passing |
-| IT-005  | Entity Merge Flow                | ✅ Passing |
-| IT-006  | Cross-User Isolation             | ✅ Passing |
-| IT-007  | Interpretation Quota Enforcement | ✅ Passing |
-| IT-008  | Merged Entity Exclusion          | ✅ Passing |
-| IT-009  | Provenance Chain                 | ✅ Passing |
-| IT-010  | Entity Redirect on Merge         | ✅ Passing |
-
-**Summary:** 11/11 integration tests passing (100%)
-
-## Migrations Applied
-1. ✅ `20251231000001_add_sources_table.sql` - Sources table with RLS
-2. ✅ `20251231000002_extend_entities_table.sql` - Entity extensions (user_id, merged_to)
-3. ✅ `20251231000003_add_interpretations_table.sql` - Interpretations
-4. ✅ `20251231000004_extend_observations_table.sql` - Observation extensions
-5. ✅ `20251231000005_extend_raw_fragments_table.sql` - Raw fragments extensions
-6. ✅ `20251231000006_add_entity_merges_table.sql` - Entity merges audit log
-7. ✅ `20251231000007_update_entity_snapshots_rls.sql` - Entity snapshots RLS
-8. ✅ `20251231000008_make_source_payload_id_nullable.sql` - Make source_payload_id nullable
-
-## Implementation Summary
-### Services Created
-- `src/services/raw_storage.ts` - Content-addressed storage with SHA-256 deduplication
-- `src/services/interpretation.ts` - Schema validation, entity resolution, unknown field routing
-- `src/services/entity_queries.ts` - Query service with merged entity exclusion and provenance
-
-### MCP Tools Added
-- `ingest()` - Raw file ingestion with optional AI interpretation
-- `ingest_structured()` - Pre-structured data ingestion
-- `reinterpret()` - Re-run interpretation with new config
-- `correct()` - High-priority correction observations
-- `merge_entities()` - Manual entity deduplication
-
-### Key Features
-- ✅ Content-addressed storage (SHA-256 per user)
-- ✅ Versioned interpretation with config logging
-- ✅ Unknown field routing to raw_fragments
-- ✅ Correction observations with priority 1000
-- ✅ Entity merge with observation rewriting
-- ✅ Cross-user data isolation (RLS)
-- ✅ Merged entity exclusion from queries
-- ✅ Complete provenance chain (source → interpretation_run → observation)
-- ✅ Interpretation quota tracking (soft limit)
-
-## Acceptance Criteria
-### Product
-- ✅ Raw files can be ingested and content-addressed
-- ✅ Interpretation runs are versioned and auditable
-- ✅ Users can correct AI-extracted fields
-- ✅ Duplicate entities can be merged manually
-- ✅ All data is user-isolated
-
-### Technical
-- ✅ All new tables created with RLS policies
-- ✅ Sources table with (user_id, content_hash) uniqueness
-- ✅ Entities extended with user_id and merge tracking
-- ✅ MCP tools functional (ingest, ingest_structured, reinterpret, correct, merge_entities)
-- ✅ Unit and integration tests passing
-- ✅ NO background workers in v0.2.0 (as planned)
-
-### Business
-- ✅ Core ingestion + correction loop validated with tests
-- ✅ User data isolation satisfies privacy requirements
-- ✅ Interpretation auditability enables trust and debugging
-
-## Success Criteria
-1. ✅ All migrations applied successfully
-2. ✅ RLS policies enforced on all user tables
-3. ✅ All MCP tools functional (synchronous operations only)
-4. ✅ Cross-user isolation verified
-5. ✅ All tests passing (11/11 - 100%)
-6. ✅ Validation goal: "Users/agents can ingest, re-interpret, correct, and merge on real data"
-
-## Decision Log
-| Date       | Decision                                  | Rationale                                                                                                   |
-| ---------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
-| 2024-12-18 | Release created as v0.2.0                 | Foundational ingestion infrastructure should precede Chat Transcript CLI (shifted to post-hardening v0.4.0) |
-| 2024-12-18 | Entity IDs remain TEXT                    | Consistency with baseline schema; avoids migration complexity                                               |
-| 2024-12-18 | Greenfield implementation                 | Pre-release, no existing user data to migrate                                                               |
-| 2025-12-31 | Made source_payload_id nullable           | v0.2.0 uses source_id instead of payload_submissions                                                        |
-| 2025-12-31 | Implementation completed in single batch  | All 13 FUs implemented and tested in one development session                                                |
-
-## Deployment Checklist
-
-### Pre-Deployment Validation
-- ✅ **Pre-release checklist completed** (see `docs/developer/pre_release_checklist.md`)
-  - ✅ TypeScript compilation passes
-  - ✅ Linting passes
-  - ✅ All migrations created for schema changes
-  - ✅ Migrations applied successfully
-  - ✅ Schema advisor checks pass
-  - ✅ MCP server starts without errors
-  - ✅ MCP configuration validated
-
-### Test Validation
-- ✅ Database migrations applied
-- ✅ All integration tests passing (11/11 integration tests)
-- ✅ Schema registry seeded with base types
-- ❌ **Manual test execution pending** (BLOCKER - required before `ready_for_deployment`)
-
-### Deployment Steps
-- ⏳ **Manual validation via Cursor/ChatGPT MCP integration** (REQUIRED - blocks deployment)
-- ⏳ Generate release report with Section 9 (Testing Guidance) containing manual test cases
-- ⏳ Execute all manual test cases and document results
-- ⏳ Deploy to production (neotoma.io) (only after manual tests pass)
-- ⏳ Mark status as `deployed`
-
-**Note:** Per `foundation/development/release_workflow.md`, release is now in `in_testing` status. Build complete, automated tests passed. Status will be updated to `ready_for_deployment` only after all manual test cases are executed and pass.
-
-## Notes
-- Pre-MVP release (not marketed)
-- Foundational infrastructure for v0.3.0 (Operational Hardening), v0.4.0 (Intelligence + Housekeeping), and v1.0.0 (MVP)
-- AI agent execution assumed for all development
-- Human review at checkpoints only
-- Architecture plan: `docs/architecture/sources_first_ingestion_final.md`
-- **Development Time**: ~4 hours (single session, 2025-12-31)
-- **Velocity**: 13 FUs in 4 hours = 3.25 FUs/hour

package/docs/releases/v0.2.1/acceptance_criteria.md

@@ -1,200 +0,0 @@
-# Release v0.2.1 — Acceptance Criteria
-**Release ID**: v0.2.1  
-**Release Name**: Documentation & Support System  
-**Release Type**: Not Marketed  
-**Deployment**: Production (neotoma.io)
-## Product Acceptance Criteria
-### P-001: One-Page Landing Website
-**Criterion:** One-page landing website accessible at root URL presenting key information from README.md with directory access to documentation and MCP setup instructions.
-**Validation:**
-- Landing page displays:
-  - What Neotoma is and problems it solves
-  - Target users and use cases
-  - Core workflow and architecture overview
-  - Key features and capabilities
-  - Directory access to documentation organized by category
-  - MCP setup instructions with step-by-step guide
-  - Quick links to essential documentation
-- Landing page follows design system (`docs/ui/design_system.md`):
-  - Colors: Neutral grays, blue accents (#0066CC primary)
-  - Typography: Inter for UI, JetBrains Mono for code/data
-  - Spacing: 4px base unit scale
-  - Components: shadcn/ui components where applicable
-  - Dark mode support
-  - Accessibility: WCAG AA compliance (keyboard nav, ARIA, contrast)
-  - Brand alignment: Minimal, technical, trustworthy, deterministic
-- Landing page is responsive and visually clear
-- Directory browsing functional
-- MCP setup instructions accessible and actionable
-**Test:** `tests/integration/landing_page.test.ts`
-**Status:** ⏳ pending
-### P-002: Comprehensive Documentation Available
-**Criterion:** Comprehensive documentation available via web interface covering all major topics.
-**Validation:**
-- Documentation covers:
-  - Architecture overview and system design
-  - API reference (all MCP actions, request/response schemas)
-  - Feature unit specifications
-  - Developer guides (getting started, workflows, testing)
-  - Integration guides (MCP setup, provider integrations)
-  - Troubleshooting and FAQ
-- Documentation accessible via web interface
-- Documentation searchable and navigable
-**Test:** `tests/integration/documentation_web.test.ts`
-**Status:** ⏳ pending
-### P-003: Support Agent Accuracy
-**Criterion:** Support agent responds accurately to common inquiries (80%+ accuracy).
-**Validation:**
-- Test query set prepared (20+ queries covering common topics)
-- Support agent accuracy >= 80% on test queries
-- Responses include citations to source documentation
-- Responses helpful and actionable
-**Test:** `tests/integration/support_agent_accuracy.test.ts`
-**Metric:** `support_agent_accuracy >= 0.80`
-**Status:** ⏳ pending
-### P-004: Documentation Search Functional
-**Criterion:** Documentation search returns relevant results quickly.
-**Validation:**
-- Search queries return relevant results
-- Search response time < 500ms (p95)
-- Results ranked by relevance
-- Matching terms highlighted
-**Test:** `tests/integration/documentation_search.test.ts`
-**Status:** ⏳ pending
-### P-005: MCP Support Endpoint Operational
-**Criterion:** MCP support endpoint operational and accessible via MCP clients.
-**Validation:**
-- MCP `query_support` action accessible via MCP clients
-- Request/response schema validated
-- Error handling functional
-- Integration with support agent working
-**Test:** `tests/integration/mcp_support_endpoint.test.ts`
-**Status:** ⏳ pending
-## Technical Acceptance Criteria
-### T-000: Design System Compliance (All UI Work)
-**Criterion:** All UI-related work MUST follow the design system (`docs/ui/design_system.md`).
-**Validation:**
-- All UI components use design system colors (neutral grays, blue accents #0066CC)
-- Typography follows design system (Inter for UI, JetBrains Mono for code/data)
-- Spacing uses 4px base unit scale
-- Components use shadcn/ui where applicable, aligned with design system
-- Dark mode support functional
-- Accessibility: WCAG AA compliance verified (keyboard navigation, ARIA labels, contrast)
-- Brand alignment: Minimal, technical, trustworthy, deterministic aesthetic
-**Test:** `tests/integration/design_system_compliance.test.ts`
-**Status:** ⏳ pending
-**Note:** This criterion applies to ALL UI work in this release, not just the landing page.
-### T-001: All P0 Feature Units Complete
-**Criterion:** All P0 Feature Units deployed and passing tests.
-**Validation:**
-- FU-300: Documentation Generation — Complete
-- FU-301: Documentation Web Server — Complete
-- FU-302: MCP Support Endpoint — Complete
-- FU-303: Support Agent System — Complete
-- All unit tests passing
-- All integration tests passing
-**Metric:** `SELECT COUNT(*) FROM feature_units WHERE priority='P0' AND status='completed' = 4`
-**Status:** ⏳ pending
-### T-002: Documentation Generation Deterministic
-**Criterion:** Documentation generation deterministic (same repo state → same docs).
-**Validation:**
-- Same repository state produces identical documentation
-- File hashes match across multiple runs
-- No non-deterministic content (timestamps, random IDs)
-**Test:** `tests/integration/documentation_determinism.test.ts`
-**Status:** ⏳ pending
-### T-003: Static Documentation Performance
-**Criterion:** Static documentation served efficiently (< 200ms page load).
-**Validation:**
-- Page load time p95 < 200ms
-- Static file serving optimized
-- Caching headers configured appropriately
-**Test:** `tests/integration/documentation_performance.test.ts`
-**Metric:** `page_load_p95 < 200`
-**Status:** ⏳ pending
-### T-004: Documentation Search Performance
-**Criterion:** Documentation search responsive (< 500ms query response).
-**Validation:**
-- Search query response time p95 < 500ms
-- Search index optimized
-- Query processing efficient
-**Test:** `tests/integration/documentation_search.test.ts`
-**Metric:** `search_query_p95 < 500`
-**Status:** ⏳ pending
-### T-005: Support Agent Citations
-**Criterion:** Support agent responses cite source documentation.
-**Validation:**
-- All responses include citations
-- Citations reference correct documentation sections
-- Citation format consistent
-**Test:** `tests/integration/support_agent.test.ts`
-**Metric:** `citation_rate >= 1.0`
-**Status:** ⏳ pending
-### T-006: MCP Protocol Compliance
-**Criterion:** MCP endpoint follows MCP protocol specification.
-**Validation:**
-- Request validation functional
-- Response format compliant with MCP spec
-- Error responses use ErrorEnvelope format
-- Protocol compliance verified
-**Test:** `tests/integration/mcp_support_endpoint.test.ts`
-**Status:** ⏳ pending
-### T-007: Build Process Integration
-**Criterion:** Documentation generation integrated into build/release process.
-**Validation:**
-- Documentation generation runs automatically on build
-- Build fails if documentation generation fails
-- Generated documentation committed or deployed appropriately
-**Metric:** `Documentation generation runs automatically on build`
-**Status:** ⏳ pending
-### T-008: Production Deployment
-**Criterion:** Release deployed to neotoma.io with all components operational.
-**Validation:**
-- Landing page accessible at neotoma.io root URL
-- Documentation accessible at neotoma.io/docs
-- MCP support endpoint operational and accessible
-- All links and navigation functional
-- Performance metrics within targets (< 200ms page load, < 500ms search)
-- SSL/TLS configured correctly
-- Domain DNS configured correctly
-**Metric:** `Deployment successful to neotoma.io`
-**Status:** ⏳ pending
-## Business Acceptance Criteria
-### B-001: User Self-Service via Documentation
-**Criterion:** Users can self-serve answers via documentation web interface.
-**Validation:**
-- Users can find answers to common questions in documentation
-- Documentation clear and actionable
-- Web interface usable and intuitive
-**Metric:** `Manual validation via web interface`
-**Status:** ⏳ pending
-### B-002: MCP Support Access
-**Criterion:** Users can query support via MCP without leaving their AI tool.
-**Validation:**
-- Users can query support via MCP from Cursor/ChatGPT
-- Responses helpful and accurate
-- Citations enable further self-service
-**Metric:** `Manual validation via MCP client`
-**Status:** ⏳ pending
-## Acceptance Criteria Summary
-| Category  | Criteria                           | Status     |
-| --------- | ---------------------------------- | ---------- |
-| Product   | P-001: One-Page Landing Website    | ⏳ pending |
-| Product   | P-002: Comprehensive Documentation | ⏳ pending |
-| Product   | P-003: Support Agent Accuracy      | ⏳ pending |
-| Product   | P-004: Documentation Search        | ⏳ pending |
-| Product   | P-005: MCP Support Endpoint        | ⏳ pending |
-| Technical | T-000: Design System Compliance    | ⏳ pending |
-| Technical | T-001: All P0 FUs Complete         | ⏳ pending |
-| Technical | T-002: Documentation Determinism   | ⏳ pending |
-| Technical | T-003: Documentation Performance   | ⏳ pending |
-| Technical | T-004: Search Performance          | ⏳ pending |
-| Technical | T-005: Support Agent Citations     | ⏳ pending |
-| Technical | T-006: MCP Protocol Compliance     | ⏳ pending |
-| Technical | T-007: Build Process Integration   | ⏳ pending |
-| Technical | T-008: Production Deployment       | ⏳ pending |
-| Business  | B-001: User Self-Service           | ⏳ pending |
-| Business  | B-002: MCP Support Access          | ⏳ pending |
-**Total:** 16 criteria  
-**Passed:** 0  
-**Pending:** 16

package/docs/releases/v0.2.1/execution_schedule.md

@@ -1,114 +0,0 @@
-# Release v0.2.1 — Execution Schedule
-**Release ID**: v0.2.1  
-**Release Name**: Documentation & Support System  
-**Release Type**: Not Marketed  
-**Deployment**: Production (neotoma.io)  
-**Generated**: 2025-12-10
-## Execution Strategy
-- **Type**: Multi-agent parallel execution
-- **Max Parallel FUs**: 2
-- **Max High-Risk FUs in Parallel**: 1
-## Batch Execution Plan
-#### Batch 0: Documentation Generation
-**Feature Units:**
-- `FU-300`: AI-Powered Documentation Analysis & Generation
-**Dependencies:**
-- None (can start immediately)
-**Description:**
-AI-powered repository analysis and comprehensive documentation generation. Analyzes codebase structure, architecture, APIs, and workflows to generate static documentation in markdown/HTML format.
-**Acceptance Criteria:**
-- Repository analysis completes successfully
-- Comprehensive documentation generated covering all major topics
-- Documentation output deterministic (same repo state → same docs)
-- Documentation includes API references, architecture diagrams, guides
-**Estimated Duration**: 8-12 hours
-#### Batch 1: Documentation Web Server
-**Feature Units:**
-- `FU-301`: Static Documentation Web Server
-**Dependencies:**
-- `FU-300` (Documentation Generation)
-**Description:**
-One-page landing website presenting key information from README.md with directory access to documentation and MCP setup instructions, plus static file serving for generated documentation with search functionality, navigation, and cross-linking.
-**Acceptance Criteria:**
-- One-page landing website accessible at root URL
-- Landing page displays key README information clearly
-- Design system compliance (colors, typography, spacing, components per `docs/ui/design_system.md`)
-- Accessibility requirements met (WCAG AA, keyboard navigation, ARIA labels)
-- Dark mode support functional
-- Directory access to documentation functional
-- MCP setup instructions available and clear
-- Static documentation served via web server
-- Page load performance < 200ms (p95)
-- Search functionality operational
-- Cross-links functional
-- Navigation usable
-**Estimated Duration**: 6-8 hours (includes landing page development with design system implementation)
-#### Batch 2: MCP Support Inquiry Endpoint
-**Feature Units:**
-- `FU-302`: MCP Support Inquiry Endpoint
-**Dependencies:**
-- `FU-200` (MCP Server Core from v0.1.0)
-**Description:**
-New MCP action `query_support` for support inquiries. Implements request/response schema, error handling, and validation following MCP protocol specification.
-**Acceptance Criteria:**
-- MCP `query_support` action implemented
-- Request/response schema validated
-- Error handling functional
-- MCP protocol compliance verified
-- Integration tests passing
-**Estimated Duration**: 4-6 hours
-#### Batch 3: Support Agent System
-**Feature Units:**
-- `FU-303`: Support Agent System
-**Dependencies:**
-- `FU-300` (Documentation Generation)
-- `FU-301` (Documentation Web Server)
-- `FU-302` (MCP Support Endpoint)
-**Description:**
-RAG-based support agent using static documentation as knowledge base. Implements query understanding, context retrieval, response generation with citations, and integration with MCP endpoint.
-**Acceptance Criteria:**
-- Support agent operational
-- Response accuracy >= 80% on test queries
-- Responses include citations to source documentation
-- Context retrieval functional
-- Integration with MCP endpoint working
-**Estimated Duration**: 8-12 hours
-## Checkpoint Schedule
-### Checkpoint 1 — Documentation Review
-**Trigger**: After Batch 1 completion
-**Review Items:**
-- Documentation completeness (all major topics covered)
-- Documentation accuracy (validated against source code)
-- Web interface usability
-- Search functionality quality
-**Gate**: Proceed to Batch 2 only if documentation quality meets standards.
-### Checkpoint 2 — Pre-Release Sign-Off
-**Trigger**: After Batch 3 completion
-**Review Items:**
-- All acceptance criteria met
-- Support agent accuracy validated
-- Integration tests passing
-- Performance metrics within targets
-- Production deployment plan reviewed (neotoma.io)
-**Gate**: Release ready for production deployment to neotoma.io if all criteria met. (No marketing activities required for not marketed release.)
-## Parallelization Opportunities
-- **Batch 0** can run independently (no dependencies)
-- **Batch 1** must wait for Batch 0
-- **Batch 2** can run in parallel with Batch 1 (depends on v0.1.0 FU-200, not Batch 1)
-- **Batch 3** must wait for Batches 1 and 2
-**Optimized Schedule:**
-- Start: Batch 0
-- After Batch 0: Start Batches 1 and 2 in parallel
-- After Batches 1 and 2: Start Batch 3
-## Risk Assessment
-**High-Risk FUs:**
-- None (all FUs are low or medium risk)
-**Medium-Risk FUs:**
-- `FU-300`: AI-powered analysis complexity
-- `FU-303`: Support agent accuracy requirements
-**Low-Risk FUs:**
-- `FU-301`: Standard static file serving
-- `FU-302`: Standard MCP action implementation
-## Estimated Total Duration
-- **Sequential**: 24-36 hours
-- **Optimized Parallel**: 16-24 hours (Batches 1 and 2 run in parallel)