neotoma 0.3.0 → 0.3.2

package/docs/releases/v0.2.1/release_plan.md

@@ -1,211 +0,0 @@
-## Release v0.2.1 — Documentation Generation System
-### 1. Release Overview
-- **Release ID**: `v0.2.1`
-- **Name**: Documentation Generation System
-- **Release Type**: Not Marketed (production deployment without marketing activities)
-- **Goal**: Deliver comprehensive, statically-generated documentation derived from AI analysis of the repository, served for web viewing with one-page landing site. Creates the documentation foundation that enables future support systems and user onboarding.
-- **Priority**: P0 (critical for user enablement)
-- **Target Ship Date**: When ready (post v0.2.0 validation)
-- **Marketing Required**: No (not marketed release)
-#### 1.1 Canonical Specs (Authoritative Sources)
-- **Manifest**: `docs/NEOTOMA_MANIFEST.md`
-- **MVP Overview**: `docs/specs/MVP_OVERVIEW.md`
-- **MCP Specification**: `docs/specs/MCP_SPEC.md`
-- **Documentation Standards**: `docs/conventions/documentation_standards.md`
-- **Design System**: `docs/ui/design_system.md`
-This release plan coordinates the documentation and support system scope into a concrete release plan.
-**Release Classification:**
-- **All releases deploy to production** at neotoma.io
-- **Release types**: "Marketed" (with marketing activities) vs "Not Marketed" (silent deployment)
-- **This release**: Not Marketed (deploys to production without marketing activities)
-### 2. Scope
-#### 2.1 Included Feature Units
-**Documentation Generation System:**
-- `FU-300`: AI-Powered Documentation Analysis & Generation
-  - Repository analysis via AI (codebase structure, architecture, APIs, workflows)
-  - Comprehensive documentation generation (API references, architecture diagrams, guides)
-  - Static markdown/HTML output for web viewing
-  - Documentation indexing and cross-referencing
-  
-- `FU-301`: Static Documentation Web Server
-  - One-page landing website presenting key information from README.md
-  - Design system compliance (follows `docs/ui/design_system.md`)
-  - Directory access to documentation (organized by category)
-  - MCP setup instructions and quick start guide
-  - Static file serving (markdown/HTML documentation)
-  - Search functionality across documentation
-  - Navigation and cross-linking
-  - Versioned documentation support
-
-#### 2.1.1 Deferred to v0.9.0
-**MCP Support System (moved to pre-v1.0.0 release):**
-- `FU-302`: MCP Support Inquiry Endpoint → Moved to v0.9.0
-- `FU-303`: Support Agent System → Moved to v0.9.0
-
-**Rationale:** Documentation foundation is critical early; interactive support system better suited for pre-MVP launch when user base exists.
-#### 2.2 Explicitly Excluded
-- Dynamic documentation generation (all docs pre-generated)
-- Real-time code analysis (analysis runs during build/release)
-- Multi-language documentation (English only for MVP)
-- User-contributed documentation (official docs only)
-- Chat UI for support (MCP-only access)
-#### 2.3 Design System Requirements
-**All UI-related work MUST follow the design system:**
-- **Design System Reference**: `docs/ui/design_system.md`
-- **Required Compliance**: All UI components, pages, and interactions must adhere to design system specifications
-- **Verification**: Design system compliance must be verified in acceptance criteria and integration tests
-- **Components**: Use shadcn/ui components where applicable, following design system guidelines
-- **Accessibility**: WCAG AA compliance required (keyboard navigation, ARIA labels, contrast ratios)
-- **Brand Alignment**: Minimal, technical, trustworthy, deterministic aesthetic
-**Design System Elements:**
-- Colors: Neutral grays with blue accents (#0066CC primary), dark mode support
-- Typography: Inter for UI text, JetBrains Mono for code/data
-- Spacing: 4px base unit scale (8px, 16px, 24px, etc.)
-- Components: shadcn/ui components aligned with design system
-- Dark Mode: Full support required
-- Responsive: Desktop-first, mobile adaptations as needed
-### 3. Release-Level Acceptance Criteria
-#### 3.1 Product
-- One-page landing website accessible at root URL presenting:
-  - Key information from README.md (what Neotoma is, problems solved, who it's for)
-  - Core workflow and architecture overview
-  - Key features and capabilities
-  - Directory access to documentation organized by category
-  - MCP setup instructions with step-by-step guide
-  - Quick links to essential documentation
-  - Design system compliance (colors, typography, spacing, components per `docs/ui/design_system.md`)
-- Comprehensive documentation available via web interface covering:
-  - Architecture overview and system design
-  - API reference (all MCP actions, request/response schemas)
-  - Feature unit specifications
-  - Developer guides (getting started, workflows, testing)
-  - Integration guides (MCP setup, provider integrations)
-  - Troubleshooting and FAQ
-- Documentation search functional (find relevant docs by keyword/topic)
-#### 3.2 Technical
-- Documentation generation deterministic (same repo state → same docs)
-- Static documentation served efficiently (< 200ms page load)
-- Documentation index searchable (< 500ms query response)
-- All documentation validated for accuracy and completeness
-- Documentation generation runs as part of release/build process
-- Static files stored in versioned directory structure
-- **All UI components follow design system** (`docs/ui/design_system.md`):
-  - Colors, typography, spacing per design system specifications
-  - shadcn/ui components used where applicable
-  - Dark mode support functional
-  - Accessibility: WCAG AA compliance (keyboard navigation, ARIA labels, contrast)
-  - Brand alignment: Minimal, technical, trustworthy, deterministic
-#### 3.3 Business
-- Users can self-serve answers via documentation web interface
-- Documentation serves as single source of truth for all Neotoma information
-- Documentation foundation enables future support systems (v0.9.0)
-### 4. Cross-FU Integration Scenarios
-These scenarios must pass end-to-end before v0.2.1 is approved:
-
-1. **Documentation Generation → Web Viewing**
-   - Run documentation generation process
-   - Verify comprehensive docs generated (architecture, APIs, guides)
-   - Access landing page at root URL
-   - Verify landing page displays key README information
-   - Verify directory access to documentation works
-   - Verify MCP setup instructions are accessible and clear
-   - Access docs via web server
-   - Verify search functionality works
-   - Verify cross-links functional
-
-2. **Documentation Update → Regeneration**
-   - Update source documentation or codebase
-   - Regenerate static documentation
-   - Verify updated content reflected in generated docs
-   - Verify search index updated
-   - Verify cross-references maintained
-
-The detailed test specifications for these flows live in `docs/releases/v0.2.1/integration_tests.md`.
-### 5. Deployment and Rollout Strategy
-- **Deployment Target**: Production (neotoma.io)
-  - All releases deploy to production at neotoma.io
-  - Deploy documentation web server to neotoma.io
-  - Deploy MCP support endpoint alongside existing MCP server
-  - Landing page accessible at neotoma.io root URL
-  - Documentation accessible at neotoma.io/docs
-  - MCP support endpoint accessible via MCP protocol
-- **Marketing Strategy**: Not Marketed
-  - No pre-launch marketing activities
-  - No post-launch marketing activities
-  - No user acquisition campaigns
-  - No announcement or promotion
-  - Release deployed silently to production
-- **Rollback Plan**:
-  - Revert code changes and redeploy to neotoma.io
-  - Restore previous documentation version if needed
-  - Disable MCP support endpoint if critical issues arise
-### 6. Post-Release Validation
-- Validate documentation completeness:
-  - All major topics covered
-  - All MCP actions documented
-  - All feature units have documentation
-  - Cross-references functional
-- Validate documentation quality:
-  - Verify accuracy of generated content
-  - Verify completeness of coverage
-  - Test documentation search accuracy
-- Validate web interface:
-  - Page load performance
-  - Search functionality
-  - Navigation usability
-  - Mobile responsiveness (basic)
-- Validate production deployment:
-  - Landing page accessible at neotoma.io
-  - Documentation accessible at neotoma.io/docs
-  - All links and navigation functional
-  - Performance metrics within targets
-### 7. Success Criteria
-**Release is Complete When:**
-1. ✅ Comprehensive documentation generated covering all major topics
-2. ✅ One-page landing website operational with key README information
-3. ✅ Directory access to documentation functional
-4. ✅ MCP setup instructions available and clear
-5. ✅ Documentation web server operational and accessible
-6. ✅ Documentation search functional
-7. ✅ Documentation generation integrated into build/release process
-8. ✅ Static documentation versioned and stored appropriately
-9. ✅ **Deployed to neotoma.io** (landing page at root, docs at /docs)
-10. ✅ All acceptance criteria met
-### 8. Status
-- **Current Status**: `planning`
-- **Release Type**: Not Marketed
-- **Deployment**: Production (neotoma.io)
-- **Owner**: Mark Hendrickson
-- **Notes**:
-  - Builds on v0.2.0 ingestion infrastructure
-  - Documentation generation uses AI analysis of repository
-  - All documentation statically generated (no runtime generation)
-  - **All UI work MUST follow design system** (`docs/ui/design_system.md`) - colors, typography, spacing, components, accessibility, dark mode
-  - **All releases deploy to production** at neotoma.io, regardless of marketing status
-  - **Support system (FU-302, FU-303) moved to v0.9.0** - enables better timing for user-facing support before v1.0.0
-### 9. Related Documentation
-- `integration_tests.md` — Cross-FU integration test specifications
-- `execution_schedule.md` — Detailed batch execution plan
-- `manifest.yaml` — Feature Unit manifest and dependencies
-- `acceptance_criteria.md` — Detailed acceptance criteria
-### 10. Feature Unit Specifications
-**New Feature Units:**
-- **FU-300**: AI-Powered Documentation Analysis & Generation
-- **FU-301**: Static Documentation Web Server
-
-**Dependencies:**
-- FU-300: No dependencies (can run independently)
-- FU-301: Depends on FU-300 (needs generated docs)
-
-**Moved to v0.9.0:**
-- **FU-302**: MCP Support Inquiry Endpoint
-- **FU-303**: Support Agent System
-
-### 11. Release Spacing Context
-
-| Release | Focus | Status |
-|---------|-------|--------|
-| v0.2.0 | Minimal ingestion + correction loop | Prerequisite |
-| **v0.2.1** | **Documentation generation system** | **This release** |
-| v0.9.0 | MCP support system | Uses v0.2.1 docs |
-| v1.0.0 | MVP (public release) | Next major |

package/docs/releases/v0.2.1/status.md

@@ -1,106 +0,0 @@
-# Release v0.2.1 — Status
-**Release ID**: v0.2.1  
-**Release Name**: Documentation Generation System  
-**Status**: `in_progress`  
-**Owner**: Mark Hendrickson  
-**Last Updated**: 2025-12-31
-## 1. Release Metadata
-- **Release ID**: `v0.2.1`
-- **Name**: Documentation Generation System
-- **Status**: `planning`
-- **Release Type**: Not Marketed
-- **Deployment**: Production (neotoma.io)
-- **Priority**: P0
-- **Target Date**: When ready (post v0.2.0 validation)
-- **Marketing**: No (not marketed release)
-- **Scope Change**: Support system features (FU-302, FU-303) moved to v0.9.0
-
-## 2. Batch Progress
-| Batch ID | Feature Units | Status     | Completion |
-| -------- | ------------- | ---------- | ---------- |
-| 0        | FU-300        | ✅ complete | 100%       |
-| 1        | FU-301        | ✅ complete | 100%       |
-
-**Summary:**
-- ✅ **Complete:** 2 batch(es)
-- ⚠️ **Partial:** 0 batch(es)
-- ❌ **Incomplete:** 0 batch(es)
-
-## 3. Feature Unit Status
-| FU ID  | Name                                           | Status     | Notes |
-| ------ | ---------------------------------------------- | ---------- | ----- |
-| FU-300 | AI-Powered Documentation Analysis & Generation | ✅ complete | Documentation generator implemented |
-| FU-301 | Static Documentation Web Server                | ✅ complete | Landing page and routes implemented |
-
-**Moved to v0.9.0:**
-| FU ID  | Name                         | Status     | Notes                  |
-| ------ | ---------------------------- | ---------- | ---------------------- |
-| FU-302 | MCP Support Inquiry Endpoint | → v0.9.0   | Pre-v1.0.0 positioning |
-| FU-303 | Support Agent System         | → v0.9.0   | Pre-v1.0.0 positioning |
-
-**Summary:**
-- ✅ **Complete:** 2
-- ⚠️ **Partial:** 0
-- ❌ **Not Started:** 0
-## 4. Checkpoints
-- **Checkpoint 1 — Documentation Review**: `pending`
-  - Trigger: After Batch 1 completion
-  - Review documentation completeness and quality
-**Completion:** 0/1 checkpoints completed
-
-**Implementation Notes:**
-- FU-300: Documentation generator service created at `src/services/documentation_generator.ts`
-  - Repository analysis engine implemented
-  - API reference generation from MCP actions
-  - Architecture documentation generation
-  - Developer and integration guides copying
-  - Search index generation
-  - Cross-reference map generation
-  - Run with: `npm run docs:generate`
-  
-- FU-301: Documentation web server routes created at `src/routes/documentation.ts`
-  - Landing page at root URL with README content
-  - Static documentation file serving
-  - Search functionality endpoint
-  - Markdown rendering
-  - Integrated into Express app
-## 5. Integration Test Status
-| Test ID | Name                                                | Status     |
-| ------- | --------------------------------------------------- | ---------- |
-| IT-001  | Documentation Generation → Web Viewing Flow         | ✅ passed  |
-| IT-002  | Support Query → Agent Response Flow                 | ⏸️ deferred (v0.9.0) |
-| IT-003  | Documentation Update → Support Agent Knowledge Flow | ⏸️ deferred (v0.9.0) |
-| IT-004  | Multi-Topic Support Query Flow                      | ⏸️ deferred (v0.9.0) |
-| IT-005  | Documentation Search Functionality                  | ✅ passed  |
-| IT-006  | Support Agent Accuracy Validation                   | ⏸️ deferred (v0.9.0) |
-| IT-007  | MCP Endpoint Protocol Compliance                    | ⏸️ deferred (v0.9.0) |
-| IT-008  | Documentation Generation Determinism                | ✅ passed  |
-| IT-009  | Landing Page Functionality                          | ✅ passed  |
-**Summary:** 4/4 applicable tests passed (5 tests deferred to v0.9.0)
-## 6. Acceptance Criteria Status
-**Product Criteria:** 0/5 met  
-**Technical Criteria:** 0/9 met (includes T-000: Design System Compliance, T-008: Production Deployment)  
-**Business Criteria:** 0/2 met
-**Total:** 0/16 criteria met
-See `acceptance_criteria.md` for detailed criteria.
-## 7. Decision Log
-| Date       | Decision                    | Rationale                                                                                    |
-| ---------- | --------------------------- | -------------------------------------------------------------------------------------------- |
-| 2025-12-10 | Created v0.2.1 release plan | Add comprehensive documentation and support system to enable users and reduce support burden |
-## 8. Risks and Blockers
-**Current Risks:**
-- None identified yet (planning phase)
-**Current Blockers:**
-- None identified yet (planning phase)
-## 9. Next Steps
-1. **Feature Unit Specification:** Create detailed specifications for FU-300, FU-301, FU-302, FU-303
-2. **Implementation Planning:** Break down each FU into implementation tasks
-3. **Dependency Validation:** Verify v0.1.0 FU-200 (MCP Server Core) is complete
-4. **Test Planning:** Expand integration test specifications with detailed test cases
-5. **Documentation Standards:** Define documentation generation standards and templates
-## 10. Related Documents
-- **Release Plan:** `docs/releases/v0.2.1/release_plan.md`
-- **Manifest:** `docs/releases/v0.2.1/manifest.yaml`
-- **Execution Schedule:** `docs/releases/v0.2.1/execution_schedule.md`
-- **Integration Tests:** `docs/releases/v0.2.1/integration_tests.md`
-- **Acceptance Criteria:** `docs/releases/v0.2.1/acceptance_criteria.md`

package/docs/releases/v0.2.15/README.md

@@ -1,79 +0,0 @@
-# Neotoma v0.2.15 - Complete Architecture Migration
-
-**Authoritative Vocabulary:** [`docs/vocabulary/canonical_terms.md`](../../vocabulary/canonical_terms.md)
-
-**Status:** ✅ Implemented (Pending Database Migrations)  
-**Breaking Changes:** Yes  
-**Migration Required:** Yes
-
-## Overview
-
-Version 0.2.15 completes Neotoma's architectural migration to a unified [source](../../vocabulary/canonical_terms.md#source)-based truth model. This release:
-
-- Unifies all [ingestion](../../vocabulary/canonical_terms.md#ingestion) into a single `ingest` MCP action
-- Eliminates the capability concept (rules moved to [entity schemas](../../vocabulary/canonical_terms.md#entity-schema))
-- Removes the legacy records table
-- Aligns all documentation to [`docs/vocabulary/canonical_terms.md`](../../vocabulary/canonical_terms.md)
-
-See `release_plan.md` for complete details.
-
-## Quick Links
-
-- **Release Plan:** [release_plan.md](./release_plan.md)
-- **Canonical Vocabulary:** [`docs/vocabulary/canonical_terms.md`](../../vocabulary/canonical_terms.md)
-- **Migration Script:** `../../scripts/migrate-records-to-sources-v0.2.15.ts`
-- **SQL Migrations:** `./migrations/`
-
-## Breaking Changes Summary
-
-### Removed APIs
-- HTTP: `/store_record`, `/update_record`, `/retrieve_records`, `/delete_record`
-- MCP: `update_record`, `retrieve_records`, `delete_record`
-
-### Deprecated APIs (Removed in v0.2.16)
-- MCP: `submit_payload`, `ingest_structured` (merged into unified `ingest`)
-
-### New APIs
-- HTTP: `/entities/query`, `/observations/create`, `/entities/merge`
-- MCP: Unified `ingest` (for all [source](../../vocabulary/canonical_terms.md#source))
-
-### Removed Database Tables
-- `records`
-- `record_relationships`
-- `record_entity_edges`
-- `record_event_edges`
-
-### Removed Concepts
-- Capabilities (rules moved to [entity schemas](../../vocabulary/canonical_terms.md#entity-schema))
-
-## Migration Quickstart
-
-```bash
-# 1. Backup database
-pg_dump neotoma > backup_$(date +%Y%m%d).sql
-
-# 2. Dry run
-npm run migrate:records-to-sources -- --user-id <uuid> --dry-run
-
-# 3. Execute
-npm run migrate:records-to-sources -- --user-id <uuid>
-
-# 4. Apply schema migrations
-npm run migrate:up
-```
-
-## Terminology
-
-This release aligns to the authoritative vocabulary defined in [`docs/vocabulary/canonical_terms.md`](../../vocabulary/canonical_terms.md):
-
-| Term | Definition |
-|------|------------|
-| [Source](../../vocabulary/canonical_terms.md#source) | Raw data (structured or unstructured) that gets [ingested](../../vocabulary/canonical_terms.md#ingestion) |
-| [Observation](../../vocabulary/canonical_terms.md#observation) | Granular facts [extracted](../../vocabulary/canonical_terms.md#extraction) from [source](../../vocabulary/canonical_terms.md#source) |
-| [Entity](../../vocabulary/canonical_terms.md#entity) | Canonical representation of a person, company, or location |
-| [Entity Snapshot](../../vocabulary/canonical_terms.md#entity-snapshot) | Deterministic [reducer](../../vocabulary/canonical_terms.md#reducer) output representing current truth |
-| [Entity Schema](../../vocabulary/canonical_terms.md#entity-schema) | Versioned definition of fields, validators, merge policies, and extraction rules |
-
-## Support
-
-Questions? See the full release plan or contact the development team.

package/docs/releases/v0.2.15/migrations/01_add_source_graph_edges.sql

@@ -1,109 +0,0 @@
--- Migration: Add source-based graph edges for v0.2.15
--- Created: 2025-01-01
--- Description: Replace record-based graph edges with source-based graph edges
-
--- Source-Entity edges table (replaces record_entity_edges)
-CREATE TABLE IF NOT EXISTS source_entity_edges (
-  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-  source_id UUID NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
-  entity_id TEXT NOT NULL REFERENCES entities(id),
-  edge_type TEXT NOT NULL DEFAULT 'EXTRACTED_FROM',
-  interpretation_id UUID REFERENCES interpretations(id),
-  created_at TIMESTAMPTZ DEFAULT NOW()
-);
-
--- Indexes for source_entity_edges
-CREATE INDEX IF NOT EXISTS idx_source_entity_edges_source ON source_entity_edges(source_id);
-CREATE INDEX IF NOT EXISTS idx_source_entity_edges_entity ON source_entity_edges(entity_id);
-CREATE INDEX IF NOT EXISTS idx_source_entity_edges_type ON source_entity_edges(edge_type);
-CREATE INDEX IF NOT EXISTS idx_source_entity_edges_interpretation ON source_entity_edges(interpretation_run_id);
-
--- RLS policies for source_entity_edges
-ALTER TABLE source_entity_edges ENABLE ROW LEVEL SECURITY;
-
-DROP POLICY IF EXISTS "Service role full access - source_entity_edges" ON source_entity_edges;
-CREATE POLICY "Service role full access - source_entity_edges" ON source_entity_edges
-  FOR ALL TO service_role USING (true) WITH CHECK (true);
-
--- Users can read edges for their own sources
-DROP POLICY IF EXISTS "Users read own source entity edges" ON source_entity_edges;
-CREATE POLICY "Users read own source entity edges"
-  ON source_entity_edges
-  FOR SELECT
-  TO authenticated
-  USING (
-    EXISTS (
-      SELECT 1 FROM sources
-      WHERE sources.id = source_entity_edges.source_id
-      AND sources.user_id = auth.uid()
-    )
-  );
-
-COMMENT ON TABLE source_entity_edges IS 'Links sources to entities extracted from them (replaces record_entity_edges)';
-COMMENT ON COLUMN source_entity_edges.interpretation_run_id IS 'Which interpretation run created this edge (for provenance)';
-
--- Source-Event edges table (replaces record_event_edges)
-CREATE TABLE IF NOT EXISTS source_event_edges (
-  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-  source_id UUID NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
-  event_id TEXT NOT NULL,
-  edge_type TEXT NOT NULL DEFAULT 'GENERATED_FROM',
-  created_at TIMESTAMPTZ DEFAULT NOW()
-);
-
--- Indexes for source_event_edges
-CREATE INDEX IF NOT EXISTS idx_source_event_edges_source ON source_event_edges(source_id);
-CREATE INDEX IF NOT EXISTS idx_source_event_edges_event ON source_event_edges(event_id);
-CREATE INDEX IF NOT EXISTS idx_source_event_edges_type ON source_event_edges(edge_type);
-
--- RLS policies for source_event_edges
-ALTER TABLE source_event_edges ENABLE ROW LEVEL SECURITY;
-
-DROP POLICY IF EXISTS "Service role full access - source_event_edges" ON source_event_edges;
-CREATE POLICY "Service role full access - source_event_edges" ON source_event_edges
-  FOR ALL TO service_role USING (true) WITH CHECK (true);
-
--- Users can read edges for their own sources
-DROP POLICY IF EXISTS "Users read own source event edges" ON source_event_edges;
-CREATE POLICY "Users read own source event edges"
-  ON source_event_edges
-  FOR SELECT
-  TO authenticated
-  USING (
-    EXISTS (
-      SELECT 1 FROM sources
-      WHERE sources.id = source_event_edges.source_id
-      AND sources.user_id = auth.uid()
-    )
-  );
-
-COMMENT ON TABLE source_event_edges IS 'Links sources to timeline events generated from them (replaces record_event_edges)';
-
--- Add source_id to timeline_events (making source_record_id optional for migration)
-DO $$ 
-BEGIN
-  IF NOT EXISTS (
-    SELECT 1 FROM information_schema.columns 
-    WHERE table_name = 'timeline_events' AND column_name = 'source_id'
-  ) THEN
-    ALTER TABLE timeline_events ADD COLUMN source_id UUID REFERENCES sources(id);
-  END IF;
-END $$;
-
--- Make source_record_id nullable (for migration period)
-ALTER TABLE timeline_events ALTER COLUMN source_record_id DROP NOT NULL;
-
--- Add index for source_id
-CREATE INDEX IF NOT EXISTS idx_timeline_events_source ON timeline_events(source_id);
-
-COMMENT ON COLUMN timeline_events.source_id IS 'Link to source that generated this event (new architecture)';
-COMMENT ON COLUMN timeline_events.source_record_id IS 'DEPRECATED: Legacy link to record (use source_id instead)';
-
--- Add check constraint to ensure at least one reference exists
-ALTER TABLE timeline_events DROP CONSTRAINT IF EXISTS timeline_events_source_check;
-ALTER TABLE timeline_events ADD CONSTRAINT timeline_events_source_check
-  CHECK (source_record_id IS NOT NULL OR source_id IS NOT NULL);
-
-
-
-

package/docs/releases/v0.2.15/migrations/02_drop_legacy_tables.sql

@@ -1,130 +0,0 @@
--- Migration: Drop legacy record-based tables for v0.2.15
--- Created: 2025-01-01
--- Description: Remove legacy records table and associated structures after migration
--- WARNING: This is a destructive migration. Ensure data is migrated before running.
-
--- ============================================================================
--- SAFETY CHECK: Verify migration is complete before running this
--- ============================================================================
-
--- This migration should only be run after:
--- 1. All records have been migrated to sources
--- 2. All graph edges have been migrated
--- 3. All timeline events have been updated
--- 4. Frontend has been updated to use new endpoints
--- 5. Full database backup has been taken
-
--- Recommended verification queries to run BEFORE this migration:
---
--- 1. Check for unmigrated records:
---    SELECT COUNT(*) FROM records WHERE id NOT IN (
---      SELECT provenance->>'migrated_from_record_id' FROM sources
---      WHERE provenance->>'migrated_from_record_id' IS NOT NULL
---    );
---
--- 2. Check for timeline events still using source_record_id:
---    SELECT COUNT(*) FROM timeline_events WHERE source_record_id IS NOT NULL AND source_id IS NULL;
---
--- 3. Check for observations still using source_payload_id:
---    SELECT COUNT(*) FROM observations WHERE source_payload_id IS NOT NULL AND source_id IS NULL;
-
--- ============================================================================
--- DROP FOREIGN KEY CONSTRAINTS
--- ============================================================================
-
--- Drop constraints from timeline_events
-ALTER TABLE timeline_events DROP CONSTRAINT IF EXISTS timeline_events_source_record_id_fkey;
-ALTER TABLE timeline_events DROP CONSTRAINT IF EXISTS timeline_events_source_check;
-
--- Drop constraints from raw_fragments
-ALTER TABLE raw_fragments DROP CONSTRAINT IF EXISTS raw_fragments_record_id_fkey;
-
--- Drop constraints from relationships
-ALTER TABLE relationships DROP CONSTRAINT IF EXISTS relationships_source_record_id_fkey;
-
--- Drop constraints from record_relationships
-ALTER TABLE record_relationships DROP CONSTRAINT IF EXISTS record_relationships_source_id_fkey;
-ALTER TABLE record_relationships DROP CONSTRAINT IF EXISTS record_relationships_target_id_fkey;
-
--- Drop constraints from record_entity_edges
-ALTER TABLE record_entity_edges DROP CONSTRAINT IF EXISTS record_entity_edges_record_id_fkey;
-
--- Drop constraints from record_event_edges
-ALTER TABLE record_event_edges DROP CONSTRAINT IF EXISTS record_event_edges_record_id_fkey;
-
--- ============================================================================
--- DROP DEPRECATED TABLES
--- ============================================================================
-
--- Drop graph edge tables (replaced by source_entity_edges, source_event_edges)
-DROP TABLE IF EXISTS record_entity_edges;
-DROP TABLE IF EXISTS record_event_edges;
-
--- Drop record relationships table
-DROP TABLE IF EXISTS record_relationships;
-
--- Drop main records table
-DROP TABLE IF EXISTS records;
-
--- ============================================================================
--- DROP DEPRECATED COLUMNS
--- ============================================================================
-
--- Drop source_record_id from timeline_events (replaced by source_id)
-ALTER TABLE timeline_events DROP COLUMN IF EXISTS source_record_id;
-
--- Drop record_id from raw_fragments (replaced by source_id)
-ALTER TABLE raw_fragments DROP COLUMN IF EXISTS record_id;
-
--- Drop source_record_id from relationships
-ALTER TABLE relationships DROP COLUMN IF EXISTS source_record_id;
-
--- Drop source_payload_id from observations (replaced by source_id)
-ALTER TABLE observations DROP COLUMN IF EXISTS source_payload_id;
-
--- ============================================================================
--- UPDATE CONSTRAINTS
--- ============================================================================
-
--- Make source_id NOT NULL in timeline_events (now required)
-ALTER TABLE timeline_events ALTER COLUMN source_id SET NOT NULL;
-
--- Make source_id NOT NULL in raw_fragments (now required)
-ALTER TABLE raw_fragments ALTER COLUMN source_id SET NOT NULL;
-
--- ============================================================================
--- CLEANUP COMMENTS
--- ============================================================================
-
-COMMENT ON TABLE timeline_events IS 'Timeline events generated from sources (v0.2.15+)';
-COMMENT ON COLUMN timeline_events.source_id IS 'Source that generated this event';
-
-COMMENT ON TABLE raw_fragments IS 'Unknown fields from interpretation runs (v0.2.15+)';
-COMMENT ON COLUMN raw_fragments.source_id IS 'Source that contained this fragment';
-
--- ============================================================================
--- VERIFICATION QUERIES (Run after migration)
--- ============================================================================
-
--- Verify no orphaned data:
--- 
--- 1. All timeline events have valid source_id:
---    SELECT COUNT(*) FROM timeline_events WHERE source_id IS NULL;
---    -- Should return 0
---
--- 2. All raw_fragments have valid source_id:
---    SELECT COUNT(*) FROM raw_fragments WHERE source_id IS NULL;
---    -- Should return 0
---
--- 3. All observations have valid source_id:
---    SELECT COUNT(*) FROM observations WHERE source_id IS NULL;
---    -- Should return 0
---
--- 4. Verify tables are dropped:
---    SELECT table_name FROM information_schema.tables 
---    WHERE table_name IN ('records', 'record_relationships', 'record_entity_edges', 'record_event_edges');
---    -- Should return 0 rows
-
-
-
-