neotoma 0.3.0 → 0.3.2

package/docs/operations/debugging/error_debug_instructions.md

@@ -1,55 +0,0 @@
-# Error Debug Instructions
-
-Please process all pending errors in the error queue.
-
-## Workflow
-
-1. **Read Error Queue:**
-   - Read `.cursor/error_reports/pending.json`
-   - Sort errors by priority: critical → high → medium → low
-   - Within same severity, oldest first
-
-2. **Process Each Error:**
-   - Load error report from file path
-   - Analyze error: category, severity, affected files
-   - Run `/debug --auto` workflow
-   - Use `/fix_feature_bug` to fix bugs
-   - Update error report with resolution status
-   - Move to resolved archive
-   - Remove from pending queue
-
-3. **Continue Until:**
-   - Pending queue is empty, OR
-   - Human intervention needed (unclear design, security-sensitive changes)
-
-4. **Human Intervention Required:**
-   - Schema conflicts requiring architectural decisions
-   - Security-sensitive changes
-   - Breaking API changes
-   - Missing context/documentation
-
-## Commands to Use
-
-- `/debug --auto` - Automated debugging workflow
-- `/fix_feature_bug` - Fix bugs using error protocol
-- Standard file operations (read, write, update)
-
-## Error Report Status Updates
-
-When processing errors, update the `resolution_status` field:
-
-- Set to `"in_progress"` when starting to work on error
-- Set to `"resolved"` when error is fixed (with `resolution_notes`)
-- Set to `"failed"` if error cannot be resolved (with `resolution_notes` explaining why)
-
-## Resolution Notes
-
-Include details about:
-- What was fixed
-- How it was fixed
-- Any follow-up actions needed
-- Why resolution failed (if status is "failed")
-
-
-
-

package/docs/operations/health_check.md

@@ -1,73 +0,0 @@
-# Health Check
-
-_(Running and interpreting `npm run doctor`)_
-
-## Scope
-
-This document covers:
-
-- How to run the Neotoma health check
-- What each check category does
-- How to interpret results and fix common issues
-
-It does not cover:
-
-- General troubleshooting (see [Troubleshooting](troubleshooting.md))
-- Deployment or runbook procedures (see [Runbook](runbook.md) in this directory)
-
----
-
-## Running Health Checks
-
-From the repository root:
-
-```bash
-npm run doctor
-```
-
-The command runs diagnostics for environment, database, storage, migrations, security, and MCP startup. Output is printed to stdout with pass (✅), fail (❌), or warning (⚠️) per check.
-
----
-
-## Check Categories
-
-| Category        | What is checked                                                                                    |
-| --------------- | -------------------------------------------------------------------------------------------------- |
-| **Environment** | Required variables present for your setup. |
-| **Database**    | Connection and ping (local SQLite).                |
-| **Tables**      | Required tables exist (sources, interpretations, observations, entities, entity_snapshots, etc.).  |
-| **RLS**         | Row-level security enabled on all user-facing tables (when applicable).                            |
-| **Storage**     | Local paths exist and are configured.                                                             |
-| **OAuth**       | OAuth configuration present (optional; warning if missing for production).                         |
-| **Migrations**  | Latest migration applied (version comparison).                                                     |
-| **MCP**         | MCP server can start (e.g. stdio mode).                                                            |
-| **Security**    | `.env` is gitignored; no service key in logs; `.env` present when required.                        |
-
-See script output and code in `scripts/doctor.ts` for the exact checks in your version.
-
----
-
-## Interpreting Results
-
-- **Overall: HEALTHY** – All critical checks passed. Warnings (e.g. OAuth not configured) are acceptable for local dev.
-- **Overall: UNHEALTHY** – One or more critical checks failed. Fix failing items before relying on the system for data or production use.
-- **Warning (⚠️)** – Recommended but not required (e.g. OAuth for production, optional env vars).
-
----
-
-## Fixing Common Issues
-
-| Failure     | Typical fix                                                                                                                                                                              |
-| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Environment | Create or fix `.env`. See [Getting started](../developer/getting_started.md). |
-| Database    | Verify SQLite path is writable. See [Troubleshooting – Database](troubleshooting.md#issue-database-connection-timeout). |
-| Tables      | Run migrations: `npm run migrate`.                                                                                    |
-| RLS         | Ensure RLS is enabled and policies exist (schema/migrations) when applicable.                                        |
-| Storage     | Ensure `NEOTOMA_DATA_DIR` / `NEOTOMA_RAW_STORAGE_DIR` exist and are writable.                                         |
-| Security    | Ensure `.env` is not committed; remove any committed secrets. Do not log service keys.                                                                                                   |
-
----
-
-## Automated Monitoring
-
-A dedicated health-check HTTP endpoint for production (e.g. for load balancers or uptime checks) may be added in a future release. Until then, use `npm run doctor` from a cron job or deployment script, or call existing API endpoints that reflect backend connectivity.

package/docs/operations/runbook.md

@@ -1,88 +0,0 @@
-# Operational Runbook
-
-_(Startup, health checks, shutdown, and common operations)_
-
-## Scope
-
-This runbook covers:
-
-- Startup procedures (local development and production)
-- Health checks and verification
-- Shutdown and cleanup
-- Common operations (migrations, backups)
-- Emergency procedures and troubleshooting pointers
-
-It does not cover:
-
-- Feature-level behavior (see subsystem docs)
-- Deployment pipeline (see [Deployment](../infrastructure/deployment.md))
-- Detailed troubleshooting (see [Troubleshooting](troubleshooting.md))
-
----
-
-## Startup Procedures
-
-### Local development
-
-1. **Prerequisites:** Node.js v18/v20, npm 9+.
-2. **Environment:** Copy or create `.env` (see [Getting started](../developer/getting_started.md)). For local storage, no env vars are required (defaults: local backend, `./data`, `./data/neotoma.db`, `./data/sources`).
-3. **Install and migrate:**
-   ```bash
-   npm install
-   npm run migrate
-   ```
-4. **Start services:**
-   - MCP server (stdio): `npm run dev`
-   - API only (MCP at /mcp): `npm run dev:server`
-   - Full stack (API + UI): `npm run dev:full`
-   - WebSocket MCP bridge: `npm run dev:ws`
-5. **Verify:** Run `npm run doctor` (see [Health check](health_check.md)).
-
-### Production
-
-1. Build: `npm run build:server`.
-2. Set production env (data paths, HTTPS, OAuth config).
-3. Run migrations: `npm run migrate` or ensure local DB path is correct.
-4. Start: `node dist/actions.js` (API) or `node dist/index.js` (stdio MCP). Use a process manager (e.g. systemd, PM2) and HTTPS in front.
-5. Verify health: `npm run doctor` and smoke-test critical endpoints.
-
----
-
-## Health Checks
-
-- **Primary:** Run `npm run doctor` for environment, database, RLS, storage, migrations, and security checks. See [Health check](health_check.md).
-- **Manual:** Run tests (`npm test`, `npm run test:integration`), type-check (`npm run type-check`), and lint (`npm run lint`).
-- **Runtime:** Hit API health/readiness endpoints if configured; confirm MCP server starts (e.g. stdio mode).
-
----
-
-## Shutdown Procedures
-
-- **Graceful:** Stop the Node process (SIGTERM). Allow in-flight requests to complete; process managers typically send SIGTERM then SIGKILL after a timeout.
-- **Cleanup:** No mandatory cleanup for normal shutdown. For local dev, ensure no stray `tsx watch` or dev servers if switching branches (ports may be in use).
-
----
-
-## Common Operations
-
-- **Migrations:** Apply with `npm run migrate`. Dry-run: `npm run migrate:dry-run`. See [Troubleshooting](troubleshooting.md) for "relation does not exist" and RLS issues.
-- **Backups:** For local SQLite, copy the DB (default `./data/neotoma.db` in dev, `./data/neotoma.prod.db` in prod, under `NEOTOMA_DATA_DIR`), `NEOTOMA_RAW_STORAGE_DIR` (default `./data/sources` in dev, `./data/sources_prod` in prod), and `NEOTOMA_LOGS_DIR` (default `./data/logs` in dev, `./data/logs_prod` in prod; includes `events.log`).
-- **User and access:** Auth and user management are in [Auth](docs/subsystems/auth.md).
-
----
-
-## Emergency Procedures
-
-- **Rollback:** Revert deployment to previous version; re-run migrations only if a migration rollback is defined (see migration docs). Restore DB/storage from backup if data corruption is suspected.
-- **Incident response:** Triage using [Troubleshooting](troubleshooting.md). Check logs and `npm run doctor` output. For security issues, see [SECURITY.md](../../SECURITY.md) in the repo root.
-
----
-
-## Troubleshooting Quick Reference
-
-| Symptom                         | See                                                                    |
-| ------------------------------- | ---------------------------------------------------------------------- |
-| Missing env / connection errors | [Troubleshooting – Setup](troubleshooting.md#setup-issues)             |
-| Database / RLS errors           | [Troubleshooting – Runtime](troubleshooting.md#runtime-issues)         |
-| Port in use                     | [Troubleshooting – Port](troubleshooting.md#issue-port-already-in-use) |
-| Health check failures           | [Health check](health_check.md)                                        |

package/docs/prompts/llm_extraction_system_prompt.md

@@ -1,31 +0,0 @@
-# LLM extraction system prompt
-
-This file is the system prompt for AI interpretation of unstructured documents. It is loaded at runtime by `src/services/llm_extraction.ts`. Edit this file to change extraction behavior; do not duplicate prompt text in code.
-
-The prompt body used by the service is the content below the horizontal rule (`---`).
-
----
-
-You are a document analysis expert. Extract structured data from the document using only what it explicitly states or shows.
-
-Rules:
-- NEVER invent data. No placeholders, examples, or inferred values. If a value is not in the document, omit that field.
-- Choose entity_type from what the document actually is. Use the criteria below so the right schema is used. Wrong type leads to wrong or missing fields.
-- Extract every stated fact as its own field. One snake_case key per fact. Do not summarize multiple facts into one field (e.g. no single "description" when the document has VIN, model, amounts, etc.).
-- Use descriptive snake_case keys. Prefer terms from the document (e.g. precio_base, numero_bastidor, total_a_pagar) or clear English equivalents (e.g. total_amount, vin, merchant_name, date_purchased). Any consistent snake_case naming is fine; schemas can be created or extended from your output.
-- Support all languages; extract labels and values as they appear.
-
-Entity type criteria (use the first that fits; other registered or new snake_case types are allowed):
-- receipt: Proof of purchase from a vendor. Has merchant/store name, total amount (and usually currency, date). Use for receipts, recibo, comprobante, till slips, purchase proofs.
-- invoice: Bill for payment with amounts, due date, and parties (seller/buyer). Use for bills, factura, invoices.
-- note: General memo, journal, or free-form text without purchase/vendor/amount structure. Use only when the document is not a receipt or invoice.
-- contract: Agreement with parties, terms, dates. Use for contracts, contrato, agreements.
-- contact / person / company: People or organizations with names, contact details, roles.
-- task: Action item or to-do with due date or status.
-- event: Dated occurrence (meeting, appointment, etc.).
-
-Output:
-- entity_type: One label that best describes the document. Prefer a type from the criteria above when it fits; otherwise use any other appropriate snake_case type (e.g. insurance_claim, prescription) or a new one if none fit.
-- fields: Object of key-value pairs for every fact in the document. Keys: snake_case. Values: as stated (numbers, strings, dates; use consistent formats).
-
-Return ONLY valid JSON. No markdown, no code blocks, no explanations.

package/docs/proposals/IMPLEMENTATION_SUMMARY.md

@@ -1,198 +0,0 @@
-# Plan Migration Implementation Summary
-
-**Implemented:** 2026-01-22
-**Status:** Complete
-
-## Overview
-
-Successfully implemented an automated plan migration system that integrates with the `/commit` workflow. The system automatically reviews plans in `.cursor/plans/`, determines relevance, and migrates them to `docs/proposals/` for future consideration.
-
-## What Was Implemented
-
-### 1. Standalone Command
-
-**File:** `.cursor/commands/migrate_plans.md`
-
-- Command specification with relevance criteria
-- Step-by-step migration process
-- Proposal format specification
-- Integration points with commit workflow
-
-### 2. Commit Workflow Integration
-
-**File:** `.cursor/commands/commit.md`
-
-- Added plan migration step after security audit, before staging
-- Auto-stages migrated proposals for inclusion in commits
-- Includes migration summary in commit messages
-- Non-blocking (if migration fails, commit proceeds)
-
-### 3. Migration Script
-
-**File:** `scripts/migrate_plans.ts`
-
-- TypeScript implementation with proper YAML parsing
-- Handles both frontmatter and legacy plan formats
-- Extracts metadata (name, overview, todos)
-- Counts todos by status (pending, completed, in_progress)
-- Detects duplicates (multiple versions of same plan)
-- Converts plans to proposal format
-- Generates migration reports
-
-### 4. Directory Structure
-
-**Created:**
-- `docs/proposals/` - Active proposals
-- `docs/proposals/archived/` - Completed proposals
-- `docs/proposals/README.md` - Directory documentation
-- `docs/proposals/PROPOSAL_TEMPLATE.md` - Template for proposal format
-
-### 5. NPM Scripts
-
-**Added to `package.json`:**
-- `npm run migrate:plans` - Run migration
-- `npm run migrate:plans:dry-run` - Preview without changes
-
-### 6. Configuration
-
-**Added to `foundation-config.yaml`:**
-```yaml
-development:
-  commit:
-    migrate_plans: true
-    migrate_plans_config:
-      proposals_dir: "docs/proposals"
-      archive_completed: true
-      auto_remove_obsolete: false
-```
-
-## Migration Results
-
-**First Migration Run (2026-01-22):**
-
-- ✅ **8 proposals migrated** (plans with pending work)
-- ✅ **1 proposal archived** (completed plan)
-- ✅ **2 obsolete identified** (duplicate versions)
-- ⚠️ **11 require manual review** (no todos to determine status)
-
-### Migrated Proposals
-
-1. `schema-evolution-scaling.md` (8 pending todos, P0 priority)
-2. `agent-trust-framework.md` (11 pending todos)
-3. `dual-path-inference-based-ingestion.md` (19 pending todos)
-4. `raw-first-ingestion-architecture.md` (13 pending todos)
-5. `sources-first-ingestion-v10.md` (7 pending todos)
-6. `test-coverage-gap-analysis.md` (7 pending todos)
-7. `document-v023-release.md` (5 pending, 9 completed)
-8. `plan-migration-command-integrated-with-commit.md` (9 pending - meta!)
-
-### Archived
-
-1. `complete_mvp_implementation_2ebb6b9d.md` (15/15 todos completed)
-
-## Features
-
-### Relevance Detection
-
-- Counts todos by status
-- Identifies pending work
-- Detects completed plans for archival
-- Finds duplicate versions (v8, v9, v10, etc.)
-
-### Format Conversion
-
-- Removes todo tracking from proposals
-- Adds proposal-specific metadata
-- Adds Proposal Context section explaining migration
-- Preserves all technical content
-- Updates references where needed
-
-### Automatic Integration
-
-- Runs during `/commit` after security audit
-- Auto-stages created proposals
-- Includes migration summary in commit message
-- Non-blocking (doesn't stop commits if migration fails)
-
-## Usage
-
-### Manual Migration
-
-```bash
-npm run migrate:plans
-```
-
-or
-
-```
-/migrate_plans
-```
-
-### Automatic Migration
-
-Runs automatically during `/commit` if enabled in `foundation-config.yaml`.
-
-### Dry Run
-
-```bash
-npm run migrate:plans:dry-run
-```
-
-Preview what would be migrated without creating files.
-
-## Next Steps
-
-1. **Review migrated proposals** in `docs/proposals/` directory
-2. **Manually review plans** flagged for manual review (11 files)
-3. **Remove obsolete plans** from `.cursor/plans/` (v8, v11 duplicates)
-4. **Prioritize proposals** for future implementation
-5. **Update proposals** if any need architecture alignment fixes
-
-## Implementation Notes
-
-### Parser Handles
-
-- ✅ YAML frontmatter with todos
-- ✅ Legacy plans without frontmatter
-- ✅ Multiple todo statuses (pending, completed, in_progress)
-- ✅ Plans without todos (flagged for manual review)
-- ✅ Malformed frontmatter (graceful error handling)
-
-### Duplicate Detection
-
-- Identifies versioned plans (v8, v9, v10, etc.)
-- Keeps only the most recent version
-- Marks others as obsolete
-
-### Archive Logic
-
-- Archives plans with 100% completed todos
-- Marks as `status: "implemented"` in frontmatter
-- Saves to `docs/proposals/archived/`
-
-## Testing
-
-Migration tested on 22 plan files:
-- ✅ Parsing works for all formats
-- ✅ Relevance detection accurate
-- ✅ Conversion format correct
-- ✅ Report generation works
-- ✅ Auto-staging works
-- ✅ Integration with commit workflow verified
-
-## Files Modified
-
-1. `.cursor/commands/migrate_plans.md` (created)
-2. `.cursor/commands/commit.md` (updated)
-3. `scripts/migrate_plans.ts` (created)
-4. `package.json` (added scripts)
-5. `foundation-config.yaml` (added config)
-6. `docs/proposals/README.md` (created)
-7. `docs/proposals/PROPOSAL_TEMPLATE.md` (created)
-
-## References
-
-- Migration report: `docs/proposals/MIGRATION_REPORT.md`
-- Command spec: `.cursor/commands/migrate_plans.md`
-- Implementation: `scripts/migrate_plans.ts`
-- Proposals directory: `docs/proposals/`

package/docs/proposals/MIGRATION_REPORT.md

@@ -1,65 +0,0 @@
-# Plan Migration Report
-
-Generated: 2026-01-22T13:05:42.054Z
-
-## Summary
-
-- **Migrated to Proposals:** 8
-- **Archived (Completed):** 1
-- **Obsolete (Logged):** 2
-- **Requires Manual Review:** 11
-
----
-
-## Migrated to Proposals
-
-- `agent_skills_analysis_aac0f1b6.plan.md` → `docs/proposals/document-v023-release.md`
-- `agent_trust_framework_eb38ca7d.plan.md` → `docs/proposals/agent-trust-framework.md`
-- `dual-path_inference-based_ingestion_26e7ee07.plan.md` → `docs/proposals/dual-path-inference-based-ingestion.md`
-- `plan_migration_command_integrated.plan.md` → `docs/proposals/plan-migration-command-integrated-with-commit.md`
-- `raw-first_ingestion_architecture_6fd45409.plan.md` → `docs/proposals/raw-first-ingestion-architecture.md`
-- `schema_evolution_scaling_777f1fc8.plan.md` → `docs/proposals/schema-evolution-scaling.md`
-- `sources-first_ingestion_v10_baef2ba2.plan.md` → `docs/proposals/sources-first-ingestion-v10.md`
-- `test_coverage_gap_analysis_9cb70022.plan.md` → `docs/proposals/test-coverage-gap-analysis.md`
-
----
-
-## Archived (Completed)
-
-- `complete_mvp_implementation_2ebb6b9d.md` → `docs/proposals/archived/complete_mvp_implementation_2ebb6b9d.md`
-
----
-
-## Obsolete (Logged for Manual Removal)
-
-- `sources-first-ingestion-v11.plan.md` - Reason: duplicate (superseded by newer version)
-- `sources-first-ingestion-v8.plan.md` - Reason: duplicate (superseded by newer version)
-
----
-
-## Requires Manual Review
-
-- `add-summary-field.plan.md` - Reason: no todos (cannot determine status)
-- `automated_error_debugging_cloud_agents.plan.md` - Reason: no todos (cannot determine status)
-- `automated_error_debugging_cursor_cli.plan.md` - Reason: no todos (cannot determine status)
-- `automated_error_debugging_localhost_clarified.plan.md` - Reason: no todos (cannot determine status)
-- `debug_pending_errors_command.plan.md` - Reason: no todos (cannot determine status)
-- `error_reporting_command_8868f27b.plan.md` - Reason: no todos (cannot determine status)
-- `gdpr_compliance_with_immutability_b51593ad.plan.md` - Reason: no todos (cannot determine status)
-- `local-first-e2ee-architecture-99397386.plan.md` - Reason: no todos (cannot determine status)
-- `neotoma_mcp_data_dir_readiness_analysis_61a3af25.plan.md` - Reason: no todos (cannot determine status)
-- `sources-first-ingestion-v12-final.plan.md` - Reason: no todos (cannot determine status)
-- `sources-first-ingestion-v9.plan.md` - Reason: no todos (cannot determine status)
-
----
-
-## Next Steps
-
-1. Review migrated proposals in `docs/proposals/`
-2. Manually review plans flagged above
-3. Remove obsolete plans from `.cursor/plans/` if desired
-4. Prioritize proposals for future implementation
-
----
-
-_Generated by: /migrate_plans command on 2026-01-22_

package/docs/proposals/PROPOSAL_TEMPLATE.md

@@ -1,58 +0,0 @@
----
-title: "[Proposal Name]"
-status: "proposal"
-source_plan: "[original-plan-filename.plan.md]"
-migrated_date: "YYYY-MM-DD"
-priority: "p0|p1|p2|p3"
-estimated_effort: "[description if available]"
----
-
-# [Proposal Name]
-
-## Proposal Context
-
-This proposal was migrated from `.cursor/plans/[original-filename]` on [date].
-
-**Original Status:** [pending/completed/partial] - [X/Y todos completed]
-**Relevance:** [Why this is still relevant for future consideration]
-**Architecture Alignment:** [Verified/Needs Review/Conflicts]
-
-## Overview
-
-[Original overview from plan - what this proposal aims to achieve]
-
-## Architecture Overview
-
-[Diagrams and high-level architecture if applicable]
-
-## Technical Details
-
-[Original plan content - architecture, implementation notes, design decisions]
-
-## Implementation Steps
-
-[Detailed implementation steps from original plan]
-
-## Testing Strategy
-
-[Testing approach from original plan]
-
-## Implementation Considerations
-
-**What's Already Done:**
-- [List completed work if plan was partially implemented]
-
-**What's Still Needed:**
-- [List remaining work]
-
-**Potential Conflicts:**
-- [Note any conflicts with current architecture or other work]
-
-**Dependencies:**
-- [List dependencies on other features or systems]
-
-## References
-
-- Original plan: `.cursor/plans/[filename]`
-- Related docs: [links to relevant architecture/spec docs]
-- Related issues: [if applicable]

package/docs/proposals/README.md

@@ -1,57 +0,0 @@
-# Proposals
-
-This directory contains future work proposals that have been migrated from `.cursor/plans/` for later consideration.
-
-## Directory Structure
-
-- `*.md` - Active proposals for future implementation
-- `archived/` - Completed proposals that have been implemented
-
-## Current proposals
-
-- `mcp-cli-action-items.md` - MCP and CLI action items proposal
-- `llm_sampling_parameters_interpretation_config.md` - Add top-K and top-P to interpretation config for LLM auditability and repetition mitigation
-- `iterative_chat_store_mcp_instructions.md` - Update MCP instructions so agents store conversation and each turn as the chat progresses (no manual end-of-chat store command); Option A implemented
-- `conversation_turn_identity_reverts_forks.md` - Follow-on: turn identity, idempotency_key convention, preserve-all-branch-data, and graceful handling of reverts and chat forks; Option B contract implemented
-
-## Proposal Format
-
-Each proposal follows this structure:
-
-```yaml
----
-title: "Proposal Name"
-status: "proposal|in_progress|implemented"
-source_plan: "original-plan-filename.plan.md"
-migrated_date: "YYYY-MM-DD"
-priority: "p0|p1|p2|p3"
-estimated_effort: "description"
----
-```
-
-See `PROPOSAL_TEMPLATE.md` for the complete format.
-
-## Proposal Lifecycle
-
-1. **proposal** - Migrated from plan, awaiting prioritization
-2. **in_progress** - Implementation has started
-3. **implemented** - Completed and moved to `archived/`
-
-## How Proposals Are Created
-
-Proposals are automatically migrated from `.cursor/plans/` during the `/commit` workflow. The migration:
-
-- Reviews all plans for relevance (pending todos, architecture alignment)
-- Converts relevant plans to proposal format
-- Archives completed plans
-- Removes obsolete/duplicate plans
-
-## Manual Migration
-
-To manually migrate plans: `/migrate_plans`
-
-## References
-
-- Original plans: `.cursor/plans/`
-- Migration command: `.cursor/commands/migrate_plans.md`
-- Proposal template: `PROPOSAL_TEMPLATE.md`