neotoma 0.3.0 → 0.3.2

package/docs/developer/resolved_blockers.md

@@ -1,49 +0,0 @@
-# Resolved Blockers for Release Build
-This document tracks blockers that have been resolved for cloud agent execution.
-## Blocker 1: CSV Fixture Loader ✅ RESOLVED
-**Issue**: Playwright specs couldn't import `.csv` fixtures directly - Vite/Playwright tried to parse `frontend/src/sample-data/sets-medium.csv` as JavaScript during test runs.
-**Solution**: Converted CSV file to TypeScript constant:
-- Created `frontend/src/sample-data/sets-medium.ts` with CSV content as string constant
-- Updated `frontend/src/sample-data/sample-records.ts` to import from `.ts` instead of `.csv?raw`
-- Changed: `import workoutCsvRaw from './sets-medium.csv?raw'` → `import { setsMediumCsv } from './sets-medium'`
-**Files Changed**:
-- `frontend/src/sample-data/sets-medium.ts` (new file)
-- `frontend/src/sample-data/sample-records.ts` (updated import)
-**Status**: ✅ Fixed - CSV content now available as TypeScript constant that can be imported without issues
-## Blocker 2: Database Credentials for Integration Tests ✅ DOCUMENTED
-**Issue**: Integration/E2E tests require database connection environment variables, but cloud agents don't have access to these.
-**Solution**: 
-1. Orchestrator now includes environment variables in agent instructions
-2. Created helper script: `scripts/setup-test-env.sh` for agents to export env vars
-3. Updated orchestrator instructions to guide agents on setting up environment variables
-**How to Use**:
-- Agents receive env vars in their instructions from orchestrator's `.env` file
-- Agents should export them before running tests
-- Or use: `source scripts/setup-test-env.sh`
-**Status**: ✅ Instructions updated - Agents need to export variables from instructions before running tests
-**Note**: If credentials are not available, integration/E2E tests will be skipped or fail gracefully.
-## Blocker 3: Infrastructure Setup Automation ✅ RESOLVED
-**Issue**: Agents were failing tests due to:
-- Database migrations not applied (missing schema objects like `records.embedding`)
-- Playwright browsers not installed (E2E tests failed)
-**Solution**: Created automated setup script that handles all infrastructure setup:
-- Created `scripts/setup_agent_environment.sh` that:
-  - Applies database migrations via `npm run migrate`
-  - Installs Playwright browsers (`npx playwright install --with-deps chromium`)
-  - Verifies npm dependencies are installed
-- Updated orchestrator (`scripts/release_orchestrator.js`) to include setup script in agent instructions
-- Agents now automatically run setup script before tests
-**Files Changed**:
-- `scripts/setup_agent_environment.sh` (new file)
-- `scripts/release_orchestrator.js` (updated `generateAgentInstructions` to include setup steps)
-**Status**: ✅ Fixed - Future agents will automatically run `./scripts/setup_agent_environment.sh` to resolve infrastructure issues
-**Usage**:
-```bash
-# Agents run this automatically, but can also run manually:
-./scripts/setup_agent_environment.sh
-```
-## Next Steps
-1. ✅ Infrastructure setup is now automated - agents run setup script automatically
-2. ✅ CSV import issue is resolved - no further action needed
-3. ✅ Environment variables provided in agent instructions
-4. Test suites should now run successfully with proper infrastructure setup

package/docs/developer/schema_initialization.md

@@ -1,182 +0,0 @@
-# Schema Initialization
-
-## Overview
-
-Entity schemas defined in `src/services/schema_definitions.ts` should be initialized in the database's `schema_registry` table for transparent, queryable schema management. This is more elegant than using TypeScript code as runtime fallbacks.
-
-## Why Initialize Schemas?
-
-**Benefits:**
-- **Transparency**: Schemas are visible and queryable in the database
-- **Consistency**: Single source of truth (database) instead of code fallbacks
-- **Versioning**: Database tracks schema versions and history
-- **Simplicity**: No fallback logic needed in runtime code
-- **Auditability**: Schema changes are tracked in database
-
-**Current Approach:**
-- Schemas in `schema_definitions.ts` are used as runtime fallbacks
-- Fallback logic adds complexity to `interpretation.ts`
-- Schemas are not visible in database until explicitly registered
-
-**Recommended Approach:**
-- Initialize all schemas from `schema_definitions.ts` into database
-- Database becomes single source of truth
-- Runtime code only queries database (no fallback needed)
-
-## Initialization Script
-
-### Usage
-
-```bash
-# Initialize all schemas (register and activate)
-npm run schema:init
-
-# Dry run (see what would be registered without making changes)
-npm run schema:init:dry-run
-
-# Force re-registration of existing schemas
-tsx scripts/initialize-schemas.ts --force
-```
-
-### What It Does
-
-1. **Reads schemas** from `src/services/schema_definitions.ts`:
-   - All schemas from `ENTITY_SCHEMAS` (unified structure)
-
-2. **Checks database** for existing schemas:
-   - If schema exists and is active → skip
-   - If schema exists but is inactive → activate it
-   - If schema doesn't exist → register and activate it
-
-3. **Activates schemas**:
-   - Only one schema version per entity type can be active
-   - Activation automatically deactivates other versions
-
-4. **Reports results**:
-   - Shows which schemas were registered, activated, or skipped
-   - Reports any errors
-
-### Example Output
-
-```
-🔧 Initializing Schema Registry from schema_definitions.ts
-
-Processing base schemas...
-  ✓ holding v1.0 - registered
-  ✓ income v1.0 - registered
-  ↻ contact v1.0 - activated
-  ⊘ account v1.0 - skipped (already active)
-  ...
-
-Processing expanded schemas...
-  ✓ transaction v1.0 - registered
-  ...
-
-============================================================
-SUMMARY
-============================================================
-Total schemas processed: 32
-  ✓ Registered: 28
-  ↻ Activated: 2
-  ⊘ Skipped (already active): 2
-
-✓ Schema initialization complete!
-💡 Schemas are now available in the database and will be used instead of code fallbacks.
-```
-
-## When to Run
-
-### Initial Setup
-Run after:
-- Setting up a new database
-- Running migrations that create `schema_registry` table
-- Cloning the repository for the first time
-
-### After Schema Changes
-Run after:
-- Adding new entity types to `schema_definitions.ts`
-- Updating existing schema definitions
-- Adding new schema versions
-
-### Development Workflow
-```bash
-# 1. Make changes to schema_definitions.ts
-# 2. Initialize schemas
-npm run schema:init
-
-# 3. Verify schemas are active
-# (query schema_registry table or check logs)
-```
-
-## Fallback Behavior
-
-The runtime code in `src/services/interpretation.ts` still includes fallback logic for backward compatibility:
-
-- **Development**: Falls back to code schemas with a warning
-- **Production**: Should rely on database schemas only
-
-**Warning in Development:**
-```
-[Schema Fallback] No database schema found for entity type "invoice". 
-Using code fallback. Run 'npm run schema:init' to register schemas in the database.
-```
-
-## Database Schema
-
-Schemas are stored in the `schema_registry` table:
-
-```sql
-CREATE TABLE schema_registry (
-  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-  entity_type TEXT NOT NULL,
-  schema_version TEXT NOT NULL,
-  schema_definition JSONB NOT NULL,
-  reducer_config JSONB NOT NULL,
-  active BOOLEAN NOT NULL DEFAULT false,
-  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  UNIQUE(entity_type, schema_version)
-);
-```
-
-## Related Commands
-
-- `npm run schema:export` - Export schema snapshots to `docs/subsystems/schema_snapshots/`
-- `npm run analyze:schemas` - Compare DATA_DIR entity types with schema definitions
-
-## Migration Path
-
-For existing deployments:
-
-1. **Run initialization script:**
-   ```bash
-   npm run schema:init
-   ```
-
-2. **Verify schemas are active:**
-   ```sql
-   SELECT entity_type, schema_version, active 
-   FROM schema_registry 
-   WHERE active = true 
-   ORDER BY entity_type;
-   ```
-
-3. **Optional: Remove fallback logic** (after verifying all schemas are registered):
-   - Update `src/services/interpretation.ts` to remove fallback code
-   - This makes the system more strict and transparent
-
-## Troubleshooting
-
-### Schema Already Exists Error
-If you see "duplicate key" or "unique constraint" errors:
-- The schema is already registered
-- Use `--force` flag to re-register: `tsx scripts/initialize-schemas.ts --force`
-
-### Schema Not Found After Initialization
-- Check that the schema was actually registered: query `schema_registry` table
-- Verify the entity type name matches exactly (case-sensitive)
-- Check for errors in the initialization output
-
-### Fallback Still Being Used
-- Verify schemas are active: `SELECT * FROM schema_registry WHERE active = true`
-- Check that `entity_type` matches exactly (case-sensitive)
-- Run initialization again: `npm run schema:init`

package/docs/developer/secrets/README.md

@@ -1,15 +0,0 @@
-# Secrets Management
-
-This directory contains documentation for the encrypted secrets manager used in the Neotoma project.
-
-## Documentation
-
-- **[Secrets Setup](secrets_setup.md)** - How to set up and use the secrets manager
-- **[Secrets Architecture](secrets_architecture.md)** - Architecture and design decisions
-- **[Secrets Secure Approach](secrets_secure_approach.md)** - Security considerations
-- **[Secrets Manager Value](secrets_manager_value.md)** - Value proposition and use cases
-
-## Related
-
-- **Script**: `scripts/secrets_manager.js` - The secrets manager implementation
-- **Storage**: `.secrets/` directory (gitignored) - Encrypted secrets storage

package/docs/developer/secrets/secrets_architecture.md

@@ -1,95 +0,0 @@
-# Secrets Manager Architecture
-
-## Current Architecture (Recommended)
-
-**Flow:**
-1. Orchestrator reads encrypted secrets file (`.secrets/secrets.enc`)
-2. Orchestrator uses encryption key (`.secrets/.key` or `NEOTOMA_SECRETS_MASTER_KEY`) to decrypt
-3. Orchestrator base64-encodes **already-decrypted** credentials for transmission
-4. Agents receive base64-encoded credentials in instructions
-5. Agents decode base64 and use credentials directly
-
-**Security Benefits:**
-- ✅ Encryption key never leaves orchestrator
-- ✅ Secrets encrypted at rest until orchestrator decrypts
-- ✅ Agents only see decrypted credentials (base64 is just encoding, not encryption)
-- ✅ No key management needed for agents
-
-**How it works:**
-```
-[Secrets Manager] → [Orchestrator (decrypts)] → [Base64 encode] → [Agent Instructions] → [Agent (decodes)]
-     (encrypted)          (has key)              (safe transport)      (base64 string)       (plaintext)
-```
-
-## Alternative: Agent-Side Decryption (Not Recommended)
-
-If you wanted agents to decrypt secrets themselves, you would need to:
-
-1. Pass the encryption key to agents (defeats security purpose)
-2. Pass encrypted secrets file to agents
-3. Agents decrypt using the key
-
-**Why this is worse:**
-- ❌ Encryption key must be passed to agents (security risk)
-- ❌ Key appears in conversation history
-- ❌ More complex for agents (must handle decryption)
-- ❌ No real security benefit
-
-## Implementation Details
-
-### Orchestrator Side (Has Encryption Key)
-
-```javascript
-// In release_orchestrator.js
-function loadCredentials() {
-  // Decrypts secrets using key from .secrets/.key or NEOTOMA_SECRETS_MASTER_KEY
-  const creds = loadCredentialsFromSecretsManager();
-  
-  // Base64 encode already-decrypted credentials
-  return Buffer.from(creds.join("\n")).toString("base64");
-}
-```
-
-### Agent Side (No Encryption Key Needed)
-
-```bash
-# Agent receives base64-encoded string in instructions
-# Agent decodes (not decrypts - it's already decrypted):
-echo "<base64_string>" | base64 -d > .env.agent
-export $(cat .env.agent | xargs)
-```
-
-## Key Management
-
-**Local Development:**
-- Key stored in `.secrets/.key` (chmod 600)
-- Orchestrator reads key automatically
-- Key never committed to git
-
-**CI/CD / Production:**
-- Set `NEOTOMA_SECRETS_MASTER_KEY` environment variable
-- Value is hex-encoded encryption key
-- Store securely in CI/CD secrets manager
-
-**Key Location Priority:**
-1. `NEOTOMA_SECRETS_MASTER_KEY` environment variable (preferred for CI/CD)
-2. `.secrets/.key` file (fallback for local development)
-
-## Security Notes
-
-1. **Base64 is not encryption** - It's encoding for safe text transmission
-2. **Decryption happens at orchestrator** - Agents only decode base64
-3. **Key never shared with agents** - Only orchestrator has decryption key
-4. **Secrets encrypted at rest** - `.secrets/secrets.enc` is encrypted
-5. **Use dedicated test credentials** - Separate from production keys
-
-## Summary
-
-**Agents do NOT need the decryption key** because:
-- Orchestrator decrypts secrets before sending to agents
-- Agents only receive already-decrypted credentials (base64-encoded for safe text transmission)
-- This is more secure than giving agents the encryption key
-
-
-
-

package/docs/developer/secrets/secrets_manager_value.md

@@ -1,53 +0,0 @@
-# Secrets Manager Value Assessment
-
-## The Problem
-
-If we decrypt secrets just to pass via an API field (which gets rejected anyway), the secrets manager provides **minimal additional security** for cloud agents specifically.
-
-## Where Secrets Manager Provides Value
-
-### 1. Encryption at Rest
-
-- ✅ Credentials not in plaintext in git repo
-- ✅ Can commit encrypted file safely
-- ✅ Better than `.env` for version control
-
-### 2. Centralized Management
-
-- ✅ Single source of truth
-- ✅ Easier credential rotation
-- ✅ Better than scattered `.env` files
-
-### 3. Local Development & CI/CD
-
-- ✅ Useful where credentials can actually be used
-- ✅ Same encrypted storage works everywhere
-
-## Where It Doesn't Help (Cloud Agents)
-
-### Current Reality
-
-- ❌ API rejects `environment` field
-- ❌ Decrypted secrets still can't be delivered
-- ⚠️ Ends up equivalent to `.env` (both get decrypted)
-
-### The Issue
-
-Both approaches:
-
-1. Secrets manager: decrypt → try API → fail → no delivery
-2. `.env` file: read → try API → fail → no delivery
-
-**Same outcome for cloud agents.**
-
-## Recommendation
-
-For **cloud agent orchestrator**: `.env` is simpler and achieves the same result (since API doesn't accept env vars anyway).
-
-For **local/CI/CD**: Secrets manager is valuable if you want encrypted storage in the repo.
-
-## Conclusion
-
-The secrets manager is over-engineered for the cloud agent use case. Since the API doesn't support environment variables, decrypting from encrypted storage doesn't provide security benefits beyond what `.env` would give us.
-
-**Simpler approach**: Use `.env` for orchestrator, keep secrets manager only if you need encrypted storage in the repo for other purposes.

package/docs/developer/secrets/secrets_secure_approach.md

@@ -1,103 +0,0 @@
-# Secure Credential Management for Cloud Agents
-
-## Problem
-
-Passing credentials in conversation text (even base64-encoded) is insecure because:
-- Credentials appear in conversation history
-- Base64 is easily reversible (not encryption)
-- Security policies block this approach
-- Credentials can be extracted from logs/conversations
-
-## Secure Approaches
-
-### Option 1: Skip Credential-Dependent Tests (Recommended for Cloud Agents)
-
-**Approach:**
-- Don't pass credentials to agents at all
-- Agents mark integration/E2E tests as skipped when credentials unavailable
-- Unit tests can still run (many don't require database)
-- Status file records which tests were skipped and why
-
-**Pros:**
-- ✅ No credentials in conversation
-- ✅ Clear audit trail (status file shows skipped tests)
-- ✅ Agents can still complete their work
-
-**Cons:**
-- ❌ Integration/E2E tests don't run automatically
-- ❌ Requires manual test execution later
-
-### Option 2: Secure Token Service (More Complex)
-
-**Approach:**
-- Orchestrator creates short-lived access tokens
-- Agents receive token (not credentials)
-- Agents fetch credentials from secure endpoint using token
-- Tokens expire quickly (e.g., 15 minutes)
-
-**Pros:**
-- ✅ Credentials never in conversation
-- ✅ Tokens can be revoked/expired
-- ✅ Better audit trail
-
-**Cons:**
-- ❌ Requires infrastructure (token service, secure endpoint)
-- ❌ More complex implementation
-- ❌ Agents need network access to token service
-
-### Option 3: Cursor Cloud Environment Variables (If Available)
-
-**Approach:**
-- Set environment variables when spawning agents
-- Agents access via `process.env`
-- No credentials in conversation
-
-**Pros:**
-- ✅ Credentials never in conversation
-- ✅ Platform-native approach
-- ✅ Simple for agents
-
-**Cons:**
-- ❌ API doesn't currently support this (tested - rejected)
-- ❌ Would need Cursor Cloud API enhancement
-
-### Option 4: Agents Create Test Credentials
-
-**Approach:**
-- Agents use local SQLite for integration tests (no external credentials)
-- Or use publicly available test credentials
-- No production credentials needed
-
-**Pros:**
-- ✅ No credentials to manage
-- ✅ Isolated test environment
-
-**Cons:**
-- ❌ May have setup complexity for some test scenarios
-- ❌ May have rate limits
-- ❌ More setup complexity
-
-## Recommended Implementation
-
-**For Cloud Agents:** Use Option 1 (Skip Tests)
-
-Update orchestrator to:
-1. NOT include credentials in agent instructions
-2. Instruct agents to skip integration/E2E tests if credentials unavailable
-3. Update status file with skipped test reasons
-4. Manual test execution can happen later with proper credentials
-
-**For Local Development:** Use secrets manager (current approach is fine)
-
-**For CI/CD:** Use platform secrets manager (GitHub Secrets, etc.)
-
-## Migration Plan
-
-1. Update orchestrator to remove credential passing
-2. Update agent instructions to skip tests gracefully
-3. Update status file schema to record skipped tests
-4. Document manual test execution process
-
-
-
-

package/docs/developer/secrets/secrets_setup.md

@@ -1,116 +0,0 @@
-# Secrets Manager Setup
-
-This project uses an encrypted file-based secrets manager for storing sensitive credentials.
-
-## Setup
-
-### 1. Initialize Secrets Storage
-
-The first time you run a secrets command, a new encryption key will be generated:
-
-```bash
-node scripts/secrets_manager.js list
-```
-
-This creates:
-- `.secrets/.key` - Encryption key (DO NOT COMMIT)
-- `.secrets/secrets.enc` - Encrypted secrets file (DO NOT COMMIT)
-
-**Important:** The `.secrets/` directory is in `.gitignore` and should never be committed.
-
-### 2. Store Secrets
-
-Store credentials for cloud agents:
-
-```bash
-# Database credentials
-node scripts/secrets_manager.js set DATABASE_URL "postgresql://user:pass@host:5432/db"
-
-# OpenAI API keys (environment-specific)
-node scripts/secrets_manager.js set DEV_OPENAI_API_KEY "sk-your-dev-key"
-node scripts/secrets_manager.js set PROD_OPENAI_API_KEY "sk-your-prod-key"
-```
-
-### 3. Retrieve Secrets
-
-For use in scripts or agent instructions:
-
-```bash
-# Get a single secret
-node scripts/secrets_manager.js get DATABASE_URL
-
-# List all stored keys
-node scripts/secrets_manager.js list
-
-# Export all secrets as key=value pairs (for agent use)
-node scripts/get_secrets_for_agents.js
-```
-
-### 4. Use with Orchestrator
-
-The orchestrator can retrieve secrets and pass them to agents:
-
-```bash
-# In orchestrator script, use:
-const { execSync } = require('child_process');
-const encodedSecrets = execSync('node scripts/get_secrets_for_agents.js', { encoding: 'utf-8' }).trim();
-```
-
-## Security
-
-### Encryption Key Management
-
-**Local Development:**
-- Encryption key is stored in `.secrets/.key`
-- Keep this file secure (chmod 600)
-- Never commit it to git
-
-**CI/CD / Cloud Agents:**
-- Set `NEOTOMA_SECRETS_MASTER_KEY` environment variable
-- Value should be the hex-encoded encryption key
-- Store this securely in your CI/CD system's secrets manager
-
-**Key Rotation:**
-1. Export all secrets: `node scripts/secrets_manager.js export > secrets_backup.txt`
-2. Delete `.secrets/` directory
-3. Re-run setup to generate new key
-4. Re-import secrets from backup
-5. Update `NEOTOMA_SECRETS_MASTER_KEY` in CI/CD
-
-### Best Practices
-
-1. **Use dedicated test credentials** - Never use production credentials
-2. **Rotate keys regularly** - Monthly rotation recommended
-3. **Limit access** - Only give access to `.secrets/` to authorized users
-4. **Monitor usage** - Check service dashboards for unusual activity
-5. **Never commit secrets** - Always verify `.secrets/` is in `.gitignore`
-
-## Integration with Release Orchestrator
-
-The orchestrator will automatically use the secrets manager if available:
-
-1. Check if secrets exist: `node scripts/secrets_manager.js list`
-2. Retrieve encoded secrets: `node scripts/get_secrets_for_agents.js`
-3. Include in agent instructions (base64-encoded)
-
-This provides better security than plain base64 encoding in conversation text, as:
-- Secrets are encrypted at rest
-- Access requires the encryption key
-- No secrets in conversation history (only encrypted references)
-
-## Migration from Base64 Approach
-
-To migrate existing credentials to the secrets manager:
-
-```bash
-# Export current credentials from .env
-source .env
-node scripts/secrets_manager.js set DATABASE_URL "$DATABASE_URL"
-# ... repeat for other credentials
-```
-
-Then update the orchestrator to use `get_secrets_for_agents.js` instead of reading from `.env` directly.
-
-
-
-