neotoma 0.3.0 → 0.3.2

package/docs/developer/environment/DEV_CONNECTOR_SECRET_KEY_EXPLANATION.md

@@ -1,80 +0,0 @@
-# DEV_CONNECTOR_SECRET_KEY Explanation
-
-## What is it?
-
-`DEV_CONNECTOR_SECRET_KEY` is a cryptographic key used to encrypt and decrypt OAuth tokens stored in the database for external connectors (integrations with third-party services).
-
-## Purpose
-
-When Neotoma connects to external services (like Plaid, Asana, Gmail, etc.) via OAuth, it receives:
-- **Access tokens** - Used to authenticate API requests
-- **Refresh tokens** - Used to obtain new access tokens when they expire
-- **Expiration times** - When tokens need to be refreshed
-
-These tokens are sensitive and must be stored securely. The `DEV_CONNECTOR_SECRET_KEY` is used to encrypt these tokens before storing them in the database.
-
-## How it works
-
-1. **Encryption**: When OAuth tokens are received, they're encrypted using AES-256-GCM encryption with the connector secret key
-2. **Storage**: The encrypted tokens (stored as a "secrets envelope") are saved in the `external_connectors` table in the database
-3. **Decryption**: When tokens are needed for API calls, they're decrypted using the same key
-
-## Requirements
-
-- **Minimum length**: 16 characters
-- **Must remain static**: The key MUST NOT change within an environment
-- **Why?**: If you change the key, all existing encrypted tokens become inaccessible (you can't decrypt them anymore)
-
-## Code Location
-
-The encryption/decryption logic is in `src/services/connectors.ts`:
-- `encryptConnectorSecrets()` - Encrypts OAuth tokens before storage
-- `decryptConnectorSecrets()` - Decrypts tokens when needed
-- `getConnectorSecretKey()` - Gets the key from config and validates it
-
-## Environment Separation
-
-- **Development**: Uses `DEV_CONNECTOR_SECRET_KEY`
-- **Production**: Uses `PROD_CONNECTOR_SECRET_KEY`
-
-This ensures that:
-- Development and production use different keys (security best practice)
-- Tokens encrypted in one environment can't be decrypted in another
-
-## Security Notes
-
-- The key is hashed (SHA-256) before use as the encryption key
-- Uses AES-256-GCM encryption (authenticated encryption)
-- Each encryption uses a random IV (initialization vector) for security
-- The key itself should be stored securely (e.g., in 1Password, not in code)
-
-## Example
-
-```typescript
-// When storing OAuth tokens after OAuth flow
-const secrets: ConnectorSecrets = {
-  accessToken: "ya29.a0AfH6...",
-  refreshToken: "1//0gX...",
-  expiresAt: "2024-01-15T12:00:00Z"
-};
-
-// Encrypted and stored in database
-const encrypted = encryptConnectorSecrets(secrets);
-// Result: {"v":1,"iv":"...","tag":"...","data":"..."}
-
-// Later, when making API calls
-const decrypted = decryptConnectorSecrets(encrypted);
-// Result: { accessToken: "ya29.a0AfH6...", ... }
-```
-
-## Adding to 1Password
-
-You should add `DEV_CONNECTOR_SECRET_KEY` to 1Password as a secure note or password field. Generate a random string that's at least 16 characters long (longer is better for security).
-
-Example generation:
-```bash
-# Generate a random 32-character key
-openssl rand -base64 24
-```
-
-Then add it to your 1Password sync mappings so it can be automatically synced to your `.env` file.

package/docs/developer/environment/ENV_MAPPINGS_STATUS.md

@@ -1,86 +0,0 @@
-# Environment Variable Mappings Status
-
-Generated: 2026-01-08
-
-## Summary
-
-- **Total mappings in parquet**: 19
-- **Neotoma-specific mappings**: 0
-- **Other service mappings**: 19 (Coinbase, Asana, Twilio, Wise, Bitcoin, OpenAI, Minted)
-
-## Required Neotoma Variables Status
-
-### ✅ Available and Configured
-- `OPENAI_API_KEY` - Exists with environment-based keys (dev/prod)
-  - ✅ **Code updated**: Now uses `OPENAI_API_KEY` (with fallback to `DEV_OPENAI_API_KEY`/`PROD_OPENAI_API_KEY` for backward compatibility)
-  - The 1Password sync script will set `OPENAI_API_KEY` based on the `ENVIRONMENT` variable
-
-### ❌ Missing Required Variables (Based on src/config.ts)
-
-#### Connector Configuration (REQUIRED when using OAuth connectors)
-- `DEV_CONNECTOR_SECRET_KEY` - **MISSING** ⚠️ **CRITICAL**
-- `PROD_CONNECTOR_SECRET_KEY` - Missing (optional, if using production)
-
-#### OpenAI Configuration (RECOMMENDED - Required for embeddings/AI features)
-- `OPENAI_API_KEY` - ✅ **AVAILABLE** (environment-based, set by 1Password sync)
-  - Code now uses `OPENAI_API_KEY` with fallback to `DEV_OPENAI_API_KEY`/`PROD_OPENAI_API_KEY`
-  - The sync script automatically selects dev/prod key based on `ENVIRONMENT` variable
-
-#### Plaid Configuration (OPTIONAL - Only if using Plaid integration)
-- `DEV_PLAID_CLIENT_ID` - Missing (optional)
-- `DEV_PLAID_SECRET` - Missing (optional)
-- `PROD_PLAID_CLIENT_ID` - Missing (optional)
-- `PROD_PLAID_SECRET` - Missing (optional)
-
-## Excluded Variables (Correctly Not Mapped)
-
-These are intentionally excluded via `foundation-config.yaml` as they're local development values:
-- `PORT`, `HTTP_PORT`, `WS_PORT`
-- `ACTIONS_BEARER_TOKEN`
-- `VITE_*` variables
-- `NEOTOMA_ENV`, `NODE_ENV`
-- `MCP_CMD`, `MCP_ARGS`
-- `PLAID_ENV`, `PLAID_ENVIRONMENT`, `PLAID_PRODUCTS`, `PLAID_COUNTRY_CODES`
-- `NEOTOMA_ACTIONS_DISABLE_AUTOSTART`, `BRANCH_PORTS_FILE`
-
-## Action Items
-
-### 🔴 Critical Priority (Required for Neotoma to work)
-These must be added when using OAuth connectors:
-1. **Add `DEV_CONNECTOR_SECRET_KEY` mapping** - Required when using OAuth connectors
-
-### 🟡 High Priority (Recommended for full functionality)
-4. ✅ **OPENAI_API_KEY** - Already configured and code updated to use it
-   - The existing `OPENAI_API_KEY` mapping works with environment-based selection
-   - Code now uses `OPENAI_API_KEY` (with backward compatibility fallback)
-
-### 🟢 Medium Priority (Optional but useful)
-5. Add production environment mappings if deploying to production:
-   - `PROD_CONNECTOR_SECRET_KEY`
-   - Note: `OPENAI_API_KEY` already handles production via environment-based selection
-
-### 🔵 Low Priority (Feature-specific)
-7. Add Plaid mappings if using Plaid integration:
-   - `DEV_PLAID_CLIENT_ID`
-   - `DEV_PLAID_SECRET`
-
-## How to Add Missing Mappings
-
-See [SETUP_ENV_MAPPINGS.md](SETUP_ENV_MAPPINGS.md) for detailed instructions.
-
-Quick example:
-```javascript
-// Add DEV_CONNECTOR_SECRET_KEY
-{
-  "data_type": "env_var_mappings",
-  "record": {
-    "env_var": "DEV_CONNECTOR_SECRET_KEY",
-    "op_reference": "op://Private/your-item-id/field-id",
-    "vault": "Private",
-    "item_name": "Neotoma Connector",
-    "service": "Connector",
-    "is_optional": false,
-    "environment_based": false
-  }
-}
-```

package/docs/developer/environment/ENV_VAR_NAMING_STRATEGY.md

@@ -1,145 +0,0 @@
-# Environment Variable Naming Strategy: Prefixes vs Environment-Based Selection
-
-## The Question
-
-Should we use:
-- **Current**: `DEV_CONNECTOR_SECRET_KEY` and `PROD_CONNECTOR_SECRET_KEY` (different variable names)
-- **Proposed**: `CONNECTOR_SECRET_KEY` (same variable name, different values per environment)
-
-## Current State
-
-### Foundation Script Support
-✅ **The foundation script already supports environment-based selection!**
-
-The `OPENAI_API_KEY` mapping demonstrates this:
-- Same variable name: `OPENAI_API_KEY`
-- Different 1Password references based on `ENVIRONMENT` variable
-- Uses `environment_based: true` and `environment_key: "development"` or `"production"`
-
-### Codebase Current Approach
-The codebase currently uses DEV_/PROD_ prefixes for some credentials:
-- `DEV_CONNECTOR_SECRET_KEY` / `PROD_CONNECTOR_SECRET_KEY`
-- `DEV_OPENAI_API_KEY` / `PROD_OPENAI_API_KEY` (with fallback to `OPENAI_API_KEY`)
-
-**Rationale in code comments**: "to prevent accidental dev/prod mixups"
-
-## Comparison
-
-### Approach 1: DEV_/PROD_ Prefixes (Current)
-
-**Pros:**
-- ✅ **Explicit safety**: Can't accidentally use wrong environment's credentials
-- ✅ **Can have both in same .env**: Useful for testing/debugging
-- ✅ **Clear intent**: Variable name shows which environment it's for
-- ✅ **Prevents mistakes**: Harder to accidentally mix dev/prod
-- ✅ **Works without ENVIRONMENT variable**: No dependency on runtime env detection
-
-**Cons:**
-- ❌ More verbose code (environment checks)
-- ❌ More variables to manage
-- ❌ More mappings in parquet file
-- ❌ Code must check environment and select variable
-
-### Approach 2: Single Variable Name (Proposed)
-
-**Pros:**
-- ✅ Simpler code (no environment checks needed)
-- ✅ Standard practice in many systems (12-factor app)
-- ✅ Less variables to manage
-- ✅ Foundation script already supports this via `environment_based`
-- ✅ Environment selection happens at sync time, not runtime
-
-**Cons:**
-- ❌ **Risk of wrong value**: If `ENVIRONMENT` is wrong, you get wrong credentials
-- ❌ Can't have both dev and prod in same `.env` file
-- ❌ Less explicit (variable name doesn't show environment)
-- ❌ Requires `ENVIRONMENT` variable to be set correctly
-- ❌ If you sync with wrong `ENVIRONMENT`, you get wrong values
-
-## Recommendation
-
-### Hybrid Approach (Best of Both Worlds)
-
-Use **single variable names** for most cases, but keep **explicit prefixes** for critical credentials where safety is paramount.
-
-**Use single names for:**
-- `OPENAI_API_KEY` ✅ (already done)
-- `PLAID_CLIENT_ID` / `PLAID_SECRET` (if using Plaid)
-
-**Keep prefixes for:**
-- `DEV_CONNECTOR_SECRET_KEY` / `PROD_CONNECTOR_SECRET_KEY` ⚠️ (critical, encrypts tokens)
-
-**Why?**
-- Connector secrets are **high-risk** if wrong
-- Mixing dev/prod for these could cause data corruption or security issues
-- The explicit prefix provides an extra safety layer
-- Less critical variables (like API keys) can use environment-based selection
-
-## Migration Path
-
-If you want to migrate to single variable names:
-
-### Step 1: Update Parquet Mappings
-Use `environment_based: true` with same `env_var` name:
-
-```json
-{
-  "env_var": "CONNECTOR_SECRET_KEY",
-  "op_reference": "op://Private/connector-item/dev-key",
-  "environment_based": true,
-  "environment_key": "development"
-}
-```
-
-### Step 2: Update Code
-Simplify `src/config.ts` to use single variable names where applicable.
-
-### Step 3: Set ENVIRONMENT Variable
-Ensure `ENVIRONMENT=development` or `ENVIRONMENT=production` is set when syncing.
-
-## Current Recommendation for Neotoma
-
-**Keep the DEV_/PROD_ prefixes** for now because:
-
-1. **Safety first**: The explicit prefixes prevent accidental credential mixing
-2. **Already implemented**: The codebase is built around this pattern
-3. **Works without ENVIRONMENT**: No dependency on `ENVIRONMENT` variable being set
-4. **Clear intent**: Variable names make it obvious which environment they're for
-5. **Can test both**: Can have dev and prod values in same `.env` for testing
-
-**Exception**: `OPENAI_API_KEY` already uses environment-based selection, which is fine since it's less critical than database credentials.
-
-## If You Do Migrate
-
-If you decide to migrate to single variable names:
-
-1. ✅ Use `environment_based: true` in parquet mappings
-2. ✅ Set `ENVIRONMENT` variable before syncing
-3. ✅ Update code to use single variable names
-4. ✅ Update documentation
-5. ⚠️ Be extra careful about `ENVIRONMENT` variable value
-6. ⚠️ Consider keeping prefixes for critical credentials (Connector)
-
-## Foundation Script Behavior
-
-The foundation script checks `ENVIRONMENT` variable (defaults to "development"):
-
-```python
-current_env = os.getenv("ENVIRONMENT", "development").lower()
-
-if record.get("environment_based", False):
-    env_key = str(record.get("environment_key", "")).lower()
-    if env_key != current_env:
-        continue  # Skip this mapping
-```
-
-So if you set `ENVIRONMENT=production` before running `npm run sync:env`, it will select production values.
-
-## Conclusion
-
-**For Neotoma**: Keep DEV_/PROD_ prefixes for critical credentials (Connector) for safety. Use single variable names for less critical ones (OpenAI API key) where the foundation script already supports it.
-
-This gives you:
-- Safety for critical credentials
-- Simplicity for less critical ones
-- Best of both worlds

package/docs/developer/environment/MIGRATION_TO_SINGLE_VAR_NAMES.md

@@ -1,89 +0,0 @@
-# Migration to Single Variable Names
-
-This document tracks the migration from `DEV_*/PROD_*` prefixed environment variables to single variable names with environment-based selection.
-
-## Status: ✅ Code Updated
-
-The codebase has been updated to support single variable names with backward compatibility:
-
-### Updated Variables
-
-- `CONNECTOR_SECRET_KEY` (replaces `DEV_CONNECTOR_SECRET_KEY` / `PROD_CONNECTOR_SECRET_KEY`)
-- `OPENAI_API_KEY` (already using environment-based selection)
-
-### How It Works
-
-1. **1Password Sync**: The sync script reads `ENVIRONMENT` variable (defaults to "development")
-2. **Environment Selection**: Mappings with `environment_based: true` and matching `environment_key` are selected
-3. **Variable Setting**: The selected 1Password value is set to the single variable name (e.g., `CONNECTOR_SECRET_KEY`)
-4. **Code Usage**: Code reads the single variable name, no environment checks needed
-
-### Backward Compatibility
-
-The code still supports `DEV_*/PROD_*` prefixed variables as fallback:
-- If `CONNECTOR_SECRET_KEY` is set, use it
-- Otherwise, fall back to `DEV_CONNECTOR_SECRET_KEY` or `PROD_CONNECTOR_SECRET_KEY` based on `NEOTOMA_ENV`
-
-## Next Steps: Add Parquet Mappings
-
-You need to add parquet mappings with `environment_based: true` for:
-
-1. **CONNECTOR_SECRET_KEY**
-   - ⚠️ **Note**: This key is now generated locally, not synced from 1Password
-   - Run `npm run generate:connector-key` to generate it
-   - See [CONNECTOR_SECRET_KEY Setup](CONNECTOR_SECRET_KEY_SETUP.md) for details
-
-### Example Mapping Format
-
-```json
-{
-  "env_var": "CONNECTOR_SECRET_KEY",
-  "op_reference": "op://Private/connector-item/dev-key-field",
-  "vault": "Private",
-  "item_name": "Neotoma Connector",
-  "service": "Connector",
-  "is_optional": false,
-  "environment_based": true,
-  "environment_key": "development"
-}
-```
-
-### Finding 1Password References
-
-Use `op item get "Neotoma Connector" --format=json` to find the exact field references for connector secret keys.
-
-## Usage
-
-### Setting Environment Before Sync
-
-**Default**: If `ENVIRONMENT` is not set, it defaults to `"development"`
-
-```bash
-# For development (default - no need to set ENVIRONMENT)
-npm run sync:env
-
-# Or explicitly:
-export ENVIRONMENT=development
-npm run sync:env
-
-# For production (must set ENVIRONMENT)
-export ENVIRONMENT=production
-npm run sync:env
-```
-
-### Result
-
-After syncing, your `.env` will have:
-```bash
-CONNECTOR_SECRET_KEY=<value-from-1password>
-```
-
-The code will automatically use these values based on the `ENVIRONMENT` variable set during sync.
-
-## Benefits
-
-1. **Simpler Code**: No environment checks needed in code
-2. **Standard Practice**: Follows 12-factor app principles
-3. **Less Variables**: One variable name instead of two
-4. **Environment Selection**: Happens at sync time, not runtime
-5. **Backward Compatible**: Old `DEV_*/PROD_*` variables still work

package/docs/developer/environment/QUICK_START_1PASSWORD.md

@@ -1,126 +0,0 @@
-# Quick Start: 1Password Environment Variable Sync
-
-## ✅ Setup Complete
-
-All infrastructure is in place:
-- ✅ Foundation script configured
-- ✅ Wrapper script created (`scripts/sync-env-from-1password.sh`)
-- ✅ NPM script added (`npm run sync:env`)
-- ✅ Configuration updated (`foundation-config.yaml`)
-- ✅ Documentation created
-
-## 🚀 Quick Start (3 Steps)
-
-### 1. Sign in to 1Password CLI
-
-```bash
-op signin
-# or if using desktop app integration:
-eval $(op signin)
-```
-
-### 2. Add Environment Variable Mappings
-
-You need to map your Neotoma environment variables to 1Password references. See [SETUP_ENV_MAPPINGS.md](SETUP_ENV_MAPPINGS.md) for detailed instructions.
-
-**Quick example using MCP tools:**
-```javascript
-// Add a mapping for DEV_CONNECTOR_SECRET_KEY
-{
-  "data_type": "env_var_mappings",
-  "record": {
-    "env_var": "DEV_CONNECTOR_SECRET_KEY",
-    "op_reference": "op://Private/your-item-id/field-id",
-    "vault": "Private",
-    "item_name": "Neotoma Connector",
-    "service": "Connector",
-    "is_optional": false
-  }
-}
-```
-
-**Required variables to map (when using OAuth connectors):**
-- `DEV_CONNECTOR_SECRET_KEY`
-- `DEV_OPENAI_API_KEY` (or use existing `OPENAI_API_KEY` mappings)
-- `DEV_PLAID_CLIENT_ID` (if using Plaid)
-- `DEV_PLAID_SECRET` (if using Plaid)
-
-### 3. Run the Sync
-
-```bash
-npm run sync:env
-```
-
-**Default**: Syncs **development** environment values (no `ENVIRONMENT` variable needed)
-
-For production:
-```bash
-export ENVIRONMENT=production
-npm run sync:env
-```
-
-This will:
-- Create a backup of your `.env` file
-- Fetch values from 1Password (based on `ENVIRONMENT`, defaults to "development")
-- Update your `.env` file with single variable names (REMOTE_DB_PROJECT_ID, REMOTE_DB_SERVICE_KEY, etc.)
-- Preserve any unmanaged variables
-
-## 📚 Documentation
-
-- **Main guide**: [README_1PASSWORD_SYNC.md](README_1PASSWORD_SYNC.md) - Complete usage documentation
-- **Setup guide**: [SETUP_ENV_MAPPINGS.md](SETUP_ENV_MAPPINGS.md) - How to add mappings
-- **Foundation script**: `foundation/scripts/op_sync_env_from_1password.py` - The actual sync script
-
-## 🔧 Configuration
-
-### Exclusions
-
-Local development variables are automatically excluded (ports, URLs, etc.). To add more exclusions:
-
-**Repo-wide** (edit `foundation-config.yaml`):
-```yaml
-tooling:
-  env_management:
-    onepassword_sync:
-      default_exclusions:
-        - YOUR_LOCAL_VAR
-```
-
-**Instance-specific** (create `foundation-config.local.yaml` - gitignored):
-```yaml
-tooling:
-  env_management:
-    onepassword_sync:
-      exclusions:
-        - MY_PERSONAL_VAR
-```
-
-## 🛠️ Troubleshooting
-
-### "1Password CLI is not authenticated"
-```bash
-op signin
-```
-
-### "No environment variable mappings found"
-- Add mappings using MCP parquet tools (see `SETUP_ENV_MAPPINGS.md`)
-- Ensure `op_reference` doesn't start with `PLACEHOLDER_`
-
-### "Failed to resolve [variable]"
-- Verify the 1Password reference is correct
-- Check you have access to the vault/item
-- Ensure the field exists in 1Password
-
-## 📝 Next Steps
-
-1. **Sign in to 1Password**: `op signin`
-2. **Add mappings**: See `SETUP_ENV_MAPPINGS.md` for the list of required variables
-3. **Test sync**: Run `npm run sync:env` to verify everything works
-4. **Regular use**: Run `npm run sync:env` whenever you need to update your `.env` from 1Password
-
-## 💡 Tips
-
-- Backups are automatically created in `.env.backups/` before each sync
-- The script never prints secret values, only variable names
-- Unmanaged variables (not in mappings) are preserved
-- Placeholder mappings (starting with `PLACEHOLDER_`) are skipped until configured

package/docs/developer/environment/README.md

@@ -1,27 +0,0 @@
-# Environment Variable Management
-
-This directory contains documentation for managing environment variables in the Neotoma project, including 1Password integration and configuration strategies.
-
-## Quick Start
-
-- **[Quick Start Guide](QUICK_START_1PASSWORD.md)** - Get started with 1Password sync in 3 steps
-- **[Main Documentation](README_1PASSWORD_SYNC.md)** - Complete usage guide for 1Password sync
-- **[Setup Environment Mappings](SETUP_ENV_MAPPINGS.md)** - How to add mappings for your environment variables
-
-## Feature Documentation
-
-- **[Inclusion List Feature](INCLUSION_LIST_FEATURE.md)** - How to use inclusion lists (whitelist)
-- **[Inclusion vs Exclusion Lists](INCLUSION_VS_EXCLUSION.md)** - Understanding the difference
-- **[Environment Variable Naming Strategy](ENV_VAR_NAMING_STRATEGY.md)** - DEV_/PROD_ prefixes vs environment-based selection
-
-## Reference
-
-- **[Environment Mappings Status](ENV_MAPPINGS_STATUS.md)** - Current status of required variables
-- **[CONNECTOR_SECRET_KEY Setup](CONNECTOR_SECRET_KEY_SETUP.md)** - How to generate the connector secret key locally
-- **[DEV_CONNECTOR_SECRET_KEY Explanation](DEV_CONNECTOR_SECRET_KEY_EXPLANATION.md)** - What this key is for and why it's needed
-
-## Related Documentation
-
-- **Configuration**: `foundation-config.yaml` - Main configuration file
-- **Scripts**: `scripts/sync-env-from-1password.sh` - Sync wrapper script
-- **Foundation Script**: `foundation/scripts/op_sync_env_from_1password.py` - The actual sync script