neotoma 0.3.0 → 0.3.2

package/docs/developer/mcp_https_testing_status.md

@@ -1,130 +0,0 @@
-# MCP HTTPS Testing Status
-
-**Date:** 2026-01-27  
-**Status:** ✅ **READY FOR TESTING**
-
-## Configuration Complete
-
-### 1. ngrok Setup ✅
-- **Authtoken configured:** `NGROK_AUTHTOKEN` synced from 1Password
-- **Tunnel active:** `https://melissia-introrse-correspondently.ngrok-free.dev`
-- **Local port:** 8080
-- **Tunnel PID:** Stored in `/tmp/ngrok-mcp.pid`
-
-### 2. Cursor Configuration ✅
-- **`.cursor/mcp.json` updated** with HTTPS URL:
-  ```json
-  {
-    "mcpServers": {
-      "neotoma": {
-        "url": "https://melissia-introrse-correspondently.ngrok-free.dev/mcp"
-      }
-    }
-  }
-  ```
-
-### 3. Server Configuration ⚠️
-- **MCP server running:** Port 8080 (confirmed)
-- **NEOTOMA_HOST_URL:** Set to HTTPS URL for OAuth redirects (or rely on auto-discovery from tunnel file)
-- **Action required:** Restart MCP server with `NEOTOMA_HOST_URL` environment variable if not using auto-discovery
-
-## Next Steps
-
-### Step 1: Restart MCP Server with HTTPS URL
-
-**Stop current server** (if running in terminal, press Ctrl+C), then:
-
-```bash
-# Set API_BASE_URL to ngrok HTTPS URL
-export NEOTOMA_HOST_URL="https://melissia-introrse-correspondently.ngrok-free.dev"
-
-# Restart MCP server
-npm run dev:api
-```
-
-**Or add to `.env` file:**
-```bash
-echo 'NEOTOMA_HOST_URL="https://melissia-introrse-correspondently.ngrok-free.dev"' >> .env
-```
-
-Then restart the server.
-
-### Step 2: Restart Cursor
-
-1. **Quit Cursor completely** (Cmd+Q on Mac)
-2. **Reopen Cursor** to load new MCP configuration
-3. **Check MCP status** - should show "Authentication needed" with Connect button
-
-### Step 3: Test Connect Button
-
-1. **Open Cursor MCP settings** or check MCP status
-2. **Verify** "Authentication needed" message appears
-3. **Click Connect button** (should appear with HTTPS URL)
-4. **Complete OAuth flow** in browser
-5. **Verify** connection is established
-
-## Tunnel Management
-
-### Check Tunnel Status
-```bash
-# View ngrok web interface
-open http://localhost:4040
-
-# Check tunnel is running
-ps aux | grep "[n]grok http"
-
-# Get tunnel URL
-cat /tmp/ngrok-mcp-url.txt
-```
-
-### Stop Tunnel
-```bash
-# Kill ngrok process
-kill $(cat /tmp/ngrok-mcp.pid 2>/dev/null) 2>/dev/null || pkill -f "ngrok http"
-rm -f /tmp/ngrok-mcp.pid /tmp/ngrok-mcp-url.txt
-```
-
-### Restart Tunnel
-```bash
-npm run tunnel:https
-```
-
-## Troubleshooting
-
-### Connect Button Still Not Appearing
-
-1. **Verify HTTPS URL is correct** in `.cursor/mcp.json`
-2. **Verify NEOTOMA_HOST_URL is set** and server restarted (or rely on auto-discovery)
-3. **Check tunnel is active:** `curl https://melissia-introrse-correspondently.ngrok-free.dev/.well-known/oauth-authorization-server`
-4. **Clear Cursor's MCP cache:** Remove and re-add server
-5. **Check server logs** for 401 responses
-
-### OAuth Redirect Fails
-
-- **Verify NEOTOMA_HOST_URL** matches ngrok URL exactly
-- **Check discovery endpoints** return HTTPS URLs:
-  ```bash
-  curl https://melissia-introrse-correspondently.ngrok-free.dev/.well-known/oauth-authorization-server
-  ```
-
-### Tunnel URL Changes
-
-**Note:** ngrok free tier assigns new URLs on each restart. If you restart ngrok:
-1. Update `.cursor/mcp.json` with new URL
-2. Update `NEOTOMA_HOST_URL` environment variable
-3. Restart MCP server
-4. Restart Cursor
-
-## Current Configuration
-
-- **HTTPS URL:** `https://melissia-introrse-correspondently.ngrok-free.dev`
-- **MCP Endpoint:** `https://melissia-introrse-correspondently.ngrok-free.dev/mcp`
-- **Tunnel Status:** Active (PID in `/tmp/ngrok-mcp.pid`)
-- **Server Status:** Running on port 8080
-- **NEOTOMA_HOST_URL:** Set to HTTPS URL (or use auto-discovery)
-
-## References
-
-- HTTPS Testing Guide: `docs/developer/mcp_https_testing.md`
-- Troubleshooting: `docs/developer/mcp_connect_button_troubleshooting.md`
-- Notion MCP Example: `https://mcp.notion.com/mcp`

package/docs/developer/mcp_https_tunnel_status.md

@@ -1,147 +0,0 @@
-# MCP HTTPS Tunnel Status
-
-**Date:** 2026-01-27  
-
-**Full reference:** For provider selection, scripts, env vars, install, and troubleshooting, see [tunnels.md](tunnels.md).
-
-**Example URLs:** ngrok free tier `https://<random>.ngrok-free.dev` (new URL each run). Cloudflare quick tunnel `https://<random>.trycloudflare.com`. Both write the current URL to `/tmp/ngrok-mcp-url.txt`.
-
-## Current Status
-
-### When the tunnel is running
-
-1. **Tunnel (Cloudflare or ngrok):** PID in `/tmp/ngrok-mcp.pid`, forwards HTTPS to `http://localhost:8080` (or 8180 for prod). ngrok only: UI at `http://localhost:4040`.
-2. **Local MCP server:** Port 8080 (or 8180), `/mcp` returns 401 when unauthenticated, discovery at `/.well-known/oauth-authorization-server`.
-3. **Config:** `NEOTOMA_HOST_URL` set to the **current** tunnel URL (or server auto-discovers from `/tmp/ngrok-mcp-url.txt`). Cursor / `.cursor/mcp.json` use `https://<tunnel>/mcp`.
-
-### "Connection refused" or "refuses to connect"
-
-**Symptom:** `https://...ngrok-free.dev/mcp` (or the tunnel URL) refuses to connect (browser ERR_CONNECTION_REFUSED, Cursor fetch failed, `curl` HTTP 000).
-
-**Most likely cause: tunnel not running.** The tunnel URL only works while the tunnel is active. Free-tier URLs (ngrok, Cloudflare quick tunnel) change every run.
-
-**Causes and fixes:**
-
-1. **Tunnel not running** – You stopped the terminal running `npm run tunnel:https` or ngrok died. **Fix:** Start the tunnel again and use the **new** URL it prints.
-2. **Stale URL** – You’re still using an old URL from a previous run. **Fix:** Run `npm run tunnel:https`, copy the current HTTPS URL, update `NEOTOMA_HOST_URL` (and Cursor).
-3. **ngrok free-tier warning** – Open the tunnel URL in a browser, accept "Visit Site" if shown, then retry.
-4. **Network/firewall** – VPN or firewall blocking ngrok.
-5. **Tunnel still starting** – Wait 10–30 seconds after starting ngrok.
-6. **ngrok domain resolves to 127.0.0.1** – `curl -v` shows `IPv4: 127.0.0.1` and connection refused to localhost:443. A local DNS override (AdGuard, Pi-hole, NextDNS app, VPN, etc.) or stale cache is mapping the ngrok host to localhost. **Fix:** Flush DNS (`sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder`), then check for apps that override DNS and disable or exclude `*.ngrok-free.dev`. Confirm with `dscacheutil -q host -a name <tunnel-host>`; IPv4 should be a public IP (e.g. 3.125.x.x), not 127.0.0.1.
-
-## Troubleshooting Steps
-
-### Step 0: Start the tunnel
-
-The tunnel must be running for the HTTPS URL to work. **`npm run dev:api` starts both the server and the tunnel** in one terminal (tunnel and API/MCP run together; Ctrl+C stops both).
-
-```bash
-npm run dev:api
-```
-
-Use the URL the tunnel script prints (or `cat /tmp/ngrok-mcp-url.txt`). The server auto-discovers it from that file, or set `NEOTOMA_HOST_URL` to it; then update Cursor config. For server-only (no tunnel), use `npm run dev:server`. For tunnel-only (e.g. server already running elsewhere), use `npm run tunnel:https` in a separate terminal.
-
-### Step 1: Verify tunnel is active
-
-```bash
-# Tunnel URL (both providers)
-cat /tmp/ngrok-mcp-url.txt
-
-# ngrok only: API and web UI
-curl -s http://localhost:4040/api/tunnels | python3 -m json.tool
-open http://localhost:4040
-
-# Cloudflare only: log
-tail -f /tmp/cloudflared-tunnel.log
-```
-
-### Step 2: Test local server
-
-```bash
-# Verify local server responds
-curl http://localhost:8080/.well-known/oauth-authorization-server
-
-# Should return JSON with OAuth endpoints
-```
-
-### Step 3: Accept ngrok warning page (free tier)
-
-1. Open the **current** tunnel URL in a browser (from `npm run tunnel:https` or `cat /tmp/ngrok-mcp-url.txt`).
-2. Accept "Visit Site" if shown.
-3. Retry Cursor or `curl` to the tunnel URL.
-
-### Step 4: ngrok resolves to 127.0.0.1
-
-If `curl -v` shows `IPv4: 127.0.0.1` for the tunnel host:
-
-```bash
-# Flush DNS cache
-sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
-
-# Check resolution (IPv4 should be a public IP, not 127.0.0.1)
-dscacheutil -q host -a name melissia-introrse-correspondently.ngrok-free.dev
-```
-
-If IPv4 is still 127.0.0.1, a local DNS override (AdGuard, Pi-hole, VPN, etc.) is redirecting the ngrok domain. Disable or exclude `*.ngrok-free.dev` in that app. As a quick check, force the real IP:
-
-```bash
-curl -s -o /dev/null -w "%{http_code}\n" -4 \
-  --resolve "melissia-introrse-correspondently.ngrok-free.dev:443:3.125.223.134" \
-  -H "ngrok-skip-browser-warning: 1" \
-  "https://melissia-introrse-correspondently.ngrok-free.dev/mcp"
-```
-
-Expect 401. Use your tunnel's actual IP from `nslookup <tunnel-host>` if different.
-
-### Step 5: Check ngrok logs
-
-```bash
-tail -f /tmp/ngrok.log
-grep -i error /tmp/ngrok.log
-```
-
-## Provider options
-
-- **Cloudflare:** Preferred when `cloudflared` is installed. No account required for quick tunnels. Install: `brew install cloudflare/cloudflare/cloudflared`. Force: `NEOTOMA_TUNNEL_PROVIDER=cloudflare`.
-- **ngrok:** Used when Cloudflare is not installed and ngrok is installed and authenticated. Free tier: new URL each run, possible "Visit Site" page. Paid: static domains. Force: `NEOTOMA_TUNNEL_PROVIDER=ngrok`.
-
-See [tunnels.md](tunnels.md) for full provider selection, install, and verification.
-
-## Configuration (use current tunnel URL)
-
-- **Tunnel URL:** From `npm run tunnel:https` output or `cat /tmp/ngrok-mcp-url.txt`
-- **MCP endpoint:** `https://<your-tunnel-url>/mcp`
-- **Local server:** `http://localhost:8080`
-- **Env:** `NEOTOMA_HOST_URL` set to tunnel URL (or rely on auto-discovery)
-- **Cursor:** `url` in config = `https://<your-tunnel-url>/mcp`
-
-## Next steps
-
-1. **Start tunnel** → copy URL → set `NEOTOMA_HOST_URL` (or let server auto-discover) → restart MCP server.
-2. **Visit tunnel URL in browser** to accept warning page (free tier) if needed.
-3. **Restart Cursor** and test Connect.
-
-## Verification commands
-
-```bash
-# Tunnel URL (both providers)
-cat /tmp/ngrok-mcp-url.txt
-
-# ngrok only: tunnel status
-curl -s http://localhost:4040/api/tunnels | python3 -m json.tool
-
-# Local server
-curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/mcp   # expect 401
-
-# Through tunnel (replace with your current URL; ngrok free tier may need header)
-curl -s -H "ngrok-skip-browser-warning: 1" "https://YOUR-TUNNEL-URL/.well-known/oauth-authorization-server"
-
-# Processes
-lsof -i :8080
-ps aux | grep "[n]grok http"
-ps aux | grep "[c]loudflared"
-```
-
-## Related
-
-- [tunnels.md](tunnels.md) — Full tunnel documentation (providers, scripts, env, troubleshooting)

package/docs/developer/mcp_oauth_migration_guide.md

@@ -1,196 +0,0 @@
-# MCP OAuth Migration Guide
-
-## Overview
-
-This guide helps you migrate from session token authentication (deprecated) to OAuth 2.0 for MCP connections.
-
-## Why Migrate?
-
-**OAuth Benefits:**
-- No manual token copying or updating
-- Tokens automatically refresh (no expiration interruptions)
-- More secure (PKCE prevents code interception)
-- Better user experience
-- Manage multiple MCP connections
-- Revoke connections anytime
-
-**Session Token Limitations:**
-- Tokens expire when you sign out
-- Requires manual copying from web UI
-- Must update MCP config when token expires
-- No automatic refresh
-- Less secure for long-lived connections
-
-## Migration Steps
-
-### Step 1: Create OAuth Connection
-
-1. **Sign in** to Neotoma web UI (http://localhost:5195)
-2. **Navigate** to MCP Setup page (click "MCP Setup" button or go to `/mcp-setup`)
-3. **Click** "OAuth Connection" tab
-4. **Generate** or enter a connection ID:
-   - Click "Generate" for auto-generated ID
-   - Or enter your own (e.g., `cursor-2025-01-21-abc123`)
-5. **Click** "Start OAuth Flow"
-6. **Click** "Open Authorization Page" button
-7. **Approve** the connection in your browser
-8. **Wait** for confirmation (page will show "Connection active")
-9. **Copy** your connection ID (you'll need this for Step 2)
-
-### Step 2: Update MCP Client Configuration
-
-#### For Cursor
-
-Edit `.cursor/mcp.json`:
-
-**Before (Session Token):**
-```json
-{
-  "mcpServers": {
-    "neotoma": {
-      "command": "node",
-      "args": ["dist/index.js"],
-      "cwd": "/absolute/path/to/neotoma",
-      "env": {
-        "NEOTOMA_SESSION_TOKEN": "eyJhbGci..."
-      }
-    }
-  }
-}
-```
-
-**After (OAuth):**
-```json
-{
-  "mcpServers": {
-    "neotoma": {
-      "command": "node",
-      "args": ["dist/index.js"],
-      "cwd": "/absolute/path/to/neotoma",
-      "env": {
-        "NEOTOMA_CONNECTION_ID": "cursor-2025-01-21-abc123"
-      }
-    }
-  }
-}
-```
-
-#### For Claude Code
-
-Edit `claude_desktop_config.json`:
-
-**Before (Session Token):**
-```json
-{
-  "mcpServers": {
-    "neotoma": {
-      "command": "/absolute/path/to/node",
-      "args": ["/absolute/path/to/neotoma/dist/index.js"],
-      "env": {
-        "NEOTOMA_SESSION_TOKEN": "eyJhbGci..."
-      }
-    }
-  }
-}
-```
-
-**After (OAuth):**
-```json
-{
-  "mcpServers": {
-    "neotoma": {
-      "command": "/absolute/path/to/node",
-      "args": ["/absolute/path/to/neotoma/dist/index.js"],
-      "env": {
-        "NEOTOMA_CONNECTION_ID": "claude-2025-01-21-abc123"
-      }
-    }
-  }
-}
-```
-
-### Step 3: Restart Your AI Assistant
-
-- **Cursor**: Quit and restart completely
-- **Claude Code**: Quit and restart completely
-
-### Step 4: Test the Connection
-
-Try using an MCP action in your AI assistant:
-```
-Can you list my Neotoma entities via MCP?
-```
-
-If successful, the migration is complete.
-
-### Step 5: Remove Old Session Token (Optional)
-
-Once OAuth is working, you can remove the `NEOTOMA_SESSION_TOKEN` line from your MCP config.
-
-## Troubleshooting
-
-### "OAuth connection failed: Connection not found"
-
-**Solution:**
-1. Verify connection exists in web UI (MCP Setup → OAuth Connection tab)
-2. Check connection status is "active"
-3. Verify connection ID matches exactly (case-sensitive)
-
-### "Authentication required"
-
-**Solution:**
-1. Verify `NEOTOMA_CONNECTION_ID` is set in MCP config
-2. Check connection hasn't been revoked
-3. Restart AI assistant after updating config
-
-### "Failed to refresh access token"
-
-**Solution:**
-1. Create new OAuth connection via web UI
-2. Update MCP config with new connection ID
-3. Old connection may have expired or been revoked
-
-### MCP Server Not Starting
-
-**Solution:**
-1. Check that `MCP_TOKEN_ENCRYPTION_KEY` is set in `.env`
-2. Generate key with: `openssl rand -hex 32`
-3. Add to `.env`: `MCP_TOKEN_ENCRYPTION_KEY=<output>`
-
-## Managing Connections
-
-### List Your Connections
-
-1. Go to Neotoma web UI → MCP Setup
-2. Scroll to "MCP Connections" section
-3. View all active connections with details
-
-### Revoke a Connection
-
-1. Go to MCP Setup → MCP Connections section
-2. Find the connection you want to revoke
-3. Click "Revoke" button
-4. Confirm revocation
-5. Create new connection if needed
-
-### Multiple Connections
-
-You can have multiple OAuth connections (e.g., one for Cursor, one for Claude):
-- Create separate connections with different IDs
-- Use appropriate connection ID in each MCP client
-- All connections work independently
-
-## Backward Compatibility
-
-Session tokens are still supported but deprecated:
-- Existing session token configs will continue to work
-- Deprecation warnings appear in logs
-- Session tokens will be removed in a future major version
-- Migrate to OAuth at your convenience
-
-## Need Help?
-
-- **Setup Issues**: See `docs/developer/mcp_cursor_setup.md` or `docs/developer/mcp_claude_code_setup.md`
-- **OAuth Details**: See `docs/developer/mcp_oauth_implementation.md`
-- **Authentication Spec**: See `docs/subsystems/auth.md`
-- **API Reference**: See `docs/api/rest_api.md`

package/docs/developer/mcp_overview.md

@@ -1,145 +0,0 @@
-# MCP overview
-## Scope
-This document covers a curated overview of Neotoma MCP usage and links to canonical MCP references. It does not replace the full MCP specification or OAuth implementation details.
-
-## Purpose
-Provide a single entry point for MCP documentation with links to setup guides, the MCP spec, and authentication references.
-
-## Invariants
-1. MCP actions MUST follow the MCP specification.
-2. MCP mutations MUST respect Truth Layer boundaries and explicit user control.
-3. MCP examples MUST be deterministic and free of credentials.
-
-## Definitions
-- **MCP**: Model Context Protocol used by AI tools to access Neotoma.
-- **Action catalog**: The list of supported MCP actions and schemas.
-- **Setup guide**: Tool specific instructions for MCP integration.
-
-## When to use CLI vs MCP
-
-- **Local (same machine as repo):** Prefer the **Neotoma CLI** for all Neotoma operations. Invoke commands directly (e.g. `neotoma entities list`). The CLI is more reliable locally (no MCP server spawn/sync issues). Use MCP only if the user explicitly configures it.
-- **Remote (tunnel, ChatGPT, cloud):** Use **MCP** over HTTP with the configured Neotoma URL (e.g. tunnel or neotoma.fly.dev).
-
-For local development and agents running in the repo, the CLI is the recommended interface; MCP is recommended for remote clients.
-
-## Transport choice: stdio (local) vs HTTP (remote)
-
-- **Stdio:** Use for local clients (Cursor, Claude Code, Codex on the same machine as the Neotoma repo). Client spawns the server; better recovery after sleep.
-- **HTTP:** Use for remote access (tunnel, ChatGPT, deployed `neotoma.fly.dev`).
-
-See setup guides for config examples and comparison table.
-
-## Canonical MCP documentation
-- **MCP specification**: `docs/specs/MCP_SPEC.md`
-- **MCP server instructions** (loaded at runtime): `docs/developer/mcp/` — `instructions.md`, `unauthenticated.md`, `tool_descriptions.yaml`
-- **Cursor setup**: `docs/developer/mcp_cursor_setup.md`
-- **ChatGPT setup**: `docs/developer/mcp_chatgpt_setup.md`
-- **Claude Code setup**: `docs/developer/mcp_claude_code_setup.md`
-- **Agent CLI config** (unified MCP for Cursor, Claude Code, Codex): `docs/developer/agent_cli_configuration.md`
-- **OAuth implementation**: `docs/developer/mcp_oauth_implementation.md`
-- **Authentication summary**: `docs/developer/mcp_authentication_summary.md`
-
-## What MCP provides
-MCP exposes structured actions for:
-1. Storing structured data and files.
-2. Retrieving entities, observations, relationships, and timeline events.
-3. Correcting and reinterpreting observations.
-4. Schema recommendations and updates.
-
-## Diagrams
-```mermaid
-flowchart TD
-    aiTool[AITool] -->|"MCP action"| mcpServer[McpServer]
-    mcpServer -->|"Validate and store"| truthLayer[TruthLayer]
-    truthLayer -->|"Response"| mcpServer
-    mcpServer -->|"MCP response"| aiTool
-```
-
-## Examples
-
-**Storing:** Use the **structured** path (store with `entities`) for conversation- or tool-sourced data; use the **unstructured** path (store with `file_content`/`file_path`) for file- or resource-sourced data. See MCP_SPEC section 2.6 for the full rule.
-
-### Store a structured entity
-```
-{
-  "action": "store",
-  "entities": [
-    {
-      "entity_type": "task",
-      "title": "Review quarterly report",
-      "status": "open"
-    }
-  ]
-}
-```
-
-### Store a file (unstructured)
-```
-{
-  "action": "store",
-  "idempotency_key": "file-invoice-001",
-  "file_path": "/path/to/invoice.pdf",
-  "mime_type": "application/pdf",
-  "original_filename": "invoice.pdf"
-}
-```
-
-### Retrieve entities
-```
-{
-  "action": "retrieve_entities",
-  "entity_type": "task",
-  "limit": 5,
-  "offset": 0
-}
-```
-
-## Testing requirements
-1. Verify MCP actions conform to `docs/specs/MCP_SPEC.md`.
-2. Validate OAuth setup using the relevant setup guide.
-
-## Agent MCP behavior tests
-Use the agent MCP behavior suite to validate that MCP instructions trigger the expected tool calls and stored entities.
-
-**Fixture matrix:** `tests/fixtures/agent_mcp_cases.json`  
-**Test entrypoint:** `tests/agent/mcp_instruction_behavior.test.ts`  
-**Command:** `npm run test:agent-mcp`
-
-**Required environment for headless runners:**
-- `AGENT_TEST_ENABLED=true` to enable the suite
-- `AGENT_TEST_RUNNER=claude_code` (default) or `cursor_cli`
-- `AGENT_CLI_COMMAND` pointing to a wrapper command that reads:
-  - `AGENT_PROMPT_PATH` (prompt file)
-  - `AGENT_ATTACHMENTS_PATH` (JSON attachments list)
-  - `AGENT_TRACE_PATH` (write NDJSON tool-call trace)
-
-**Trace format:**
-Each NDJSON line must be JSON with at least:
-```json
-{"tool":"store","arguments":{...},"result":{...}}
-```
-
-## Agent Instructions
-### When to Load This Document
-Load this document when providing MCP entry points or curating MCP documentation navigation.
-
-### Required Co-Loaded Documents
-- `docs/NEOTOMA_MANIFEST.md`
-- `docs/specs/MCP_SPEC.md`
-- `docs/conventions/documentation_standards.md`
-
-### Constraints Agents Must Enforce
-1. Links MUST point to canonical MCP docs.
-2. Examples MUST be deterministic and credential free.
-3. Terminology MUST match the MCP spec.
-
-### Forbidden Patterns
-- Duplicating the MCP spec content in this overview
-- Using real tokens or credentials in examples
-- Omitting required sections
-
-### Validation Checklist
-- [ ] Purpose, Scope, Invariants, Definitions present
-- [ ] Links to MCP spec and setup guides included
-- [ ] Examples are deterministic and credential free
-- [ ] Agent Instructions included