neotoma 0.3.0 → 0.3.2

package/docs/developer/tunnels.md

@@ -1,327 +0,0 @@
-# HTTPS Tunnels for MCP Remote Access
-
-## Scope
-
-This document is the canonical reference for using HTTPS tunnels with the Neotoma MCP server. It covers:
-
-- When and why to use tunnels
-- Supported providers (Cloudflare, ngrok)
-- Provider selection and override
-- npm scripts and ports
-- Environment variables
-- Installation and setup per provider
-- Configuration and verification
-- Troubleshooting
-
-It does not cover: MCP protocol details, OAuth flows (see [mcp_oauth_implementation.md](mcp_oauth_implementation.md)), or production deployment without tunnels.
-
----
-
-## Purpose
-
-Tunnels expose your local MCP/API server over HTTPS so that:
-
-- Remote clients (e.g. Cursor on another machine, ChatGPT) can connect to `https://<tunnel-host>/mcp`
-- OAuth and "Add to Cursor" flows use a public HTTPS URL
-- The Connect button and discovery endpoints work as expected
-
-Without a tunnel, only clients on the same host can use `http://localhost:PORT/mcp`.
-
----
-
-## When to Use a Tunnel
-
-Use a tunnel when:
-
-- You want **remote MCP access** (Cursor or other IDEs connecting from another machine or over the internet)
-- You need **HTTPS** for OAuth or discovery (e.g. Cursor Connect)
-- You are testing **production-like** behavior with a local server (e.g. `watch:prod:tunnel`)
-
-Do not use a tunnel when:
-
-- You only use MCP over stdio or localhost
-- You deploy to a host that already has a public HTTPS URL
-
----
-
-## Security
-
-**Transport:** The tunnel provider (Cloudflare or ngrok) terminates TLS. Clients connect to `https://<tunnel-host>/mcp`; the provider forwards traffic to your local server over HTTP on localhost. Only the tunnel URL is exposed publicly.
-
-**Authentication:** Requests that reach the server via the tunnel have a **non-local Host** header (the tunnel hostname). The server never treats them as "local," so they do **not** get no-auth access:
-
-- **Unauthenticated** requests to `/mcp` receive **401 Unauthorized**. Cursor uses this to show the **Connect** button and start OAuth.
-- **OAuth (key-gated):** Use the Connect flow only after completing `/mcp/oauth/key-auth` with private key hex or mnemonic. PKCE and encrypted refresh tokens apply; see [auth.md](../subsystems/auth.md) and [mcp_oauth_implementation.md](mcp_oauth_implementation.md).
-- **Optional shared secret:** Set `NEOTOMA_BEARER_TOKEN` in `.env`; clients can send `Authorization: Bearer <token>` (useful for scripts or non-OAuth clients).
-- **Encryption on:** When `NEOTOMA_ENCRYPTION_ENABLED=true`, only the key-derived MCP token is accepted (`neotoma auth mcp-token`); OAuth is disabled.
-
-**Summary:** Tunnel traffic must always authenticate. OAuth is key-gated; non-key users should use `NEOTOMA_BEARER_TOKEN`.
-
-**Local OAuth over a public tunnel:** With encryption off, OAuth still uses a built-in dev account **after** key-auth preflight succeeds. When the server is reached **via a tunnel** (non-local Host), it requires **explicit approval** (an "Approve this connection" page) before completing OAuth, and it only accepts **allowlisted redirect URIs** (e.g. `cursor://`, `http://localhost`, `http://127.0.0.1`) so the authorization code cannot be sent to an arbitrary third-party site. If users cannot complete key-authenticated OAuth, use bearer token access:
-
-- **`NEOTOMA_BEARER_TOKEN`** in `.env` and send it as `Authorization: Bearer <token>` from the client (no OAuth; good for scripts or single-user).
-- **Encryption on** (`NEOTOMA_ENCRYPTION_ENABLED=true`): OAuth is disabled; use the key-derived token from `neotoma auth mcp-token` only.
-
----
-
-## Supported Providers
-
-| Provider    | Binary      | Auto-selected when        | Override                    |
-|------------|-------------|----------------------------|-----------------------------|
-| ngrok      | `ngrok`     | Installed and authenticated | `NEOTOMA_TUNNEL_PROVIDER=ngrok` |
-| Cloudflare | `cloudflared` | ngrok not available, cloudflared installed | `NEOTOMA_TUNNEL_PROVIDER=cloudflare` |
-
-**Selection order (when `NEOTOMA_TUNNEL_PROVIDER` is not set):**
-
-1. If `ngrok` is installed and authenticated → **ngrok**
-2. Else if `cloudflared` is installed → **Cloudflare**
-3. Else → **ngrok** (script will error if ngrok is not usable)
-
-To force a provider, set in `.env` or export before running:
-
-```bash
-NEOTOMA_TUNNEL_PROVIDER=cloudflare   # use cloudflared
-NEOTOMA_TUNNEL_PROVIDER=ngrok        # use ngrok
-```
-
----
-
-## npm Scripts and Ports
-
-| Script                 | Port | What runs                          | Use case                          |
-|------------------------|------|------------------------------------|-----------------------------------|
-| `npm run tunnel:https` | 8080 | Tunnel only                        | Server already running elsewhere  |
-| `npm run watch:dev:tunnel` / `npm run dev:api` | 8080 | Tunnel + API (dev)                 | Dev + remote MCP                 |
-| `npm run watch:server+api` | 8080 | Tunnel + API + `tsc --watch`       | Dev + tunnel + rebuild            |
-| `npm run watch:prod:tunnel` | 8180 | Tunnel + API + `tsc --watch` (prod) | Prod env + remote MCP             |
-
-Port is controlled by `NEOTOMA_HTTP_PORT` or `HTTP_PORT` (default 8080 for dev, 8180 for prod scripts). The tunnel forwards HTTPS to `http://localhost:${HTTP_PORT}`. Prefer `NEOTOMA_HTTP_PORT` when running Neotoma as MCP inside a host workspace to avoid port conflicts.
-
-**Combined commands** (e.g. `dev:api`, `watch:prod:tunnel`) start both tunnel and server in one terminal; the tunnel script writes the URL to `/tmp/ngrok-mcp-url.txt` and the server reads it for auto-discovery or explicit `NEOTOMA_HOST_URL`. Ctrl+C stops both.
-
-**CLI:** To start the API with a tunnel from the Neotoma CLI: `neotoma api start --background --env dev --tunnel` (or `--env prod --tunnel`). See [cli_reference.md](cli_reference.md).
-
-**Testing OAuth over the tunnel:** With the API and tunnel running, run `neotoma auth login --tunnel`. The CLI reads the tunnel URL from `/tmp/ngrok-mcp-url.txt`, opens the key-auth page in your browser, then completes the OAuth flow. Use this to create an OAuth connection for Cursor (or to verify the flow). If the tunnel URL file is missing, the command prints guidance to start the API with `--tunnel` first.
-
----
-
-## Environment Variables
-
-| Variable | Purpose |
-|----------|---------|
-| `NEOTOMA_HOST_URL` | Base URL for API and MCP (OAuth, discovery, "Add to Cursor"). If unset, server auto-discovers from `/tmp/ngrok-mcp-url.txt`. |
-| `NEOTOMA_REQUIRE_KEY_FOR_OAUTH` | Default `true`. Requires `/mcp/oauth/key-auth` preflight before OAuth authorize/local-login. |
-| `NEOTOMA_BEARER_TOKEN` | Shared bearer token fallback for remote users who are not using key-authenticated OAuth. |
-| `NEOTOMA_TUNNEL_PROVIDER` | Force provider: `cloudflare` or `ngrok`. |
-| `NEOTOMA_HTTP_PORT` | Port the server listens on; prefer when running inside a host workspace (avoids conflicts). |
-| `HTTP_PORT` | Legacy; tunnel forwards to this (default 8080; prod scripts use 8180). |
-| `TUNNEL_NONINTERACTIVE=1` | Used by npm scripts to skip "port not in use" prompt. |
-| `NEOTOMA_MCP_PROXY_URL` | Override MCP URL only when it must differ from host (rare). Prefer when running inside a host workspace. |
-| `MCP_PROXY_URL` | Legacy alias for `NEOTOMA_MCP_PROXY_URL`. |
-| `TUNNEL_TOKEN` or `CLOUDFLARE_TUNNEL_TOKEN` | Optional; for Cloudflare named tunnels. Not required for quick tunnels. |
-| `NEOTOMA_OAUTH_REDIRECT_BASE_URL` | **Advanced.** Override OAuth callback base (defaults to `NEOTOMA_HOST_URL`). Prefer when running inside a host workspace. |
-| `OAUTH_REDIRECT_BASE_URL` | Legacy alias for `NEOTOMA_OAUTH_REDIRECT_BASE_URL`. |
-
----
-
-## Cloudflare (cloudflared)
-
-**Install:**
-
-```bash
-brew install cloudflare/cloudflare/cloudflared
-# Or: https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/downloads/
-```
-
-No Cloudflare account is required for **quick tunnels** (ephemeral URL).
-
-**Behavior:**
-
-- Quick tunnel runs: `cloudflared tunnel --url "http://localhost:${HTTP_PORT}"`
-- URL pattern: `https://<random>.trycloudflare.com`
-- Log file: `/tmp/cloudflared-tunnel.log`
-- Tunnel URL is written to `/tmp/ngrok-mcp-url.txt` (same as ngrok so server and Cursor config flow are unchanged)
-- PID is stored in `/tmp/ngrok-mcp.pid` (same file name for both providers)
-
-**Verification:**
-
-```bash
-# Tunnel URL
-cat /tmp/ngrok-mcp-url.txt
-
-# Cloudflare log
-tail -f /tmp/cloudflared-tunnel.log
-```
-
----
-
-## ngrok
-
-**Install:**
-
-```bash
-brew install ngrok/ngrok/ngrok
-# Or: https://ngrok.com/download
-```
-
-**Authenticate (required):**
-
-```bash
-# Sign up: https://dashboard.ngrok.com/signup
-# Get authtoken: https://dashboard.ngrok.com/get-started/your-authtoken
-ngrok config add-authtoken YOUR_AUTHTOKEN
-```
-
-**Behavior:**
-
-- Free tier: new URL each run (e.g. `https://<random>.ngrok-free.dev`); may show a "Visit Site" warning in the browser.
-- PID: `/tmp/ngrok-mcp.pid`
-- URL: `/tmp/ngrok-mcp-url.txt`
-- Web UI: `http://localhost:4040` (when ngrok is running)
-- Optional fixed domain: if `NEOTOMA_HOST_URL` in `.env` contains a hostname, the script uses it for `ngrok http --domain=...` (requires ngrok reserved/custom domain).
-
-**Verification:**
-
-```bash
-# Tunnel URL
-cat /tmp/ngrok-mcp-url.txt
-
-# ngrok API (tunnel status)
-curl -s http://localhost:4040/api/tunnels | python3 -m json.tool
-
-# Logs
-tail -f /tmp/ngrok.log
-```
-
----
-
-## Tunnel URL Auto-Discovery
-
-**The server automatically discovers the tunnel URL** when `NEOTOMA_HOST_URL` is not set:
-
-1. **Tunnel script** (`setup-https-tunnel.sh`) writes URL to `/tmp/ngrok-mcp-url.txt`
-2. **Server** (`src/config.ts`) reads this file on startup if `NEOTOMA_HOST_URL` not in environment
-3. **OAuth and discovery** automatically use the tunnel URL
-
-**Priority order for `config.apiBase`:**
-1. `NEOTOMA_HOST_URL` (explicit environment variable)
-2. Auto-discovered from `/tmp/ngrok-mcp-url.txt`
-3. `http://localhost:${httpPort}` (fallback)
-
-**This means:**
-- Combined scripts (`dev:api`, `watch:prod:tunnel`) work without manual URL configuration
-- Starting tunnel first, then server separately works automatically
-- No `.env` changes needed for most tunnel use cases
-
-**To use a fixed domain:** Set `NEOTOMA_HOST_URL` in `.env` to your reserved ngrok domain or custom URL.
-
----
-
-## Configuration After Tunnel Starts
-
-1. **Tunnel URL**  
-   From script output or `cat /tmp/ngrok-mcp-url.txt`.
-
-2. **Server base URL**  
-   **Auto-discovery (no action needed):** Combined scripts (`dev:api`, `watch:prod:tunnel`) write the tunnel URL to a file; the server auto-discovers from `/tmp/ngrok-mcp-url.txt`. If you start server separately, it reads that file on startup.  
-   **Manual:** Set `NEOTOMA_HOST_URL` to the tunnel URL and restart the server.
-
-3. **Cursor / MCP client**  
-   Use `https://<tunnel-url>/mcp` in `.cursor/mcp.json` or the Neotoma UI "Add to Cursor" flow.
-
-4. **Optional**  
-   Visit the tunnel URL in a browser once; ngrok free tier may require accepting the "Visit Site" page.
-
----
-
-## Stopping the Tunnel
-
-- **Combined command (e.g. dev:api, watch:prod:tunnel):** Ctrl+C in the terminal stops both tunnel and server.
-- **Tunnel-only (`tunnel:https`):** Ctrl+C in the tunnel terminal.
-- **Manual kill (either provider):**
-  ```bash
-  kill $(cat /tmp/ngrok-mcp.pid 2>/dev/null) 2>/dev/null || true
-  rm -f /tmp/ngrok-mcp.pid /tmp/ngrok-mcp-url.txt
-  ```
-
----
-
-## Troubleshooting
-
-### Connection refused to tunnel URL
-
-- **Tunnel not running:** Start it again (`npm run tunnel:https` or a combined command). Use the **current** URL from the script or `cat /tmp/ngrok-mcp-url.txt`.
-- **Stale URL:** Free-tier URLs change each run. Update `NEOTOMA_HOST_URL` (if set explicitly in .env) and Cursor config with the new URL. If using auto-discovery, restart the server to pick up the new URL.
-- **ngrok free-tier:** Open the tunnel URL in a browser and accept "Visit Site" if shown, then retry.
-- **Port:** Ensure the server is listening on the port the tunnel uses (8080 or 8180). Check with `lsof -i :8080` (or 8180).
-
-### ngrok: domain resolves to 127.0.0.1
-
-If `curl -v` shows the tunnel host resolving to `127.0.0.1`, a local DNS override (AdGuard, Pi-hole, VPN, etc.) may be mapping the host to localhost.
-
-- Flush DNS: `sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder`
-- Exclude `*.ngrok-free.dev` (or the tunnel host) in the DNS app.
-- Confirm: `dscacheutil -q host -a name <tunnel-host>`; IPv4 should be a public IP, not 127.0.0.1.
-
-### Cloudflare: no URL in log
-
-- Wait a few seconds; the script retries for the URL in `/tmp/cloudflared-tunnel.log`.
-- Check: `tail -f /tmp/cloudflared-tunnel.log` for errors.
-
-### OAuth / discovery issues
-
-- Server auto-discovers tunnel URL from `/tmp/ngrok-mcp-url.txt`. If using explicit `NEOTOMA_HOST_URL` in `.env`, ensure it matches the tunnel URL exactly and restart the server after changing it.
-- See [mcp_oauth_implementation.md](mcp_oauth_implementation.md) and [mcp_oauth_troubleshooting.md](mcp_oauth_troubleshooting.md).
-
-### Verification commands (summary)
-
-```bash
-# Tunnel URL (both providers)
-cat /tmp/ngrok-mcp-url.txt
-
-# ngrok only: tunnel status
-curl -s http://localhost:4040/api/tunnels | python3 -m json.tool
-
-# Local server
-curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/mcp   # expect 401
-
-# Through tunnel (replace with your URL; ngrok free tier may need header)
-curl -s -H "ngrok-skip-browser-warning: 1" "https://YOUR-TUNNEL-URL/.well-known/oauth-authorization-server"
-```
-
----
-
-## Related Documents
-
-- [mcp_https_tunnel_status.md](mcp_https_tunnel_status.md) — Current status and quick troubleshooting
-- [mcp_https_testing.md](mcp_https_testing.md) — HTTPS testing for MCP/Cursor Connect
-- [mcp_cursor_setup.md](mcp_cursor_setup.md) — Cursor MCP setup and tunnel usage
-- [mcp_oauth_implementation.md](mcp_oauth_implementation.md) — OAuth redirect and base URL
-- [getting_started.md](getting_started.md) — Server and tunnel commands overview
-- [cli_reference.md](cli_reference.md) — npm scripts summary
-
----
-
-## Agent Instructions
-
-### When to Load This Document
-
-Load when adding or changing tunnel behavior, documenting tunnel scripts or env vars, or troubleshooting remote MCP/HTTPS access.
-
-### Required Co-Loaded Documents
-
-- [getting_started.md](getting_started.md) for server/tunnel command context
-- [mcp_oauth_implementation.md](mcp_oauth_implementation.md) when editing OAuth or base URL behavior
-
-### Constraints Agents Must Enforce
-
-1. Tunnel provider selection must match `scripts/setup-https-tunnel.sh`: Cloudflare when cloudflared installed, else ngrok when installed and authenticated.
-2. Document both Cloudflare and ngrok for install, verification, and troubleshooting.
-3. Use `NEOTOMA_HOST_URL` as the base URL config.
-4. Server auto-discovers tunnel URL from `/tmp/ngrok-mcp-url.txt` if `NEOTOMA_HOST_URL` not set.
-
-### Forbidden Patterns
-
-- Documenting only one provider without the other
-- Requiring a Cloudflare token for quick tunnels
-- Stating that ngrok is always preferred over Cloudflare (Cloudflare is preferred when cloudflared is installed)

package/docs/developer/vite_troubleshooting.md

@@ -1,111 +0,0 @@
-# Vite Troubleshooting Guide
-## Common Issues and Solutions
-### Outdated Optimize Dep Error
-**Error:** `GET http://localhost:XXXX/@fs/... net::ERR_ABORTED 504 (Outdated Optimize Dep)`
-**Cause:** Vite's dependency optimization cache is outdated, typically after:
-- Adding new npm packages
-- Updating dependencies
-- Adding new component imports (especially Radix UI components)
-**Solution 1: Restart Dev Server (Recommended)**
-```bash
-# Stop the dev server (Ctrl+C)
-# Then restart
-npm run dev:ui
-```
-Vite will automatically detect the outdated cache and re-optimize dependencies.
-**Solution 2: Force Cache Clear**
-```bash
-# Stop the dev server first, then:
-rm -rf node_modules/.vite
-# Restart the dev server
-npm run dev:ui
-```
-**Solution 3: Permission Issues (EACCES Error)**
-If you see `EACCES: permission denied` errors, the cache files may be owned by root:
-```bash
-# Fix ownership (may require password)
-sudo chown -R $(whoami):staff node_modules/.vite
-# Then remove cache
-rm -rf node_modules/.vite
-# Restart dev server
-npm run dev:ui
-```
-**Alternative:** If sudo is not available, you can work around this by:
-1. Stopping all dev processes completely
-2. Waiting a few seconds for file locks to release
-3. Manually deleting the cache directory in Finder/File Manager
-4. Restarting the dev server
-**Solution 3: Force Re-optimization**
-```bash
-# Set environment variable to force re-optimization
-VITE_FORCE_OPTIMIZE=true npm run dev:ui
-```
-### Port Already in Use
-**Error:** `Port XXXX is already in use`
-**Solution:**
-```bash
-# Find and kill the process using the port
-lsof -ti:5173 | xargs kill -9
-# Or use a different port
-VITE_PORT=5174 npm run dev:ui
-```
-### HMR (Hot Module Replacement) Not Working
-**Symptoms:** Changes to files don't reflect in browser automatically
-**Solutions:**
-1. Check browser console for WebSocket errors
-2. Verify HMR port matches server port in `vite.config.ts`
-3. Clear browser cache and hard refresh (Cmd+Shift+R / Ctrl+Shift+R)
-4. Restart dev server
-### Module Not Found Errors
-**Error:** `Cannot find module '@/components/...'`
-**Solution:**
-1. Verify the file exists at the correct path
-2. Check `vite.config.ts` alias configuration
-3. Restart dev server (path resolution is cached)
-### Build Errors
-**Error:** Build fails with dependency errors
-**Solution:**
-```bash
-# Clear all caches and reinstall
-rm -rf node_modules/.vite
-rm -rf node_modules
-npm install
-npm run build:ui
-```
-## Best Practices
-1. **Always restart dev server** after:
-   - Installing new packages
-   - Adding new Radix UI / shadcn components
-   - Changing `vite.config.ts`
-2. **Clear cache** if you see persistent errors:
-   ```bash
-   rm -rf node_modules/.vite
-   ```
-3. **Check browser console** for specific error messages - Vite provides helpful error details
-4. **Use browser DevTools Network tab** to see which files are failing to load
-## Environment Variables
-- `VITE_PORT`: Override default dev server port (default: 5173)
-- `VITE_FORCE_OPTIMIZE`: Force dependency re-optimization (set to `true`)
-- `HTTP_PORT`: Backend API port (default: 8080)
-- `WS_PORT`: WebSocket port (default: 8280)
-## Agent Instructions
-### When to Load This Document
-Load `docs/developer/vite_troubleshooting.md` when:
-- Frontend (Vite) dev server is failing or behaving unexpectedly
-- HMR, ports, or optimize-deps issues block UI work
-- You need to debug local UI environment problems
-### Required Co-Loaded Documents
-- `docs/developer/getting_started.md` (environment setup)
-- `vite.config.ts` (Vite configuration)
-### Constraints Agents Must Enforce
-1. MUST NOT modify Vite configuration or scripts without cross-checking architecture and UI docs
-2. MUST preserve existing dev workflows and branch-aware port behavior
-3. MUST NOT add non-deterministic behavior to build/dev scripts
-### Forbidden Patterns
-- Editing troubleshooting guide to change product behavior (code lives in configs/scripts, not here)
-- Suggesting global `sudo` fixes that break local environments
-- Recommending cache clears that delete user data beyond dev caches
-### Validation Checklist
-- [ ] Troubleshooting steps keep existing dev workflows intact
-- [ ] No changes conflict with `docs/developer/development_workflow.md`
-- [ ] No instructions violate determinism or environment isolation