@cerefox/memory 0.4.2 → 0.5.0

package/docs/guides/setup-cloud-run.md

@@ -0,0 +1,117 @@
+# Google Cloud Run Deployment
+
+Deploy the Cerefox web UI to Google Cloud Run for a lightweight, serverless hosting option. This guide uses Supabase (free tier) for the database.
+
+---
+
+## Prerequisites
+
+- Google Cloud account with billing enabled
+- `gcloud` CLI installed and authenticated (`gcloud auth login`)
+- Supabase project set up (see `setup-supabase.md`)
+- Docker installed locally
+
+---
+
+## Step 1 — Set up environment variables
+
+```bash
+export PROJECT_ID=your-gcp-project-id
+export REGION=us-central1
+export SERVICE_NAME=cerefox
+export IMAGE=gcr.io/$PROJECT_ID/$SERVICE_NAME
+```
+
+---
+
+## Step 2 — Build and push the Docker image
+
+```bash
+# Authenticate Docker to GCR
+gcloud auth configure-docker
+
+# Build and push
+docker build -t $IMAGE .
+docker push $IMAGE
+```
+
+Or use Cloud Build for fully cloud-side building:
+
+```bash
+gcloud builds submit --tag $IMAGE .
+```
+
+---
+
+## Step 3 — Deploy to Cloud Run
+
+```bash
+gcloud run deploy $SERVICE_NAME \
+  --image $IMAGE \
+  --region $REGION \
+  --platform managed \
+  --allow-unauthenticated \
+  --port 8000 \
+  --memory 512Mi \
+  --cpu 1 \
+  --set-env-vars \
+    CEREFOX_SUPABASE_URL=https://YOUR-REF.supabase.co,\
+    CEREFOX_SUPABASE_KEY=YOUR-SERVICE-ROLE-KEY,\
+    OPENAI_API_KEY=YOUR-OPENAI-KEY,\
+    CEREFOX_MAX_RESPONSE_BYTES=200000
+```
+
+**Memory**: Cerefox uses cloud embeddings (OpenAI API) — no local model is loaded. `--memory 512Mi` is sufficient for the web server alone.
+
+**CPU**: `--cpu 1` is sufficient for personal use. Scale up for concurrent users.
+
+---
+
+## Step 4 — Verify the deployment
+
+```bash
+gcloud run services describe $SERVICE_NAME --region $REGION --format="value(status.url)"
+```
+
+Open the printed URL — the Cerefox dashboard should load.
+
+---
+
+## Updating the deployment
+
+After making code changes:
+
+```bash
+docker build -t $IMAGE .
+docker push $IMAGE
+gcloud run deploy $SERVICE_NAME --image $IMAGE --region $REGION --platform managed
+```
+
+Cloud Run performs a zero-downtime rolling update.
+
+---
+
+## Cost
+
+For a typical single-user personal knowledge base, Cloud Run comfortably fits within its
+always-free tier limits. The main variable expense is OpenAI embedding calls, shared with
+all other deployment options. See `docs/guides/operational-cost.md` for a full breakdown.
+
+---
+
+## Securing access
+
+By default `--allow-unauthenticated` makes the URL public. To restrict access:
+
+### Option 1 — Cloud Run IAM
+
+Remove `--allow-unauthenticated` and use `gcloud run services add-iam-policy-binding` to add specific Google accounts.
+
+### Option 2 — Cloud Run proxy
+
+Use Identity-Aware Proxy (IAP) in front of Cloud Run for Google SSO.
+
+### Option 3 — Add HTTP basic auth
+
+In `src/cerefox/api/app.py`, add a middleware that checks `Authorization: Basic ...` headers. Use `CEREFOX_BASIC_AUTH_USER` / `CEREFOX_BASIC_AUTH_PASSWORD` env vars.
+

package/docs/guides/setup-local.md

@@ -0,0 +1,178 @@
+# Local Setup Guide
+
+Run the Cerefox web server and database on your own machine using Docker for Postgres+pgvector. Embeddings use the OpenAI API — an `OPENAI_API_KEY` is required even for local setups.
+
+---
+
+## Prerequisites
+
+- Docker and Docker Compose
+- Python 3.11+ with `uv` (`pip install uv`)
+- An OpenAI API key (for embeddings — [platform.openai.com/api-keys](https://platform.openai.com/api-keys))
+
+---
+
+## Step 1 — Clone and install
+
+```bash
+git clone https://github.com/fstamatelopoulos/cerefox.git
+cd cerefox
+uv sync
+```
+
+---
+
+## Step 2 — Start Postgres with pgvector
+
+The included `docker-compose.yml` spins up a Postgres 16 instance with the pgvector extension pre-installed:
+
+```bash
+docker compose up -d postgres
+```
+
+Default connection details (overridable in `.env`):
+
+| Setting | Default |
+|---------|---------|
+| Host | `localhost` |
+| Port | `5432` |
+| User | `cerefox` |
+| Password | `cerefox` |
+| Database | `cerefox` |
+
+---
+
+## Step 3 — Create a `.env` file
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` for local Docker:
+
+```env
+# Local Postgres (Docker)
+CEREFOX_DATABASE_URL=postgresql://cerefox:cerefox@localhost:5432/cerefox
+
+# For local-only use, Supabase keys are not required.
+# The web UI and CLI will work without them if you skip the Supabase MCP integration.
+CEREFOX_SUPABASE_URL=
+CEREFOX_SUPABASE_KEY=
+
+# OpenAI API key for embeddings (text-embedding-3-small)
+OPENAI_API_KEY=sk-...
+```
+
+---
+
+## Step 4 — Deploy the schema
+
+```bash
+python scripts/db_deploy.py
+```
+
+This creates all tables, indexes, and RPC functions. Run with `--dry-run` to preview SQL without executing.
+
+To start fresh:
+
+```bash
+python scripts/db_deploy.py --reset   # drops all cerefox_ tables first
+```
+
+---
+
+## Step 5 — Verify the setup
+
+```bash
+python scripts/db_status.py
+```
+
+You should see all tables (cerefox_documents, cerefox_chunks, cerefox_projects) and RPC functions listed as ✓.
+
+---
+
+## Step 6 — Ingest your first document
+
+```bash
+# Ingest a markdown file
+cerefox ingest my-notes.md --project-name "personal"
+
+# Or paste content from stdin
+echo "# Quick Note\n\nThis is a quick note." | cerefox ingest --paste --title "Quick Note"
+```
+
+Each ingest calls the OpenAI embedding API once per batch of chunks (fast, typically under a second).
+
+---
+
+## Step 7 — Start the web UI
+
+```bash
+cerefox web
+```
+
+Open [http://localhost:8000](http://localhost:8000) in your browser.
+
+For development with auto-reload:
+
+```bash
+cerefox web --reload
+```
+
+---
+
+## Step 8 — Search from the CLI
+
+```bash
+# Hybrid search (recommended)
+cerefox search "what did I write about project planning?"
+
+# Keyword-only search
+cerefox search "meeting notes" --mode fts
+
+# Semantic search
+cerefox search "ideas about creativity" --mode semantic
+```
+
+---
+
+## Running everything at once
+
+The `docker-compose.yml` also includes a `cerefox` service that runs the web UI:
+
+```bash
+docker compose up -d
+```
+
+Web UI will be at [http://localhost:8000](http://localhost:8000).
+
+---
+
+## Stopping services
+
+```bash
+docker compose down          # stop, keep data
+docker compose down -v       # stop and delete database volume
+```
+
+---
+
+## Updating the schema
+
+When a new version of Cerefox introduces schema changes, run:
+
+```bash
+python scripts/db_migrate.py
+```
+
+This applies incremental migrations without losing data. Always back up first (see `ops-scripts.md`).
+
+---
+
+## Troubleshooting
+
+**pgvector extension not found**
+Make sure you're using the `pgvector/pgvector:pg16` Docker image (included in `docker-compose.yml`). Raw Postgres images do not include pgvector.
+
+**"Supabase is not configured" error**
+The CLI and web UI show this error if `CEREFOX_SUPABASE_URL` / `CEREFOX_SUPABASE_KEY` are empty. For local Docker setups, the app uses the direct Postgres URL (`CEREFOX_DATABASE_URL`) for schema deployment but the Supabase client for queries. Set up a local Supabase instance or use the hosted free tier (see `setup-supabase.md`).

package/docs/guides/setup-supabase.md

@@ -0,0 +1,370 @@
+# Setting Up Cerefox with Supabase
+
+This guide walks you from a blank Supabase project to a fully deployed Cerefox schema, ready to ingest documents and serve AI agents via MCP.
+
+**Time required**: ~15 minutes
+
+---
+
+## Prerequisites
+
+- [uv](https://docs.astral.sh/uv/getting-started/installation/) installed
+- A Supabase account (free tier is enough): [supabase.com](https://supabase.com)
+- Python 3.11 or higher
+
+---
+
+## Step 1 — Create a Supabase Project
+
+1. Go to [app.supabase.com](https://app.supabase.com) and sign in
+2. Click **New project**
+3. Choose a name (e.g. `cerefox`), set a strong database password, pick a region close to you
+4. Click **Create new project** and wait ~2 minutes for it to provision
+
+---
+
+## Step 2 — Enable the pgvector Extension
+
+Supabase includes pgvector but you need to activate it:
+
+1. In your project dashboard, go to **Database → Extensions**
+2. Search for `vector` and enable it
+
+The deploy script also runs `CREATE EXTENSION IF NOT EXISTS vector` automatically, but enabling it in the UI first prevents permission issues.
+
+---
+
+## Step 3 — Collect Your Credentials
+
+You need three values from Supabase: a URL, an API key, and a direct Postgres connection string. The API-key panel and the connection-string panel both went through significant changes in 2026 — read the two reference sections below for the up-to-date guidance.
+
+### 3a. API URL
+- **Project Settings → API → Project URL** → `CEREFOX_SUPABASE_URL`
+
+### 3b. API key — `CEREFOX_SUPABASE_KEY`
+
+See the **[Supabase API keys (2026)](#supabase-api-keys-2026)** section near the end of this guide for the full picture. The short version:
+
+- For `CEREFOX_SUPABASE_KEY` (this guide, Python web app, CLI): use the new **secret key** (`sb_secret_…`) from **Project Settings → API Keys → Secret key**. The legacy `service_role` JWT also still works during the transition.
+- For `CEREFOX_SUPABASE_ANON_KEY` (only if you'll use Edge Functions / MCP / GPT Actions; not needed for this guide's deployment step): you must use the **legacy anon JWT** (`eyJ…`). The new `sb_publishable_…` key fails at the Edge Function gateway. See the reference section for why.
+
+Either way: keep this key secret — it bypasses Row Level Security and grants full database access.
+
+### 3c. Direct database URL — `CEREFOX_DATABASE_URL`
+
+This is used by `db_deploy.py`, `db_migrate.py`, and `db_status.py`. See the **[Connection pooling in 2026](#connection-pooling-2026)** reference section near the end of this guide for context. The short version:
+
+1. Open **Project Settings → Database → Connection pooling** (not the "Connect" dialog — that one usually omits the Session Pooler in the new UI).
+2. Copy the **Session Pooler** URI (host ends in `.pooler.supabase.com`, port `5432`).
+3. Confirm the username has the form `postgres.<project-ref>` — without that suffix you'll get "Tenant or user not found".
+4. Append `?sslmode=require` to enforce TLS explicitly.
+
+If you only see Direct Connection and Transaction Pooler in your dashboard, take the Transaction Pooler URI and change `:6543` → `:5432`. That gives you the Session Pooler. **Do not use port 6543** — Transaction Pooler does not support DDL and `db_deploy.py` will fail mid-schema.
+
+The Direct Connection (`db.<project-ref>.supabase.co:5432`) is IPv6-only on the free tier and unusable on most home/office networks. The dashboard now warns about this directly.
+
+---
+
+## Step 4 — Configure Your Environment
+
+```bash
+# In the cerefox project root:
+cp .env.example .env
+```
+
+Edit `.env` and fill in your values. A minimal working configuration looks like:
+
+```bash
+CEREFOX_SUPABASE_URL=https://your-project-ref.supabase.co
+CEREFOX_SUPABASE_KEY=sb_secret_...your-secret-key...
+CEREFOX_DATABASE_URL=postgresql://postgres.yourref:yourpassword@aws-N-region.pooler.supabase.com:5432/postgres?sslmode=require
+# CEREFOX_SUPABASE_ANON_KEY only needed if you'll deploy Edge Functions (Step 8)
+# Must be the legacy anon JWT, NOT sb_publishable_... — see "Supabase API keys (2026)" below
+# CEREFOX_SUPABASE_ANON_KEY=eyJ...your-legacy-anon-jwt...
+```
+
+Leave all other settings at their defaults for now.
+
+---
+
+## Step 5 — Install Dependencies
+
+```bash
+uv sync
+```
+
+This installs all Python dependencies defined in `pyproject.toml`, including `supabase`, `psycopg2-binary`, and `pydantic-settings`.
+
+---
+
+## Step 6 — Deploy the Schema
+
+```bash
+# Preview what will happen (no changes made):
+python scripts/db_deploy.py --dry-run
+
+# Apply the schema:
+python scripts/db_deploy.py
+```
+
+Expected output:
+```
+╔══════════════════════════════════════╗
+║  Cerefox DB Deploy                   ║
+╚══════════════════════════════════════╝
+
+Connecting to database...
+
+▶  Enable extensions (uuid-ossp, vector/pgvector)...
+   ✓  Done
+
+▶  Apply schema (tables, indexes, triggers)...
+   ✓  Done
+
+▶  Apply RPCs (search functions)...
+   ✓  Done
+
+──────────────────────────────────────────
+✓  Deployment complete. 3 steps applied.
+
+Next step: verify the schema with:
+    python scripts/db_status.py
+```
+
+---
+
+## Step 7 — Verify the Schema
+
+```bash
+python scripts/db_status.py
+```
+
+Expected output:
+```
+╔══════════════════════════════════════╗
+║  Cerefox DB Status                   ║
+╚══════════════════════════════════════╝
+
+Extensions:
+  ✓  uuid-ossp
+  ✓  vector
+
+Tables:
+  ✓  cerefox_projects
+  ✓  cerefox_documents
+  ✓  cerefox_chunks
+  ✓  cerefox_migrations
+
+Functions / RPCs:
+  ✓  cerefox_set_updated_at()
+  ✓  cerefox_hybrid_search()
+  ✓  cerefox_fts_search()
+  ✓  cerefox_semantic_search()
+  ✓  cerefox_reconstruct_doc()
+  ✓  cerefox_save_note()
+  ✓  cerefox_search_docs()
+  ✓  cerefox_context_expand()
+
+Indexes:
+  ✓  idx_cerefox_chunks_fts
+  ✓  idx_cerefox_chunks_emb_primary
+  ✓  idx_cerefox_chunks_emb_upgrade
+  ✓  idx_cerefox_chunks_document
+  ✓  idx_cerefox_docs_metadata
+  ✓  idx_cerefox_docs_project
+
+Row counts:
+  ℹ  cerefox_projects: 0 rows
+  ℹ  cerefox_documents: 0 rows
+  ℹ  cerefox_chunks: 0 rows
+
+──────────────────────────────────────────
+✓  All checks passed. Schema looks healthy.
+```
+
+All checks should show ✓. If any show ✗, re-run `python scripts/db_deploy.py`.
+
+---
+
+## Step 8 — Deploy Edge Functions
+
+The Edge Functions run server-side on Supabase. `cerefox-search` and `cerefox-ingest` handle
+embedding for agents; `cerefox-mcp` wraps them as a remote MCP endpoint (recommended for
+Claude Code, Cursor, and Claude Desktop). Deploy using the Supabase CLI via `npx` — no
+separate install needed, just Node.js.
+
+**First time only — authenticate and link your project:**
+
+```bash
+npx supabase login        # opens a browser tab; click "Confirm" to generate an access token
+npx supabase link         # prompts for your project ref (the ID in your Supabase dashboard URL)
+```
+
+Your project ref is in the Supabase dashboard URL:
+`https://supabase.com/dashboard/project/<project-ref>`
+
+**Deploy all three functions** (from the cerefox project root):
+
+```bash
+npx supabase functions deploy cerefox-ingest
+npx supabase functions deploy cerefox-search
+npx supabase functions deploy cerefox-mcp
+```
+
+Expected output for each:
+```
+Bundling Function: cerefox-ingest
+Deploying Function: cerefox-ingest (script size: ~880kB)
+Deployed Functions on project <your-project-ref>: cerefox-ingest
+```
+
+You can verify in the Supabase Dashboard → **Edge Functions** — all three functions should
+appear with a green "Active" status.
+
+> **Re-deploying after updates**: run the same `npx supabase functions deploy` commands
+> again from the project root. `npx supabase login` and `npx supabase link` only need to be
+> run once per machine.
+
+---
+
+## Step 9 — Run the Tests
+
+Confirm everything is wired up correctly:
+
+```bash
+uv run pytest
+```
+
+These are unit tests only (no real database connection needed). You should see all tests pass.
+
+To also run the integration tests against your live Supabase instance:
+```bash
+uv run pytest -m integration
+```
+
+---
+
+## Step 11 — Connect an AI agent (optional)
+
+Cerefox ships a built-in MCP server that gives desktop agents named tools
+(`cerefox_search`, `cerefox_ingest`) with full hybrid search.
+
+**For Claude Desktop / ChatGPT Desktop / Cursor** — add to the client's MCP config:
+```json
+{
+  "mcpServers": {
+    "cerefox": {
+      "command": "uv",
+      "args": ["--directory", "/path/to/cerefox", "run", "cerefox", "mcp"]
+    }
+  }
+}
+```
+
+**For cloud Claude.ai** — connect to the Supabase remote MCP (FTS keyword search only):
+1. In Supabase Dashboard → Project Settings → Integrations → MCP, get your project ref
+2. In Claude.ai Settings → Integrations, add `https://mcp.supabase.com/sse?project_ref=<ref>`
+
+**For cloud ChatGPT** — create a Custom GPT with GPT Actions pointing at the Edge Functions.
+
+See `docs/guides/connect-agents.md` for the complete guide including system prompts,
+architecture explanation, and ChatGPT GPT Actions setup.
+
+---
+
+## Troubleshooting
+
+### "could not connect to server"
+- Check that `CEREFOX_DATABASE_URL` is correct and the password doesn't contain special characters that need URL-encoding
+- Try pasting the URL directly into `psql` to verify it works
+
+### "extension 'vector' does not exist"
+- Go to Supabase Dashboard → Database → Extensions → enable `vector`
+- Then re-run `python scripts/db_deploy.py`
+
+### "permission denied for table"
+- Make sure `CEREFOX_SUPABASE_KEY` is your secret key (`sb_secret_…`) or legacy `service_role` JWT — not the publishable / anon key. See [Supabase API keys (2026)](#supabase-api-keys-2026) below.
+
+### Schema already exists (re-deploying)
+- All schema objects use `CREATE ... IF NOT EXISTS` / `CREATE OR REPLACE`, so re-running is safe
+- To start completely fresh: `python scripts/db_deploy.py --reset` (⚠️ deletes all data)
+
+---
+
+## Supabase API keys (2026) <a id="supabase-api-keys-2026"></a>
+
+In 2026 Supabase rolled out a new API key system. The dashboard now shows two key types side by side, and the migration is **asymmetric** — one half is fully migrated, the other is not. Cerefox needs values from both halves.
+
+### The two key families
+
+| Family | What you'll see in the dashboard | Use it for |
+|---|---|---|
+| **New (recommended)** | "Publishable key" (`sb_publishable_…`) and "Secret key" (`sb_secret_…`) | `CEREFOX_SUPABASE_KEY` — works end-to-end through the Data API. |
+| **Legacy** | "anon" and "service_role" JWTs (`eyJ…`), filed under a "Legacy" section | `CEREFOX_SUPABASE_ANON_KEY` — **still required** for any Edge Function call (MCP, GPT Actions, e2e tests, direct curl). |
+
+### What goes where in `.env`
+
+| Variable | Recommended value | Why |
+|---|---|---|
+| `CEREFOX_SUPABASE_KEY` | New **secret key** (`sb_secret_…`). Legacy `service_role` JWT also works. | Used by `db/client.py` to reach the Data API (PostgREST). Both formats are accepted by the gateway. |
+| `CEREFOX_SUPABASE_ANON_KEY` | **Legacy anon JWT** (`eyJ…`). | Used as the `Authorization: Bearer …` header for Edge Function calls. The Supabase Edge Function gateway still validates this token as a JWT — it parses `header.payload.signature` and rejects non-JWT keys with `UNAUTHORIZED_INVALID_JWT_FORMAT`. The new `sb_publishable_…` key is not a JWT and fails. |
+
+### Why we cannot just use the new keys everywhere
+
+The Data API gateway was migrated in 2026 to accept both the new and legacy key formats. The Edge Function gateway was not. A Supabase team member [confirmed this](https://github.com/orgs/supabase/discussions/41834): the only way to call Edge Functions with the new keys today is to deploy each function with `verify_jwt = false` and validate the key inside the function. Cerefox is not yet doing that (a future migration; see Decision Log 2026 Q2 for context and triggers).
+
+### Is the legacy key going away?
+
+Not yet. The Supabase docs explicitly state: *"You can still use old anon and service-role API keys after enabling the publishable and secret keys."* The "Legacy" label in the dashboard suggests deprecation, but **no end-of-life date has been published** as of May 2026. When Supabase announces one, Cerefox will migrate.
+
+### Sources
+
+- [Edge Function returning "JWT is invalid" after migrating to 2026 API Keys — supabase/discussions#41834](https://github.com/orgs/supabase/discussions/41834)
+- [Understanding API keys — Supabase Docs](https://supabase.com/docs/guides/api/api-keys)
+- [Securing Edge Functions — Supabase Docs](https://supabase.com/docs/guides/functions/auth)
+
+---
+
+## Connection pooling in 2026 <a id="connection-pooling-2026"></a>
+
+Supabase's "Connect" dialog was redesigned in 2026 and the **Session Pooler** is no longer a first-class tab in many projects. The other two surfaces (Direct Connection and Transaction Pooler) are present but neither works for Cerefox's deployment scripts. Here's the full picture.
+
+### The three Postgres connection types
+
+| Type | Host / port | DDL support | IPv4 compatible? | Use for Cerefox? |
+|---|---|---|---|---|
+| **Direct Connection** | `db.<project-ref>.supabase.co:5432` | ✓ | **IPv6 only on free tier.** Dashboard warns about this directly. | No — most networks can't reach it. |
+| **Transaction Pooler** | `aws-N-region.pooler.supabase.com:6543` (note port) | **✗ no DDL** | ✓ | No — `db_deploy.py` fails mid-schema. |
+| **Session Pooler** | `aws-N-region.pooler.supabase.com:5432` | ✓ | ✓ | **Yes — use this.** |
+
+The Transaction Pooler runs PgBouncer in transaction mode and does not maintain a session between statements, breaking DDL, prepared statements, `SET LOCAL`, and advisory locks. The Session Pooler is the same hostname on a different port and keeps a full session per connection.
+
+### Finding the Session Pooler URI
+
+Two reliable paths in the current dashboard:
+
+1. **Project Settings → Database → Connection pooling.** The Session Pooler URI is listed here.
+2. **Quick shortcut**: take whatever Transaction Pooler URI the "Connect" dialog shows you and change `:6543` → `:5432`. Same host, same username, just session mode instead of transaction mode.
+
+### Required pieces of the connection string
+
+- **Host**: `aws-N-region.pooler.supabase.com` (substituted by your project's region — e.g. `aws-1-us-east-1`).
+- **Port**: `5432` — **never `6543`** for Cerefox.
+- **Username**: `postgres.<project-ref>` — the `.<project-ref>` suffix is mandatory. Without it Supabase returns "Tenant or user not found".
+- **Password**: the database password set when you created the project (or rotated under Project Settings → Database). URL-encode special characters (`@` → `%40`, etc.) or pick a URL-safe password to avoid the gotcha.
+- **Query string**: append `?sslmode=require` to enforce TLS explicitly. Supabase enforces TLS server-side anyway; being explicit doesn't hurt.
+
+Final shape:
+
+```
+postgresql://postgres.<project-ref>:<password>@aws-N-<region>.pooler.supabase.com:5432/postgres?sslmode=require
+```
+
+### Common errors and what they mean
+
+| Error | Likely cause |
+|---|---|
+| `Tenant or user not found` | Username is missing the `.<project-ref>` suffix. |
+| `nodename nor servname provided` | You used the Direct Connection on an IPv4-only network. Switch to Session Pooler. |
+| `cannot create function ... transaction mode does not support ...` (or similar mid-schema failure) | You used the Transaction Pooler (port 6543). Switch to Session Pooler (port 5432). |
+| `password authentication failed` | Wrong password, or special characters not URL-encoded. Reset under Project Settings → Database. |