@cerefox/memory 0.4.3 → 0.5.1

package/docs/guides/ops-scripts.md

@@ -0,0 +1,271 @@
+# Operations Scripts
+
+Reference guide for the operational scripts in `scripts/`. Run these from the project root.
+
+> Looking for `cerefox <subcommand>` reference (ingest, search, get-doc, etc.)? See [`docs/guides/cli.md`](cli.md). This guide covers the `scripts/` directory only.
+
+## Two languages, one directory
+
+As of v0.3.0, Cerefox scripts come in two flavors:
+
+| Script | Language | Run with |
+|---|---|---|
+| `db_status.ts` | TypeScript (v0.3.0+) | `bun scripts/db_status.ts` |
+| `sync_docs.ts` | TypeScript (v0.3.0+) | `bun scripts/sync_docs.ts` |
+| `db_deploy.py` | Python | `uv run python scripts/db_deploy.py` |
+| `db_migrate.py` | Python | `uv run python scripts/db_migrate.py` |
+| `backup_create.py` | Python | `uv run python scripts/backup_create.py` |
+| `backup_restore.py` | Python | `uv run python scripts/backup_restore.py` |
+| `reindex_all.py` | Python | `uv run python scripts/reindex_all.py` |
+
+The TS scripts require [Bun](https://bun.sh) — install with `curl -fsSL https://bun.sh/install | bash`. They are functionally equivalent to their previous Python forms; **the legacy `db_status.py` and `sync_docs.py` are deprecation shims that exit non-zero with a pointer to the TS replacement**. The shims are kept indefinitely as a migration aid — there's no scheduled removal date, but the exit code stays non-zero, so update any cron jobs / CI / make targets that invoke them.
+
+The remaining `.py` scripts stay Python until their scheduled port (v0.5 for `db_deploy` / `db_migrate`; v0.7 for `backup_*` / `reindex_all`) — per the §12f script-language policy in [`CONTRIBUTING.md`](../../CONTRIBUTING.md), Python scripts get ported when they're extended.
+
+### TS scripts and `.env` resolution
+
+`bun scripts/<name>.ts` reads the same `.env` the Python CLI does. Precedence:
+
+1. `CEREFOX_CONFIG_DIR` env var (explicit override; supports `~`).
+2. `./.env` in the current working directory (dev mode).
+3. `~/.cerefox/.env` (user-state root).
+
+See [`docs/specs/polish-and-distribution-design.md` §7b](../specs/polish-and-distribution-design.md) for the full rule.
+
+---
+
+## db_deploy.py — Schema deployment
+
+Applies the full Cerefox schema (tables, indexes, RPC functions) to a Postgres database. Use this for **fresh installs** or to re-apply the schema after a Cerefox update.
+
+```bash
+uv run python scripts/db_deploy.py [OPTIONS]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--dry-run` | Print the SQL that would be executed, without running it |
+| `--reset` | Drop all `cerefox_*` tables before deploying (destructive) |
+
+**Requires**: `CEREFOX_DATABASE_URL` — a direct Postgres connection URL (not the Supabase API URL).
+
+After applying the schema, `db_deploy.py` automatically stamps any migration files in `src/cerefox/db/migrations/` into the `cerefox_migrations` table. This ensures `db_migrate.py` does not re-apply changes that are already incorporated in the base schema.
+
+Example:
+```bash
+# Deploy to local Docker Postgres
+CEREFOX_DATABASE_URL=postgresql://cerefox:cerefox@localhost:5432/cerefox \
+  uv run python scripts/db_deploy.py
+```
+
+---
+
+## db_status.ts — Schema verification
+
+**TypeScript (v0.3.0+).** Checks that the schema is correctly deployed and reports table statistics. Replaces the legacy `db_status.py`, which now prints a deprecation notice and exits non-zero.
+
+```bash
+bun scripts/db_status.ts          # human-readable report
+bun scripts/db_status.ts --json   # structured JSON output
+```
+
+Reports:
+- Tables: `cerefox_documents`, `cerefox_chunks`, `cerefox_document_versions`, `cerefox_projects`, `cerefox_document_projects`, `cerefox_audit_log`, `cerefox_migrations`
+- RPC functions: hybrid_search, fts_search, semantic_search, reconstruct_doc, save_note, search_docs, context_expand, snapshot_version, get_document, list_document_versions, ingest_document, delete_document, create_audit_entry, list_audit_entries, list_metadata_keys, update_chunk_fts, **`cerefox_schema_version`** (new in v0.3.0), **`cerefox_pg_function_exists`** (new in v0.3.0)
+- Row counts per table
+- **Schema-version mismatch**: compares the `@version` marker in the bundled `schema.sql` against the deployed `cerefox_schema_version()` RPC. Non-zero exit if they differ (the same check powers the web UI's schema-mismatch banner).
+
+Exit code 0 if everything is healthy; 1 if any check fails; 2 on configuration error.
+
+**Function-existence detection** routes through the `cerefox_pg_function_exists()` introspection RPC for reliability. Legacy deployments missing that RPC fall back to a naive "call with no args" probe — the legacy fallback will misreport RPCs that take required parameters as missing, which is itself a signal that the deployment needs `db_deploy.py`.
+
+---
+
+## db_migrate.py — Schema migrations
+
+Applies incremental migration files to an **existing** database with data. Use this when upgrading Cerefox on a database that already has documents — it applies only the changes that haven't been applied yet.
+
+```bash
+uv run python scripts/db_migrate.py [OPTIONS]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--dry-run` | Show which migrations would run, without applying them |
+| `--status` | List all migration files and whether each has been applied |
+
+**When to use `db_deploy.py` vs `db_migrate.py`:**
+
+| Situation | Use |
+|-----------|-----|
+| Fresh database, no data | `db_deploy.py` |
+| Existing database, upgrading to a new version | `db_migrate.py` |
+
+On a freshly deployed database, `db_migrate.py` is always a no-op — `db_deploy.py` has already stamped all existing migrations.
+
+Migration files live in `src/cerefox/db/migrations/` and are applied in filename order (`0001_...`, `0002_...`). Each file is applied exactly once; applied filenames are recorded in the `cerefox_migrations` table.
+
+Always run a backup before migrating:
+
+```bash
+uv run python scripts/backup_create.py && uv run python scripts/db_migrate.py
+```
+
+---
+
+## backup_create.py — Create a backup
+
+Exports all documents, chunks, and metadata to a JSON file in the backup directory.
+
+```bash
+uv run python scripts/backup_create.py [OPTIONS]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--label LABEL` | Optional label appended to the filename (e.g. `pre-migration`) |
+| `--dir DIR` | Directory to write backup to (default: `./backup-data`) |
+| `--git-commit` | Stage and commit the backup file to git after writing |
+
+Backup filename format: `cerefox-{YYYYMMDDTHHMMSSZ}[-{label}].json`
+
+**Versioning note**: Backups capture only **current** chunks (those not yet archived). Archived version history (previous content snapshots) is intentionally excluded — backups represent the present state of your knowledge base, not its history. Archived versions remain in the database and continue to be accessible via the versioning API until they expire.
+
+Example:
+```bash
+uv run python scripts/backup_create.py --label before-v2-migration
+```
+
+Output: `backup-data/cerefox-20260308T143022Z-before-v2-migration.json`
+
+---
+
+## backup_restore.py — Restore from a backup
+
+Restores documents and chunks from a previously created backup file. Idempotent — documents with the same content hash are skipped.
+
+```bash
+uv run python scripts/backup_restore.py BACKUP_FILE [OPTIONS]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--dry-run` | Show what would be restored without writing |
+
+Example:
+```bash
+# Preview what will be restored
+uv run python scripts/backup_restore.py backup-data/cerefox-20260308T143022Z.json --dry-run
+
+# Restore
+uv run python scripts/backup_restore.py backup-data/cerefox-20260308T143022Z.json
+```
+
+Restore output shows counts of restored / skipped / error documents.
+
+---
+
+## Backup format
+
+Backups are JSON files with the following structure:
+
+```json
+{
+  "version": 1,
+  "created_at": "2026-03-08T14:30:22.000Z",
+  "document_count": 42,
+  "chunk_count": 317,
+  "documents": [
+    {
+      "id": "uuid",
+      "title": "My Note",
+      "source": "file",
+      "content_hash": "sha256hex",
+      "metadata": {},
+      "chunks": [
+        {
+          "chunk_index": 0,
+          "heading_path": ["My Note", "Section"],
+          "heading_level": 2,
+          "title": "Section",
+          "content": "...",
+          "char_count": 120,
+          "embedder_primary": "text-embedding-3-small",
+          "embedding_primary": [0.012, -0.034, ...],
+          "embedding_upgrade": null
+        }
+      ]
+    }
+  ]
+}
+```
+
+**Embeddings are included** in backups. This means a restored database is immediately searchable — no `cerefox reindex` required after restore.
+
+The backup directory (`./backup-data/` by default) is gitignored. Back up the backup files separately if you want off-site copies (e.g. copy to cloud storage).
+
+---
+
+## sync_docs.ts — Sync project documentation into Cerefox
+
+**TypeScript (v0.3.0+).** Ingests `README.md`, `AGENT_GUIDE.md`, `AGENT_QUICK_REFERENCE.md`, and every Markdown file under `docs/` into your Cerefox knowledge base, updating existing documents in-place. Run this any time after editing documentation so AI agents always have access to the current state of the project.
+
+Replaces the legacy `sync_docs.py`, which now prints a deprecation notice and exits non-zero.
+
+```bash
+bun scripts/sync_docs.ts [OPTIONS]
+```
+
+| Option | Description |
+|--------|-------------|
+| `--project NAME`, `-p NAME` | Project to assign documents to (default: `cerefox`) |
+| `--dry-run`, `-n` | List files that would be synced without ingesting anything |
+
+**Requires**: `CEREFOX_SUPABASE_URL` and `CEREFOX_SUPABASE_ANON_KEY` (the legacy anon JWT — `eyJ…` — used to invoke Edge Functions). Embedding happens server-side inside the `cerefox-ingest` Edge Function, so you don't need an OpenAI / Fireworks key in your local env for the TS script.
+
+The target project must already exist (create it with `uv run cerefox create-project cerefox` if needed).
+
+**What gets synced**: `README.md` + `AGENT_GUIDE.md` + `AGENT_QUICK_REFERENCE.md` + all `.md` files under `docs/` (including `docs/research/` and `docs/specs/`). Research notes are included because Cerefox is a shared memory layer for multiple agents — exploratory notes, experiments, and decision rationale are exactly the kind of context agents benefit from. Files are matched to existing documents by their relative path (`source_path`), so re-running the script updates content in-place rather than creating duplicates.
+
+Example output:
+```
+Syncing 22 file(s) → project "cerefox"
+  =  README.md  (Cerefox)                            [unchanged]
+  ↑  docs/plan.md  (Cerefox Implementation Plan)     [re-embedded]
+  =  docs/guides/quickstart.md  (Quickstart)         [unchanged]
+  ...
+Done. 0 new · 1 updated · 21 unchanged · 0 errors
+```
+
+---
+
+## Recommended backup schedule
+
+For a personal knowledge base, a simple daily cron is sufficient:
+
+```cron
+0 3 * * * cd /path/to/cerefox && uv run python scripts/backup_create.py --label daily
+```
+
+Backups include embeddings so they are larger than pure-text exports, but for a personal knowledge base they typically remain well under 100 MB.
+
+---
+
+## CLI commands
+
+The `cerefox` CLI also provides data management commands:
+
+| Command | Description |
+|---------|-------------|
+| `uv run cerefox ingest FILE` | Ingest a markdown file |
+| `uv run cerefox ingest --paste --title TITLE` | Ingest text from stdin |
+| `uv run cerefox search QUERY` | Search the knowledge base |
+| `uv run cerefox list-docs` | List all documents |
+| `uv run cerefox delete-doc ID` | Delete a document by ID |
+| `uv run cerefox list-projects` | List all projects |
+| `uv run cerefox list-versions ID` | List all archived versions of a document |
+| `uv run cerefox get-doc ID` | Retrieve current content of a document |
+| `uv run cerefox get-doc ID --version VERSION_ID` | Retrieve a specific archived version |
+| `uv run cerefox web` | Start the web UI |
+
+Run `uv run cerefox --help` or `uv run cerefox COMMAND --help` for details.

package/docs/guides/quickstart.md

@@ -0,0 +1,165 @@
+# Quickstart -- Zero to First Document in 15 Minutes
+
+Get Cerefox running locally and ingest your first document.
+
+> **Upgrading from a previous version?** See the [Upgrading Guide](upgrading.md) for migration steps instead.
+
+---
+
+## 1. Prerequisites (2 min)
+
+- Python 3.11+ (`python3 --version`)
+- Node.js 18+ and npm (`node --version`)
+- `uv` package manager (`pip install uv`)
+- A Supabase account -- [supabase.com](https://supabase.com) (free tier works)
+- An OpenAI API key -- [platform.openai.com/api-keys](https://platform.openai.com/api-keys)
+
+---
+
+## 2. Install Cerefox (2 min)
+
+```bash
+git clone https://github.com/fstamatelopoulos/cerefox.git
+cd cerefox
+uv sync
+```
+
+> No heavy ML model downloads needed -- embeddings are handled by the OpenAI API.
+
+---
+
+## 3. Set up Supabase (5 min)
+
+1. Create a new Supabase project at [app.supabase.com](https://app.supabase.com).
+2. Go to **Project Settings → API → Project URL** and copy it. Also note your project ref (the slug in the URL, e.g. `abcd1234`).
+3. Go to **Project Settings → API Keys** and copy the **Secret key** (`sb_secret_…`). The legacy `service_role` JWT also works if you prefer; either goes into `CEREFOX_SUPABASE_KEY`. See [`setup-supabase.md` → Supabase API keys (2026)](setup-supabase.md#supabase-api-keys-2026) for the full key story (including why the anon key, if you ever need it, must currently stay as the legacy JWT — `sb_publishable_…` does not work for Edge Functions).
+4. Go to **Project Settings → Database → Connection pooling** and copy the **Session Pooler** URI (host ends `.pooler.supabase.com`, port `5432`). If you only see the Transaction Pooler in the dashboard, take that URI and change `:6543` → `:5432`. **Do not use port 6543** — Transaction Pooler does not support DDL. See [`setup-supabase.md` → Connection pooling (2026)](setup-supabase.md#connection-pooling-2026) for context.
+
+Create a `.env` file:
+
+```env
+CEREFOX_SUPABASE_URL=https://your-project-ref.supabase.co
+CEREFOX_SUPABASE_KEY=sb_secret_...your-supabase-secret-key...
+CEREFOX_DATABASE_URL=postgresql://postgres.your-project-ref:your-db-password@aws-N-region.pooler.supabase.com:5432/postgres?sslmode=require
+OPENAI_API_KEY=sk-...your-openai-key...
+```
+
+The username must include the `.<project-ref>` suffix (e.g. `postgres.abcd1234`) — without it, Supabase returns "Tenant or user not found".
+
+---
+
+## 4. Deploy the schema (1 min)
+
+```bash
+uv run python scripts/db_deploy.py
+```
+
+You should see all steps complete with a final `Done` message.
+
+Verify:
+```bash
+uv run python scripts/db_status.py
+```
+
+This should show all checks passed.
+
+---
+
+## 5. Ingest your first document (2 min)
+
+Have a markdown file? Ingest it:
+
+```bash
+uv run cerefox ingest my-notes.md
+```
+
+Or paste directly from the terminal:
+
+```bash
+echo "# My First Note
+
+This is the beginning of my personal knowledge base." | uv run cerefox ingest --paste --title "First Note"
+```
+
+---
+
+## 6. Build and start the web app (1 min)
+
+Build the React frontend:
+
+```bash
+cd frontend && npm install && npm run build && cd ..
+```
+
+Start the web app:
+
+```bash
+uv run cerefox web
+```
+
+Open [http://localhost:8000/app/](http://localhost:8000/app/) -- your dashboard is live.
+
+> The root URL (`http://localhost:8000/`) redirects to `/app/` automatically.
+
+---
+
+## 7. Search your knowledge (30 sec)
+
+From the CLI:
+
+```bash
+uv run cerefox search "my first note"
+```
+
+Or use the web UI search page at [http://localhost:8000/app/search](http://localhost:8000/app/search).
+
+---
+
+## 8. Connect an AI agent (optional, 5 min)
+
+Cerefox ships a built-in MCP server. Add it to Claude Desktop's config file
+(`~/Library/Application Support/Claude/claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "cerefox": {
+      "command": "uv",
+      "args": ["--directory", "/path/to/cerefox", "run", "cerefox", "mcp"]
+    }
+  }
+}
+```
+
+Replace `/path/to/cerefox` with the absolute path to this checkout. Restart Claude Desktop.
+
+> **Recommended: remote MCP** -- if you deployed the Edge Functions (see the main
+> README), use the remote MCP path instead -- no Python install needed on the client machine.
+> See `docs/guides/connect-agents.md` for Path A-Remote.
+>
+> **ChatGPT** does not support MCP -- use a Custom GPT with
+> Edge Functions instead (see `docs/guides/connect-agents.md`, Path B).
+
+For full setup details (remote MCP, Cursor, cloud clients, GPT Actions), see `docs/guides/connect-agents.md`.
+
+---
+
+## You're done!
+
+**What's next:**
+- Ingest a directory of notes: `cerefox ingest-dir ./notes/ --recursive`
+- Re-embed existing content: `cerefox reindex`
+- Create a backup: `python scripts/backup_create.py`
+- Sync project docs into your knowledge base: `python scripts/sync_docs.py`
+  (this also ingests the agent reference guides -- `AGENT_GUIDE.md` and `AGENT_QUICK_REFERENCE.md` --
+  so your AI agents can search for "How AI Agents Use Cerefox" and learn how to use the tools)
+- See all commands: `cerefox --help`
+
+**More guides:**
+- `AGENT_GUIDE.md` -- comprehensive reference for AI agents using Cerefox tools
+- `AGENT_QUICK_REFERENCE.md` -- minimal quick reference card for AI agents
+- `docs/guides/setup-supabase.md` -- detailed Supabase setup
+- `docs/guides/configuration.md` -- all configuration options
+- `docs/guides/connect-agents.md` -- connecting AI agents via MCP and Edge Functions
+- `docs/guides/setup-local.md` -- local Docker setup (no Supabase account needed)
+- `docs/guides/upgrading.md` -- upgrading from a previous version

package/docs/guides/response-limits.md

@@ -0,0 +1,151 @@
+# Cerefox Response Size Limits
+
+Cerefox returns content from your knowledge base — documents can be large, and returning
+too much in a single search response can overwhelm an AI agent's context window. This guide
+explains how response size limits work and how to tune them.
+
+---
+
+## The key principle: opt-in limits, never truncate the web UI
+
+The web UI and CLI never truncate results. They have no size limit — the browser or terminal
+can handle arbitrarily large responses and there is no LLM context window to worry about.
+
+Limits are **opt-in per call**, used only on the MCP and Edge Function paths where an AI
+agent's context window matters. Callers always choose whether to apply a limit.
+
+---
+
+## How each access path handles response size
+
+| Path | Limit behaviour |
+|------|----------------|
+| Web UI (`/search`) | **No limit** — all results returned |
+| CLI (`cerefox search`) | **No limit** — all results returned |
+| Local MCP server (`cerefox mcp`) | Defaults to `CEREFOX_MAX_RESPONSE_BYTES` (200 000); agent can request less |
+| Edge Function (`cerefox-search`) | Defaults to 200 000 bytes; agent can request less via `max_bytes` body param |
+| Remote MCP (`cerefox-mcp` Edge Function) | Defaults to 200 000 bytes; agent can request less via `max_bytes` tool param |
+
+---
+
+## How limits are applied
+
+Truncation is always **whole-document**: results are dropped in full once adding the next
+document would exceed the budget. Cerefox never cuts a document mid-content.
+
+When truncation occurs:
+- The local MCP server appends `[Results truncated at N bytes — ...]` to the response text.
+- The Edge Function includes `"truncated": true` and `"response_bytes": N` in the JSON response.
+
+---
+
+## The server ceiling — agents can request less, never more
+
+For both the local MCP server and the `cerefox-search` Edge Function, the server-side
+maximum acts as a hard ceiling. An agent can pass a smaller `max_bytes` value; a larger
+value is silently capped.
+
+```
+effective_max = min(agent_requested_max, SERVER_MAX)
+```
+
+The Edge Function's `SERVER_MAX` is `200 000` bytes (hardcoded TypeScript constant).
+The local MCP server's ceiling is `CEREFOX_MAX_RESPONSE_BYTES` from `.env`.
+
+---
+
+## Configuring the local MCP server limit
+
+Set `CEREFOX_MAX_RESPONSE_BYTES` in `.env`:
+
+```env
+CEREFOX_MAX_RESPONSE_BYTES=200000
+```
+
+This value is used as both the **default** and the **ceiling** for the local MCP server.
+Agents can pass a smaller `max_bytes` in the tool call, but never larger.
+
+When should you lower this?
+- Your MCP client (Claude Desktop, Cursor) has a small context window
+- You want tighter, more focused responses at the cost of potentially seeing fewer results
+
+When should you raise it?
+- You use high `match_count` values (e.g. 20) and want all results returned
+- Your documents are large and you want full content even for large-document results
+
+---
+
+## Passing `max_bytes` as an agent
+
+The `cerefox_search` MCP tool accepts an optional `max_bytes` parameter in both the local
+and remote MCP paths. Pass it when you want the response to fit within a specific budget:
+
+```json
+{
+  "query": "knowledge management",
+  "max_bytes": 50000
+}
+```
+
+Values above the server ceiling are silently capped. Omitting `max_bytes` uses the server
+default (200 000).
+
+The `cerefox-search` Edge Function (Path B / GPT Actions) also accepts `max_bytes` as a
+JSON body field:
+
+```http
+POST https://<project>.supabase.co/functions/v1/cerefox-search
+Authorization: Bearer <legacy-anon-jwt>   # see docs/guides/setup-supabase.md#supabase-api-keys-2026
+Content-Type: application/json
+
+{
+  "query": "knowledge management",
+  "max_bytes": 50000
+}
+```
+
+---
+
+## Why 200 000 bytes?
+
+200 KB is a safe ceiling that prevents pathologically large responses (e.g. very high
+`match_count` combined with many large documents) while never cutting legitimate results
+at the default `match_count=5`.
+
+**Worst-case budget at default settings:**
+5 documents × 20 000 chars each (the small-to-big threshold) ≈ 100 KB — comfortably under
+200 KB. In practice, most documents are shorter and the limit is rarely reached.
+
+The original 65 KB default was driven by the Supabase MCP protocol limit, which no longer
+applies (Cerefox now uses its own `cerefox-mcp` Edge Function for remote MCP access).
+
+---
+
+## How small-to-big retrieval complements the limit
+
+For large documents (over 20 000 chars by default), `cerefox_search_docs` returns only the
+matched chunks plus their immediate neighbours, not the full document text. This means a
+single large document contributes only a few kilobytes to the response rather than tens of
+kilobytes.
+
+This **small-to-big threshold** acts as a complementary guard that keeps individual document
+contributions compact. The response size limit then governs the total across all returned
+documents.
+
+See `docs/guides/configuration.md` → "RPC-level retrieval parameters" to change the
+threshold (it is a SQL DEFAULT in `rpcs.sql`, changed via `db_deploy.py`).
+
+---
+
+## Summary
+
+| Question | Answer |
+|----------|--------|
+| Does the web UI truncate results? | No — unlimited |
+| Does the CLI truncate results? | No — unlimited |
+| What is the default MCP response limit? | 200 000 bytes |
+| Can an agent request a smaller limit? | Yes — `max_bytes` tool parameter |
+| Can an agent exceed the server ceiling? | No — always capped |
+| Where is the ceiling configured? | `.env` for local MCP; TypeScript constant in Edge Functions |
+| How are limits applied? | Whole-document drop; never mid-content truncation |
+| Is truncation signalled? | Yes — `truncated: true` in responses |