@cerefox/memory 0.11.1 → 1.0.0-beta.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/AGENT_GUIDE.md +2 -2
  2. package/AGENT_QUICK_REFERENCE.md +1 -1
  3. package/dist/bin/cerefox.js +1079 -836
  4. package/dist/frontend/assets/index-D3FshoP3.js +125 -0
  5. package/dist/frontend/assets/index-D3FshoP3.js.map +1 -0
  6. package/dist/frontend/index.html +1 -1
  7. package/dist/server-assets/_shared/ef-auth/index.ts +134 -0
  8. package/dist/server-assets/_shared/ef-meta/index.ts +1 -1
  9. package/dist/server-assets/_shared/embeddings/index.ts +42 -2
  10. package/dist/server-assets/_shared/ingest/chunker.ts +210 -0
  11. package/dist/server-assets/_shared/ingest/index.ts +32 -0
  12. package/dist/server-assets/_shared/ingest/pipeline-helpers.ts +135 -0
  13. package/dist/server-assets/_shared/mcp-auth/index.ts +352 -0
  14. package/dist/server-assets/_shared/mcp-tools/_chunker.ts +16 -170
  15. package/dist/server-assets/_shared/mcp-tools/get-help-content.ts +2 -2
  16. package/dist/server-assets/_shared/mcp-tools/ingest.ts +13 -4
  17. package/dist/server-assets/db/migrations/0012_content_format.sql +20 -0
  18. package/dist/server-assets/db/rpcs.sql +76 -8
  19. package/dist/server-assets/db/schema.sql +7 -1
  20. package/dist/server-assets/supabase/functions/cerefox-get-audit-log/index.ts +8 -0
  21. package/dist/server-assets/supabase/functions/cerefox-get-document/index.ts +8 -0
  22. package/dist/server-assets/supabase/functions/cerefox-ingest/index.ts +39 -173
  23. package/dist/server-assets/supabase/functions/cerefox-list-projects/index.ts +8 -0
  24. package/dist/server-assets/supabase/functions/cerefox-list-versions/index.ts +8 -0
  25. package/dist/server-assets/supabase/functions/cerefox-mcp/index.ts +54 -0
  26. package/dist/server-assets/supabase/functions/cerefox-mcp/oauth.ts +121 -0
  27. package/dist/server-assets/supabase/functions/cerefox-metadata/index.ts +8 -0
  28. package/dist/server-assets/supabase/functions/cerefox-metadata-search/index.ts +8 -0
  29. package/dist/server-assets/supabase/functions/cerefox-search/index.ts +11 -1
  30. package/docs/guides/access-paths.md +84 -35
  31. package/docs/guides/cli.md +29 -0
  32. package/docs/guides/configuration.md +10 -8
  33. package/docs/guides/connect-agents.md +99 -101
  34. package/docs/guides/content-format.md +55 -0
  35. package/docs/guides/migration-1.0.md +96 -0
  36. package/docs/guides/ops-scripts.md +5 -7
  37. package/docs/guides/quickstart.md +22 -2
  38. package/docs/guides/setup-cloud-run.md +5 -9
  39. package/docs/guides/setup-supabase.md +157 -16
  40. package/docs/guides/upgrading.md +7 -8
  41. package/package.json +1 -1
  42. package/dist/frontend/assets/index-ojNhWSxm.js +0 -125
  43. package/dist/frontend/assets/index-ojNhWSxm.js.map +0 -1
@@ -30,24 +30,27 @@ Three top-level paths plus a few special cases:
30
30
  | Client | Path | Search | Requirements / caveats |
31
31
  |--------|------|--------|-----------------------|
32
32
  | Claude Desktop (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | Node.js for `npx supergateway` or `npx mcp-remote`; no Python needed |
33
- | Claude Code (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
34
- | Cursor (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
35
- | OpenAI Codex CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key env var; TOML config |
33
+ | Claude Code (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
34
+ | Cursor (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
35
+ | OpenAI Codex CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token env var; TOML config. Advanced/fallback — prefer Path A-Local |
36
36
  | ChatGPT (chatgpt.com or desktop) | Path B — Custom GPT → Edge Functions | Hybrid | ChatGPT Plus required |
37
37
  | Claude Desktop (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; Node.js; zero Edge Function invocations |
38
38
  | Claude Code (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; zero Edge Function invocations |
39
39
  | Cursor (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; zero Edge Function invocations |
40
- | Cloud Claude (claude.ai web) | Remote Supabase MCP | FTS only | No install; search quality limited |
41
- | Gemini CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
40
+ | Cloud Claude (claude.ai web + mobile) | Path A-Remote — `cerefox-mcp` over **OAuth** | Hybrid | No install; **optional** one-time OAuth setup + a free Cloudflare Worker consent page ([setup-supabase Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)) |
41
+ | Gemini CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
42
42
  | Local coding agents (Claude Code, Codex CLI, opencode, OpenClaw, Hermes, …) | Path C — Shell CLI (Bash tool) | Hybrid | `npm install -g @cerefox/memory`; agent runs `cerefox …` as a shell command. Useful when MCP setup is friction. |
43
43
  | curl / scripts | Path B — Edge Functions directly | Hybrid | Direct HTTP; no client needed |
44
- | Custom Python agents | Python SDK directly (legacy) | Hybrid | Local Python + repo clone; the Python path is legacy/frozen |
45
44
 
46
45
  > **"Hybrid"** = FTS + semantic, document-level (complete reconstructed notes, not isolated chunks).
47
46
  > **"FTS only"** = keyword search only; no semantic/vector search.
48
47
 
49
- > **Cloud hybrid for all clients (future)**: deploying the MCP server to Cloud Run would give
50
- > cloud clients (claude.ai, chatgpt.com) full hybrid search. Tracked in `docs/TODO.md`.
48
+ > **Cloud hybrid for claude.ai / mobile (iter-28A)**: `cerefox-mcp` is now an OAuth 2.1
49
+ > protected resource, so claude.ai web and the Claude mobile app get **full hybrid search**
50
+ > over the standard tool surface — no Cloud Run needed. Setup:
51
+ > [setup-supabase Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
52
+ > + [Cloud Claude](#cloud-claude-claudeai-web--mobile-oauth) below. (An OAuth connector for
53
+ > ChatGPT becomes possible on the same server but is not yet documented.)
51
54
 
52
55
  > **Perplexity** supports stdio-only MCP on macOS Desktop (via Helper App). Remote MCP is
53
56
  > "coming soon." Perplexity's CTO has signalled a strategic shift away from MCP (March 2026),
@@ -88,20 +91,24 @@ in the container.
88
91
  in the per-client configs below if your client can't see the file)
89
92
  - `.env` must define `CEREFOX_SUPABASE_URL`, `CEREFOX_SUPABASE_KEY`, and your
90
93
  embedding API key (`OPENAI_API_KEY`)
91
- - A frozen Python MCP server still exists as a standalone fallback (`uv run cerefox mcp` from a
92
- repo clone), but the npm package is the maintained path.
93
-
94
- > **Important — which anon key to use (2026):** Path A-Remote and Path B both require an
95
- > "anon key" as a Bearer token. As of 2026, you **must** use the **legacy anon JWT**
96
- > (`eyJ…`) — the new `sb_publishable_…` key is rejected by the Supabase Edge Function
97
- > gateway with `UNAUTHORIZED_INVALID_JWT_FORMAT`. Find the legacy key in **Project
98
- > Settings → API Keys → Legacy → anon**. This is a Supabase platform constraint;
99
- > see [`setup-supabase.md` → Supabase API keys (2026)](setup-supabase.md#supabase-api-keys-2026)
100
- > for the full story.
101
-
102
- **For Path A-Remote (remote MCP Edge Function) — recommended:**
94
+
95
+ > **Important — the Cerefox access token (iter-28E):** Path A-Remote and Path B both require a
96
+ > **Bearer token** on every request. That credential is now the **Cerefox access token**
97
+ > (`cfx_pat_…`), a random, Cerefox-managed secret validated in-function — **not** the legacy
98
+ > Supabase anon JWT (which is retired for all Edge Function paths). Generate it with
99
+ > `cerefox token generate`: it sets the accepted token set on Supabase (the
100
+ > `CEREFOX_ACCESS_TOKENS` Function secret) and writes the value to your local `.env` as
101
+ > `CEREFOX_ACCESS_TOKEN`, printing it once so you can paste it into client configs. Lose it →
102
+ > `cerefox token rotate`. See [`setup-supabase.md` → Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
103
+ > for the server-side setup.
104
+
105
+ **For Path A-Remote (remote MCP Edge Function) — advanced / fallback:**
106
+ > For **local agents** (Claude Code, Cursor, Codex, Gemini, Claude Desktop) the **local MCP**
107
+ > (Path A-Local, via `cerefox configure-agent`) is the preferred path — zero Edge Function
108
+ > cost and no token to distribute. Use Path A-Remote when you specifically want a hosted URL
109
+ > (multiple machines, cloud dev environments) or as a fallback.
103
110
  - `cerefox-mcp` Edge Function deployed (`npx supabase functions deploy cerefox-mcp`)
104
- - Your **legacy anon JWT** (see callout above): Supabase Dashboard → Project Settings → API Keys → Legacy → anon
111
+ - Your **Cerefox access token** (see callout above): run `cerefox token generate`
105
112
  - For Claude Desktop: [Node.js](https://nodejs.org) installed (for `npx supergateway` or `npx mcp-remote`)
106
113
  - For Claude Code: [Node.js](https://nodejs.org) for `npx mcp-remote` (recommended), or no extra deps for native HTTP
107
114
 
@@ -111,7 +118,7 @@ in the container.
111
118
  `cerefox-get-audit-log`, `cerefox-metadata-search`, `cerefox-list-projects`, `cerefox-mcp`.
112
119
  End-user path: `cerefox server deploy`. Contributor/manual path: `npx supabase functions
113
120
  deploy` (see `setup-supabase.md`).
114
- - Your **legacy anon JWT** (see callout above): Supabase Dashboard → Project Settings → API Keys → Legacy → anon
121
+ - Your **Cerefox access token** (see callout above): run `cerefox token generate`
115
122
  - Your **project ref**: visible in the Supabase Dashboard URL
116
123
  (`app.supabase.com/project/<project-ref>`)
117
124
 
@@ -130,12 +137,6 @@ Edge Function, communicating with clients over stdio.
130
137
  The local server ships as an npm package — **[`@cerefox/memory`](https://www.npmjs.com/package/@cerefox/memory)** — built with the official `@modelcontextprotocol/sdk`.
131
138
  The bin entry is `cerefox` (run as `cerefox mcp`). The recommended client config is `npx -y --package=@cerefox/memory cerefox mcp`, or if you've installed the package globally, just `cerefox mcp`.
132
139
 
133
- A separate, **frozen** Python MCP server still exists as a standalone fallback — invoke it
134
- explicitly with `uv run cerefox mcp` from a Cerefox repo clone. It is independent and
135
- unmaintained; the npm package is the maintained path. (A "soft wrapper" that auto-delegated
136
- the Python `cerefox mcp` to the npm package was removed in v0.5.2 — the two are now fully
137
- separate. Pick one explicitly in your MCP client config.)
138
-
139
140
  - Embeddings are computed locally using your `.env` key (no extra credentials)
140
141
  - Works offline except for the OpenAI embedding API call per query
141
142
  - One setup, all compatible local clients (Claude Desktop, Cursor, Claude Code, Codex CLI, …)
@@ -317,8 +318,8 @@ https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp
317
318
 
318
319
  | Scenario | Prefer |
319
320
  |----------|--------|
320
- | Default / new setup | Path A-Remote -- no Python, no local clone, one URL works everywhere |
321
- | Multiple machines / cloud dev environments | Path A-Remote |
321
+ | Default local agent (Claude Code, Cursor, Codex, Gemini, Claude Desktop) | Path A-Local -- preferred; `cerefox configure-agent`, zero Edge Function cost, no token to distribute |
322
+ | Multiple machines / cloud dev environments | Path A-Remote -- one hosted URL works everywhere (needs a Cerefox token) |
322
323
  | Minimise Supabase Edge Function usage (free tier limits) | Path A-Local -- zero Edge Function invocations |
323
324
  | Offline use or development on the cerefox codebase | Path A-Local -- no network dependency |
324
325
  | Lowest latency (same machine, no HTTPS round-trip) | Path A-Local -- slightly faster |
@@ -355,7 +356,7 @@ Add to your project's `.mcp.json` (or copy
355
356
  "mcp-remote",
356
357
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
357
358
  "--header",
358
- "Authorization: Bearer <your-anon-key>"
359
+ "Authorization: Bearer <your-cerefox-token>"
359
360
  ]
360
361
  }
361
362
  }
@@ -370,7 +371,7 @@ overhead (fixed in v0.1.12). However, `mcp-remote` is still preferred for the OA
370
371
  ```bash
371
372
  claude mcp add --transport http cerefox \
372
373
  https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp \
373
- --header "Authorization: Bearer <your-anon-key>"
374
+ --header "Authorization: Bearer <your-cerefox-token>"
374
375
  ```
375
376
 
376
377
  Verify:
@@ -382,7 +383,7 @@ For a user-scoped server (available in all projects), add `--scope user`:
382
383
  ```bash
383
384
  claude mcp add --transport http --scope user cerefox \
384
385
  https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp \
385
- --header "Authorization: Bearer <your-anon-key>"
386
+ --header "Authorization: Bearer <your-cerefox-token>"
386
387
  ```
387
388
 
388
389
  ---
@@ -400,7 +401,7 @@ Cursor supports remote MCP servers natively via `url` + `headers` in `mcp.json`.
400
401
  "cerefox": {
401
402
  "url": "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
402
403
  "headers": {
403
- "Authorization": "Bearer <your-anon-key>"
404
+ "Authorization": "Bearer <your-cerefox-token>"
404
405
  }
405
406
  }
406
407
  }
@@ -441,19 +442,19 @@ Add (or merge into) the file:
441
442
  "args": [
442
443
  "-y", "supergateway",
443
444
  "--streamableHttp", "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
444
- "--oauth2Bearer", "<your-anon-key>"
445
+ "--oauth2Bearer", "<your-cerefox-token>"
445
446
  ]
446
447
  }
447
448
  }
448
449
  }
449
450
  ```
450
451
 
451
- Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
452
+ Replace `<your-project-ref>` and `<your-cerefox-token>` with your actual values.
452
453
 
453
454
  **Important:**
454
455
  - Restart Claude Desktop fully (Cmd+Q on macOS) after saving the config.
455
456
  - `-y` tells npx to auto-install `supergateway` without prompting.
456
- - No Python, no local repo clone, no `.env` file needed — just the URL and anon key.
457
+ - No Python, no local repo clone, no `.env` file needed — just the URL and Cerefox token.
457
458
 
458
459
  ---
459
460
 
@@ -462,13 +463,13 @@ Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
462
463
  [Codex](https://github.com/openai/codex) supports remote MCP servers natively via Streamable
463
464
  HTTP. Configuration uses TOML (not JSON like most other MCP clients).
464
465
 
465
- **Step 1 — Set the anon key as an environment variable:**
466
+ **Step 1 — Set the Cerefox token as an environment variable:**
466
467
 
467
468
  Codex references Bearer tokens by environment variable name, not by value. Add to your
468
469
  `~/.zshrc` (or `~/.bashrc`):
469
470
 
470
471
  ```bash
471
- export CEREFOX_ANON_KEY="<your-anon-key>"
472
+ export CEREFOX_ACCESS_TOKEN="<your-cerefox-token>"
472
473
  ```
473
474
 
474
475
  Then reload: `source ~/.zshrc`
@@ -478,7 +479,7 @@ Then reload: `source ~/.zshrc`
478
479
  ```toml
479
480
  [mcp_servers.cerefox]
480
481
  url = "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp"
481
- bearer_token_env_var = "CEREFOX_ANON_KEY"
482
+ bearer_token_env_var = "CEREFOX_ACCESS_TOKEN"
482
483
  ```
483
484
 
484
485
  Replace `<your-project-ref>` with your Supabase project ref.
@@ -489,9 +490,9 @@ Launch Codex and use the `/mcp` slash command to confirm the `cerefox` server is
489
490
  and all 10 tools are listed.
490
491
 
491
492
  **Notes:**
492
- - `bearer_token_env_var` is the **name** of the env var (e.g. `"CEREFOX_ANON_KEY"`), not the
493
+ - `bearer_token_env_var` is the **name** of the env var (e.g. `"CEREFOX_ACCESS_TOKEN"`), not the
493
494
  token itself. Codex reads the value at runtime.
494
- - No Python, no local repo clone needed — just the URL and anon key.
495
+ - No Python, no local repo clone needed — just the URL and Cerefox token.
495
496
  - No idle SSE polling cost — the 405 GET fix in `cerefox-mcp` prevents it.
496
497
 
497
498
  ---
@@ -514,14 +515,14 @@ Add (or merge into) the file:
514
515
  "cerefox": {
515
516
  "httpUrl": "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
516
517
  "headers": {
517
- "Authorization": "Bearer <your-anon-key>"
518
+ "Authorization": "Bearer <your-cerefox-token>"
518
519
  }
519
520
  }
520
521
  }
521
522
  }
522
523
  ```
523
524
 
524
- Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
525
+ Replace `<your-project-ref>` and `<your-cerefox-token>` with your actual values.
525
526
 
526
527
  **Verify:**
527
528
 
@@ -554,11 +555,11 @@ stored in Supabase.
554
555
  All Edge Function calls require:
555
556
 
556
557
  ```
557
- Authorization: Bearer <your-anon-key>
558
+ Authorization: Bearer <your-cerefox-token>
558
559
  Content-Type: application/json
559
560
  ```
560
561
 
561
- Find your anon key: **Supabase Dashboard → Project Settings → API Keys → Legacy → anon** (use the legacy JWT, not the new `sb_publishable_…` — see the API keys callout in the Prerequisites section).
562
+ Generate your token: run `cerefox token generate` (it prints the `cfx_pat_…` Cerefox access token once and sets it on Supabase). The legacy Supabase anon JWT is no longer accepted for Edge Function calls — see the token callout in the Prerequisites section.
562
563
 
563
564
  ### Path B system prompt
564
565
 
@@ -580,7 +581,7 @@ overwrite blindly.
580
581
  ```bash
581
582
  curl -s -X POST \
582
583
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-search" \
583
- -H "Authorization: Bearer <your-anon-key>" \
584
+ -H "Authorization: Bearer <your-cerefox-token>" \
584
585
  -H "Content-Type: application/json" \
585
586
  -d '{"query": "second brain", "match_count": 3}'
586
587
  ```
@@ -609,7 +610,7 @@ In the action editor, paste this schema (replace `<your-project-ref>`):
609
610
  openapi: 3.1.0
610
611
  info:
611
612
  title: Cerefox Knowledge Base
612
- version: 2.1.0
613
+ version: 3.0.0
613
614
  servers:
614
615
  - url: https://<your-project-ref>.supabase.co/functions/v1
615
616
  paths:
@@ -1001,11 +1002,15 @@ paths:
1001
1002
  In the action's **Authentication** settings:
1002
1003
  - Type: **API Key**
1003
1004
  - Auth type: **Bearer**
1004
- - API key: your Supabase **anon key**
1005
+ - API key: your **Cerefox access token** (`cfx_pat_…`). Generate it with
1006
+ `cerefox token generate` (it prints the token once and sets it on Supabase).
1007
+ The legacy Supabase anon key is no longer accepted (iter-28E).
1005
1008
 
1006
- > **Important:** ChatGPT may reset the API key when you update the action schema.
1007
- > If you get a 403 error after changing the schema, re-enter the anon key in the
1008
- > authentication settings — the functions themselves are fine.
1009
+ > **Important:** ChatGPT resets the stored API key when you update the action
1010
+ > schema. This release bumps the schema `info.version`, so on your next schema
1011
+ > paste ChatGPT will clear the key — re-enter your **Cerefox access token** in the
1012
+ > authentication settings. (That reset is expected here: it's what swaps you off
1013
+ > the old anon key.)
1009
1014
 
1010
1015
  **Step 4 — Save and test**
1011
1016
 
@@ -1025,7 +1030,7 @@ Direct HTTP access — useful for shell scripts, CI pipelines, or one-off querie
1025
1030
  ```bash
1026
1031
  curl -s -X POST \
1027
1032
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-search" \
1028
- -H "Authorization: Bearer <your-anon-key>" \
1033
+ -H "Authorization: Bearer <your-cerefox-token>" \
1029
1034
  -H "Content-Type: application/json" \
1030
1035
  -d '{"query": "knowledge management", "match_count": 5}'
1031
1036
  ```
@@ -1034,7 +1039,7 @@ curl -s -X POST \
1034
1039
  ```bash
1035
1040
  curl -s -X POST \
1036
1041
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-ingest" \
1037
- -H "Authorization: Bearer <your-anon-key>" \
1042
+ -H "Authorization: Bearer <your-cerefox-token>" \
1038
1043
  -H "Content-Type: application/json" \
1039
1044
  -d '{
1040
1045
  "title": "Meeting Notes 2026-03-11",
@@ -1101,20 +1106,50 @@ See `docs/guides/configuration.md` → "Response size limit" for full details.
1101
1106
 
1102
1107
  ---
1103
1108
 
1104
- ### Cloud Claude (claude.ai web)
1109
+ ### Cloud Claude (claude.ai web + mobile — OAuth) <a id="cloud-claude-claudeai-web--mobile-oauth"></a>
1110
+
1111
+ > **Optional feature.** This is the only path that needs the OAuth setup + a hosted
1112
+ > consent page. If you don't use claude.ai web / mobile, skip it — every other client
1113
+ > works without it. Design + rationale:
1114
+ > [`docs/specs/oauth-mcp-server-design.md`](../specs/oauth-mcp-server-design.md).
1105
1115
 
1106
- Claude.ai web can connect to the Supabase-hosted remote MCP (no local install):
1116
+ Claude.ai web **and the Claude mobile app** connect to your own `cerefox-mcp` Edge
1117
+ Function over **OAuth**, with the **full hybrid-search tool surface** — no local install.
1118
+ This replaces the old FTS-only `mcp.supabase.com` approach (raw keyword search over the
1119
+ tables, no semantic search, no Cerefox tool ergonomics).
1107
1120
 
1108
- 1. In Claude.ai: **Settings → Integrations → Add integration**
1109
- 2. Enter the MCP URL:
1121
+ **Prerequisite:** complete the one-time Supabase OAuth setup in
1122
+ [`setup-supabase.md` → Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
1123
+ — enable the OAuth 2.1 Server, deploy the Cloudflare Worker consent page, create the owner
1124
+ user + `CEREFOX_OAUTH_OWNER_ID` pin, and register the Claude OAuth App
1125
+ (**`client_secret_post`**) whose Client ID/Secret you paste below.
1126
+
1127
+ 1. In Claude.ai (web): **Settings → Connectors → Add custom connector**.
1128
+ 2. **Name**: `CerefoxMCP` (distinct from the local `cerefox` server, so both can coexist).
1129
+ 3. **URL** (must be the `*.supabase.co` host — custom domains break OAuth discovery):
1110
1130
  ```
1111
- https://mcp.supabase.com/sse?project_ref=<your-project-ref>
1131
+ https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp
1112
1132
  ```
1113
- 3. Authenticate with your Personal Access Token when prompted.
1114
-
1115
- > **Limitation**: The cloud Supabase MCP only supports **FTS keyword search** — no hybrid or
1116
- > semantic search. For full hybrid search from the web, deploy the MCP server to Cloud Run
1117
- > (see `docs/TODO.md` → "Remote HTTP MCP server").
1133
+ 4. **Advanced settings → OAuth Client ID / OAuth Client Secret**: paste the Client ID and
1134
+ Client Secret from the pre-registered OAuth App (setup-supabase Step 7d).
1135
+ 5. Save. Claude runs the OAuth flow → redirects you to the **Cerefox consent page** (sign
1136
+ in with the owner email/password from Step 7c, then **Allow**) → returns to Claude. The
1137
+ connector shows as connected with **10 tools**. (If you've approved before, Supabase
1138
+ auto-consents and the page just flashes through — that's expected.)
1139
+ 6. **Mobile**: connectors are account-level, so `CerefoxMCP` appears in the Claude mobile
1140
+ app automatically — run one search from your phone to confirm.
1141
+
1142
+ **Verify**: in a new chat with the connector enabled, ask *"Using CerefoxMCP, search the
1143
+ Cerefox Decision Log and give the title of the latest part."* A correct answer proves
1144
+ search + document reconstruction over the OAuth path.
1145
+
1146
+ > **If the connect fails right after consent** with an `ofid_…` reference: the OAuth App's
1147
+ > token endpoint auth method is wrong. It must be **`request body` (`client_secret_post`)**,
1148
+ > not HTTP Basic — see setup-supabase Step 7d. This is the most common mistake.
1149
+
1150
+ > **Cost note**: each cloud tool call is ~1 Edge Function invocation. Fine for interactive
1151
+ > use; for heavy/automated work prefer the local stdio server (zero EF cost). Your local
1152
+ > and static-Bearer clients (Claude Code, Cursor, Codex, Gemini, Desktop) are unchanged.
1118
1153
 
1119
1154
  ---
1120
1155
 
@@ -1237,39 +1272,6 @@ There is nothing Cerefox-specific to configure for the agent itself — just the
1237
1272
 
1238
1273
  ---
1239
1274
 
1240
- ## Custom agents (Python SDK — legacy)
1241
-
1242
- > **Legacy/frozen path.** The Python codebase is no longer the maintained runtime — the CLI,
1243
- > local MCP server, and deploy tooling have all moved to TypeScript (`@cerefox/memory`). The
1244
- > Python client below still works from a repo clone but is unmaintained. Prefer Path A (MCP),
1245
- > Path B (Edge Functions HTTP), or Path C (the `cerefox` CLI) for new integrations.
1246
-
1247
- Use the Cerefox Python client directly for scripted or embedded agents (from a repo clone):
1248
-
1249
- ```python
1250
- from cerefox.config import Settings
1251
- from cerefox.db.client import CerefoxClient
1252
- from cerefox.embeddings.cloud import CloudEmbedder
1253
- from cerefox.retrieval.search import SearchClient
1254
-
1255
- settings = Settings() # reads from .env
1256
- client = CerefoxClient(settings)
1257
- embedder = CloudEmbedder(
1258
- api_key=settings.get_embedder_api_key(),
1259
- base_url=settings.get_embedder_base_url(),
1260
- model=settings.get_embedder_model(),
1261
- dimensions=settings.get_embedder_dimensions(),
1262
- )
1263
- sc = SearchClient(client, embedder, settings)
1264
-
1265
- resp = sc.search_docs("what did I write about Rust?", match_count=5)
1266
- for hit in resp.results:
1267
- print(f"[{hit.best_score:.2f}] {hit.doc_title}")
1268
- print(hit.full_content[:400])
1269
- ```
1270
-
1271
- ---
1272
-
1273
1275
  ## Keeping both paths in sync
1274
1276
 
1275
1277
  Both paths use the same Postgres RPCs and the same stored embeddings, but embed queries
@@ -1539,7 +1541,3 @@ claude mcp add --scope user cerefox \
1539
1541
  If `.env` isn't resolvable from your shell's CWD, edit the resulting JSON config to add an
1540
1542
  `env` block (see the Claude Desktop block above). The **Code** tab inside Claude Desktop uses
1541
1543
  this same config — run the `claude mcp add` above and it picks it up automatically.
1542
-
1543
- > The legacy Python MCP server can be wired manually as a standalone fallback by pointing the
1544
- > `command` at `uv` with args `["--directory", "/path/to/cerefox", "run", "cerefox", "mcp"]`.
1545
- > It is frozen and unmaintained — prefer the npm entry above.
@@ -0,0 +1,55 @@
1
+ # Document content format (chunk reconstruction)
2
+
3
+ Cerefox stores each document as a set of **chunks** (for search and small-to-big
4
+ retrieval) and reassembles the full text from those chunks whenever you read a
5
+ document. The **content format** records *how* a document's chunks reassemble.
6
+ You normally never need to think about it: it exists so that a fix shipped in a
7
+ later version can't change how your older documents read back.
8
+
9
+ There are two formats:
10
+
11
+ | Format | Meaning | When |
12
+ |---|---|---|
13
+ | **1 — legacy** | Chunk contents were trimmed sections; the full document is rebuilt by joining them with a blank line (`\n\n`) on read. | Every document written before the fix (Cerefox ≤ 0.11.x). |
14
+ | **2 — blind-stitch** | Chunk contents are an exact, gapless slice-by-slice partition of the document; the full document is rebuilt by plain concatenation (nothing is inserted between chunks). | Documents written by the newer exact-partition chunker. |
15
+
16
+ ## Why it exists
17
+
18
+ The older reconstruction always inserted a `\n\n` between chunks on read — a
19
+ separator that was **never actually stored**. That is only correct if a chunk
20
+ boundary falls exactly on a paragraph break. When a boundary fell elsewhere (for
21
+ example, inside a large markdown table that has no blank lines), reconstruction
22
+ injected a spurious blank line mid-content — splitting a table row, or turning
23
+ `Source` into `Sour` + blank line + `ce`. A handful of large documents were
24
+ corrupted this way.
25
+
26
+ Format 2 fixes this at the root: because the stored chunks are an exact partition
27
+ of the document, reconstruction just concatenates them and gets the original back
28
+ byte-for-byte, so a chunk boundary can fall anywhere with zero corruption.
29
+
30
+ The format is stored **per chunk** (on `cerefox_chunks`), not per document, so an
31
+ archived version of a document always reconstructs with the format it was written
32
+ in — even after the current version has moved to format 2.
33
+
34
+ ## Do I need to do anything?
35
+
36
+ **No.** The migration is lazy and safe:
37
+
38
+ - **Existing documents keep format 1** and reconstruct exactly as they always did.
39
+ Nothing is re-processed, nothing re-embeds.
40
+ - A document **moves to format 2 automatically the next time it is edited/saved**
41
+ (it gets re-chunked by the new chunker).
42
+ - If you want to convert everything now rather than on next edit, run
43
+ `cerefox server reindex` (re-chunks + re-embeds the whole knowledge base).
44
+
45
+ `cerefox doctor` reports how many documents still use the legacy format — purely
46
+ informational, never a failure. A fresh install shows zero.
47
+
48
+ ## For contributors
49
+
50
+ - Column: `cerefox_chunks.content_format SMALLINT NOT NULL DEFAULT 1`.
51
+ - The ingest RPC (`cerefox_ingest_document`) takes `p_content_format` and stamps it
52
+ on every chunk it writes; the exact-partition chunker path passes `2`.
53
+ - The reconstruction RPCs branch on `MAX(content_format) >= 2` per aggregated group
54
+ (blind concat vs the legacy `\n\n`-join).
55
+ - Design: `docs/specs/chunk-reconstruction-design.md`.
@@ -0,0 +1,96 @@
1
+ # Migrating to Cerefox 1.0.0
2
+
3
+ Cerefox 1.0.0 is the first stable release. Two changes need attention when you
4
+ upgrade an existing Supabase deployment; a third is automatic. **Nothing here
5
+ affects the local/self-hosted (World B) backend.**
6
+
7
+ Pre-releases are published on the `1.0.0-beta.N` line (breaking changes may still
8
+ occur between betas) leading up to `1.0.0`.
9
+
10
+ ## 1. Edge Function auth: anon key → Cerefox access token (action required)
11
+
12
+ The legacy Supabase **anon JWT** is no longer accepted for calling Cerefox Edge
13
+ Functions. Every function now validates a rotatable, Cerefox-managed **access token**
14
+ (`cfx_pat_…`) in-function. This is a **breaking change** for two client classes:
15
+
16
+ - **ChatGPT Custom GPTs** using GPT Actions.
17
+ - **Remote HTTP MCP** clients (any client pointed at `cerefox-mcp` with a static Bearer).
18
+
19
+ **Local agents and cloud Claude are unaffected** (local MCP uses the Data API; cloud
20
+ Claude uses OAuth).
21
+
22
+ ### `.env` / secrets changes
23
+
24
+ | Variable | Change |
25
+ |---|---|
26
+ | `CEREFOX_ACCESS_TOKENS` | **New** — Supabase **Function secret**, the server-side accepted set (comma-separated; enables rotation). Set by `cerefox token generate`. |
27
+ | `CEREFOX_ACCESS_TOKEN` | **New** — in your local `.env`, the token this machine presents (used by `cerefox doctor`, the live tests, and the optional remote-MCP client). Written by `cerefox token generate`. |
28
+ | `CEREFOX_MCP_STATIC_BEARER` | **Remove** — the legacy anon static-Bearer for `cerefox-mcp` is gone. |
29
+ | `CEREFOX_SUPABASE_ANON_KEY` | **No longer used** for Edge Function auth. Safe to delete from `.env`. |
30
+ | `CEREFOX_SUPABASE_URL`, `CEREFOX_SUPABASE_KEY` | **Unchanged** — the Data API / local MCP path is not affected. |
31
+
32
+ ### Steps (in order — there is a brief cutover window)
33
+
34
+ 1. **Generate the token:** `cerefox token generate` — mints it, sets the
35
+ `CEREFOX_ACCESS_TOKENS` Function secret on Supabase, writes `CEREFOX_ACCESS_TOKEN`
36
+ into your `.env`, and prints it once.
37
+ 2. **Deploy the token-gated functions** (this also applies change #2 below):
38
+ `cerefox server deploy`. (If `CEREFOX_ACCESS_TOKENS` is unset, the functions reject
39
+ every caller, so keep step 1 first.)
40
+ 3. **Update your clients** (promptly after step 2):
41
+ - **Custom GPT (GPT Actions):** re-paste the OpenAPI schema (its `info.version` bumped,
42
+ so ChatGPT clears the stored key), then set **Authentication → API Key** to your
43
+ Cerefox access token (Bearer).
44
+ - **Remote HTTP MCP:** replace the anon key in the `Authorization: Bearer` header.
45
+ 4. **Verify:** `cerefox doctor` (the edge-functions check should pass).
46
+ 5. **Revoke the legacy anon key** in the Supabase dashboard (Project Settings → API Keys →
47
+ Legacy). Nothing at runtime uses it anymore.
48
+ 6. **Remove the retired consent Edge Function** (if you deployed it):
49
+ `npx supabase functions delete cerefox-oauth-consent`. The OAuth consent page is now
50
+ served solely by the Cloudflare Worker (`cloudflare/cerefox-consent/`).
51
+
52
+ Rotating the token later: `cerefox token rotate` (accepts new + old for zero-downtime),
53
+ then `cerefox token rotate --finalize` once every client is on the new token.
54
+
55
+ ## 2. New schema (0.7.0 → 0.8.0): document reconstruction fix (redeploy required, no data action)
56
+
57
+ 1.0.0 fixes a document-reconstruction bug that could corrupt documents containing large
58
+ tables or blank-line-free paragraphs. It adds a `content_format` column on
59
+ `cerefox_chunks`. **You just redeploy** — `cerefox server deploy` (step 2 above) applies
60
+ the schema change.
61
+
62
+ The migration is **lazy and safe**:
63
+
64
+ - Existing documents keep the legacy format and reconstruct **exactly as before** — nothing
65
+ re-processes, nothing re-embeds.
66
+ - A document moves to the new format automatically the next time it is edited/saved. To
67
+ convert everything now instead, run `cerefox server reindex`.
68
+ - `cerefox doctor` shows how many documents still use the legacy format.
69
+
70
+ Details: `cerefox guides show content-format` (or [`content-format.md`](content-format.md)).
71
+
72
+ ## 3. Python is fully removed at 1.0.0 (breaking)
73
+
74
+ **The Python implementation is deleted in 1.0.0** — including the frozen MCP-server fallback
75
+ (`uv run cerefox mcp`), the husked Python CLI / web / ingestion packages, and
76
+ `pyproject.toml`. There is no Python code left to run.
77
+
78
+ If you still invoke `uv run cerefox mcp`, switch to the maintained local server:
79
+
80
+ ```bash
81
+ npx --package=@cerefox/memory cerefox mcp
82
+ ```
83
+
84
+ (or stay on 0.11.x until you have migrated). The TypeScript CLI, local MCP, remote MCP, and
85
+ web app are the only maintained paths. The SQL schema assets under `src/cerefox/db/` are
86
+ unaffected — they are not Python and remain the source of truth for the schema + RPCs.
87
+
88
+ ## Upgrade order summary
89
+
90
+ ```bash
91
+ cerefox self-update # or: npm install -g @cerefox/memory@latest
92
+ cerefox token generate # change #1: mint + set the access token
93
+ cerefox server deploy # changes #1 + #2: token-gated EFs + schema 0.8.0
94
+ cerefox doctor # verify (edge-functions green; content-format ℹ)
95
+ # then: update GPT Actions / remote MCP clients to the token; revoke the anon key
96
+ ```
@@ -30,10 +30,8 @@ The canonical scripts are **TypeScript**, run with [Bun](https://bun.sh) (instal
30
30
  | `backup_restore.ts` | `bun scripts/backup_restore.ts` |
31
31
  | `reindex_all.ts` | `bun scripts/reindex_all.ts` |
32
32
 
33
- The `.py` equivalents are **legacy** — they still exist as a migration aid but are no longer
34
- maintained; use the `.ts` scripts. The legacy `db_status.py` and `sync_docs.py` are
35
- deprecation shims that exit non-zero with a pointer to the TS replacement, so update any cron
36
- jobs / CI / make targets that invoke them.
33
+ Use the `.ts` scripts. The former `.py` equivalents were **removed at v1.0.0** — update any
34
+ cron jobs / CI / make targets that still invoke a `python scripts/*.py` path.
37
35
 
38
36
  ### TS scripts and `.env` resolution
39
37
 
@@ -75,7 +73,7 @@ CEREFOX_DATABASE_URL=postgresql://cerefox:cerefox@localhost:5432/cerefox \
75
73
 
76
74
  ## db_status.ts — Schema verification
77
75
 
78
- **TypeScript (v0.3.0+).** Checks that the schema is correctly deployed and reports table statistics. Replaces the legacy `db_status.py`, which now prints a deprecation notice and exits non-zero.
76
+ **TypeScript (v0.3.0+).** Checks that the schema is correctly deployed and reports table statistics. Replaces the former `db_status.py` (removed at v1.0.0).
79
77
 
80
78
  ```bash
81
79
  bun scripts/db_status.ts # human-readable report
@@ -223,7 +221,7 @@ The backup directory (`./backup-data/` by default) is gitignored. Back up the ba
223
221
 
224
222
  Ingests `README.md`, `AGENT_GUIDE.md`, `AGENT_QUICK_REFERENCE.md`, and every Markdown file under `docs/` into your Cerefox knowledge base, updating existing documents in-place. Run this any time after editing documentation so AI agents always have access to the current state of the project.
225
223
 
226
- Replaces the legacy `sync_docs.py`, which now prints a deprecation notice and exits non-zero.
224
+ Replaces the former `sync_docs.py` (removed at v1.0.0).
227
225
 
228
226
  ```bash
229
227
  bun scripts/sync_docs.ts [OPTIONS]
@@ -234,7 +232,7 @@ bun scripts/sync_docs.ts [OPTIONS]
234
232
  | `--project NAME`, `-p NAME` | Project to assign documents to (default: `cerefox`) |
235
233
  | `--dry-run`, `-n` | List files that would be synced without ingesting anything |
236
234
 
237
- **Requires**: `CEREFOX_SUPABASE_URL` and `CEREFOX_SUPABASE_ANON_KEY` (the legacy anon JWT — `eyJ…` — used to invoke Edge Functions). Embedding happens server-side inside the `cerefox-ingest` Edge Function, so you don't need an OpenAI / Fireworks key in your local env for the TS script.
235
+ **Requires**: `CEREFOX_SUPABASE_URL` and `CEREFOX_ACCESS_TOKEN` (the Cerefox access token — `cfx_pat_…`, from `cerefox token generate` — the Bearer used to invoke Edge Functions). Embedding happens server-side inside the `cerefox-ingest` Edge Function, so you don't need an OpenAI / Fireworks key in your local env for the TS script.
238
236
 
239
237
  The target project must already exist (create it with `cerefox project create cerefox` if needed).
240
238
 
@@ -80,6 +80,26 @@ On an existing database this applies pending migrations and re-applies RPCs
80
80
  in place, so re-run it after upgrading. Detailed walkthrough:
81
81
  [`setup-supabase.md`](setup-supabase.md).
82
82
 
83
+ ## 3b. Generate the Edge Function access token
84
+
85
+ The Edge Functions authenticate callers with a **Cerefox access token**. You need it
86
+ if you'll connect a **Custom GPT** (GPT Actions) or a **remote HTTP MCP** client, and
87
+ for a fully-green `cerefox doctor`. The **local MCP** (local agents), **cloud Claude**
88
+ (OAuth), and the CLI/web reach Supabase over the Data API and don't use it — so this
89
+ step is optional if that's your whole setup, but it's quick and harmless to run:
90
+
91
+ ```bash
92
+ cerefox token generate
93
+ ```
94
+
95
+ This creates a `cfx_pat_…` token, sets it as the `CEREFOX_ACCESS_TOKENS` secret on
96
+ Supabase, and writes `CEREFOX_ACCESS_TOKEN` into your `.env` (so `cerefox doctor`,
97
+ your tools, and any remote client can use it). It prints the token **once** — store
98
+ it. You only need to paste it somewhere by hand if you connect a **Custom GPT**
99
+ (into the Action's Authentication → API Key) or a **remote HTTP MCP** client; the
100
+ local MCP and cloud Claude (OAuth) don't use it. Lose it → `cerefox token rotate`.
101
+ Run `cerefox doctor` again — the "edge functions" check should now be green.
102
+
83
103
  ## 4. Wire up an AI agent
84
104
 
85
105
  ```bash
@@ -113,8 +133,8 @@ You should see results from the bundled self-docs.
113
133
 
114
134
  The path above is for **end users** (no clone). If you want to hack on Cerefox,
115
135
  clone the repo, run `bun install`, and use the contributor scripts
116
- (`bun scripts/db_deploy.ts`, `bun scripts/db_migrate.ts`). `uv` is only needed
117
- for the legacy Python MCP fallback. See [`CONTRIBUTING.md`](../../CONTRIBUTING.md).
136
+ (`bun scripts/db_deploy.ts`, `bun scripts/db_migrate.ts`). See
137
+ [`CONTRIBUTING.md`](../../CONTRIBUTING.md).
118
138
  (Want a no-cloud install instead? That's the self-hosted Docker backend —
119
139
  [`setup-local.md`](setup-local.md).)
120
140