@cerefox/memory 0.11.1 → 1.0.0-beta.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/dist/bin/cerefox.js +1077 -834
  2. package/dist/frontend/assets/index-CCkg5PXt.js +125 -0
  3. package/dist/frontend/assets/index-CCkg5PXt.js.map +1 -0
  4. package/dist/frontend/index.html +1 -1
  5. package/dist/server-assets/_shared/ef-auth/index.ts +134 -0
  6. package/dist/server-assets/_shared/ef-meta/index.ts +1 -1
  7. package/dist/server-assets/_shared/embeddings/index.ts +42 -2
  8. package/dist/server-assets/_shared/ingest/chunker.ts +210 -0
  9. package/dist/server-assets/_shared/ingest/index.ts +32 -0
  10. package/dist/server-assets/_shared/ingest/pipeline-helpers.ts +135 -0
  11. package/dist/server-assets/_shared/mcp-auth/index.ts +352 -0
  12. package/dist/server-assets/_shared/mcp-tools/_chunker.ts +16 -170
  13. package/dist/server-assets/_shared/mcp-tools/ingest.ts +13 -4
  14. package/dist/server-assets/db/migrations/0012_content_format.sql +20 -0
  15. package/dist/server-assets/db/rpcs.sql +76 -8
  16. package/dist/server-assets/db/schema.sql +7 -1
  17. package/dist/server-assets/supabase/functions/cerefox-get-audit-log/index.ts +8 -0
  18. package/dist/server-assets/supabase/functions/cerefox-get-document/index.ts +8 -0
  19. package/dist/server-assets/supabase/functions/cerefox-ingest/index.ts +22 -171
  20. package/dist/server-assets/supabase/functions/cerefox-list-projects/index.ts +8 -0
  21. package/dist/server-assets/supabase/functions/cerefox-list-versions/index.ts +8 -0
  22. package/dist/server-assets/supabase/functions/cerefox-mcp/index.ts +54 -0
  23. package/dist/server-assets/supabase/functions/cerefox-mcp/oauth.ts +121 -0
  24. package/dist/server-assets/supabase/functions/cerefox-metadata/index.ts +8 -0
  25. package/dist/server-assets/supabase/functions/cerefox-metadata-search/index.ts +8 -0
  26. package/dist/server-assets/supabase/functions/cerefox-search/index.ts +11 -1
  27. package/docs/guides/access-paths.md +81 -30
  28. package/docs/guides/cli.md +29 -0
  29. package/docs/guides/configuration.md +4 -1
  30. package/docs/guides/connect-agents.md +98 -54
  31. package/docs/guides/content-format.md +55 -0
  32. package/docs/guides/migration-1.0.md +87 -0
  33. package/docs/guides/ops-scripts.md +1 -1
  34. package/docs/guides/quickstart.md +20 -0
  35. package/docs/guides/setup-supabase.md +154 -13
  36. package/docs/guides/upgrading.md +4 -3
  37. package/package.json +1 -1
  38. package/dist/frontend/assets/index-ojNhWSxm.js +0 -125
  39. package/dist/frontend/assets/index-ojNhWSxm.js.map +0 -1
@@ -30,15 +30,15 @@ Three top-level paths plus a few special cases:
30
30
  | Client | Path | Search | Requirements / caveats |
31
31
  |--------|------|--------|-----------------------|
32
32
  | Claude Desktop (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | Node.js for `npx supergateway` or `npx mcp-remote`; no Python needed |
33
- | Claude Code (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
34
- | Cursor (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
35
- | OpenAI Codex CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key env var; TOML config |
33
+ | Claude Code (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
34
+ | Cursor (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
35
+ | OpenAI Codex CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token env var; TOML config. Advanced/fallback — prefer Path A-Local |
36
36
  | ChatGPT (chatgpt.com or desktop) | Path B — Custom GPT → Edge Functions | Hybrid | ChatGPT Plus required |
37
37
  | Claude Desktop (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; Node.js; zero Edge Function invocations |
38
38
  | Claude Code (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; zero Edge Function invocations |
39
39
  | Cursor (local) | Path A-Local — `@cerefox/memory` via `npx` | Hybrid | Local alternative; zero Edge Function invocations |
40
- | Cloud Claude (claude.ai web) | Remote Supabase MCP | FTS only | No install; search quality limited |
41
- | Gemini CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + anon key only; no local install |
40
+ | Cloud Claude (claude.ai web + mobile) | Path A-Remote — `cerefox-mcp` over **OAuth** | Hybrid | No install; **optional** one-time OAuth setup + a free Cloudflare Worker consent page ([setup-supabase Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)) |
41
+ | Gemini CLI (remote) | Path A-Remote — `cerefox-mcp` Edge Function | Hybrid | URL + Cerefox token only; no local install. Advanced/fallback — prefer Path A-Local |
42
42
  | Local coding agents (Claude Code, Codex CLI, opencode, OpenClaw, Hermes, …) | Path C — Shell CLI (Bash tool) | Hybrid | `npm install -g @cerefox/memory`; agent runs `cerefox …` as a shell command. Useful when MCP setup is friction. |
43
43
  | curl / scripts | Path B — Edge Functions directly | Hybrid | Direct HTTP; no client needed |
44
44
  | Custom Python agents | Python SDK directly (legacy) | Hybrid | Local Python + repo clone; the Python path is legacy/frozen |
@@ -46,8 +46,12 @@ Three top-level paths plus a few special cases:
46
46
  > **"Hybrid"** = FTS + semantic, document-level (complete reconstructed notes, not isolated chunks).
47
47
  > **"FTS only"** = keyword search only; no semantic/vector search.
48
48
 
49
- > **Cloud hybrid for all clients (future)**: deploying the MCP server to Cloud Run would give
50
- > cloud clients (claude.ai, chatgpt.com) full hybrid search. Tracked in `docs/TODO.md`.
49
+ > **Cloud hybrid for claude.ai / mobile (iter-28A)**: `cerefox-mcp` is now an OAuth 2.1
50
+ > protected resource, so claude.ai web and the Claude mobile app get **full hybrid search**
51
+ > over the standard tool surface — no Cloud Run needed. Setup:
52
+ > [setup-supabase Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
53
+ > + [Cloud Claude](#cloud-claude-claudeai-web--mobile-oauth) below. (An OAuth connector for
54
+ > ChatGPT becomes possible on the same server but is not yet documented.)
51
55
 
52
56
  > **Perplexity** supports stdio-only MCP on macOS Desktop (via Helper App). Remote MCP is
53
57
  > "coming soon." Perplexity's CTO has signalled a strategic shift away from MCP (March 2026),
@@ -91,17 +95,23 @@ in the container.
91
95
  - A frozen Python MCP server still exists as a standalone fallback (`uv run cerefox mcp` from a
92
96
  repo clone), but the npm package is the maintained path.
93
97
 
94
- > **Important — which anon key to use (2026):** Path A-Remote and Path B both require an
95
- > "anon key" as a Bearer token. As of 2026, you **must** use the **legacy anon JWT**
96
- > (`eyJ…`) — the new `sb_publishable_…` key is rejected by the Supabase Edge Function
97
- > gateway with `UNAUTHORIZED_INVALID_JWT_FORMAT`. Find the legacy key in **Project
98
- > Settings → API Keys → Legacy → anon**. This is a Supabase platform constraint;
99
- > see [`setup-supabase.md` → Supabase API keys (2026)](setup-supabase.md#supabase-api-keys-2026)
100
- > for the full story.
101
-
102
- **For Path A-Remote (remote MCP Edge Function) — recommended:**
98
+ > **Important — the Cerefox access token (iter-28E):** Path A-Remote and Path B both require a
99
+ > **Bearer token** on every request. That credential is now the **Cerefox access token**
100
+ > (`cfx_pat_…`), a random, Cerefox-managed secret validated in-function — **not** the legacy
101
+ > Supabase anon JWT (which is retired for all Edge Function paths). Generate it with
102
+ > `cerefox token generate`: it sets the accepted token set on Supabase (the
103
+ > `CEREFOX_ACCESS_TOKENS` Function secret) and writes the value to your local `.env` as
104
+ > `CEREFOX_ACCESS_TOKEN`, printing it once so you can paste it into client configs. Lose it →
105
+ > `cerefox token rotate`. See [`setup-supabase.md` → Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
106
+ > for the server-side setup.
107
+
108
+ **For Path A-Remote (remote MCP Edge Function) — advanced / fallback:**
109
+ > For **local agents** (Claude Code, Cursor, Codex, Gemini, Claude Desktop) the **local MCP**
110
+ > (Path A-Local, via `cerefox configure-agent`) is the preferred path — zero Edge Function
111
+ > cost and no token to distribute. Use Path A-Remote when you specifically want a hosted URL
112
+ > (multiple machines, cloud dev environments) or as a fallback.
103
113
  - `cerefox-mcp` Edge Function deployed (`npx supabase functions deploy cerefox-mcp`)
104
- - Your **legacy anon JWT** (see callout above): Supabase Dashboard → Project Settings → API Keys → Legacy → anon
114
+ - Your **Cerefox access token** (see callout above): run `cerefox token generate`
105
115
  - For Claude Desktop: [Node.js](https://nodejs.org) installed (for `npx supergateway` or `npx mcp-remote`)
106
116
  - For Claude Code: [Node.js](https://nodejs.org) for `npx mcp-remote` (recommended), or no extra deps for native HTTP
107
117
 
@@ -111,7 +121,7 @@ in the container.
111
121
  `cerefox-get-audit-log`, `cerefox-metadata-search`, `cerefox-list-projects`, `cerefox-mcp`.
112
122
  End-user path: `cerefox server deploy`. Contributor/manual path: `npx supabase functions
113
123
  deploy` (see `setup-supabase.md`).
114
- - Your **legacy anon JWT** (see callout above): Supabase Dashboard → Project Settings → API Keys → Legacy → anon
124
+ - Your **Cerefox access token** (see callout above): run `cerefox token generate`
115
125
  - Your **project ref**: visible in the Supabase Dashboard URL
116
126
  (`app.supabase.com/project/<project-ref>`)
117
127
 
@@ -317,8 +327,8 @@ https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp
317
327
 
318
328
  | Scenario | Prefer |
319
329
  |----------|--------|
320
- | Default / new setup | Path A-Remote -- no Python, no local clone, one URL works everywhere |
321
- | Multiple machines / cloud dev environments | Path A-Remote |
330
+ | Default local agent (Claude Code, Cursor, Codex, Gemini, Claude Desktop) | Path A-Local -- preferred; `cerefox configure-agent`, zero Edge Function cost, no token to distribute |
331
+ | Multiple machines / cloud dev environments | Path A-Remote -- one hosted URL works everywhere (needs a Cerefox token) |
322
332
  | Minimise Supabase Edge Function usage (free tier limits) | Path A-Local -- zero Edge Function invocations |
323
333
  | Offline use or development on the cerefox codebase | Path A-Local -- no network dependency |
324
334
  | Lowest latency (same machine, no HTTPS round-trip) | Path A-Local -- slightly faster |
@@ -355,7 +365,7 @@ Add to your project's `.mcp.json` (or copy
355
365
  "mcp-remote",
356
366
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
357
367
  "--header",
358
- "Authorization: Bearer <your-anon-key>"
368
+ "Authorization: Bearer <your-cerefox-token>"
359
369
  ]
360
370
  }
361
371
  }
@@ -370,7 +380,7 @@ overhead (fixed in v0.1.12). However, `mcp-remote` is still preferred for the OA
370
380
  ```bash
371
381
  claude mcp add --transport http cerefox \
372
382
  https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp \
373
- --header "Authorization: Bearer <your-anon-key>"
383
+ --header "Authorization: Bearer <your-cerefox-token>"
374
384
  ```
375
385
 
376
386
  Verify:
@@ -382,7 +392,7 @@ For a user-scoped server (available in all projects), add `--scope user`:
382
392
  ```bash
383
393
  claude mcp add --transport http --scope user cerefox \
384
394
  https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp \
385
- --header "Authorization: Bearer <your-anon-key>"
395
+ --header "Authorization: Bearer <your-cerefox-token>"
386
396
  ```
387
397
 
388
398
  ---
@@ -400,7 +410,7 @@ Cursor supports remote MCP servers natively via `url` + `headers` in `mcp.json`.
400
410
  "cerefox": {
401
411
  "url": "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
402
412
  "headers": {
403
- "Authorization": "Bearer <your-anon-key>"
413
+ "Authorization": "Bearer <your-cerefox-token>"
404
414
  }
405
415
  }
406
416
  }
@@ -441,19 +451,19 @@ Add (or merge into) the file:
441
451
  "args": [
442
452
  "-y", "supergateway",
443
453
  "--streamableHttp", "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
444
- "--oauth2Bearer", "<your-anon-key>"
454
+ "--oauth2Bearer", "<your-cerefox-token>"
445
455
  ]
446
456
  }
447
457
  }
448
458
  }
449
459
  ```
450
460
 
451
- Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
461
+ Replace `<your-project-ref>` and `<your-cerefox-token>` with your actual values.
452
462
 
453
463
  **Important:**
454
464
  - Restart Claude Desktop fully (Cmd+Q on macOS) after saving the config.
455
465
  - `-y` tells npx to auto-install `supergateway` without prompting.
456
- - No Python, no local repo clone, no `.env` file needed — just the URL and anon key.
466
+ - No Python, no local repo clone, no `.env` file needed — just the URL and Cerefox token.
457
467
 
458
468
  ---
459
469
 
@@ -462,13 +472,13 @@ Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
462
472
  [Codex](https://github.com/openai/codex) supports remote MCP servers natively via Streamable
463
473
  HTTP. Configuration uses TOML (not JSON like most other MCP clients).
464
474
 
465
- **Step 1 — Set the anon key as an environment variable:**
475
+ **Step 1 — Set the Cerefox token as an environment variable:**
466
476
 
467
477
  Codex references Bearer tokens by environment variable name, not by value. Add to your
468
478
  `~/.zshrc` (or `~/.bashrc`):
469
479
 
470
480
  ```bash
471
- export CEREFOX_ANON_KEY="<your-anon-key>"
481
+ export CEREFOX_ACCESS_TOKEN="<your-cerefox-token>"
472
482
  ```
473
483
 
474
484
  Then reload: `source ~/.zshrc`
@@ -478,7 +488,7 @@ Then reload: `source ~/.zshrc`
478
488
  ```toml
479
489
  [mcp_servers.cerefox]
480
490
  url = "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp"
481
- bearer_token_env_var = "CEREFOX_ANON_KEY"
491
+ bearer_token_env_var = "CEREFOX_ACCESS_TOKEN"
482
492
  ```
483
493
 
484
494
  Replace `<your-project-ref>` with your Supabase project ref.
@@ -489,9 +499,9 @@ Launch Codex and use the `/mcp` slash command to confirm the `cerefox` server is
489
499
  and all 10 tools are listed.
490
500
 
491
501
  **Notes:**
492
- - `bearer_token_env_var` is the **name** of the env var (e.g. `"CEREFOX_ANON_KEY"`), not the
502
+ - `bearer_token_env_var` is the **name** of the env var (e.g. `"CEREFOX_ACCESS_TOKEN"`), not the
493
503
  token itself. Codex reads the value at runtime.
494
- - No Python, no local repo clone needed — just the URL and anon key.
504
+ - No Python, no local repo clone needed — just the URL and Cerefox token.
495
505
  - No idle SSE polling cost — the 405 GET fix in `cerefox-mcp` prevents it.
496
506
 
497
507
  ---
@@ -514,14 +524,14 @@ Add (or merge into) the file:
514
524
  "cerefox": {
515
525
  "httpUrl": "https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp",
516
526
  "headers": {
517
- "Authorization": "Bearer <your-anon-key>"
527
+ "Authorization": "Bearer <your-cerefox-token>"
518
528
  }
519
529
  }
520
530
  }
521
531
  }
522
532
  ```
523
533
 
524
- Replace `<your-project-ref>` and `<your-anon-key>` with your actual values.
534
+ Replace `<your-project-ref>` and `<your-cerefox-token>` with your actual values.
525
535
 
526
536
  **Verify:**
527
537
 
@@ -554,11 +564,11 @@ stored in Supabase.
554
564
  All Edge Function calls require:
555
565
 
556
566
  ```
557
- Authorization: Bearer <your-anon-key>
567
+ Authorization: Bearer <your-cerefox-token>
558
568
  Content-Type: application/json
559
569
  ```
560
570
 
561
- Find your anon key: **Supabase Dashboard → Project Settings → API Keys → Legacy → anon** (use the legacy JWT, not the new `sb_publishable_…` — see the API keys callout in the Prerequisites section).
571
+ Generate your token: run `cerefox token generate` (it prints the `cfx_pat_…` Cerefox access token once and sets it on Supabase). The legacy Supabase anon JWT is no longer accepted for Edge Function calls — see the token callout in the Prerequisites section.
562
572
 
563
573
  ### Path B system prompt
564
574
 
@@ -580,7 +590,7 @@ overwrite blindly.
580
590
  ```bash
581
591
  curl -s -X POST \
582
592
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-search" \
583
- -H "Authorization: Bearer <your-anon-key>" \
593
+ -H "Authorization: Bearer <your-cerefox-token>" \
584
594
  -H "Content-Type: application/json" \
585
595
  -d '{"query": "second brain", "match_count": 3}'
586
596
  ```
@@ -609,7 +619,7 @@ In the action editor, paste this schema (replace `<your-project-ref>`):
609
619
  openapi: 3.1.0
610
620
  info:
611
621
  title: Cerefox Knowledge Base
612
- version: 2.1.0
622
+ version: 3.0.0
613
623
  servers:
614
624
  - url: https://<your-project-ref>.supabase.co/functions/v1
615
625
  paths:
@@ -1001,11 +1011,15 @@ paths:
1001
1011
  In the action's **Authentication** settings:
1002
1012
  - Type: **API Key**
1003
1013
  - Auth type: **Bearer**
1004
- - API key: your Supabase **anon key**
1014
+ - API key: your **Cerefox access token** (`cfx_pat_…`). Generate it with
1015
+ `cerefox token generate` (it prints the token once and sets it on Supabase).
1016
+ The legacy Supabase anon key is no longer accepted (iter-28E).
1005
1017
 
1006
- > **Important:** ChatGPT may reset the API key when you update the action schema.
1007
- > If you get a 403 error after changing the schema, re-enter the anon key in the
1008
- > authentication settings — the functions themselves are fine.
1018
+ > **Important:** ChatGPT resets the stored API key when you update the action
1019
+ > schema. This release bumps the schema `info.version`, so on your next schema
1020
+ > paste ChatGPT will clear the key — re-enter your **Cerefox access token** in the
1021
+ > authentication settings. (That reset is expected here: it's what swaps you off
1022
+ > the old anon key.)
1009
1023
 
1010
1024
  **Step 4 — Save and test**
1011
1025
 
@@ -1025,7 +1039,7 @@ Direct HTTP access — useful for shell scripts, CI pipelines, or one-off querie
1025
1039
  ```bash
1026
1040
  curl -s -X POST \
1027
1041
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-search" \
1028
- -H "Authorization: Bearer <your-anon-key>" \
1042
+ -H "Authorization: Bearer <your-cerefox-token>" \
1029
1043
  -H "Content-Type: application/json" \
1030
1044
  -d '{"query": "knowledge management", "match_count": 5}'
1031
1045
  ```
@@ -1034,7 +1048,7 @@ curl -s -X POST \
1034
1048
  ```bash
1035
1049
  curl -s -X POST \
1036
1050
  "https://<your-project-ref>.supabase.co/functions/v1/cerefox-ingest" \
1037
- -H "Authorization: Bearer <your-anon-key>" \
1051
+ -H "Authorization: Bearer <your-cerefox-token>" \
1038
1052
  -H "Content-Type: application/json" \
1039
1053
  -d '{
1040
1054
  "title": "Meeting Notes 2026-03-11",
@@ -1101,20 +1115,50 @@ See `docs/guides/configuration.md` → "Response size limit" for full details.
1101
1115
 
1102
1116
  ---
1103
1117
 
1104
- ### Cloud Claude (claude.ai web)
1118
+ ### Cloud Claude (claude.ai web + mobile — OAuth) <a id="cloud-claude-claudeai-web--mobile-oauth"></a>
1119
+
1120
+ > **Optional feature.** This is the only path that needs the OAuth setup + a hosted
1121
+ > consent page. If you don't use claude.ai web / mobile, skip it — every other client
1122
+ > works without it. Design + rationale:
1123
+ > [`docs/specs/oauth-mcp-server-design.md`](../specs/oauth-mcp-server-design.md).
1105
1124
 
1106
- Claude.ai web can connect to the Supabase-hosted remote MCP (no local install):
1125
+ Claude.ai web **and the Claude mobile app** connect to your own `cerefox-mcp` Edge
1126
+ Function over **OAuth**, with the **full hybrid-search tool surface** — no local install.
1127
+ This replaces the old FTS-only `mcp.supabase.com` approach (raw keyword search over the
1128
+ tables, no semantic search, no Cerefox tool ergonomics).
1107
1129
 
1108
- 1. In Claude.ai: **Settings → Integrations → Add integration**
1109
- 2. Enter the MCP URL:
1130
+ **Prerequisite:** complete the one-time Supabase OAuth setup in
1131
+ [`setup-supabase.md` → Step 7](setup-supabase.md#step-7--oauth-for-cloud-agents-claudeai--mobile-optional)
1132
+ — enable the OAuth 2.1 Server, deploy the Cloudflare Worker consent page, create the owner
1133
+ user + `CEREFOX_OAUTH_OWNER_ID` pin, and register the Claude OAuth App
1134
+ (**`client_secret_post`**) whose Client ID/Secret you paste below.
1135
+
1136
+ 1. In Claude.ai (web): **Settings → Connectors → Add custom connector**.
1137
+ 2. **Name**: `CerefoxMCP` (distinct from the local `cerefox` server, so both can coexist).
1138
+ 3. **URL** (must be the `*.supabase.co` host — custom domains break OAuth discovery):
1110
1139
  ```
1111
- https://mcp.supabase.com/sse?project_ref=<your-project-ref>
1140
+ https://<your-project-ref>.supabase.co/functions/v1/cerefox-mcp
1112
1141
  ```
1113
- 3. Authenticate with your Personal Access Token when prompted.
1114
-
1115
- > **Limitation**: The cloud Supabase MCP only supports **FTS keyword search** — no hybrid or
1116
- > semantic search. For full hybrid search from the web, deploy the MCP server to Cloud Run
1117
- > (see `docs/TODO.md` → "Remote HTTP MCP server").
1142
+ 4. **Advanced settings → OAuth Client ID / OAuth Client Secret**: paste the Client ID and
1143
+ Client Secret from the pre-registered OAuth App (setup-supabase Step 7d).
1144
+ 5. Save. Claude runs the OAuth flow → redirects you to the **Cerefox consent page** (sign
1145
+ in with the owner email/password from Step 7c, then **Allow**) → returns to Claude. The
1146
+ connector shows as connected with **10 tools**. (If you've approved before, Supabase
1147
+ auto-consents and the page just flashes through — that's expected.)
1148
+ 6. **Mobile**: connectors are account-level, so `CerefoxMCP` appears in the Claude mobile
1149
+ app automatically — run one search from your phone to confirm.
1150
+
1151
+ **Verify**: in a new chat with the connector enabled, ask *"Using CerefoxMCP, search the
1152
+ Cerefox Decision Log and give the title of the latest part."* A correct answer proves
1153
+ search + document reconstruction over the OAuth path.
1154
+
1155
+ > **If the connect fails right after consent** with an `ofid_…` reference: the OAuth App's
1156
+ > token endpoint auth method is wrong. It must be **`request body` (`client_secret_post`)**,
1157
+ > not HTTP Basic — see setup-supabase Step 7d. This is the most common mistake.
1158
+
1159
+ > **Cost note**: each cloud tool call is ~1 Edge Function invocation. Fine for interactive
1160
+ > use; for heavy/automated work prefer the local stdio server (zero EF cost). Your local
1161
+ > and static-Bearer clients (Claude Code, Cursor, Codex, Gemini, Desktop) are unchanged.
1118
1162
 
1119
1163
  ---
1120
1164
 
@@ -0,0 +1,55 @@
1
+ # Document content format (chunk reconstruction)
2
+
3
+ Cerefox stores each document as a set of **chunks** (for search and small-to-big
4
+ retrieval) and reassembles the full text from those chunks whenever you read a
5
+ document. The **content format** records *how* a document's chunks reassemble.
6
+ You normally never need to think about it: it exists so that a fix shipped in a
7
+ later version can't change how your older documents read back.
8
+
9
+ There are two formats:
10
+
11
+ | Format | Meaning | When |
12
+ |---|---|---|
13
+ | **1 — legacy** | Chunk contents were trimmed sections; the full document is rebuilt by joining them with a blank line (`\n\n`) on read. | Every document written before the fix (Cerefox ≤ 0.11.x). |
14
+ | **2 — blind-stitch** | Chunk contents are an exact, gapless slice-by-slice partition of the document; the full document is rebuilt by plain concatenation (nothing is inserted between chunks). | Documents written by the newer exact-partition chunker. |
15
+
16
+ ## Why it exists
17
+
18
+ The older reconstruction always inserted a `\n\n` between chunks on read — a
19
+ separator that was **never actually stored**. That is only correct if a chunk
20
+ boundary falls exactly on a paragraph break. When a boundary fell elsewhere (for
21
+ example, inside a large markdown table that has no blank lines), reconstruction
22
+ injected a spurious blank line mid-content — splitting a table row, or turning
23
+ `Source` into `Sour` + blank line + `ce`. A handful of large documents were
24
+ corrupted this way.
25
+
26
+ Format 2 fixes this at the root: because the stored chunks are an exact partition
27
+ of the document, reconstruction just concatenates them and gets the original back
28
+ byte-for-byte, so a chunk boundary can fall anywhere with zero corruption.
29
+
30
+ The format is stored **per chunk** (on `cerefox_chunks`), not per document, so an
31
+ archived version of a document always reconstructs with the format it was written
32
+ in — even after the current version has moved to format 2.
33
+
34
+ ## Do I need to do anything?
35
+
36
+ **No.** The migration is lazy and safe:
37
+
38
+ - **Existing documents keep format 1** and reconstruct exactly as they always did.
39
+ Nothing is re-processed, nothing re-embeds.
40
+ - A document **moves to format 2 automatically the next time it is edited/saved**
41
+ (it gets re-chunked by the new chunker).
42
+ - If you want to convert everything now rather than on next edit, run
43
+ `cerefox server reindex` (re-chunks + re-embeds the whole knowledge base).
44
+
45
+ `cerefox doctor` reports how many documents still use the legacy format — purely
46
+ informational, never a failure. A fresh install shows zero.
47
+
48
+ ## For contributors
49
+
50
+ - Column: `cerefox_chunks.content_format SMALLINT NOT NULL DEFAULT 1`.
51
+ - The ingest RPC (`cerefox_ingest_document`) takes `p_content_format` and stamps it
52
+ on every chunk it writes; the exact-partition chunker path passes `2`.
53
+ - The reconstruction RPCs branch on `MAX(content_format) >= 2` per aggregated group
54
+ (blind concat vs the legacy `\n\n`-join).
55
+ - Design: `docs/specs/chunk-reconstruction-design.md`.
@@ -0,0 +1,87 @@
1
+ # Migrating to Cerefox 1.0.0
2
+
3
+ Cerefox 1.0.0 is the first stable release. Two changes need attention when you
4
+ upgrade an existing Supabase deployment; a third is automatic. **Nothing here
5
+ affects the local/self-hosted (World B) backend.**
6
+
7
+ Pre-releases are published on the `1.0.0-beta.N` line (breaking changes may still
8
+ occur between betas) leading up to `1.0.0`.
9
+
10
+ ## 1. Edge Function auth: anon key → Cerefox access token (action required)
11
+
12
+ The legacy Supabase **anon JWT** is no longer accepted for calling Cerefox Edge
13
+ Functions. Every function now validates a rotatable, Cerefox-managed **access token**
14
+ (`cfx_pat_…`) in-function. This is a **breaking change** for two client classes:
15
+
16
+ - **ChatGPT Custom GPTs** using GPT Actions.
17
+ - **Remote HTTP MCP** clients (any client pointed at `cerefox-mcp` with a static Bearer).
18
+
19
+ **Local agents and cloud Claude are unaffected** (local MCP uses the Data API; cloud
20
+ Claude uses OAuth).
21
+
22
+ ### `.env` / secrets changes
23
+
24
+ | Variable | Change |
25
+ |---|---|
26
+ | `CEREFOX_ACCESS_TOKENS` | **New** — Supabase **Function secret**, the server-side accepted set (comma-separated; enables rotation). Set by `cerefox token generate`. |
27
+ | `CEREFOX_ACCESS_TOKEN` | **New** — in your local `.env`, the token this machine presents (used by `cerefox doctor`, the live tests, and the optional remote-MCP client). Written by `cerefox token generate`. |
28
+ | `CEREFOX_MCP_STATIC_BEARER` | **Remove** — the legacy anon static-Bearer for `cerefox-mcp` is gone. |
29
+ | `CEREFOX_SUPABASE_ANON_KEY` | **No longer used** for Edge Function auth. Safe to delete from `.env`. |
30
+ | `CEREFOX_SUPABASE_URL`, `CEREFOX_SUPABASE_KEY` | **Unchanged** — the Data API / local MCP path is not affected. |
31
+
32
+ ### Steps (in order — there is a brief cutover window)
33
+
34
+ 1. **Generate the token:** `cerefox token generate` — mints it, sets the
35
+ `CEREFOX_ACCESS_TOKENS` Function secret on Supabase, writes `CEREFOX_ACCESS_TOKEN`
36
+ into your `.env`, and prints it once.
37
+ 2. **Deploy the token-gated functions** (this also applies change #2 below):
38
+ `cerefox server deploy`. (If `CEREFOX_ACCESS_TOKENS` is unset, the functions reject
39
+ every caller, so keep step 1 first.)
40
+ 3. **Update your clients** (promptly after step 2):
41
+ - **Custom GPT (GPT Actions):** re-paste the OpenAPI schema (its `info.version` bumped,
42
+ so ChatGPT clears the stored key), then set **Authentication → API Key** to your
43
+ Cerefox access token (Bearer).
44
+ - **Remote HTTP MCP:** replace the anon key in the `Authorization: Bearer` header.
45
+ 4. **Verify:** `cerefox doctor` (the edge-functions check should pass).
46
+ 5. **Revoke the legacy anon key** in the Supabase dashboard (Project Settings → API Keys →
47
+ Legacy). Nothing at runtime uses it anymore.
48
+ 6. **Remove the retired consent Edge Function** (if you deployed it):
49
+ `npx supabase functions delete cerefox-oauth-consent`. The OAuth consent page is now
50
+ served solely by the Cloudflare Worker (`cloudflare/cerefox-consent/`).
51
+
52
+ Rotating the token later: `cerefox token rotate` (accepts new + old for zero-downtime),
53
+ then `cerefox token rotate --finalize` once every client is on the new token.
54
+
55
+ ## 2. New schema (0.7.0 → 0.8.0): document reconstruction fix (redeploy required, no data action)
56
+
57
+ 1.0.0 fixes a document-reconstruction bug that could corrupt documents containing large
58
+ tables or blank-line-free paragraphs. It adds a `content_format` column on
59
+ `cerefox_chunks`. **You just redeploy** — `cerefox server deploy` (step 2 above) applies
60
+ the schema change.
61
+
62
+ The migration is **lazy and safe**:
63
+
64
+ - Existing documents keep the legacy format and reconstruct **exactly as before** — nothing
65
+ re-processes, nothing re-embeds.
66
+ - A document moves to the new format automatically the next time it is edited/saved. To
67
+ convert everything now instead, run `cerefox server reindex`.
68
+ - `cerefox doctor` shows how many documents still use the legacy format.
69
+
70
+ Details: `cerefox guides show content-format` (or [`content-format.md`](content-format.md)).
71
+
72
+ ## 3. Python is retired at 1.0.0
73
+
74
+ The frozen Python MCP fallback (`uv run cerefox mcp`) is retired in 1.0.0. If you still rely
75
+ on it, migrate to the `@cerefox/memory` npm package (`npx --package=@cerefox/memory cerefox
76
+ mcp`), or stay on 0.11.x until you have. The TypeScript CLI, local MCP, remote MCP, and web
77
+ app are the maintained paths.
78
+
79
+ ## Upgrade order summary
80
+
81
+ ```bash
82
+ cerefox self-update # or: npm install -g @cerefox/memory@latest
83
+ cerefox token generate # change #1: mint + set the access token
84
+ cerefox server deploy # changes #1 + #2: token-gated EFs + schema 0.8.0
85
+ cerefox doctor # verify (edge-functions green; content-format ℹ)
86
+ # then: update GPT Actions / remote MCP clients to the token; revoke the anon key
87
+ ```
@@ -234,7 +234,7 @@ bun scripts/sync_docs.ts [OPTIONS]
234
234
  | `--project NAME`, `-p NAME` | Project to assign documents to (default: `cerefox`) |
235
235
  | `--dry-run`, `-n` | List files that would be synced without ingesting anything |
236
236
 
237
- **Requires**: `CEREFOX_SUPABASE_URL` and `CEREFOX_SUPABASE_ANON_KEY` (the legacy anon JWT — `eyJ…` — used to invoke Edge Functions). Embedding happens server-side inside the `cerefox-ingest` Edge Function, so you don't need an OpenAI / Fireworks key in your local env for the TS script.
237
+ **Requires**: `CEREFOX_SUPABASE_URL` and `CEREFOX_ACCESS_TOKEN` (the Cerefox access token — `cfx_pat_…`, from `cerefox token generate` — the Bearer used to invoke Edge Functions). Embedding happens server-side inside the `cerefox-ingest` Edge Function, so you don't need an OpenAI / Fireworks key in your local env for the TS script.
238
238
 
239
239
  The target project must already exist (create it with `cerefox project create cerefox` if needed).
240
240
 
@@ -80,6 +80,26 @@ On an existing database this applies pending migrations and re-applies RPCs
80
80
  in place, so re-run it after upgrading. Detailed walkthrough:
81
81
  [`setup-supabase.md`](setup-supabase.md).
82
82
 
83
+ ## 3b. Generate the Edge Function access token
84
+
85
+ The Edge Functions authenticate callers with a **Cerefox access token**. You need it
86
+ if you'll connect a **Custom GPT** (GPT Actions) or a **remote HTTP MCP** client, and
87
+ for a fully-green `cerefox doctor`. The **local MCP** (local agents), **cloud Claude**
88
+ (OAuth), and the CLI/web reach Supabase over the Data API and don't use it — so this
89
+ step is optional if that's your whole setup, but it's quick and harmless to run:
90
+
91
+ ```bash
92
+ cerefox token generate
93
+ ```
94
+
95
+ This creates a `cfx_pat_…` token, sets it as the `CEREFOX_ACCESS_TOKENS` secret on
96
+ Supabase, and writes `CEREFOX_ACCESS_TOKEN` into your `.env` (so `cerefox doctor`,
97
+ your tools, and any remote client can use it). It prints the token **once** — store
98
+ it. You only need to paste it somewhere by hand if you connect a **Custom GPT**
99
+ (into the Action's Authentication → API Key) or a **remote HTTP MCP** client; the
100
+ local MCP and cloud Claude (OAuth) don't use it. Lose it → `cerefox token rotate`.
101
+ Run `cerefox doctor` again — the "edge functions" check should now be green.
102
+
83
103
  ## 4. Wire up an AI agent
84
104
 
85
105
  ```bash