@cerefox/memory 1.15.2 → 1.15.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/bin/cerefox.js +126 -108
- package/dist/server-assets/_shared/ef-meta/index.ts +3 -3
- package/dist/server-assets/_shared/embeddings/onnx-embedder.ts +34 -0
- package/dist/server-assets/_shared/mcp-tools/ingest.ts +7 -6
- package/dist/server-assets/supabase/functions/cerefox-ingest/index.ts +2 -2
- package/docs/api/openapi.json +5790 -0
- package/docs/guides/api.md +297 -0
- package/docs/guides/linking.md +118 -0
- package/docs/guides/securing-local-access.md +253 -0
- package/docs/guides/staging-env.md +334 -0
- package/package.json +1 -1
|
@@ -0,0 +1,297 @@
|
|
|
1
|
+
# The `/api/v1` HTTP API
|
|
2
|
+
|
|
3
|
+
`cerefox web` serves a JSON API at `/api/v1` alongside the web UI. It was built
|
|
4
|
+
as the UI's own backend, and since **v1.11.0** it is also usable by other
|
|
5
|
+
clients: any caller can identify itself, so its reads and writes are attributed
|
|
6
|
+
to it rather than to the web app. Since **v1.12.0** it is authenticated for
|
|
7
|
+
callers that are not on the same machine (see below).
|
|
8
|
+
|
|
9
|
+
This guide covers what you need to call it from something that is not the
|
|
10
|
+
bundled UI. It is not an exhaustive endpoint reference; the routes live in
|
|
11
|
+
`packages/memory/src/web/routes/` and the response shapes are pinned by the zod
|
|
12
|
+
schemas in `_shared/schemas/`.
|
|
13
|
+
|
|
14
|
+
**If you are choosing a surface, read [`access-paths.md`](access-paths.md)
|
|
15
|
+
first.** MCP is the right answer for most agents. This API is the right answer
|
|
16
|
+
when you want plain HTTP, are already running `cerefox web` (or Cerefox Local),
|
|
17
|
+
and do not want an MCP client in the loop.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Security posture: local by default, key for anything else
|
|
22
|
+
|
|
23
|
+
Since **v1.12.0** this surface is authenticated for callers that are not on the
|
|
24
|
+
same machine:
|
|
25
|
+
|
|
26
|
+
> **A request arriving on the loopback interface (`127.0.0.1`) is allowed
|
|
27
|
+
> without a credential. A request arriving on any other interface must present
|
|
28
|
+
> the server's API key** as `Authorization: Bearer <key>`.
|
|
29
|
+
|
|
30
|
+
So the web UI, agents on the same machine and localhost scripts need nothing,
|
|
31
|
+
while a caller on a Docker network or across a widened bind needs a key:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
cerefox api-key generate # cerefox web
|
|
35
|
+
cerefox-local api-key # Cerefox Local (minted at first boot)
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Without a valid key the server answers `401` with a `WWW-Authenticate: Bearer`
|
|
39
|
+
challenge and a `detail` explaining what to do. With no key configured at all,
|
|
40
|
+
the gate is off and the server behaves exactly as it did before v1.12.0, so
|
|
41
|
+
upgrading never breaks a working install.
|
|
42
|
+
|
|
43
|
+
**→ [`securing-local-access.md`](securing-local-access.md) is the full guide**:
|
|
44
|
+
whether you need a key at all, why it is drawn at the network boundary, the
|
|
45
|
+
container and reverse-proxy recipes, rotation, and what this deliberately does
|
|
46
|
+
not protect.
|
|
47
|
+
|
|
48
|
+
Two things to carry into the rest of this document:
|
|
49
|
+
|
|
50
|
+
- **There is no TLS here.** The key authenticates; it does not encrypt. This is
|
|
51
|
+
still not an internet-facing API. If you need Cerefox reachable over a
|
|
52
|
+
network you do not control, use the Edge Functions.
|
|
53
|
+
- **`X-Forwarded-For` is never consulted** to decide where a request came from.
|
|
54
|
+
A same-host reverse proxy therefore makes every request look local; that
|
|
55
|
+
topology sets `CEREFOX_API_REQUIRE_KEY=1`.
|
|
56
|
+
|
|
57
|
+
## Identifying your client
|
|
58
|
+
|
|
59
|
+
By default every call is recorded as the web app: author `web-ui`, author type
|
|
60
|
+
`user`, access path `webapp`. Supply an identity and it is recorded as you.
|
|
61
|
+
|
|
62
|
+
**This identity is declared, not verified.** It is a label the caller chooses,
|
|
63
|
+
recorded for attribution and record-keeping: so the audit trail says which
|
|
64
|
+
harness wrote a document, so usage analytics can tell your bot's reads from the
|
|
65
|
+
web app's, and so an agent-authored ingest is queued for review (while the
|
|
66
|
+
review workflow is on). It is not a
|
|
67
|
+
credential, it grants nothing, and nothing checks it. See
|
|
68
|
+
[Attribution is not authentication](#attribution-is-not-authentication) below
|
|
69
|
+
before relying on it for anything else.
|
|
70
|
+
|
|
71
|
+
Three optional fields:
|
|
72
|
+
|
|
73
|
+
| Field | Header | Recorded as | Default |
|
|
74
|
+
|---|---|---|---|
|
|
75
|
+
| author | `X-Cerefox-Author` | `cerefox_audit_log.author` on writes | `web-ui` |
|
|
76
|
+
| requestor | `X-Cerefox-Requestor` | `cerefox_usage_log.requestor` on reads and writes | `web-ui` |
|
|
77
|
+
| author type | `X-Cerefox-Author-Type` | `cerefox_audit_log.author_type` | `user` |
|
|
78
|
+
|
|
79
|
+
`author_type` must be `user` or `agent`. Anything else is a `400` naming the
|
|
80
|
+
two valid values.
|
|
81
|
+
|
|
82
|
+
**Headers work on every method.** A `GET` cannot carry a body, and reads are
|
|
83
|
+
half the point of attribution, so headers are the primary mechanism. Endpoints
|
|
84
|
+
that take a JSON body (or a multipart form) also accept the same three as
|
|
85
|
+
fields in it, for clients that find that more natural. If both appear, the
|
|
86
|
+
header wins.
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
# A write, attributed
|
|
90
|
+
curl -X POST http://127.0.0.1:8000/api/v1/ingest \
|
|
91
|
+
-H 'Content-Type: application/json' \
|
|
92
|
+
-H 'X-Cerefox-Author: my-bot' \
|
|
93
|
+
-H 'X-Cerefox-Author-Type: agent' \
|
|
94
|
+
-d '{"title": "Meeting notes", "content": "# Notes\n\n..."}'
|
|
95
|
+
|
|
96
|
+
# A read, attributed
|
|
97
|
+
curl http://127.0.0.1:8000/api/v1/documents/<uuid> \
|
|
98
|
+
-H 'X-Cerefox-Requestor: my-bot'
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
`author` and `requestor` stand in for each other. They name one actor; the
|
|
102
|
+
split is historical (the MCP tools used `requestor` on reads and `author` on
|
|
103
|
+
writes until v1.13.1, which collapsed them to `author` with `requestor` kept as
|
|
104
|
+
an alias). Supplying either identifies you for both, so a client that only sets
|
|
105
|
+
`X-Cerefox-Author` still has its reads attributed correctly.
|
|
106
|
+
|
|
107
|
+
### Two consequences worth knowing before you use it
|
|
108
|
+
|
|
109
|
+
**`author_type: agent` marks your documents as pending review.** An
|
|
110
|
+
agent-authored ingest is recorded `pending_review` rather than `approved`,
|
|
111
|
+
exactly as it is over MCP. That equivalence is the point of the feature: the
|
|
112
|
+
same actor is recorded, and treated, identically whichever transport it used.
|
|
113
|
+
If you want your writes approved on arrival, send `user` (or send nothing).
|
|
114
|
+
|
|
115
|
+
Whether anyone *sees* that status is the store's `review_workflow_enabled`
|
|
116
|
+
flag. With the workflow **off** (the default on a fresh v1.13.0+ install) the
|
|
117
|
+
API does not expose review status at all: no row carries a `review_status`
|
|
118
|
+
key, `GET /search?review_status=…` is a `400`, and
|
|
119
|
+
`POST /documents/{id}/review-status` is a `404`. The status is still recorded
|
|
120
|
+
(v1.13.1), so turning the workflow on later shows exactly what the store would
|
|
121
|
+
have had all along. Read the flag from `GET /config/review_workflow_enabled`
|
|
122
|
+
if your client needs to know. Attribution is unaffected either way. See
|
|
123
|
+
[configuration.md → Review Workflow](configuration.md#review-workflow).
|
|
124
|
+
|
|
125
|
+
**Identifying yourself changes the access path.** There is deliberately no way
|
|
126
|
+
to ask for a particular `access_path`: the server derives it. Supply any
|
|
127
|
+
identity field and the operation is logged as `api`; supply none and it is
|
|
128
|
+
logged as `webapp`. The reasoning is that `access_path` is the one field in the
|
|
129
|
+
usage log the server still sets itself, so accepting it from the caller would
|
|
130
|
+
let a client misreport which transport it used. In the Analytics dashboard,
|
|
131
|
+
`api` operations count toward the agent total; `webapp` does not.
|
|
132
|
+
|
|
133
|
+
### Attribution is not authentication
|
|
134
|
+
|
|
135
|
+
`author`, `requestor` and `author_type` are **declared by the caller and taken
|
|
136
|
+
at face value**. Nothing verifies them, here or anywhere else in Cerefox: MCP
|
|
137
|
+
takes `author` as a client-declared string, and the Edge Functions accept
|
|
138
|
+
whatever `requestor` arrives. Any client can send any name, including a name
|
|
139
|
+
another client uses.
|
|
140
|
+
|
|
141
|
+
They exist for **attribution and record-keeping**: a legible audit trail, usage
|
|
142
|
+
analytics that can tell callers apart, and the review-queue behaviour of
|
|
143
|
+
`author_type: agent` while the review workflow is on. They are not a security measure, they do not establish an
|
|
144
|
+
authenticated identity, and no access decision is made on them. Treat a name in
|
|
145
|
+
the audit log as "what the caller said", never as "who the caller was".
|
|
146
|
+
|
|
147
|
+
The API key added in v1.12.0 (#229) does not change this. It proves a caller is
|
|
148
|
+
*allowed to talk to this server*; it does not prove the caller is *who it says
|
|
149
|
+
it is*. A remote client holding the key can still send any `author` it likes.
|
|
150
|
+
Authentication and attribution stay separate on purpose — conflating them would
|
|
151
|
+
produce a design that does neither well.
|
|
152
|
+
|
|
153
|
+
`require_requestor_identity` and `requestor_identity_format` do **not** apply to
|
|
154
|
+
this surface. They are enforced by the Edge Functions only. See
|
|
155
|
+
[`configuration.md`](configuration.md).
|
|
156
|
+
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
## Endpoints
|
|
160
|
+
|
|
161
|
+
**A machine-readable OpenAPI 3.1 document lives at
|
|
162
|
+
[`docs/api/openapi.json`](../api/openapi.json).** If you are generating a client
|
|
163
|
+
or a tool set, read that rather than this table: it carries the request and
|
|
164
|
+
response *shapes*, the path parameters and the error model, which a prose table
|
|
165
|
+
cannot. Point any OpenAPI tool at it.
|
|
166
|
+
|
|
167
|
+
It is **generated** (`bun scripts/gen_openapi.ts`) from three sources that are
|
|
168
|
+
each kept honest on their own: the route registrations, the summaries in the
|
|
169
|
+
table below, and the zod schemas in `_shared/schemas/`. Those schemas are
|
|
170
|
+
asserted against a running server by `api-schema-truth.test.ts`, so the spec
|
|
171
|
+
describes what the API returns rather than what someone believed it returned.
|
|
172
|
+
|
|
173
|
+
Coverage is stated in the document itself under `x-cerefox-coverage`. Endpoints
|
|
174
|
+
with no modelled shape are listed there and get no response schema rather than
|
|
175
|
+
an invented one — a spec that guesses is worse than one that admits a gap.
|
|
176
|
+
|
|
177
|
+
Base URL is wherever `cerefox web` is listening (`http://127.0.0.1:8000` by
|
|
178
|
+
default; Cerefox Local picks its own port and `cerefox-local status` prints it).
|
|
179
|
+
|
|
180
|
+
| Method + path | Purpose |
|
|
181
|
+
|---|---|
|
|
182
|
+
| `GET /version` | Server version and environment label. |
|
|
183
|
+
| `GET /schema-version` | Deployed schema version. |
|
|
184
|
+
| `GET /openapi.json` | This API's OpenAPI 3.1 description, served by the version you are talking to. Every `/api/v1` response points at it with `Link: </api/v1/openapi.json>; rel="service-desc"` (RFC 8631). |
|
|
185
|
+
| `GET /search?q=…` | Hybrid search (FTS + semantic). `&review_status=approved\|pending_review` filters server-side; `400` while the review workflow is off. |
|
|
186
|
+
| `GET /dashboard`, `GET /dashboard/recent-docs` | Dashboard aggregates. |
|
|
187
|
+
| `GET /documents/{id}` | Full document, with metadata, projects and versions. |
|
|
188
|
+
| `GET /documents/{id}/chunks` | The document's chunks. |
|
|
189
|
+
| `GET /documents/{id}/versions` | Version history. |
|
|
190
|
+
| `POST /documents/{id}/versions/{version_id}/archive` | Archive or unarchive one stored version (body `{ "archived": true\|false }`). Archiving strips a version's search artifacts; its content is kept as the safety copy. |
|
|
191
|
+
| `GET /documents/{id}/download` | Raw markdown. |
|
|
192
|
+
| `GET /documents/trash` | Soft-deleted documents, newest first, `limit` ≤ 500. The exact total is in the `X-Total-Count` response header (v1.14.1). |
|
|
193
|
+
| `POST /documents/metadata-search` | Query by metadata / project / time, no text query. |
|
|
194
|
+
| `GET /metadata-keys` | Metadata keys with counts and example values. |
|
|
195
|
+
| `GET /resolve-link`, `GET /check-filename` | Link and title resolution helpers. |
|
|
196
|
+
| `POST /ingest` | Create or update from JSON (`title`, `content`, …). Unknown fields are a `400`. |
|
|
197
|
+
| `POST /ingest/file` | Create from a multipart file upload. |
|
|
198
|
+
| `POST /documents/{id}/upload` | Replace a document's content from a file. |
|
|
199
|
+
| `POST /documents/{id}/edit` | Update title, content, metadata, projects. Unknown fields are a `400`. |
|
|
200
|
+
| `DELETE /documents/{id}` | Soft-delete (to trash). |
|
|
201
|
+
| `POST /documents/{id}/restore` | Restore from trash. |
|
|
202
|
+
| `DELETE /documents/{id}/purge` | Permanent delete. Irreversible. |
|
|
203
|
+
| `POST /documents/{id}/review-status` | Approve or re-queue for review. `404` while the review workflow is off. |
|
|
204
|
+
| `GET /projects`, `POST /projects`, `PUT /projects/{id}`, `DELETE /projects/{id}` | Project CRUD. |
|
|
205
|
+
| `GET /projects/{id}/documents` | A project's documents. |
|
|
206
|
+
| `GET /config`, `GET /config/{key}`, `PUT /config/{key}` | Runtime config. |
|
|
207
|
+
| `GET /audit-log` | Audit trail, filterable. |
|
|
208
|
+
| `GET /preferences`, `PUT /preferences` | Web UI preferences (currently the colour scheme), stored in a file in the user-state dir, not the database. Registers even when Supabase is not configured. |
|
|
209
|
+
| `GET /docs`, `GET /docs/{path}` | The guides bundled with this server: a listing, and one document's markdown. Serves the shipped copies, so it answers the same content regardless of what is in the knowledge base. |
|
|
210
|
+
| `GET /usage-log`, `GET /usage-log/summary`, `GET /usage-log/export.csv` | Usage queries. |
|
|
211
|
+
|
|
212
|
+
Paths are shown without the `/api/v1` prefix for width; every one carries it.
|
|
213
|
+
The writes and the two reads that log usage (`/search`, `/documents/{id}`)
|
|
214
|
+
honour the identity headers. `/version` needs nothing.
|
|
215
|
+
|
|
216
|
+
### Every mutation needs a concurrency token
|
|
217
|
+
|
|
218
|
+
Cerefox uses optimistic locking, and this API is no exception. Read the
|
|
219
|
+
document first: `GET /documents/{id}` returns its `content_hash`. Then send
|
|
220
|
+
that hash back with the write. A stale hash is a `409` carrying `current_hash`,
|
|
221
|
+
so you can re-read without a second round trip. Sending no hash (and no
|
|
222
|
+
`last_write_wins`) is a `400` with `CEREFOX_TOKEN_REQUIRED`.
|
|
223
|
+
|
|
224
|
+
| Write | Where the hash goes | `last_write_wins` |
|
|
225
|
+
|---|---|---|
|
|
226
|
+
| `POST /ingest` with `document_id`, or with `update_if_exists` matching a document | `expected_content_hash` in the body | Accepted |
|
|
227
|
+
| `POST /documents/{id}/edit` | `expected_content_hash` in the body. Checked on every save, including a metadata-only one | Accepted |
|
|
228
|
+
| `POST /documents/{id}/upload`, `POST /ingest/file` with `update_existing` | `expected_content_hash` form field | `last_write_wins=true` form field |
|
|
229
|
+
| `DELETE /documents/{id}` | `X-Cerefox-Expected-Content-Hash` header, or `expected_content_hash` query parameter | Not accepted |
|
|
230
|
+
| `POST /documents/{id}/restore`, `DELETE /documents/{id}/purge` | No hash | Not applicable |
|
|
231
|
+
|
|
232
|
+
`last_write_wins` skips the check. Use it only when an external source of truth
|
|
233
|
+
(a file you are syncing, say) makes a conflict meaningless; there is no implicit
|
|
234
|
+
default. Creating a new document needs no hash.
|
|
235
|
+
|
|
236
|
+
**Check the status code, not just the body.** Every refusal is a real HTTP
|
|
237
|
+
status: `400` for a malformed or incomplete request (including a body field the
|
|
238
|
+
route does not know, which is refused rather than ignored), `404` for a missing
|
|
239
|
+
document or version, `409` for a concurrency conflict or a trashed document,
|
|
240
|
+
`422` for content that links to document ids that do not exist, and `503` when
|
|
241
|
+
the embedder is unavailable. The reason is in the body's `detail` (and, on the
|
|
242
|
+
ingest and edit routes, also in `error`, which they have always used). Until
|
|
243
|
+
v1.15.2 a delete with a stale hash answered `500`, and `POST /ingest` answered
|
|
244
|
+
`200` with `success: false` for a missing title or empty content; both are
|
|
245
|
+
fixed.
|
|
246
|
+
|
|
247
|
+
**Delete follows a read too.** A caller that identifies itself must send the
|
|
248
|
+
hash on `DELETE /documents/{id}`, exactly as `cerefox_delete_document` requires
|
|
249
|
+
it over MCP. A caller that sends no identity is the bundled web UI, which
|
|
250
|
+
confirms with the human in a dialog instead, and it keeps working unchanged.
|
|
251
|
+
|
|
252
|
+
The full contract of every endpoint (parameters, headers, bodies, error codes)
|
|
253
|
+
is in [`docs/api/openapi.json`](../api/openapi.json), generated from the route
|
|
254
|
+
handlers themselves. A running server serves its own copy at
|
|
255
|
+
`GET /api/v1/openapi.json` (since v1.15.3), so a client always gets the
|
|
256
|
+
contract of the version it is actually talking to. It does not need to know
|
|
257
|
+
that path: every `/api/v1` response carries
|
|
258
|
+
`Link: </api/v1/openapi.json>; rel="service-desc"`, the standard pointer to a
|
|
259
|
+
machine-readable API description (RFC 8631).
|
|
260
|
+
|
|
261
|
+
Purge (`DELETE /documents/{id}/purge`) is irreversible and takes no token. It is
|
|
262
|
+
reachable by anything that can reach the port, which is another reason the
|
|
263
|
+
warning at the top of this guide is not boilerplate.
|
|
264
|
+
|
|
265
|
+
The response is `{ "success": true, "purged": <bool> }` (v1.14.0): `purged` is
|
|
266
|
+
`false` when the document was no longer in the trash when the call landed (a
|
|
267
|
+
restore raced it) and is still live; the RPC is a silent no-op in that case.
|
|
268
|
+
|
|
269
|
+
**There is no bulk purge.** The web UI's "Empty trash" (v1.14.0) is a loop in
|
|
270
|
+
the browser over this per-document endpoint, one call and one audit entry per
|
|
271
|
+
document, with a confirmation and a Stop button in front of it. It purges only
|
|
272
|
+
the documents listed when the count was confirmed; anything trashed afterwards
|
|
273
|
+
stays in the trash. A single
|
|
274
|
+
request that emptied the trash would be a footgun on a surface anything with
|
|
275
|
+
the key can reach, so it was not added and should not be.
|
|
276
|
+
|
|
277
|
+
---
|
|
278
|
+
|
|
279
|
+
## Choosing between this and MCP
|
|
280
|
+
|
|
281
|
+
Both surfaces run the same code underneath and write the same rows. The
|
|
282
|
+
practical differences:
|
|
283
|
+
|
|
284
|
+
| | `/api/v1` | MCP |
|
|
285
|
+
|---|---|---|
|
|
286
|
+
| Transport | Plain HTTP | stdio or Streamable HTTP |
|
|
287
|
+
| Client needs | An HTTP client | An MCP client |
|
|
288
|
+
| Identity | Optional, defaults to `web-ui` | Per-call `author` |
|
|
289
|
+
| Section-level edits | None: `POST /documents/{id}/edit` replaces the whole content | `cerefox_insert`, `cerefox_edit` |
|
|
290
|
+
| Guidance for agents | This guide | `cerefox_get_help()`, in-band |
|
|
291
|
+
| Authentication | None on loopback; API key otherwise | None locally; token or OAuth remotely |
|
|
292
|
+
|
|
293
|
+
If an agent is doing the calling, MCP is usually better: the tools carry their
|
|
294
|
+
own documentation, and `cerefox_get_help()` keeps the conventions in front of
|
|
295
|
+
the model. Reach for the API when the caller is a program you are writing
|
|
296
|
+
yourself, or when you want one transport for a harness that already speaks
|
|
297
|
+
HTTP.
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# Document links: capabilities and guardrails
|
|
2
|
+
|
|
3
|
+
Cerefox documents can link to each other with ordinary markdown links. This
|
|
4
|
+
guide covers how linking works, and the deterministic guardrails (v1.7.0+)
|
|
5
|
+
that protect links from the most common way they break: an AI agent
|
|
6
|
+
corrupting a long document id while rewriting content.
|
|
7
|
+
|
|
8
|
+
## How linking works
|
|
9
|
+
|
|
10
|
+
Write a normal markdown link whose target is the document's UUID:
|
|
11
|
+
|
|
12
|
+
```markdown
|
|
13
|
+
See the [Opportunity Index](c937b70f-77af-43d3-b9bc-9f31e0d2041d) for details.
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The Cerefox web UI resolves these at click time — the stored markdown is
|
|
17
|
+
untouched, and the link keeps working if the target is renamed. Every search
|
|
18
|
+
result and every write response includes the document id, so an id is always
|
|
19
|
+
one lookup away.
|
|
20
|
+
|
|
21
|
+
Four link forms are accepted; one is recommended:
|
|
22
|
+
|
|
23
|
+
| Form | Example target | Use |
|
|
24
|
+
|---|---|---|
|
|
25
|
+
| **Document UUID** | `c937b70f-…` | **Always, for cross-references.** Stable, unambiguous, encoding-safe, validated (below). |
|
|
26
|
+
| Repo-relative path | `docs/guides/quickstart.md` | Repo-ingested files only (their markdown naturally uses paths). |
|
|
27
|
+
| Basename | `quickstart.md` | Best-effort fallback for the same. |
|
|
28
|
+
| `[[Wikilink]]` | `[[Future Doc Name]]` | A deliberate **dangling** reference — "this document should exist someday." Never validated. |
|
|
29
|
+
|
|
30
|
+
Agent-specific writing rules live in `AGENT_GUIDE.md` → "Writing linkable
|
|
31
|
+
content".
|
|
32
|
+
|
|
33
|
+
## Why links need a guardrail: LLMs and long ids
|
|
34
|
+
|
|
35
|
+
This is worth understanding, because it explains the design and it is not a
|
|
36
|
+
bug in any particular AI model — it is structural.
|
|
37
|
+
|
|
38
|
+
A language model editing a document does not copy text the way a program
|
|
39
|
+
does. It **regenerates** the text token by token from an internal
|
|
40
|
+
representation. Normal prose survives this well because language is
|
|
41
|
+
redundant — if a word drifts slightly, the surrounding context constrains it
|
|
42
|
+
back. A UUID has **zero redundancy**: 32 hex characters, each independent,
|
|
43
|
+
carrying no meaning that could catch a mistake. `508e4c21` regenerated as
|
|
44
|
+
`502041d8` looks equally plausible to the model, and to anyone reading past
|
|
45
|
+
it.
|
|
46
|
+
|
|
47
|
+
Now compound that. A document with a table of 25–30 links makes an agent
|
|
48
|
+
regenerate 25–30 UUIDs *every time it rewrites that table*. Even an
|
|
49
|
+
excellent per-character accuracy, multiplied across hundreds of characters
|
|
50
|
+
of pure entropy, several times per session, makes occasional corruption
|
|
51
|
+
**expected, not unlucky** — and observed repeatedly in real use. Each
|
|
52
|
+
corrupted id silently becomes a
|
|
53
|
+
dead link that read-back review may or may not catch.
|
|
54
|
+
|
|
55
|
+
Two defenses exist, and they compose:
|
|
56
|
+
|
|
57
|
+
1. **Reduce how much gets regenerated**: the partial-edit tools
|
|
58
|
+
(`cerefox_insert`, `cerefox_edit`) change only the targeted section, so
|
|
59
|
+
untouched links are never retyped and literally cannot corrupt. Give
|
|
60
|
+
link-heavy tables their own heading so appends never rewrite existing
|
|
61
|
+
rows.
|
|
62
|
+
2. **Catch what still gets retyped**: the write-time validation below.
|
|
63
|
+
|
|
64
|
+
## The write-time guardrail (v1.7.0+)
|
|
65
|
+
|
|
66
|
+
Every write — create, full update, or partial edit, on every surface (MCP,
|
|
67
|
+
CLI, web, GPT Actions) — validates the `[Text](uuid)` links in the content
|
|
68
|
+
against the store, inside the same database transaction as the write:
|
|
69
|
+
|
|
70
|
+
- **A link to a document id that does not exist rejects the whole write**,
|
|
71
|
+
and the error lists every unresolvable id. Nothing is stored; the
|
|
72
|
+
corruption never lands. The check is deterministic and costs ~1–2ms.
|
|
73
|
+
- **The agent can self-correct in the same turn**: the error says to re-read
|
|
74
|
+
the source each link was copied from, fix the id(s), and resend — and
|
|
75
|
+
explicitly not to retry unchanged.
|
|
76
|
+
- **Examples escape via code formatting.** A link inside backticks or a
|
|
77
|
+
fenced code block is an example, not a link, and is not validated — which
|
|
78
|
+
is also just correct markdown, since a bare link-syntax example would
|
|
79
|
+
render as a real link. There is deliberately **no bypass flag**.
|
|
80
|
+
- **A trashed target still resolves** (the id denotes a document; restoring
|
|
81
|
+
decides its fate). **`[[Wikilinks]]` are never validated** — that is the
|
|
82
|
+
sanctioned form for a forward reference.
|
|
83
|
+
|
|
84
|
+
### Updates validate only what the write introduces
|
|
85
|
+
|
|
86
|
+
On an update, only **newly-introduced** link ids are validated. A dead link
|
|
87
|
+
the document *already carried* — typically because its target was purged
|
|
88
|
+
after the link was written — does not block an unrelated edit or a file
|
|
89
|
+
re-sync. Without this, one purge could make every document that ever linked
|
|
90
|
+
the purged target permanently unwritable through automated paths.
|
|
91
|
+
|
|
92
|
+
The corollary: a link that was mangled *before* v1.7.0, or whose target was
|
|
93
|
+
purged later, sits undetected in the content. That is what the sweep is for.
|
|
94
|
+
|
|
95
|
+
## The dead-link sweep (v1.7.1+)
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
cerefox document dead-links # human-readable report
|
|
99
|
+
cerefox document dead-links --json # machine-readable
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
One server-side pass over the whole knowledge base, reporting every
|
|
103
|
+
`[Text](uuid)` link whose target no longer exists — with the linking
|
|
104
|
+
document and an occurrence count. Same scanning rules as the write guard
|
|
105
|
+
(the two share one implementation, so they cannot disagree). Trashed
|
|
106
|
+
*linker* documents are excluded until restored. Run it after a purge, or
|
|
107
|
+
periodically; it is a full content scan, so it is deliberately on demand
|
|
108
|
+
rather than part of `cerefox doctor`.
|
|
109
|
+
|
|
110
|
+
Fix each hit by editing the linking document: correct the id, remove the
|
|
111
|
+
link, or backtick it as an example.
|
|
112
|
+
|
|
113
|
+
## Design record
|
|
114
|
+
|
|
115
|
+
The full rationale — including the rejected bypass-flag design, why fence
|
|
116
|
+
handling splits on lines instead of pairing with a regex, and the
|
|
117
|
+
probability analysis of a mangled id colliding with a real document — is in
|
|
118
|
+
[`docs/specs/link-integrity-design.md`](../specs/link-integrity-design.md).
|
|
@@ -0,0 +1,253 @@
|
|
|
1
|
+
# Securing local access (the API key)
|
|
2
|
+
|
|
3
|
+
`cerefox web` and Cerefox Local serve an HTTP API on a port. This guide is
|
|
4
|
+
about who is allowed to use it.
|
|
5
|
+
|
|
6
|
+
**Most people need to do nothing.** Read the first section, confirm you are in
|
|
7
|
+
the common case, and stop.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## The rule
|
|
12
|
+
|
|
13
|
+
> **A request arriving on the loopback interface (`127.0.0.1`) is allowed
|
|
14
|
+
> without a credential. A request arriving on any other interface must present
|
|
15
|
+
> the server's API key.**
|
|
16
|
+
|
|
17
|
+
Loopback means "a connection that started on this machine". Your browser, an
|
|
18
|
+
agent running on your laptop, a script you typed into your own terminal: all
|
|
19
|
+
loopback, all exempt, all work with no configuration.
|
|
20
|
+
|
|
21
|
+
Introduced in **v1.12.0** ([#229](https://github.com/fstamatelopoulos/cerefox/issues/229)).
|
|
22
|
+
|
|
23
|
+
## Do you need a key?
|
|
24
|
+
|
|
25
|
+
**It depends on how Cerefox itself is running**, so start there.
|
|
26
|
+
|
|
27
|
+
### If you run Cerefox Local (Docker)
|
|
28
|
+
|
|
29
|
+
| Your publish address | Cerefox checks credentials? | What protects the port | You do |
|
|
30
|
+
|---|---|---|---|
|
|
31
|
+
| `127.0.0.1` (the default) | **No** | **Docker.** The port is bound to this machine's loopback only, so nothing outside it can open a connection at all. | Nothing |
|
|
32
|
+
| Anything wider (`CEREFOX_LOCAL_BIND=0.0.0.0`) | **Yes, for every caller** | The key | Give the key to every client, including your own browser |
|
|
33
|
+
|
|
34
|
+
**Read the middle column carefully.** On a default install Cerefox is *not*
|
|
35
|
+
refusing remote callers — it never sees them. Docker declines the connection at
|
|
36
|
+
the socket, before Cerefox is involved. "No key required" and "protected" are
|
|
37
|
+
both true at the same time, for different reasons, and it is worth knowing
|
|
38
|
+
which is doing the work: widen the publish address and the first protection
|
|
39
|
+
disappears instantly, which is why the key turns on in the same move.
|
|
40
|
+
|
|
41
|
+
There is no middle setting here, and that is deliberate rather than a
|
|
42
|
+
limitation we forgot to lift. **Docker rewrites the source address of every
|
|
43
|
+
request that comes through a published port**, so a server inside a container
|
|
44
|
+
cannot tell a request from your own machine apart from one that crossed the
|
|
45
|
+
network — both arrive from the same bridge address. Since the distinction
|
|
46
|
+
cannot be made, the gate does not pretend to make it: it is either off, with
|
|
47
|
+
the publish address as the boundary, or on for everybody.
|
|
48
|
+
|
|
49
|
+
`cerefox-local` sets this for you from `CEREFOX_LOCAL_BIND`. To force the gate
|
|
50
|
+
on while still publishing to loopback:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
echo 'CEREFOX_API_REQUIRE_KEY=1' >> ~/.cerefox/local/.env
|
|
54
|
+
cerefox-local restart
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
> **v1.12.0 got this wrong** and injected a key into every container without
|
|
58
|
+
> require-mode, so the loopback exemption never matched and every caller — the
|
|
59
|
+
> web UI included — got a `401`. Fixed in **v1.12.1**. If you are on v1.12.0
|
|
60
|
+
> and seeing 401s from a container, upgrade.
|
|
61
|
+
|
|
62
|
+
### If you run `cerefox web` directly (npm install)
|
|
63
|
+
|
|
64
|
+
Here the server sees real client addresses, so the loopback rule applies as
|
|
65
|
+
written.
|
|
66
|
+
|
|
67
|
+
| Your situation | Need a key? |
|
|
68
|
+
|---|---|
|
|
69
|
+
| You use the web UI in your browser | **No** |
|
|
70
|
+
| An agent or script runs on the same machine | **No** |
|
|
71
|
+
| A container reaches your host server via `host.docker.internal` | **No** — it arrives on the host's loopback |
|
|
72
|
+
| You bind `--host 0.0.0.0` to reach Cerefox from another machine | **Yes** |
|
|
73
|
+
| A reverse proxy sits in front on the same machine | **Yes**, plus `CEREFOX_API_REQUIRE_KEY=1` (see below) |
|
|
74
|
+
|
|
75
|
+
If every row that applies to you says No, you are done.
|
|
76
|
+
|
|
77
|
+
**Not sure?** Ask for the version endpoint the way your client will. `200`
|
|
78
|
+
means you are on the exempt path; `401` means you need a key.
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
docker exec <your-client-container> \
|
|
82
|
+
curl -s -o /dev/null -w '%{http_code}\n' http://host.docker.internal:8010/api/v1/version
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
## Why it works this way
|
|
86
|
+
|
|
87
|
+
The attacker worth defending against can reach the port but **cannot read your
|
|
88
|
+
filesystem**: a machine on the café wifi after you widened the bind, a
|
|
89
|
+
container on a shared Docker network, a malicious page open in your browser.
|
|
90
|
+
|
|
91
|
+
Someone who *can* read your filesystem already has the key file, your `.env`,
|
|
92
|
+
and your database credentials. Demanding a key from them would protect nothing
|
|
93
|
+
while making the common case worse for everyone. On a single-user machine,
|
|
94
|
+
loopback access and key-file access are the same trust boundary, so the gate is
|
|
95
|
+
drawn at the place where the trust actually changes: the network.
|
|
96
|
+
|
|
97
|
+
This is also why **the browser never holds a key**. The web UI is a static
|
|
98
|
+
file, so a key embedded in the page could be read by anything that can load the
|
|
99
|
+
page — precisely the attacker the key exists to stop. A key in an
|
|
100
|
+
unauthenticated page is not a secret. The UI works because it is local, not
|
|
101
|
+
because it holds a credential.
|
|
102
|
+
|
|
103
|
+
## Getting a key
|
|
104
|
+
|
|
105
|
+
### `cerefox web` (npm install)
|
|
106
|
+
|
|
107
|
+
No key exists until you ask for one. Until then the gate is off and the server
|
|
108
|
+
behaves exactly as it did before v1.12.0, which is what makes upgrading safe.
|
|
109
|
+
|
|
110
|
+
```bash
|
|
111
|
+
cerefox api-key generate # mint one, write it to your .env, print it once
|
|
112
|
+
cerefox api-key show # print it again, in full
|
|
113
|
+
cerefox api-key rotate # replace it
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`generate` refuses if a key already exists. Silently replacing one would give
|
|
117
|
+
every client you had already configured a `401` with no explanation.
|
|
118
|
+
|
|
119
|
+
### Cerefox Local (Docker)
|
|
120
|
+
|
|
121
|
+
The container mints a key **automatically at first boot** and persists it on
|
|
122
|
+
the data volume, so it survives `cerefox-local upgrade`. You never create it;
|
|
123
|
+
you only read it.
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
cerefox-local api-key # print it (and say whether it is enforced)
|
|
127
|
+
cerefox-local api-key --rotate # mint a new one and restart
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The key exists whether or not the gate is on, so turning the gate on later
|
|
131
|
+
never changes the value your clients were given. `cerefox-local api-key` tells
|
|
132
|
+
you which state you are in — printing a key while implying it is being enforced
|
|
133
|
+
when it is not would be worse than printing nothing.
|
|
134
|
+
|
|
135
|
+
## Using it
|
|
136
|
+
|
|
137
|
+
A remote caller sends it as a bearer token:
|
|
138
|
+
|
|
139
|
+
```bash
|
|
140
|
+
curl http://<host>:8010/api/v1/documents/<uuid> \
|
|
141
|
+
-H 'Authorization: Bearer cfx_lak_…' \
|
|
142
|
+
-H 'X-Cerefox-Requestor: my-bot'
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Without a valid key the server answers `401` with a `WWW-Authenticate: Bearer`
|
|
146
|
+
challenge and a `detail` field explaining what to do.
|
|
147
|
+
|
|
148
|
+
The key authenticates; it does not identify. `X-Cerefox-Requestor` and
|
|
149
|
+
`X-Cerefox-Author` are still declared labels for attribution, exactly as
|
|
150
|
+
described in [`api.md`](api.md#attribution-is-not-authentication). Holding the
|
|
151
|
+
key does not make the name you send true.
|
|
152
|
+
|
|
153
|
+
## Recipe: a harness in its own container
|
|
154
|
+
|
|
155
|
+
The case where a key is genuinely needed, and the one worth automating.
|
|
156
|
+
|
|
157
|
+
**The ordering that avoids a chicken-and-egg problem.** You could boot Cerefox,
|
|
158
|
+
read its key, then configure the client — but that is two steps and a manual
|
|
159
|
+
copy every time either side is rebuilt. Instead, mint one key on the host and
|
|
160
|
+
give the same value to both containers. Cerefox prefers an injected key over
|
|
161
|
+
the one it would generate, so this pins it across recreates.
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
# 1. Mint once, on the host.
|
|
165
|
+
KEY="cfx_lak_$(openssl rand -base64 32 | tr -d '=+/' | cut -c1-43)"
|
|
166
|
+
|
|
167
|
+
# 2. Give it to Cerefox Local. Both variables are on the passthrough allowlist,
|
|
168
|
+
# so they survive `cerefox-local upgrade` and any recreate. REQUIRE_KEY is
|
|
169
|
+
# what turns the gate on: inside a container there is no loopback exemption
|
|
170
|
+
# to fall back on, so the gate is all-or-nothing.
|
|
171
|
+
{ echo "CEREFOX_API_KEY=$KEY"; echo "CEREFOX_API_REQUIRE_KEY=1"; } >> ~/.cerefox/local/.env
|
|
172
|
+
cerefox-local restart
|
|
173
|
+
|
|
174
|
+
# Note: with the gate on, YOUR BROWSER needs the key too. If you use the web UI
|
|
175
|
+
# daily, prefer leaving Cerefox published on loopback with the gate off and
|
|
176
|
+
# giving your harness host networking instead.
|
|
177
|
+
|
|
178
|
+
# 3. Give the same value to your client container, however it takes config.
|
|
179
|
+
docker run -e CEREFOX_API_KEY="$KEY" … your-harness
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
**Rotating** is the same two writes plus two restarts, in this order: update
|
|
183
|
+
Cerefox first, then the client. Between them the client gets `401`s, so keep
|
|
184
|
+
the window short, or accept both keys briefly by setting a comma-separated list
|
|
185
|
+
(`CEREFOX_API_KEY=new,old`), migrating the client, then dropping the old value.
|
|
186
|
+
|
|
187
|
+
## Recipe: a reverse proxy on the same machine
|
|
188
|
+
|
|
189
|
+
**This is the case the loopback rule cannot handle on its own, so read it if it
|
|
190
|
+
applies to you.**
|
|
191
|
+
|
|
192
|
+
A proxy on the same host terminates the client's connection and opens its own
|
|
193
|
+
to Cerefox, from `127.0.0.1`. So *every* proxied request looks local and is
|
|
194
|
+
exempt, including requests that came from the internet.
|
|
195
|
+
|
|
196
|
+
Cerefox deliberately does **not** read `X-Forwarded-For` to recover the
|
|
197
|
+
original address. That header is set by the caller, so trusting it would let
|
|
198
|
+
anyone claim to be local — strictly worse than having no gate at all.
|
|
199
|
+
|
|
200
|
+
The answer is to stop exempting loopback:
|
|
201
|
+
|
|
202
|
+
```bash
|
|
203
|
+
CEREFOX_API_REQUIRE_KEY=1
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Now every caller presents the key, proxy included. Your browser will need it
|
|
207
|
+
too, so this mode suits headless deployments rather than daily UI use.
|
|
208
|
+
|
|
209
|
+
## What this does not protect
|
|
210
|
+
|
|
211
|
+
**Other programs on your own machine.** Any local process can reach a loopback
|
|
212
|
+
port. This is a network boundary, not process isolation.
|
|
213
|
+
|
|
214
|
+
**Traffic on the wire.** There is no TLS. The key authenticates; it does not
|
|
215
|
+
encrypt. Anyone who can observe the connection can read the key and everything
|
|
216
|
+
else.
|
|
217
|
+
|
|
218
|
+
**Fine-grained access.** One key, all-or-nothing. No per-client scoping, no
|
|
219
|
+
read-only keys, no expiry, no revocation beyond rotating the single value.
|
|
220
|
+
|
|
221
|
+
**Being on the internet.** A key makes a widened bind survivable. It does not
|
|
222
|
+
turn this into an internet-facing API: there is no rate limiting, no TLS, no
|
|
223
|
+
audit of failed attempts. If you need Cerefox reachable over a network you do
|
|
224
|
+
not control, use the Edge Functions — that is the surface built for it, and it
|
|
225
|
+
authenticates every request against a rotatable token. See
|
|
226
|
+
[`access-paths.md`](access-paths.md).
|
|
227
|
+
|
|
228
|
+
## Troubleshooting
|
|
229
|
+
|
|
230
|
+
**Everything returns 401 after upgrading.** You have `CEREFOX_API_REQUIRE_KEY=1`
|
|
231
|
+
set, which removes the loopback exemption. Unset it unless you are running a
|
|
232
|
+
reverse proxy.
|
|
233
|
+
|
|
234
|
+
**My container gets 401 but curl on the host works.** With `cerefox web` on the
|
|
235
|
+
host, that is the gate behaving correctly: your container reaches Cerefox over
|
|
236
|
+
a Docker network, not loopback. Give it the key (see the recipe above).
|
|
237
|
+
|
|
238
|
+
**Everything gets 401 from a Cerefox Local container, including the web UI.**
|
|
239
|
+
That is the v1.12.0 bug, not a configuration problem. Run
|
|
240
|
+
`cerefox-local upgrade` to get v1.12.1 or newer.
|
|
241
|
+
|
|
242
|
+
**`cerefox-local api-key` says no key was found.** The image predates v1.12.0.
|
|
243
|
+
Run `cerefox-local upgrade`.
|
|
244
|
+
|
|
245
|
+
**I rotated and now everything is broken.** Rotation invalidates the old key at
|
|
246
|
+
the next restart. Update every remote client. Local callers are unaffected,
|
|
247
|
+
which is a useful way to confirm the server itself is healthy: if
|
|
248
|
+
`curl http://127.0.0.1:<port>/api/v1/version` works from the host, the server
|
|
249
|
+
is fine and the problem is client configuration.
|
|
250
|
+
|
|
251
|
+
**I lost the key.** You cannot recover it from the server, but you can read it
|
|
252
|
+
from where it is stored: `cerefox api-key show`, or `cerefox-local api-key`.
|
|
253
|
+
Failing that, rotate and reconfigure your clients.
|