@cerefox/memory 1.15.1 → 1.15.3

package/docs/guides/api.md

@@ -0,0 +1,297 @@
+# The `/api/v1` HTTP API
+
+`cerefox web` serves a JSON API at `/api/v1` alongside the web UI. It was built
+as the UI's own backend, and since **v1.11.0** it is also usable by other
+clients: any caller can identify itself, so its reads and writes are attributed
+to it rather than to the web app. Since **v1.12.0** it is authenticated for
+callers that are not on the same machine (see below).
+
+This guide covers what you need to call it from something that is not the
+bundled UI. It is not an exhaustive endpoint reference; the routes live in
+`packages/memory/src/web/routes/` and the response shapes are pinned by the zod
+schemas in `_shared/schemas/`.
+
+**If you are choosing a surface, read [`access-paths.md`](access-paths.md)
+first.** MCP is the right answer for most agents. This API is the right answer
+when you want plain HTTP, are already running `cerefox web` (or Cerefox Local),
+and do not want an MCP client in the loop.
+
+---
+
+## Security posture: local by default, key for anything else
+
+Since **v1.12.0** this surface is authenticated for callers that are not on the
+same machine:
+
+> **A request arriving on the loopback interface (`127.0.0.1`) is allowed
+> without a credential. A request arriving on any other interface must present
+> the server's API key** as `Authorization: Bearer <key>`.
+
+So the web UI, agents on the same machine and localhost scripts need nothing,
+while a caller on a Docker network or across a widened bind needs a key:
+
+```bash
+cerefox api-key generate     # cerefox web
+cerefox-local api-key        # Cerefox Local (minted at first boot)
+```
+
+Without a valid key the server answers `401` with a `WWW-Authenticate: Bearer`
+challenge and a `detail` explaining what to do. With no key configured at all,
+the gate is off and the server behaves exactly as it did before v1.12.0, so
+upgrading never breaks a working install.
+
+**→ [`securing-local-access.md`](securing-local-access.md) is the full guide**:
+whether you need a key at all, why it is drawn at the network boundary, the
+container and reverse-proxy recipes, rotation, and what this deliberately does
+not protect.
+
+Two things to carry into the rest of this document:
+
+- **There is no TLS here.** The key authenticates; it does not encrypt. This is
+  still not an internet-facing API. If you need Cerefox reachable over a
+  network you do not control, use the Edge Functions.
+- **`X-Forwarded-For` is never consulted** to decide where a request came from.
+  A same-host reverse proxy therefore makes every request look local; that
+  topology sets `CEREFOX_API_REQUIRE_KEY=1`.
+
+## Identifying your client
+
+By default every call is recorded as the web app: author `web-ui`, author type
+`user`, access path `webapp`. Supply an identity and it is recorded as you.
+
+**This identity is declared, not verified.** It is a label the caller chooses,
+recorded for attribution and record-keeping: so the audit trail says which
+harness wrote a document, so usage analytics can tell your bot's reads from the
+web app's, and so an agent-authored ingest is queued for review (while the
+review workflow is on). It is not a
+credential, it grants nothing, and nothing checks it. See
+[Attribution is not authentication](#attribution-is-not-authentication) below
+before relying on it for anything else.
+
+Three optional fields:
+
+| Field | Header | Recorded as | Default |
+|---|---|---|---|
+| author | `X-Cerefox-Author` | `cerefox_audit_log.author` on writes | `web-ui` |
+| requestor | `X-Cerefox-Requestor` | `cerefox_usage_log.requestor` on reads and writes | `web-ui` |
+| author type | `X-Cerefox-Author-Type` | `cerefox_audit_log.author_type` | `user` |
+
+`author_type` must be `user` or `agent`. Anything else is a `400` naming the
+two valid values.
+
+**Headers work on every method.** A `GET` cannot carry a body, and reads are
+half the point of attribution, so headers are the primary mechanism. Endpoints
+that take a JSON body (or a multipart form) also accept the same three as
+fields in it, for clients that find that more natural. If both appear, the
+header wins.
+
+```bash
+# A write, attributed
+curl -X POST http://127.0.0.1:8000/api/v1/ingest \
+  -H 'Content-Type: application/json' \
+  -H 'X-Cerefox-Author: my-bot' \
+  -H 'X-Cerefox-Author-Type: agent' \
+  -d '{"title": "Meeting notes", "content": "# Notes\n\n..."}'
+
+# A read, attributed
+curl http://127.0.0.1:8000/api/v1/documents/<uuid> \
+  -H 'X-Cerefox-Requestor: my-bot'
+```
+
+`author` and `requestor` stand in for each other. They name one actor; the
+split is historical (the MCP tools used `requestor` on reads and `author` on
+writes until v1.13.1, which collapsed them to `author` with `requestor` kept as
+an alias). Supplying either identifies you for both, so a client that only sets
+`X-Cerefox-Author` still has its reads attributed correctly.
+
+### Two consequences worth knowing before you use it
+
+**`author_type: agent` marks your documents as pending review.** An
+agent-authored ingest is recorded `pending_review` rather than `approved`,
+exactly as it is over MCP. That equivalence is the point of the feature: the
+same actor is recorded, and treated, identically whichever transport it used.
+If you want your writes approved on arrival, send `user` (or send nothing).
+
+Whether anyone *sees* that status is the store's `review_workflow_enabled`
+flag. With the workflow **off** (the default on a fresh v1.13.0+ install) the
+API does not expose review status at all: no row carries a `review_status`
+key, `GET /search?review_status=…` is a `400`, and
+`POST /documents/{id}/review-status` is a `404`. The status is still recorded
+(v1.13.1), so turning the workflow on later shows exactly what the store would
+have had all along. Read the flag from `GET /config/review_workflow_enabled`
+if your client needs to know. Attribution is unaffected either way. See
+[configuration.md → Review Workflow](configuration.md#review-workflow).
+
+**Identifying yourself changes the access path.** There is deliberately no way
+to ask for a particular `access_path`: the server derives it. Supply any
+identity field and the operation is logged as `api`; supply none and it is
+logged as `webapp`. The reasoning is that `access_path` is the one field in the
+usage log the server still sets itself, so accepting it from the caller would
+let a client misreport which transport it used. In the Analytics dashboard,
+`api` operations count toward the agent total; `webapp` does not.
+
+### Attribution is not authentication
+
+`author`, `requestor` and `author_type` are **declared by the caller and taken
+at face value**. Nothing verifies them, here or anywhere else in Cerefox: MCP
+takes `author` as a client-declared string, and the Edge Functions accept
+whatever `requestor` arrives. Any client can send any name, including a name
+another client uses.
+
+They exist for **attribution and record-keeping**: a legible audit trail, usage
+analytics that can tell callers apart, and the review-queue behaviour of
+`author_type: agent` while the review workflow is on. They are not a security measure, they do not establish an
+authenticated identity, and no access decision is made on them. Treat a name in
+the audit log as "what the caller said", never as "who the caller was".
+
+The API key added in v1.12.0 (#229) does not change this. It proves a caller is
+*allowed to talk to this server*; it does not prove the caller is *who it says
+it is*. A remote client holding the key can still send any `author` it likes.
+Authentication and attribution stay separate on purpose — conflating them would
+produce a design that does neither well.
+
+`require_requestor_identity` and `requestor_identity_format` do **not** apply to
+this surface. They are enforced by the Edge Functions only. See
+[`configuration.md`](configuration.md).
+
+---
+
+## Endpoints
+
+**A machine-readable OpenAPI 3.1 document lives at
+[`docs/api/openapi.json`](../api/openapi.json).** If you are generating a client
+or a tool set, read that rather than this table: it carries the request and
+response *shapes*, the path parameters and the error model, which a prose table
+cannot. Point any OpenAPI tool at it.
+
+It is **generated** (`bun scripts/gen_openapi.ts`) from three sources that are
+each kept honest on their own: the route registrations, the summaries in the
+table below, and the zod schemas in `_shared/schemas/`. Those schemas are
+asserted against a running server by `api-schema-truth.test.ts`, so the spec
+describes what the API returns rather than what someone believed it returned.
+
+Coverage is stated in the document itself under `x-cerefox-coverage`. Endpoints
+with no modelled shape are listed there and get no response schema rather than
+an invented one — a spec that guesses is worse than one that admits a gap.
+
+Base URL is wherever `cerefox web` is listening (`http://127.0.0.1:8000` by
+default; Cerefox Local picks its own port and `cerefox-local status` prints it).
+
+| Method + path | Purpose |
+|---|---|
+| `GET /version` | Server version and environment label. |
+| `GET /schema-version` | Deployed schema version. |
+| `GET /openapi.json` | This API's OpenAPI 3.1 description, served by the version you are talking to. Every `/api/v1` response points at it with `Link: </api/v1/openapi.json>; rel="service-desc"` (RFC 8631). |
+| `GET /search?q=…` | Hybrid search (FTS + semantic). `&review_status=approved\|pending_review` filters server-side; `400` while the review workflow is off. |
+| `GET /dashboard`, `GET /dashboard/recent-docs` | Dashboard aggregates. |
+| `GET /documents/{id}` | Full document, with metadata, projects and versions. |
+| `GET /documents/{id}/chunks` | The document's chunks. |
+| `GET /documents/{id}/versions` | Version history. |
+| `POST /documents/{id}/versions/{version_id}/archive` | Archive or unarchive one stored version (body `{ "archived": true\|false }`). Archiving strips a version's search artifacts; its content is kept as the safety copy. |
+| `GET /documents/{id}/download` | Raw markdown. |
+| `GET /documents/trash` | Soft-deleted documents, newest first, `limit` ≤ 500. The exact total is in the `X-Total-Count` response header (v1.14.1). |
+| `POST /documents/metadata-search` | Query by metadata / project / time, no text query. |
+| `GET /metadata-keys` | Metadata keys with counts and example values. |
+| `GET /resolve-link`, `GET /check-filename` | Link and title resolution helpers. |
+| `POST /ingest` | Create or update from JSON (`title`, `content`, …). Unknown fields are a `400`. |
+| `POST /ingest/file` | Create from a multipart file upload. |
+| `POST /documents/{id}/upload` | Replace a document's content from a file. |
+| `POST /documents/{id}/edit` | Update title, content, metadata, projects. Unknown fields are a `400`. |
+| `DELETE /documents/{id}` | Soft-delete (to trash). |
+| `POST /documents/{id}/restore` | Restore from trash. |
+| `DELETE /documents/{id}/purge` | Permanent delete. Irreversible. |
+| `POST /documents/{id}/review-status` | Approve or re-queue for review. `404` while the review workflow is off. |
+| `GET /projects`, `POST /projects`, `PUT /projects/{id}`, `DELETE /projects/{id}` | Project CRUD. |
+| `GET /projects/{id}/documents` | A project's documents. |
+| `GET /config`, `GET /config/{key}`, `PUT /config/{key}` | Runtime config. |
+| `GET /audit-log` | Audit trail, filterable. |
+| `GET /preferences`, `PUT /preferences` | Web UI preferences (currently the colour scheme), stored in a file in the user-state dir, not the database. Registers even when Supabase is not configured. |
+| `GET /docs`, `GET /docs/{path}` | The guides bundled with this server: a listing, and one document's markdown. Serves the shipped copies, so it answers the same content regardless of what is in the knowledge base. |
+| `GET /usage-log`, `GET /usage-log/summary`, `GET /usage-log/export.csv` | Usage queries. |
+
+Paths are shown without the `/api/v1` prefix for width; every one carries it.
+The writes and the two reads that log usage (`/search`, `/documents/{id}`)
+honour the identity headers. `/version` needs nothing.
+
+### Every mutation needs a concurrency token
+
+Cerefox uses optimistic locking, and this API is no exception. Read the
+document first: `GET /documents/{id}` returns its `content_hash`. Then send
+that hash back with the write. A stale hash is a `409` carrying `current_hash`,
+so you can re-read without a second round trip. Sending no hash (and no
+`last_write_wins`) is a `400` with `CEREFOX_TOKEN_REQUIRED`.
+
+| Write | Where the hash goes | `last_write_wins` |
+|---|---|---|
+| `POST /ingest` with `document_id`, or with `update_if_exists` matching a document | `expected_content_hash` in the body | Accepted |
+| `POST /documents/{id}/edit` | `expected_content_hash` in the body. Checked on every save, including a metadata-only one | Accepted |
+| `POST /documents/{id}/upload`, `POST /ingest/file` with `update_existing` | `expected_content_hash` form field | `last_write_wins=true` form field |
+| `DELETE /documents/{id}` | `X-Cerefox-Expected-Content-Hash` header, or `expected_content_hash` query parameter | Not accepted |
+| `POST /documents/{id}/restore`, `DELETE /documents/{id}/purge` | No hash | Not applicable |
+
+`last_write_wins` skips the check. Use it only when an external source of truth
+(a file you are syncing, say) makes a conflict meaningless; there is no implicit
+default. Creating a new document needs no hash.
+
+**Check the status code, not just the body.** Every refusal is a real HTTP
+status: `400` for a malformed or incomplete request (including a body field the
+route does not know, which is refused rather than ignored), `404` for a missing
+document or version, `409` for a concurrency conflict or a trashed document,
+`422` for content that links to document ids that do not exist, and `503` when
+the embedder is unavailable. The reason is in the body's `detail` (and, on the
+ingest and edit routes, also in `error`, which they have always used). Until
+v1.15.2 a delete with a stale hash answered `500`, and `POST /ingest` answered
+`200` with `success: false` for a missing title or empty content; both are
+fixed.
+
+**Delete follows a read too.** A caller that identifies itself must send the
+hash on `DELETE /documents/{id}`, exactly as `cerefox_delete_document` requires
+it over MCP. A caller that sends no identity is the bundled web UI, which
+confirms with the human in a dialog instead, and it keeps working unchanged.
+
+The full contract of every endpoint (parameters, headers, bodies, error codes)
+is in [`docs/api/openapi.json`](../api/openapi.json), generated from the route
+handlers themselves. A running server serves its own copy at
+`GET /api/v1/openapi.json` (since v1.15.3), so a client always gets the
+contract of the version it is actually talking to. It does not need to know
+that path: every `/api/v1` response carries
+`Link: </api/v1/openapi.json>; rel="service-desc"`, the standard pointer to a
+machine-readable API description (RFC 8631).
+
+Purge (`DELETE /documents/{id}/purge`) is irreversible and takes no token. It is
+reachable by anything that can reach the port, which is another reason the
+warning at the top of this guide is not boilerplate.
+
+The response is `{ "success": true, "purged": <bool> }` (v1.14.0): `purged` is
+`false` when the document was no longer in the trash when the call landed (a
+restore raced it) and is still live; the RPC is a silent no-op in that case.
+
+**There is no bulk purge.** The web UI's "Empty trash" (v1.14.0) is a loop in
+the browser over this per-document endpoint, one call and one audit entry per
+document, with a confirmation and a Stop button in front of it. It purges only
+the documents listed when the count was confirmed; anything trashed afterwards
+stays in the trash. A single
+request that emptied the trash would be a footgun on a surface anything with
+the key can reach, so it was not added and should not be.
+
+---
+
+## Choosing between this and MCP
+
+Both surfaces run the same code underneath and write the same rows. The
+practical differences:
+
+| | `/api/v1` | MCP |
+|---|---|---|
+| Transport | Plain HTTP | stdio or Streamable HTTP |
+| Client needs | An HTTP client | An MCP client |
+| Identity | Optional, defaults to `web-ui` | Per-call `author` |
+| Section-level edits | None: `POST /documents/{id}/edit` replaces the whole content | `cerefox_insert`, `cerefox_edit` |
+| Guidance for agents | This guide | `cerefox_get_help()`, in-band |
+| Authentication | None on loopback; API key otherwise | None locally; token or OAuth remotely |
+
+If an agent is doing the calling, MCP is usually better: the tools carry their
+own documentation, and `cerefox_get_help()` keeps the conventions in front of
+the model. Reach for the API when the caller is a program you are writing
+yourself, or when you want one transport for a harness that already speaks
+HTTP.

package/docs/guides/linking.md

@@ -0,0 +1,118 @@
+# Document links: capabilities and guardrails
+
+Cerefox documents can link to each other with ordinary markdown links. This
+guide covers how linking works, and the deterministic guardrails (v1.7.0+)
+that protect links from the most common way they break: an AI agent
+corrupting a long document id while rewriting content.
+
+## How linking works
+
+Write a normal markdown link whose target is the document's UUID:
+
+```markdown
+See the [Opportunity Index](c937b70f-77af-43d3-b9bc-9f31e0d2041d) for details.
+```
+
+The Cerefox web UI resolves these at click time — the stored markdown is
+untouched, and the link keeps working if the target is renamed. Every search
+result and every write response includes the document id, so an id is always
+one lookup away.
+
+Four link forms are accepted; one is recommended:
+
+| Form | Example target | Use |
+|---|---|---|
+| **Document UUID** | `c937b70f-…` | **Always, for cross-references.** Stable, unambiguous, encoding-safe, validated (below). |
+| Repo-relative path | `docs/guides/quickstart.md` | Repo-ingested files only (their markdown naturally uses paths). |
+| Basename | `quickstart.md` | Best-effort fallback for the same. |
+| `[[Wikilink]]` | `[[Future Doc Name]]` | A deliberate **dangling** reference — "this document should exist someday." Never validated. |
+
+Agent-specific writing rules live in `AGENT_GUIDE.md` → "Writing linkable
+content".
+
+## Why links need a guardrail: LLMs and long ids
+
+This is worth understanding, because it explains the design and it is not a
+bug in any particular AI model — it is structural.
+
+A language model editing a document does not copy text the way a program
+does. It **regenerates** the text token by token from an internal
+representation. Normal prose survives this well because language is
+redundant — if a word drifts slightly, the surrounding context constrains it
+back. A UUID has **zero redundancy**: 32 hex characters, each independent,
+carrying no meaning that could catch a mistake. `508e4c21` regenerated as
+`502041d8` looks equally plausible to the model, and to anyone reading past
+it.
+
+Now compound that. A document with a table of 25–30 links makes an agent
+regenerate 25–30 UUIDs *every time it rewrites that table*. Even an
+excellent per-character accuracy, multiplied across hundreds of characters
+of pure entropy, several times per session, makes occasional corruption
+**expected, not unlucky** — and observed repeatedly in real use. Each
+corrupted id silently becomes a
+dead link that read-back review may or may not catch.
+
+Two defenses exist, and they compose:
+
+1. **Reduce how much gets regenerated**: the partial-edit tools
+   (`cerefox_insert`, `cerefox_edit`) change only the targeted section, so
+   untouched links are never retyped and literally cannot corrupt. Give
+   link-heavy tables their own heading so appends never rewrite existing
+   rows.
+2. **Catch what still gets retyped**: the write-time validation below.
+
+## The write-time guardrail (v1.7.0+)
+
+Every write — create, full update, or partial edit, on every surface (MCP,
+CLI, web, GPT Actions) — validates the `[Text](uuid)` links in the content
+against the store, inside the same database transaction as the write:
+
+- **A link to a document id that does not exist rejects the whole write**,
+  and the error lists every unresolvable id. Nothing is stored; the
+  corruption never lands. The check is deterministic and costs ~1–2ms.
+- **The agent can self-correct in the same turn**: the error says to re-read
+  the source each link was copied from, fix the id(s), and resend — and
+  explicitly not to retry unchanged.
+- **Examples escape via code formatting.** A link inside backticks or a
+  fenced code block is an example, not a link, and is not validated — which
+  is also just correct markdown, since a bare link-syntax example would
+  render as a real link. There is deliberately **no bypass flag**.
+- **A trashed target still resolves** (the id denotes a document; restoring
+  decides its fate). **`[[Wikilinks]]` are never validated** — that is the
+  sanctioned form for a forward reference.
+
+### Updates validate only what the write introduces
+
+On an update, only **newly-introduced** link ids are validated. A dead link
+the document *already carried* — typically because its target was purged
+after the link was written — does not block an unrelated edit or a file
+re-sync. Without this, one purge could make every document that ever linked
+the purged target permanently unwritable through automated paths.
+
+The corollary: a link that was mangled *before* v1.7.0, or whose target was
+purged later, sits undetected in the content. That is what the sweep is for.
+
+## The dead-link sweep (v1.7.1+)
+
+```bash
+cerefox document dead-links          # human-readable report
+cerefox document dead-links --json   # machine-readable
+```
+
+One server-side pass over the whole knowledge base, reporting every
+`[Text](uuid)` link whose target no longer exists — with the linking
+document and an occurrence count. Same scanning rules as the write guard
+(the two share one implementation, so they cannot disagree). Trashed
+*linker* documents are excluded until restored. Run it after a purge, or
+periodically; it is a full content scan, so it is deliberately on demand
+rather than part of `cerefox doctor`.
+
+Fix each hit by editing the linking document: correct the id, remove the
+link, or backtick it as an example.
+
+## Design record
+
+The full rationale — including the rejected bypass-flag design, why fence
+handling splits on lines instead of pairing with a regex, and the
+probability analysis of a mangled id colliding with a real document — is in
+[`docs/specs/link-integrity-design.md`](../specs/link-integrity-design.md).

package/docs/guides/ops-scripts.md

@@ -29,6 +29,16 @@ The canonical scripts are **TypeScript**, run with [Bun](https://bun.sh) (instal
 | `backup_create.ts` | `bun scripts/backup_create.ts` |
 | `backup_restore.ts` | `bun scripts/backup_restore.ts` |
 | `reindex_all.ts` | `bun scripts/reindex_all.ts` |
+| `cerefox_export.ts` | `bun scripts/cerefox_export.ts` |
+| `gen_openapi.ts` | `bun scripts/gen_openapi.ts` |
+| `cut_release.ts` | `bun scripts/cut_release.ts` (see [RELEASING.md](../../RELEASING.md)) |
+
+The remaining files under `scripts/` are not run by hand: `bundle_help.ts`,
+`bundle_package_docs.ts` and `bundle_server_assets.ts` are `prepublishOnly` steps,
+and `check_ef_parity.ts` and `check_help_bundle.ts` are CI checks.
+`_shared/__tests__/ops-scripts-documented.test.ts` holds that split, so a new
+script has to be either documented here or deliberately classified — it cannot be
+silently forgotten.
 
 Use the `.ts` scripts. The former `.py` equivalents were **removed at v1.0.0** — update any
 cron jobs / CI / make targets that still invoke a `python scripts/*.py` path.
@@ -279,6 +289,43 @@ Two naming details worth knowing:
   numeric suffix and neither can overwrite the other. A real store contains such
   a pair.
 
+## gen_openapi.ts — Regenerate the OpenAPI document
+
+`docs/api/openapi.json` describes `/api/v1` in OpenAPI 3.1, for callers that embed
+Cerefox instead of using MCP or the CLI. Point any OpenAPI tool at it to generate
+a client or a tool set.
+
+```bash
+bun scripts/gen_openapi.ts             # write docs/api/openapi.json
+bun scripts/gen_openapi.ts --check     # fail if the committed file is stale (CI)
+bun scripts/gen_openapi.ts --stdout    # print, write nothing
+```
+
+**Run it after changing a route, a summary in [`api.md`](api.md), or a zod schema
+in `_shared/schemas/`.** You will not forget silently:
+`_shared/__tests__/api-openapi-spec.test.ts` fails when the committed artifact is
+not what the generator produces.
+
+It is assembled from three sources rather than written, so no fact about the API
+is authored twice:
+
+| Part of the spec | Comes from |
+|---|---|
+| paths and methods | the route registrations in `packages/memory/src/web/routes/` |
+| summaries | the endpoint table in `docs/guides/api.md` |
+| request/response shapes | the zod schemas in `_shared/schemas/` |
+
+Two things worth knowing about what it does *not* claim:
+
+- **Coverage is stated, not implied.** `x-cerefox-coverage` in the document lists
+  how many routes carry a response schema, which routes legitimately return
+  something other than JSON, and which are unmodelled. An unmodelled route gets
+  **no** schema rather than a guessed one.
+- **`info.version` is the API's, not the release's.** It moves when `/api/v1`
+  changes. Binding it to the package version made the committed artifact stale at
+  every release cut — which broke the v1.15.1 npm publish — and churned every
+  generated client on patch releases that did not touch the API.
+
 ## sync_docs.ts — Sync project documentation into Cerefox
 
 Ingests `README.md`, `AGENT_GUIDE.md`, `AGENT_QUICK_REFERENCE.md`, and every Markdown file under `docs/` into your Cerefox knowledge base, updating existing documents in-place. Run this any time after editing documentation so AI agents always have access to the current state of the project.