@pyxmate/memory 0.20.5 → 0.21.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,526 +0,0 @@
1
- # pyx-memory HTTP API Reference
2
-
3
- ## Authentication
4
-
5
- When the server has `API_KEY` configured, all requests (except `/health` and enrichment routes) require one of:
6
-
7
- ```
8
- Authorization: Bearer <your-api-key>
9
- X-API-Key: <your-api-key>
10
- ```
11
-
12
- Destructive operations (DELETE, forget, decay, consolidate, reindex) require `ADMIN_API_KEY` when configured (falls back to `API_KEY`).
13
-
14
- **Enrichment routes** (`/api/memory/files/*`) use HMAC token auth instead of API keys. Tokens are issued by the server during file ingestion and are short-lived (30 minutes).
15
-
16
- `MemoryClient` handles this automatically when `apiKey` is passed to the constructor:
17
- ```typescript
18
- const client = new MemoryClient('http://localhost:7822', process.env.MEMORY_API_KEY);
19
- ```
20
-
21
- ## Core (13 endpoints)
22
-
23
- | Method | Endpoint | Description |
24
- |--------|----------|-------------|
25
- | GET | `/health` | Public health check (status only — no internals exposed) |
26
- | GET | `/admin/health` | Admin health check (version, uptime, embedding provider, memory stats) |
27
- | POST | `/api/memory/ingest` | Store a memory (JSON: `{ content, type, metadata, namespaceId?, agentId?, sessionId?, targets?, entities?, relationships?, importance?, source?, eventTime?, id?, parentId?, ingestTime?, pinned? }`) |
28
- | POST | `/api/memory/ingest/file` | Upload file (multipart, 100MB limit; optional `namespaceId` field or `X-Namespace-Id` header). For PDFs with images, returns an `enrichment` block with HMAC token and image metadata for two-phase enrichment. |
29
- | GET | `/api/memory/files/download/:filename` | Download uploaded file binary by filename. Returns the original file with proper Content-Type. Images served inline, documents as attachment. |
30
- | GET | `/api/memory/files/:fileId/images/:imageId?token=...` | Fetch an extracted PDF image (binary). Requires HMAC token from ingest response. |
31
- | POST | `/api/memory/files/:fileId/enrich` | Submit image descriptions + entities after LLM vision processing. Requires `X-Enrichment-Token` header. |
32
- | GET | `/api/memory/search?query=...&strategy=...&limit=...` | Search memories — **does NOT support** filters, enableHyDE, enableRerank |
33
- | GET | `/api/memory/stats` | Memory statistics |
34
- | GET | `/api/memory/entries?page=...&limit=...` | List entries (paginated) |
35
- | GET | `/api/memory/entries/:id` | Get entry by ID |
36
- | DELETE | `/api/memory/entries/:id` | Delete entry |
37
- | DELETE | `/api/memory/sessions/:sessionId` | Clear session |
38
-
39
- ## Graph (5 endpoints)
40
-
41
- | Method | Endpoint | Description |
42
- |--------|----------|-------------|
43
- | GET | `/api/memory/graph/nodes?name=...&type=...` | Find graph nodes |
44
- | GET | `/api/memory/graph/edges` | Graph stats |
45
- | GET | `/api/memory/graph/relationships` | List relationships |
46
- | POST | `/api/memory/graph/query` | Traverse (JSON: `{ nodeId, depth? }`) |
47
- | POST | `/api/memory/graph/clear` | Clear all graph nodes and edges (admin) |
48
-
49
- ## Lifecycle (13 endpoints)
50
-
51
- | Method | Endpoint | Description |
52
- |--------|----------|-------------|
53
- | POST | `/api/memory/consolidate` | Run consolidation pipeline |
54
- | POST | `/api/memory/forget/:id` | Soft-delete (JSON: `{ reason? }`) |
55
- | POST | `/api/memory/sessions/:sid/summarize` | Summarize session |
56
- | POST | `/api/memory/decay` | Run decay pass |
57
- | POST | `/api/memory/reindex` | Rebuild FTS5 + vector indices |
58
- | DELETE | `/api/memory/source/:source` | Delete by source |
59
- | GET | `/api/memory/consolidation-log` | Audit trail |
60
- | GET | `/api/memory/query-as-of?asOf=...` | Bi-temporal point-in-time query (asOf, type, agentId, source, limit) |
61
- | GET | `/api/memory/query-by-event-time?startTime=...&endTime=...` | Bi-temporal event time range query (startTime, endTime, type, agentId, source, limit) |
62
- | GET | `/api/memory/lint` | Read-only memory-wide lint report — graph orphans, decay candidates, stale syntheses, dedup candidates (v0.18.0) |
63
- | GET | `/api/memory/log?since=...` | Chronological feed (Karpathy `log.md`). Cursor-style via `since` created_at lower bound; also `limit`, `type`, `agentId`, `source`. Returns `{ entries, count, since? }` (v0.18.1) |
64
- | POST | `/api/memory/synthesis/entity` | Synthesize a per-entity profile (Karpathy "wiki page"). JSON: `{ name }`. LLM-driven; stores a pinned `_kind:'entity-summary'` entry with `_search_visibility:'on-demand'`. Returns the entry, 201 (v0.19.0) |
65
- | GET | `/api/memory/synthesis/entity?name=...` | Fetch a previously-synthesized entity profile. Returns `{ entry }` (`null` if none) (v0.19.0) |
66
-
67
- ## ReBAC Admin (11 endpoints)
68
-
69
- Fine-grained access control inside a tenant — namespaces, principals, group membership, and authz tuples. All routes require `ADMIN_API_KEY` and an `X-Tenant-Id` header. See [`patterns/access-control.md`](../patterns/access-control.md) Pattern 15 for the full guide.
70
-
71
- | Method | Endpoint | Description |
72
- |--------|----------|-------------|
73
- | POST | `/api/admin/namespaces` | Create namespace (JSON: `{ name, parentId?, metadata?, createdBy? }`) |
74
- | GET | `/api/admin/namespaces` | List namespaces in this tenant |
75
- | DELETE | `/api/admin/namespaces/:id` | Delete namespace + tuples referencing it (RESTRICT on children) |
76
- | POST | `/api/admin/principals` | Upsert principal (JSON: `{ kind, externalId?, displayName?, metadata? }`) |
77
- | GET | `/api/admin/principals` | List principals in this tenant |
78
- | DELETE | `/api/admin/principals/:id` | Delete principal + cascade memberships |
79
- | POST | `/api/admin/principal-members` | Add member to group (JSON: `{ groupId, memberId }`) — cycle-checked |
80
- | DELETE | `/api/admin/principal-members/:groupId/:memberId` | Remove membership |
81
- | POST | `/api/admin/authz-tuples` | Write tuple (JSON: `{ subjectKind, subjectId, subjectRelation?, relation, objectKind, objectId, createdBy? }`) |
82
- | GET | `/api/admin/authz-tuples?objectId=&subjectId=` | List tuples (filterable) |
83
- | DELETE | `/api/admin/authz-tuples` | Delete tuple (JSON: same body shape as POST) |
84
-
85
- `kind`: `'user' | 'team' | 'group' | 'agent' | 'service' | 'everyone'`. `objectKind` is `'namespace'` only in v1. `relation` is freeform (built-in rewrite: `owner ⊇ editor ⊇ viewer`).
86
-
87
- Cross-tenant safety: every route verifies the referenced resource belongs to the caller's tenant; mismatches return `404` (not `403`) to avoid existence disclosure.
88
-
89
- ## Namespaced Ingest (v0.17.1)
90
-
91
- `POST /api/memory/ingest` accepts `namespaceId` either in the JSON body or the `X-Namespace-Id` header. `POST /api/memory/ingest/file` accepts the same header or a multipart `namespaceId` field. The header is canonical; sending both channels with different values returns `400 namespace_id_conflict`. Sending JSON `namespaceId: null` is invalid; omit the field to keep the legacy NULL namespace bucket.
92
-
93
- When a namespace is supplied, the server resolves it once against the caller tenant before storing. Missing or cross-tenant namespaces return `404 namespace_not_found` so callers cannot distinguish "does not exist" from "exists in another tenant".
94
-
95
- ```bash
96
- curl -X POST {{ENDPOINT}}/api/memory/ingest \
97
- -H "Authorization: Bearer {{API_KEY}}" \
98
- -H "X-Tenant-Id: tenant-acme" \
99
- -H "X-Namespace-Id: ns-engineering" \
100
- -H "Content-Type: application/json" \
101
- -d '{"content":"Q4 revenue projections","type":"long-term"}'
102
- ```
103
-
104
- ## Strict Topology Isolation (v0.17.0)
105
-
106
- | Method | Endpoint | Description |
107
- |--------|----------|-------------|
108
- | POST | `/api/admin/namespaces/:id/isolation` | Toggle isolation. Body: `{ isolation: 'shared' \| 'strict' }`. Returns `{ namespace, changed }` — `changed: false` means the no-op path (already in the requested mode); revision unchanged. |
109
-
110
- Effect of `strict`:
111
-
112
- - Graph traversal (`/api/memory/graph/query` and indirect uses inside `/api/memory/search`) drops edges whose `namespace_id` matches `AuthzPlan.forbiddenStrictNamespaceIds`.
113
- - `/api/memory/graph/nodes` and `/api/memory/graph/relationships` apply the AuthzPlan at the HTTP boundary. Node projection intersects `memoryEntryIds` with the visible set and strips provenance properties (`source`, `sourceUrl`, `sourceTitle`, `sourceUri`, `url`, `title`).
114
- - Effect propagates immediately — the toggle bumps the per-tenant AuthzPlan revision, so cached plans pick up `forbiddenStrictNamespaceIds` on the next request.
115
-
116
- ## Idempotent Admin Mutations (v0.17.0)
117
-
118
- All `/api/admin/*` mutation routes (POST / DELETE / PUT) honor the `X-Idempotency-Key` request header. A replay with the same key — within the 24h TTL — returns the previously recorded response body + status without re-executing the handler. Set the header from the BFF / sweeper:
119
-
120
- ```bash
121
- curl -X POST "$ENDPOINT/api/admin/namespaces" \
122
- -H "Authorization: Bearer $ADMIN_API_KEY" \
123
- -H "X-Tenant-Id: $TENANT" \
124
- -H "X-Idempotency-Key: $(uuidgen)" \
125
- -H "Content-Type: application/json" \
126
- -d '{"name":"engineering"}'
127
- ```
128
-
129
- GET / OPTIONS / HEAD bypass the wrapper entirely (read-only methods are safe to retry). The header is OPTIONAL — callers that don't set it get the legacy fire-and-forget behavior.
130
-
131
- ## Entry Move Admin (v0.16.1)
132
-
133
- Reassign existing entries to a target namespace — the migration path for legacy NULL-namespace data and any later reorganization. Both routes are gated by `ADMIN_API_KEY` and require `X-Tenant-Id`. See [`patterns/access-control.md`](../patterns/access-control.md) Pattern 15 § "Migrating legacy entries" for the workflow.
134
-
135
- | Method | Endpoint | Description |
136
- |--------|----------|-------------|
137
- | POST | `/api/admin/entries/:id/move` | Single move (JSON: `{ namespaceId: string \| null }`) |
138
- | POST | `/api/admin/entries/move-batch` | Batch move with cursor + dryRun (JSON: `{ filter, target, limit?, dryRun? }`) |
139
-
140
- **Single response** — `200 OK` with the standard `{ success: true, data: T }` envelope wrapping a `MoveResult`:
141
-
142
- ```json
143
- {
144
- "success": true,
145
- "data": {
146
- "entryId": "entry-abc",
147
- "success": true,
148
- "fromNamespaceId": null,
149
- "toNamespaceId": "ns-engineering"
150
- }
151
- }
152
- ```
153
-
154
- The inner `success: false` flag carries `failureReason` ∈ `{ not_found, cross_tenant_forbidden, target_namespace_not_found, sqlite_update_failed, vector_update_failed, graph_update_failed, compensation_failed }`. The route never returns 2xx with a failed inner result — instead it maps the failureReason to HTTP status as: `not_found` → 404, `cross_tenant_forbidden` / `target_namespace_not_found` → 400, store failures (`sqlite_update_failed` / `vector_update_failed` / `graph_update_failed`) → 502, `compensation_failed` → 500. The error body uses the project's standard `{ success: false, error: string }` envelope.
155
-
156
- **Batch body shape**:
157
-
158
- ```json
159
- {
160
- "filter": {
161
- "fromNamespaceId": "ns-old" | null,
162
- "entryIds": ["..."],
163
- "agentId": "...", "teamId": "...", "userId": "...", "source": "...",
164
- "limit": 100,
165
- "cursor": "<opaque>"
166
- },
167
- "target": { "namespaceId": "ns-new" | null },
168
- "dryRun": true,
169
- "limit": 100
170
- }
171
- ```
172
-
173
- `filter.fromNamespaceId` is required (footgun guard — there is no "move every entry I can see" form). `target.namespaceId: null` reverts entries to tenant-root. `dryRun: true` returns the prospective `MoveResult[]` without touching any store; `dryRun: false` (default) executes. Response carries `nextCursor` when more rows match — feed it back via `filter.cursor` for the next page.
174
-
175
- ## File Ingestion (Images + Documents)
176
-
177
- `POST /api/memory/ingest/file` accepts multipart/form-data with:
178
-
179
- | Field | Type | Required | Description |
180
- |-------|------|----------|-------------|
181
- | `file` | File | Yes | The file to ingest (max 100MB) |
182
- | `description` | string | No | Agent-provided description (e.g., from LLM vision). Used instead of parser output for images. |
183
- | `namespaceId` | string | No | ReBAC namespace to stamp on every chunk. `X-Namespace-Id` header is preferred when both are present. |
184
-
185
- **Supported formats**: `.txt`, `.md`, `.csv`, `.tsv`, `.log`, `.pdf`, `.docx`, `.xlsx`, `.pptx`, `.json`, `.jsonl`, `.html`, `.htm`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.bmp`, `.tiff`, `.svg`
186
-
187
- **Memory behavior, by format**:
188
-
189
- - **Plain-text + structured-text** (`csv`, `tsv`, `txt`, `log`, `json`, `jsonl`, `html`, `htm`, `md`) — fully streaming; peak server memory ≈ a few MB regardless of file size up to the 100 MB cap.
190
- - **`pdf`** — streaming via `pdftotext` (poppler-utils); fall-back path (`pdf-parse`) buffers the file, so install poppler-utils on the server for streaming behavior.
191
- - **`xlsx`** — `ExcelJS.stream.xlsx.WorkbookReader` yields rows as it parses, but the shared-string table is cached for the whole workbook. Peak memory grows with shared-string count; dense workbooks can exceed "a few MB" significantly.
192
- - **`pptx`** — the full ZIP is decompressed in memory (~3× file size peak for typical decks). One slide's XML is decoded at a time. Bounded by the 100 MB file cap and a 200 MB decompressed cap.
193
- - **`docx`** — hard-capped at 10 MB. mammoth has no streaming API; above 10 MB the server returns a `MemoryError` asking you to pre-extract the text upstream and re-upload as `.txt` or `.md`.
194
-
195
- For deterministic peak memory or timing (production UX), consumers should pre-extract `.pptx` and large `.xlsx` upstream and upload the text as `.txt` — see `patterns/file-uploads.md`.
196
-
197
- **What happens on upload**:
198
- 1. Original file is saved to `{DATA_DIR}/files/{filename}` (persistent across restarts)
199
- 2. Text is extracted (documents) or description is used (images)
200
- 3. Content is chunked and stored in SQLite + vector for semantic search
201
- 4. Source-aware dedup: re-uploading the same filename replaces the previous version
202
- 5. **PDFs with images**: Images are extracted via poppler (`pdfimages` for embedded objects, `pdftoppm` for page renders on scanned PDFs), saved to a temp directory, and an `enrichment` block is returned with HMAC-signed tokens for two-phase enrichment. `pdf-parse` is used only as a dev fallback for small files (<5MB) when poppler isn't installed.
203
-
204
- **Image ingestion**: Images cannot have text extracted. Pass a `description` field with a natural-language description (e.g., from an LLM with vision). Without a description, images get a minimal placeholder (`[Image] filename (size KB)`).
205
-
206
- **PDF enrichment**: When a PDF contains images (≥50x50px), the server emits an `enrichment` block on the terminal `result` event. The SDK's `ingestFileEvents()` with `EnrichmentCallbacks` handles the full flow automatically — see [sdk-guide.md](sdk-guide.md#two-phase-pdf-enrichment).
207
-
208
- ### NDJSON Wire Format (the only response shape)
209
-
210
- `POST /api/memory/ingest/file` always streams `Content-Type: application/x-ndjson`. Each line is one typed event with `schemaVersion: 1`. The terminal event is exactly one of `result` (success) or `error` (failure):
211
-
212
- ```
213
- {"schemaVersion":1,"type":"progress","stage":"parsing","filename":"large-report.pdf","message":"Extracting text..."}
214
- {"schemaVersion":1,"type":"progress","stage":"storing","filename":"large-report.pdf","chunksStored":10,"totalCharacters":4800}
215
- {"schemaVersion":1,"type":"heartbeat","stage":"storing","message":"File ingest still running"}
216
- {"schemaVersion":1,"type":"progress","stage":"enrichment","filename":"large-report.pdf"}
217
- {"schemaVersion":1,"type":"result","stage":"complete","filename":"large-report.pdf","fileType":".pdf","chunks":24,"entryIds":["…"],"totalCharacters":11520}
218
- ```
219
-
220
- **Stable stages**: `parsing` (text/image extraction), `storing` (chunk + vector writes), `enrichment` (LLM image-describe + entity-extract + `/enrich` POST), `complete` (terminal result only). Finer detail goes in optional counters (`chunksStored`, `totalCharacters`) and `message`, not in new stage names.
221
-
222
- **Heartbeat**: emitted every 20s while the pipeline is doing slow work. Sized below undici's default 300s `headersTimeout` and Envoy's default 5m stream-idle so an LLM call or graph batch never trips an upstream socket timeout.
223
-
224
- **Errors before the stream starts** (multipart parse, file size cap, validation) come back as a single-line NDJSON `error` event with the appropriate HTTP status code on the response.
225
-
226
- Pre-v0.15.0 servers returned `application/json` for this endpoint. v0.15.0 SDK consumers will see a terminal `error` event with the message `Memory server returned application/json instead of application/x-ndjson — server is older than v0.15.0` if they hit such a server; upgrade the server.
227
-
228
- ### Example: ingest a document
229
-
230
- ```bash
231
- curl -N -X POST {{ENDPOINT}}/api/memory/ingest/file \
232
- -H "Authorization: Bearer {{API_KEY}}" \
233
- -F "file=@report.pdf"
234
- ```
235
-
236
- `-N` disables curl buffering so events appear as the server emits them.
237
-
238
- ### Example: ingest an image with description
239
-
240
- ```bash
241
- curl -N -X POST {{ENDPOINT}}/api/memory/ingest/file \
242
- -H "Authorization: Bearer {{API_KEY}}" \
243
- -F "file=@screenshot.png" \
244
- -F "description=A screenshot of the dashboard showing memory usage at 85%"
245
- ```
246
-
247
- ### SDK usage
248
-
249
- ```typescript
250
- import { MemoryClient, type FileIngestResult } from '@pyxmate/memory';
251
-
252
- // Iterate the AsyncIterable. The terminal event is `result` (success) or `error` (failure).
253
- async function ingestAndCollect(memory: MemoryClient, file: File): Promise<FileIngestResult> {
254
- for await (const event of memory.ingestFileEvents(file)) {
255
- if (event.type === 'progress' || event.type === 'heartbeat') continue;
256
- if (event.type === 'error') throw new Error(event.message ?? event.error);
257
- // event.type === 'result'
258
- const { schemaVersion: _v, type: _t, stage: _s, message: _m, ...result } = event;
259
- return result;
260
- }
261
- throw new Error('memory ingest stream ended without a terminal event');
262
- }
263
-
264
- // Document (text-only)
265
- const doc = new File([buffer], 'report.pdf', { type: 'application/pdf' });
266
- await ingestAndCollect(memory, doc);
267
-
268
- // Image with description
269
- const img = new File([imgBuffer], 'photo.png', { type: 'image/png' });
270
- await ingestAndCollect(memory, img); // pass via options when needed
271
- for await (const _ of memory.ingestFileEvents(img, {
272
- description: 'A photo of the whiteboard from the architecture meeting',
273
- })) { /* drain */ }
274
-
275
- // PDF with two-phase enrichment (images described via LLM vision)
276
- for await (const event of memory.ingestFileEvents(pdfFile, {
277
- enrichment: {
278
- describeImage: async (buffer, meta) => 'A bar chart showing Q4 revenue growth of 23%',
279
- extractEntitiesV2: async ({ textWindows, imageDescriptions, filename, mimeType }) => ({
280
- entities: [{ name: 'Q4 Revenue', type: 'METRIC' }],
281
- relationships: [],
282
- }),
283
- },
284
- })) {
285
- if (event.type === 'progress') {
286
- console.log(`[${event.stage}] ${event.message ?? ''}`);
287
- }
288
- }
289
- ```
290
-
291
- ## File Downloads
292
-
293
- `GET /api/memory/files/download/:filename` serves uploaded file binaries.
294
-
295
- - **Images** (`image/*`): served inline (`Content-Disposition: inline`)
296
- - **Documents**: served as attachment (`Content-Disposition: attachment; filename="..."`)
297
- - **Caching**: `Cache-Control: public, max-age=86400, immutable`
298
- - **Auth**: standard API key auth (same as other endpoints)
299
- - **Security**: path traversal prevention, extension validation against ingestion allowlist
300
-
301
- ### Example: download a file
302
-
303
- ```bash
304
- curl -o report.pdf "{{ENDPOINT}}/api/memory/files/download/report.pdf" \
305
- -H "Authorization: Bearer {{API_KEY}}"
306
- ```
307
-
308
- ### SDK usage
309
-
310
- ```typescript
311
- // Get download URL (useful for markdown images, browser rendering)
312
- const url = memory.getFileDownloadUrl('screenshot.png');
313
- // → 'http://localhost:7822/api/memory/files/download/screenshot.png'
314
-
315
- // Download file binary
316
- const response = await memory.downloadFile('report.pdf');
317
- const buffer = await response.arrayBuffer();
318
- ```
319
-
320
- ## Two-Phase PDF Enrichment
321
-
322
- When a PDF contains extractable images, the server's terminal `result` event carries an `enrichment` block. The SDK's `ingestFileEvents()` with `EnrichmentCallbacks` runs the full three-phase flow automatically, emitting its own `enrichment` progress + heartbeat events while it works.
323
-
324
- ### Flow
325
-
326
- ```
327
- Phase 1: POST /api/memory/ingest/file → text stored immediately, enrichment block emitted on terminal `result`
328
- Result event: { schemaVersion: 1, type: 'result', stage: 'complete', ...ingestResult, enrichment: { fileId, token, expiresAt, images: [...] } }
329
-
330
- Phase 2: GET /api/memory/files/{fileId}/images/{imageId}?token=... → binary image
331
- (SDK fetches each image, calls describeImage callback)
332
-
333
- Phase 3: POST /api/memory/files/{fileId}/enrich
334
- Header: X-Enrichment-Token: {token}:{expiresAt}
335
- Body: { imageDescriptions: [...], entities?: [...], relationships?: [...] }
336
- → descriptions stored as memory entries, temp images cleaned up
337
- ```
338
-
339
- ### Image fetch endpoint
340
-
341
- `GET /api/memory/files/:fileId/images/:imageId?token=<hmac>`
342
-
343
- Returns the binary image with `Content-Type: image/png` or `image/jpeg`. The HMAC token is issued by the server during Phase 1 and expires after 30 minutes.
344
-
345
- ### Enrich endpoint
346
-
347
- `POST /api/memory/files/:fileId/enrich`
348
-
349
- | Header | Required | Description |
350
- |--------|----------|-------------|
351
- | `X-Enrichment-Token` | Yes | Format: `{hmac_token}:{expiresAt_iso}` |
352
-
353
- Body:
354
- ```json
355
- {
356
- "imageDescriptions": [
357
- { "imageId": "uuid-img-0", "description": "A chart showing..." }
358
- ],
359
- "entities": [{ "name": "Revenue", "type": "METRIC" }],
360
- "relationships": []
361
- }
362
- ```
363
-
364
- Input limits: descriptions max 10,000 chars each, entities max 200, relationships max 500.
365
-
366
- ### Security
367
-
368
- - Tokens are HMAC-SHA256 signed (server secret + fileId + expiresAt)
369
- - `fileId` must be a valid UUIDv4
370
- - `imageId` must match `^[a-zA-Z0-9_-]+$`
371
- - Image paths are sandboxed via `resolve()` + `startsWith()` to prevent path traversal
372
- - Enrichment routes bypass API key auth — they use their own HMAC verification
373
-
374
- ---
375
-
376
- ## Entity Extraction (Knowledge Graph)
377
-
378
- When storing memories that mention named subjects, include `entities` and `relationships` in the ingest request to populate the knowledge graph. This enables graph-based RAG and dashboard visualization.
379
-
380
- ### Entity types
381
-
382
- `PERSON`, `ORGANIZATION`, `CONCEPT`, `TOOL`, `LOCATION`, `EVENT`
383
-
384
- ### Relationship types
385
-
386
- `USES`, `OWNS`, `DEPENDS_ON`, `RELATED_TO`, `CREATED_BY`, `PART_OF`, `IS_A`, `WORKS_AT`, `LOCATED_IN`
387
-
388
- ### Example: store with entities
389
-
390
- ```bash
391
- curl -s -X POST {{ENDPOINT}}/api/memory/ingest \
392
- -H "Authorization: Bearer {{API_KEY}}" \
393
- -H "Content-Type: application/json" \
394
- -d '{
395
- "content":"Alice switched the frontend from JavaScript to TypeScript for type safety",
396
- "type":"long-term",
397
- "metadata":{"source":"agent","topic":"tech-stack","project":"my-project"},
398
- "entities":[
399
- {"name":"Alice","type":"PERSON"},
400
- {"name":"TypeScript","type":"TOOL"},
401
- {"name":"JavaScript","type":"TOOL"}
402
- ],
403
- "relationships":[
404
- {"source":"Alice","target":"TypeScript","type":"USES"},
405
- {"source":"TypeScript","target":"JavaScript","type":"RELATED_TO"}
406
- ]
407
- }'
408
- ```
409
-
410
- **When to extract**: Always extract when content mentions specific people, tools, technologies, organizations, locations, or events by name. Skip only for abstract observations with no named subjects (e.g., "prefer tabs over spaces").
411
-
412
- **Entity fields**: `name` (required), `type` (required), `metadata` (optional properties object)
413
- **Relationship fields**: `source` (source entity name), `target` (target entity name), `type` (required). Legacy `{from, to}` aliases are also accepted by the pyx-cloud hosted wrapper at `memory.api.pyxmate.com` for backward compatibility, but `{source, target}` is canonical.
414
-
415
- ---
416
-
417
- ## Multi-Tenant Isolation
418
-
419
- When `TENANT_MODE=multi`, the server requires `X-Tenant-Id` on all operations. Requests without it are rejected with HTTP 400.
420
-
421
- ### Tenant Headers
422
-
423
- | Header | Description |
424
- |--------|-------------|
425
- | `X-Tenant-Id` | **Required** in multi-tenant mode. Tenant ID for data isolation. |
426
- | `X-User-Id` | Optional. User ID within the tenant. |
427
- | `X-Team-Id` | Optional. Team/group ID within the tenant. |
428
-
429
- As a fallback, `tenantId` can be extracted from JWT Bearer token claims (`tenantId`, `tenant_id`, or `tid` field in the payload).
430
-
431
- ### Example: multi-tenant ingest
432
-
433
- ```bash
434
- curl -X POST {{ENDPOINT}}/api/memory/ingest \
435
- -H "Authorization: Bearer {{API_KEY}}" \
436
- -H "Content-Type: application/json" \
437
- -H "X-Tenant-Id: tenant-abc" \
438
- -H "X-User-Id: user-123" \
439
- -H "X-Team-Id: team-eng" \
440
- -d '{"content":"Tenant-scoped memory.","type":"long-term","metadata":{}}'
441
- ```
442
-
443
- Tenant scoping is enforced on all operations: ingest, search, list, get, delete, clearSession, stats.
444
-
445
- When `TENANT_MODE=single` (default), no tenant filtering applies (backward compatible). Tenant fields in the request body are still stored but not enforced.
446
-
447
- ---
448
-
449
- ## Sensitivity Classification & Encryption
450
-
451
- pyx-memory auto-classifies content sensitivity and optionally encrypts sensitive data at rest.
452
-
453
- ### Sensitivity levels
454
-
455
- | Level | Auto-detected when |
456
- |-------|--------------------|
457
- | `public` | Neither credentials nor PII detected |
458
- | `internal` | PII detected (email, phone, SSN, etc.) |
459
- | `secret` | API keys, tokens, connection strings, private keys, passwords detected |
460
-
461
- ### Sensitivity policy (`SENSITIVITY_POLICY` env var)
462
-
463
- | Policy | Behavior |
464
- |--------|----------|
465
- | `flag` (default) | Detect and classify. Store `sensitivity` field on entry. No content modification. |
466
- | `redact` | Replace detected credentials with `[REDACTED]` before storage. |
467
- | `block` | Reject ingest requests containing credentials (HTTP 400). |
468
- | `encrypt` | Entries classified as `secret` are encrypted at rest with AES-256-GCM. |
469
-
470
- ### Search access control
471
-
472
- The `maxSensitivity` search parameter (or `X-Caller-Access-Level` header) filters results by sensitivity level. Entries above the caller's access level have their content replaced with `[REDACTED: sensitive content]`.
473
-
474
- ```bash
475
- # Only see public and internal entries (secret entries are redacted)
476
- curl '{{ENDPOINT}}/api/memory/search?query=config' \
477
- -H "Authorization: Bearer {{API_KEY}}" \
478
- -H "X-Caller-Access-Level: internal"
479
- ```
480
-
481
- ---
482
-
483
- ## Confidence / Abstention
484
-
485
- Search results include optional confidence scoring via the `abstentionThreshold` parameter.
486
-
487
- ```bash
488
- # Get confidence scoring with custom threshold (0-1, default 0.3)
489
- curl '{{ENDPOINT}}/api/memory/search?query=user+preferences&strategy=hybrid&abstentionThreshold=0.5'
490
- # Response includes: { confidence: { confidence: 0.72, shouldAbstain: false, signals: { ... } } }
491
- ```
492
-
493
- When `shouldAbstain` is `true`, the retrieval confidence is below the threshold — the agent should respond with "I don't know" rather than using low-confidence results.
494
-
495
- ---
496
-
497
- ## Response Format
498
-
499
- All responses follow: `{ success: boolean, data?: T, error?: string }`
500
-
501
- ---
502
-
503
- ## Server Environment Variables
504
-
505
- | Variable | Default | Description |
506
- |----------|---------|-------------|
507
- | `MEMORY_SERVER_PORT` | `7822` | HTTP server port |
508
- | `DATA_DIR` | `./data` | Storage directory |
509
- | ~~`EMBEDDING_PROVIDER`~~ | — | **Removed** — embedding is now internal (BGE-M3 via LocalEmbeddingProvider) |
510
- | ~~`EMBEDDING_API_KEY`~~ | — | **Removed** — no external embedding provider needed |
511
- | ~~`EMBEDDING_MODEL`~~ | — | **Removed** — model is always BGE-M3 (ONNX int8 quantized) |
512
- | `EMBEDDING_DIMENSIONS` | `1024` | Dimension override for the internal LocalEmbeddingProvider (default: 1024) |
513
- | `NEO4J_URL` | — | Neo4j bolt URL (enables Neo4j graph store) |
514
- | `NEO4J_USERNAME` | `neo4j` | Neo4j username |
515
- | `NEO4J_PASSWORD` | — | Neo4j password (never logged) |
516
- | `API_KEY` | — | API key for authenticating requests. Unset = open access |
517
- | `ADMIN_API_KEY` | — | Separate admin key for destructive ops (DELETE, forget, decay, consolidate, reindex). Falls back to `API_KEY` |
518
- | `CORS_ORIGIN` | `*` | CORS allowed origin. Set to specific domain in production |
519
- | `MAX_REQUEST_BODY_MB` | `512` | Maximum request body size in MB |
520
- | `NODE_ENV` | `development` | Set to `production` to mask 5xx error details and enable HSTS |
521
- | `PII_POLICY` | `flag` | PII handling: `flag` (detect + tag), `redact` (replace with [REDACTED]), `block` (reject 400) |
522
- | `SENSITIVITY_POLICY` | `flag` | Credential sensitivity handling: `flag` (detect + classify), `redact` (replace with [REDACTED]), `block` (reject 400), `encrypt` (AES-256-GCM at rest) |
523
- | `ENCRYPTION_KEY` | — | AES-256-GCM encryption key for sensitive content at rest (32 bytes as 64 hex chars or 44 base64 chars). Required when `SENSITIVITY_POLICY=encrypt` |
524
- | `RATE_LIMIT_RPM` | `0` | Requests per minute per IP. 0 = disabled |
525
- | `TENANT_MODE` | `single` | Tenant isolation mode: `single` (no tenant filtering, backward compatible) or `multi` (require `X-Tenant-Id` header on all operations) |
526
- | `ENRICHMENT_SECRET` | (auto-generated) | HMAC secret for enrichment token signing. Auto-generated if unset (tokens won't survive restarts). Min 32 bytes for production. |
@@ -1,74 +0,0 @@
1
- # Feature Parity: Embedded vs Sidecar
2
-
3
- **Most features are available in both modes.** The HTTP API and MemoryClient forward all core `StoreInput` fields. A few advanced search parameters remain embedded-only.
4
-
5
- ## store() Field Parity
6
-
7
- | Field | Embedded `Memory.store()` | Sidecar `MemoryClient.store()` |
8
- |-------|--------------------------|-------------------------------|
9
- | content | yes | yes |
10
- | type | yes | yes |
11
- | metadata | yes | yes |
12
- | agentId | yes | yes |
13
- | sessionId | yes | yes |
14
- | targets | yes | yes |
15
- | entities | yes | yes |
16
- | relationships | yes | yes |
17
- | importance | yes | yes |
18
- | source | yes | yes |
19
- | eventTime | yes | yes |
20
- | id (custom) | yes | yes |
21
- | parentId | yes | yes |
22
- | ingestTime | yes | yes |
23
- | tenantId | yes | yes |
24
- | userId | yes | yes |
25
- | teamId | yes | yes |
26
- | sensitivity | yes | yes (auto-classified) |
27
-
28
- **All StoreInput fields are forwarded.** Full parity.
29
-
30
- ## search() Param Parity
31
-
32
- | Param | Embedded `Memory.search()` | Sidecar `MemoryClient.search()` |
33
- |-------|---------------------------|--------------------------------|
34
- | query, limit, type, agentId, strategy | yes | yes |
35
- | tenantId, userId, teamId | yes | yes (via `X-Tenant-Id`/`X-User-Id`/`X-Team-Id` headers or `defaultHeaders` — NOT forwarded from per-request params) |
36
- | maxSensitivity | yes | yes (via `X-Caller-Access-Level` header or `defaultHeaders` — NOT forwarded from per-request params) |
37
- | abstentionThreshold | yes | yes |
38
- | eventTimeRange (bi-temporal search) | yes | yes |
39
- | asOf (point-in-time search) | yes | yes |
40
- | **filters** (source, importanceMin, parentId, contentType) | yes | **NO** — not forwarded |
41
- | **enableHyDE** | yes | **NO** — not forwarded |
42
- | **enableRerank** | yes | **NO** — not forwarded |
43
-
44
- **Impact**: Sidecar consumers cannot use advanced search filters, HyDE query expansion, or reranking. Temporal search filters (eventTimeRange, asOf) ARE supported.
45
-
46
- ## Endpoint Coverage by Client
47
-
48
- | Server Endpoint | MemoryClient | DashboardClient |
49
- |----------------|-------------|-----------------|
50
- | All core (9) | yes | yes (inherited) |
51
- | Graph nodes/edges/query (3) | yes (concrete methods) | yes (inherited) |
52
- | Graph clear (admin) | yes (`graphClear()`) | yes (inherited) |
53
- | **Graph relationships** | **NO** | yes (`graphRelationships()`) |
54
- | All lifecycle (7) | yes | yes (inherited) |
55
- | **Consolidation log** | **NO** | yes (`consolidationLog()`) |
56
- | Query as-of (bi-temporal) | yes (`queryAsOf()`) | yes (inherited) |
57
- | Query by event time | yes (`queryByEventTime()`) | yes (inherited) |
58
-
59
- ## Security Features (Server-side)
60
-
61
- | Feature | Configuration |
62
- |---------|--------------|
63
- | API key auth | `API_KEY` env var (unset = open access) |
64
- | Admin key for destructive ops | `ADMIN_API_KEY` env var |
65
- | Rate limiting | `RATE_LIMIT_RPM` env var (0 = disabled) |
66
- | CORS | `CORS_ORIGIN` env var (default: `*`) |
67
- | Security headers | Always on (CSP, X-Frame-Options, nosniff) |
68
- | HSTS | Auto-enabled when `NODE_ENV=production` |
69
- | PII policy | `PII_POLICY` env var (`flag` / `redact` / `block`) |
70
- | Sensitivity policy | `SENSITIVITY_POLICY` env var (`flag` / `redact` / `block` / `encrypt`) |
71
- | Encryption at rest | `ENCRYPTION_KEY` env var (32 bytes, hex or base64) |
72
- | Multi-tenant isolation | `TENANT_MODE` env var (`single` / `multi`) |
73
- | Body size limit | `MAX_REQUEST_BODY_MB` env var (default: 512) |
74
- | Error masking | 5xx details hidden when `NODE_ENV=production` |