@pyxmate/memory 0.6.3 → 0.9.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pyxmate/memory",
-  "version": "0.6.3",
+  "version": "0.9.0",
   "type": "module",
   "description": "SDK for pyx-memory — Memory as a Service for AI agents",
   "license": "MIT",

package/skills/pyx-memory/SKILL.md

@@ -91,7 +91,9 @@ curl -s -X POST {{ENDPOINT}}/api/memory/ingest/file \
   -F "description=What this file contains or shows"
 ```
 
-Supported formats: txt, md, csv, pdf, docx, json, jsonl, html, png, jpg, jpeg, webp, gif, bmp, tiff, svg (50MB limit).
+Supported formats: txt, md, csv, tsv, log, pdf, docx, xlsx, pptx, json, jsonl, html, htm, png, jpg, jpeg, webp, gif, bmp, tiff, svg (100MB limit).
+
+Ingestion streams from disk — parsers read row-by-row (csv/xlsx), line-by-line (txt/jsonl), page-by-page (pdf), or slide-by-slide (pptx) so peak memory is bounded regardless of file size. The only exception is `.docx`, which is hard-capped at 10MB because the underlying library (mammoth) loads the whole document into memory; above 10MB the server returns an error asking you to pre-extract the text upstream and re-upload as `.txt` or `.md`.
 
 **Images require a `description`** — this is how the content gets embedded and becomes searchable. Without it, the image is stored but not findable. Use your vision capabilities to generate the description when the user doesn't provide one.
 

package/skills/pyx-memory/reference/http-api.md

@@ -25,7 +25,7 @@ const client = new MemoryClient('http://localhost:7822', process.env.MEMORY_API_
 | GET | `/health` | Public health check (status only — no internals exposed) |
 | GET | `/admin/health` | Admin health check (version, uptime, embedding provider, memory stats) |
 | POST | `/api/memory/ingest` | Store a memory (JSON: `{ content, type, metadata, agentId?, sessionId?, targets?, entities?, relationships?, importance?, source?, eventTime?, id?, parentId?, ingestTime? }`) |
-| POST | `/api/memory/ingest/file` | Upload file (multipart, 50MB limit). For PDFs with images, returns an `enrichment` block with HMAC token and image metadata for two-phase enrichment. |
+| POST | `/api/memory/ingest/file` | Upload file (multipart, 100MB limit). For PDFs with images, returns an `enrichment` block with HMAC token and image metadata for two-phase enrichment. |
 | GET | `/api/memory/files/download/:filename` | Download uploaded file binary by filename. Returns the original file with proper Content-Type. Images served inline, documents as attachment. |
 | GET | `/api/memory/files/:fileId/images/:imageId?token=...` | Fetch an extracted PDF image (binary). Requires HMAC token from ingest response. |
 | POST | `/api/memory/files/:fileId/enrich` | Submit image descriptions + entities after LLM vision processing. Requires `X-Enrichment-Token` header. |
@@ -66,22 +66,57 @@ const client = new MemoryClient('http://localhost:7822', process.env.MEMORY_API_
 
 | Field | Type | Required | Description |
 |-------|------|----------|-------------|
-| `file` | File | Yes | The file to ingest (max 50MB) |
+| `file` | File | Yes | The file to ingest (max 100MB) |
 | `description` | string | No | Agent-provided description (e.g., from LLM vision). Used instead of parser output for images. |
 
-**Supported formats**: `.txt`, `.md`, `.csv`, `.pdf`, `.docx`, `.json`, `.jsonl`, `.html`, `.htm`, `.log`, `.tsv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.bmp`, `.tiff`, `.svg`
+**Supported formats**: `.txt`, `.md`, `.csv`, `.tsv`, `.log`, `.pdf`, `.docx`, `.xlsx`, `.pptx`, `.json`, `.jsonl`, `.html`, `.htm`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.bmp`, `.tiff`, `.svg`
+
+**Memory behavior**: All text parsers (csv, tsv, txt, log, pdf, json, jsonl, html, xlsx, pptx) stream from disk during ingestion — peak server memory is bounded to a few MB regardless of file size, up to the 100MB file limit. `.docx` is the exception: mammoth (the parser) has no streaming API, so `.docx` is hard-capped at 10MB on the server. Above 10MB the server returns a `MemoryError` asking you to pre-extract the text upstream and re-upload as `.txt` or `.md`.
 
 **What happens on upload**:
 1. Original file is saved to `{DATA_DIR}/files/{filename}` (persistent across restarts)
 2. Text is extracted (documents) or description is used (images)
 3. Content is chunked and stored in SQLite + vector for semantic search
 4. Source-aware dedup: re-uploading the same filename replaces the previous version
-5. **PDFs with images**: Images are extracted via pdfjs-dist, saved to a temp directory, and an `enrichment` block is returned with HMAC-signed tokens for two-phase enrichment
+5. **PDFs with images**: Images are extracted via poppler (`pdfimages` for embedded objects, `pdftoppm` for page renders on scanned PDFs), saved to a temp directory, and an `enrichment` block is returned with HMAC-signed tokens for two-phase enrichment. `pdf-parse` is used only as a dev fallback for small files (<5MB) when poppler isn't installed.
 
 **Image ingestion**: Images cannot have text extracted. Pass a `description` field with a natural-language description (e.g., from an LLM with vision). Without a description, images get a minimal placeholder (`[Image] filename (size KB)`).
 
 **PDF enrichment**: When a PDF contains images (≥50x50px), the response includes an `enrichment` block. The SDK's `ingestFile()` with `EnrichmentCallbacks` handles the full flow automatically — see [sdk-guide.md](sdk-guide.md#two-phase-pdf-enrichment).
 
+### Streaming Progress (NDJSON)
+
+For real-time ingestion progress, request NDJSON streaming by setting the `Accept` header or a query parameter:
+
+```bash
+# Via Accept header
+curl -X POST {{ENDPOINT}}/api/memory/ingest/file \
+  -H "Authorization: Bearer {{API_KEY}}" \
+  -H "Accept: application/x-ndjson" \
+  -F "file=@large-report.pdf"
+
+# Via query parameter
+curl -X POST "{{ENDPOINT}}/api/memory/ingest/file?stream=true" \
+  -H "Authorization: Bearer {{API_KEY}}" \
+  -F "file=@large-report.pdf"
+```
+
+The response is streamed as newline-delimited JSON (`Content-Type: application/x-ndjson`). Each line is a JSON object:
+
+```
+{"type":"progress","stage":"parsing","filename":"large-report.pdf","message":"Extracting text..."}
+{"type":"progress","stage":"storing","filename":"large-report.pdf","chunksStored":10,"totalCharacters":4800,"message":"Storing chunk 10..."}
+{"type":"progress","stage":"storing","filename":"large-report.pdf","chunksStored":20,"totalCharacters":9600,"message":"Storing chunk 20..."}
+{"type":"progress","stage":"complete","filename":"large-report.pdf","chunksStored":24,"totalCharacters":11520,"message":"Ingestion complete"}
+{"type":"result","filename":"large-report.pdf","fileType":".pdf","chunks":24,"entryIds":[...],"totalCharacters":11520}
+```
+
+**Progress stages**: `parsing` (text extraction), `storing` (chunk storage — emitted every batch), `enrichment` (PDF image extraction), `complete` (done).
+
+The last line always has `"type":"result"` and contains the full `IngestionResult` (same shape as the non-streaming JSON response). On error, the last line has `"type":"error"` with an `error` message string.
+
+Without the `Accept: application/x-ndjson` header (or `?stream=true`), the endpoint behaves exactly as before — returning a single JSON response after ingestion completes.
+
 ### Example: ingest a document
 
 ```bash
@@ -249,7 +284,7 @@ curl -s -X POST {{ENDPOINT}}/api/memory/ingest \
 **When to extract**: Always extract when content mentions specific people, tools, technologies, organizations, locations, or events by name. Skip only for abstract observations with no named subjects (e.g., "prefer tabs over spaces").
 
 **Entity fields**: `name` (required), `type` (required), `metadata` (optional properties object)
-**Relationship fields**: `source` (source entity name), `target` (target entity name), `type` (required)
+**Relationship fields**: `source` (source entity name), `target` (target entity name), `type` (required). Legacy `{from, to}` aliases are also accepted by the pyx-cloud hosted wrapper at `memory.api.pyxmate.com` for backward compatibility, but `{source, target}` is canonical.
 
 ---