@mindstudio-ai/agent 0.1.138 → 0.1.139

package/llms.txt

@@ -1811,15 +1811,26 @@ const chunks = await Policies.chunks(docs[0].id);       // why a document does o
 // "N passages mention these words", never "N relevant results": no such total exists,
 // and search's hits are not this set (semantic hits need not contain the words).
 const { chunks: mentions } = await Policies.count({ contains: 'client secret rotation' });
+
+// what the matching passages are about: your own ids, asserted by add(), a mapper or a manifest
+const { entities } = await Policies.entities({ filter: { entities: ['vendor-acme'] } });  // [{ id, chunks }], most first
+
+// the source's dictionary: your ids and their names, so passages are tagged wherever the text names them
+await Policies.pushDictionary(vendors.map((v) => ({ id: v.id, names: [v.name, ...v.aliases] })));
+const { mentions } = await Policies.link('Renewal terms agreed with ACME Corp.');  // what it finds: [{ ids, text }]
 ```
 
-`SearchHit`: `{ score, text, citation: { documentId, filename, pageNumber, chunkIndex, headingPath, boundingBox?, url } }`. `citation.url` is a stable on-domain link to the source document for a signed-in user — drop it in an `<a href>`. A document is a cousin of a file and shares its share verbs: for a public app or another origin, `search(q, { shareCitations: 3600 })` returns every `citation.url` as an absolute signed link that needs no session, and `Policies.shareUrl(hit.citation, { expiresIn })` signs one document on demand (default 24h). `(documentId, chunkIndex)` is a chunk's stable identity — key an eval set on that, not on `text`.
+`SearchHit`: `{ score, text, entities, citation: { documentId, filename, pageNumber, chunkIndex, headingPath, boundingBox?, url } }`. `citation.url` is a stable on-domain link to the source document for a signed-in user — drop it in an `<a href>`. A document is a cousin of a file and shares its share verbs: for a public app or another origin, `search(q, { shareCitations: 3600 })` returns every `citation.url` as an absolute signed link that needs no session, and `Policies.shareUrl(hit.citation, { expiresIn })` signs one document on demand (default 24h). `(documentId, chunkIndex)` is a chunk's stable identity — key an eval set on that, not on `text`.
 
 Also on each hit: `retrievalRank`/`retrievalScore` (position before reranking — compare with the final position to see what reranking did). Three opt-in search options for debugging and display: `explain: true` adds `explain.{dense,lexical,matchedVia}` so you can see which half of hybrid found a hit; `expand: 1` adds `neighbors.{before,after}` for surrounding context; `highlight: true` adds `matches` — `{start, end}` offsets of query terms in `text`, for rendering highlights. None of them changes the results or their order.
 
 Score semantics: `score`'s scale depends on the mode — cosine similarity (`semantic`), a rank-fusion reciprocal (`hybrid`; small numbers, not similarities), keyword-overlap weight (`lexical`) — and when reranking ran, it's the reranker's 0–1 relevance instead (the retrieval value stays in `retrievalScore`). So with reranking on (the default), put quality cutoffs on `score`. `scoreThreshold` cuts on the retrieval branch BEFORE fusion/reranking — a per-embedding-model cosine floor, not a relevance cutoff — so leave it unset unless you've measured it on your corpus.
 
-`filter` fields (all AND together, and only narrow): `metadata` (per-key equals, any-of via an array, or a numeric range via `{gte?, lte?}`), `filename`, `documentIds`, `pages: {min?, max?}`, `contains` (all words, any order), `phrase` (exact adjacent sequence). Document metadata is set at add time: scalars only, ≤16 keys, keys `[a-zA-Z0-9_-]`. Ranges are numeric only — to filter by date, store dates as sortable integers (`20250315`-style YYYYMMDD, or epoch seconds) and range on those. Re-adding the same bytes with different metadata updates the tags in place — no re-processing.
+`filter` fields (all AND together, and only narrow): `metadata` (per-key equals, any-of via an array, or a numeric range via `{gte?, lte?}`), `filename`, `documentIds`, `entities` (any of these ids), `pages: {min?, max?}`, `contains` (all words, any order), `phrase` (exact adjacent sequence). Document metadata is set at add time: scalars only, ≤16 keys, keys `[a-zA-Z0-9_-]`. Ranges are numeric only — to filter by date, store dates as sortable integers (`20250315`-style YYYYMMDD, or epoch seconds) and range on those. Re-adding the same bytes with different metadata or entities updates them in place — no re-processing.
+
+A document's `entities` name what it is about as the app's own ids (an asset tag, a person's row id), whether or not its text says so: set by `add(bytes, { entities })`, a mapper's `entities` or a job manifest line's, up to 256 per document, each 1–128 printable characters with no spaces or commas, and carried by every passage. Metadata describes a document; entities say what it is about, and the app's own tables say how those things relate. `filter.entities` narrows to passages about any of the ids. `entities({ filter?, limit? })` lists which entities the matching passages are about, most first, as `{ id, chunks }`: approximate counts of passages, not documents (limit default 20, max 100). Filter on one entity to see what appears beside it.
+
+A source's DICTIONARY tags passages by what their text says: `pushDictionary(entries)` sends every `{ id, names }` the app has (a projection of its own tables: up to 250,000 entries, 16 names each, a million names in all) and replaces the dictionary; from then on every passage carries the ids whose names it mentions, beside its document's asserted ones, and `filter.entities`, hits and `entities()` see both. Matching is deterministic and model-free: case and a tag's separators don't matter (`P-101A`, `P101A`, `p 101a`), a name matches only as whole words (`P-101AB` is not `P-101A`), the longest name wins, and a name two ids claim tags both. A name needs three letters or digits (two in Chinese, Japanese or Korean). Passages indexed before a push are re-tagged in the background (a relink, nothing re-embedded; `dictionary()` shows it). Push it whole after each load of the tables it comes from: an unchanged dictionary commits nothing (`unchanged: true`). The result reports `ambiguousNames`, `mostClaimed` and names that can't match their own text (`unmatchable`). `link(text)` shows what the current dictionary finds in any text; use it to check a dictionary before relying on it. A mapper may not push one.
 
 Search is deterministic for a fixed corpus and configuration — same query, same order — so regression checks are meaningful. `search()` also returns `latencyMs`.
 
@@ -1827,13 +1838,15 @@ Search is deterministic for a fixed corpus and configuration — same query, sam
 
 On dedicated capacity `search()` throws `index_building` (HTTP 503) instead, while a bulk load runs and while the index is built after one: the index is built once at the end of a load rather than behind every write, and the message carries "N of M vectors indexed". Handle it like `index_warming` — the knowledge base is being built, never empty; say so and retry later.
 
-Other methods: `stats()`, `documents()` (the first thousand, or `documents({ ids })` to poll what `add` returned), `documentsPage({ cursor?, limit?, order? })` (one page; `order: "newest"` is what just arrived) and `allDocuments()` (walk a corpus of any size, oldest first — how an app builds its own timeline or index of a big source after ingest, in a background task), `chunks(documentId, {vectors?})`, `remove(documentId)`, `removeWhere(selector)`, `ensure(name?)` (rarely needed — `add` and `search` create-on-reference), and `DataSource.contentHash(bytes)` to check for an existing document before adding.
+Other methods: `stats()`, `documents()` (the first thousand, or `documents({ ids })` to poll what `add` returned), `documentsPage({ cursor?, limit?, order? })` (one page; `order: "newest"` is what just arrived) and `allDocuments()` (walk a corpus of any size, oldest first; to write what each document says into the app's own tables, give the source a READER instead), `chunks(documentId, {vectors?})`, `remove(documentId)`, `removeWhere(selector)`, `ensure(name?)` (rarely needed — `add` and `search` create-on-reference), and `DataSource.contentHash(bytes)` to check for an existing document before adding.
 
-`removeWhere({ metadata?, filename?, documentIds?, externalIdPrefix? })` removes every matching document (vectors, row and bytes) in pages of a thousand and returns `{ deleted, remaining }`. Chunk-level filter fields and an empty selector are refused; deleting a whole source is an owner operation (`remy-admin datasources delete`), not something app code does.
+`removeWhere({ metadata?, filename?, documentIds?, externalIdPrefix? })` removes every matching document (vectors, row and bytes) in pages of a thousand and returns `{ deleted, remaining }`. Chunk-level filter fields, `entities` and an empty selector are refused; deleting a whole source is an owner operation (`remy-admin datasources delete`), not something app code does.
 
 Keeping a corpus in sync with an S3 bucket: the owner connects the bucket once with `remy-admin datasources connect --source <slug> --bucket <b> --region <r> --prefix <p> --access-key-secret <NAME> --secret-key-secret <NAME>` (the two names are app secrets set with `remy-admin secrets set`; credentials are never written in code or pasted in chat). From then on `await Source.sync()` lists the bucket, diffs it by ETag against what was ingested, and runs a job over what changed — returning `{ job }` immediately; poll `Source.job(id)` or list `Source.jobs()`. A nightly sync is an ordinary cron interface job whose method calls `sync()`; nothing is spent when nothing changed. The job auto-approves under the connector's per-sync budget; a plan over it waits in `planned` for the owner to `remy-admin datasources jobs approve <id>`.
 
-When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, replaces? }])`, `passthrough({ metadata? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several) and throws `mapper_skipped` for an object the mapper refused. A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
+When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, entities, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, entities?, replaces? }])`, `passthrough({ metadata?, entities? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several) and throws `mapper_skipped` for an object the mapper refused. A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
+
+To write what each document says into the app's own tables (the equipment a passage names, the relations it states, the people it quotes), give the source a READER: `datasources/<slug>.reader.ts` exporting `defineReader(Source, { read, removed, cleared? })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "reader": { "path": "dist/datasources/<slug>.reader.ts", "timeoutMs"?: 60000, "concurrency"?: 16 } }]` (beside or instead of a mapper). The platform runs it over every document the source builds, however it arrived (`add()`, a sync or a job), with its own batching, retries, budgets and pauses: `read(document)` gets `{ id, sourceId, filename, contentType, externalId, metadata, entities, pageCount, chunkCount, createdAt, chunks() }`, where `chunks()` returns the passages as indexed, each with its entities; `removed(documentId)` runs when a document leaves the source; `cleared(sourceId)` once when the source is deleted. A document can be read more than once, so `read` replaces that document's rows as a set, keyed by its id, and `removed` deletes them. A throw fails that document and it is retried. The reader always runs against the LIVE release (its code and its database) and starts reading once the app publishes it; it may call models and `fetch`, and every call is spend per document. A reader may not add to, remove from or push a dictionary to its own source. The owner tests one with `remy-admin datasources reader test --source <slug> --dev`, sees its deliveries with `datasources reader status`, and reads what it hasn't with `datasources reread --source <slug> --unread`, a job priced before it runs. See the dataSources skill for the authoring loop.
 
 Supported: pdf, docx, pptx, xlsx, odt, rtf, epub, images, txt, md, json, csv, tsv, log, html.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/agent",
-  "version": "0.1.138",
+  "version": "0.1.139",
   "description": "TypeScript SDK for MindStudio direct step execution",
   "type": "module",
   "main": "./dist/index.js",