@mindstudio-ai/agent 0.1.139 → 0.1.141

package/llms.txt

@@ -1828,9 +1828,9 @@ Score semantics: `score`'s scale depends on the mode — cosine similarity (`sem
 
 `filter` fields (all AND together, and only narrow): `metadata` (per-key equals, any-of via an array, or a numeric range via `{gte?, lte?}`), `filename`, `documentIds`, `entities` (any of these ids), `pages: {min?, max?}`, `contains` (all words, any order), `phrase` (exact adjacent sequence). Document metadata is set at add time: scalars only, ≤16 keys, keys `[a-zA-Z0-9_-]`. Ranges are numeric only — to filter by date, store dates as sortable integers (`20250315`-style YYYYMMDD, or epoch seconds) and range on those. Re-adding the same bytes with different metadata or entities updates them in place — no re-processing.
 
-A document's `entities` name what it is about as the app's own ids (an asset tag, a person's row id), whether or not its text says so: set by `add(bytes, { entities })`, a mapper's `entities` or a job manifest line's, up to 256 per document, each 1–128 printable characters with no spaces or commas, and carried by every passage. Metadata describes a document; entities say what it is about, and the app's own tables say how those things relate. `filter.entities` narrows to passages about any of the ids. `entities({ filter?, limit? })` lists which entities the matching passages are about, most first, as `{ id, chunks }`: approximate counts of passages, not documents (limit default 20, max 100). Filter on one entity to see what appears beside it.
+A document's `entities` name what it is about as the app's own ids (an asset tag, a person's row id), whether or not its text says so: set by `add(bytes, { entities })`, a mapper's `entities` or a job manifest line's, up to 256 per document, each 1–128 printable characters with no spaces or commas, and carried by every passage. Metadata describes a document; entities say what it is about, and the app's own tables say how those things relate. `filter.entities` narrows to passages about any of the ids. `entities({ filter?, limit? })` lists which entities the matching passages are about, most first, as `{ id, chunks }`: approximate counts of passages, not documents (limit default 20, max 1,000; `filter: { documentIds: [id] }` for one document's). Filter on one entity to see what appears beside it.
 
-A source's DICTIONARY tags passages by what their text says: `pushDictionary(entries)` sends every `{ id, names }` the app has (a projection of its own tables: up to 250,000 entries, 16 names each, a million names in all) and replaces the dictionary; from then on every passage carries the ids whose names it mentions, beside its document's asserted ones, and `filter.entities`, hits and `entities()` see both. Matching is deterministic and model-free: case and a tag's separators don't matter (`P-101A`, `P101A`, `p 101a`), a name matches only as whole words (`P-101AB` is not `P-101A`), the longest name wins, and a name two ids claim tags both. A name needs three letters or digits (two in Chinese, Japanese or Korean). Passages indexed before a push are re-tagged in the background (a relink, nothing re-embedded; `dictionary()` shows it). Push it whole after each load of the tables it comes from: an unchanged dictionary commits nothing (`unchanged: true`). The result reports `ambiguousNames`, `mostClaimed` and names that can't match their own text (`unmatchable`). `link(text)` shows what the current dictionary finds in any text; use it to check a dictionary before relying on it. A mapper may not push one.
+A source's DICTIONARY tags passages by what their text says: `pushDictionary(entries)` sends every `{ id, names }` the app has (a projection of its own tables: up to 250,000 entries, 16 names each, a million names in all) and replaces the dictionary; from then on every passage carries the ids whose names it mentions, beside its document's asserted ones, and `filter.entities`, hits and `entities()` see both. Matching is deterministic and model-free: case and a tag's separators don't matter (`P-101A`, `P101A`, `p 101a`), nor a size's inch or foot mark before one (`8"-P-1057-A1A` is `8-P-1057-A1A`), a name matches only as whole words (`P-101AB` is not `P-101A`), the longest name wins, and a name two ids claim tags both. A name needs three letters or digits (two in Chinese, Japanese or Korean). Passages indexed before a push are re-tagged in the background (a relink, nothing re-embedded; `dictionary()` shows it). Push it whole after each load of the tables it comes from: an unchanged dictionary commits nothing (`unchanged: true`). The result reports `ambiguousNames`, `mostClaimed` (each `{ key, name, ids }`: the folded key and one name as pushed) and names that can't match their own text (`unmatchable`). `link(text)` shows what the current dictionary finds in any text; use it to check a dictionary before relying on it. A mapper may not push one.
 
 Search is deterministic for a fixed corpus and configuration — same query, same order — so regression checks are meaningful. `search()` also returns `latencyMs`.
 
@@ -1838,15 +1838,15 @@ Search is deterministic for a fixed corpus and configuration — same query, sam
 
 On dedicated capacity `search()` throws `index_building` (HTTP 503) instead, while a bulk load runs and while the index is built after one: the index is built once at the end of a load rather than behind every write, and the message carries "N of M vectors indexed". Handle it like `index_warming` — the knowledge base is being built, never empty; say so and retry later.
 
-Other methods: `stats()`, `documents()` (the first thousand, or `documents({ ids })` to poll what `add` returned), `documentsPage({ cursor?, limit?, order? })` (one page; `order: "newest"` is what just arrived) and `allDocuments()` (walk a corpus of any size, oldest first; to write what each document says into the app's own tables, give the source a READER instead), `chunks(documentId, {vectors?})`, `remove(documentId)`, `removeWhere(selector)`, `ensure(name?)` (rarely needed — `add` and `search` create-on-reference), and `DataSource.contentHash(bytes)` to check for an existing document before adding.
+Other methods: `stats()`, `documents()` (the first thousand, or `documents({ ids })` to poll what `add` returned), `documentsPage({ cursor?, limit?, order? })` (one page; `order: "newest"` is what just arrived) and `allDocuments()` (walk a corpus of any size, oldest first; to write what each document says into the app's own tables, give the source a READER instead), `passages({ filter?, cursor?, limit? })` and `allPassages(filter?)` (every passage matching a filter, unranked, document by document, as `{ text, entities, citation }`: every page naming an entity rather than search's best few; limit default 50, max 200), `chunks(documentId, {vectors?})`, `remove(documentId)`, `removeWhere(selector)`, `ensure(name?)` (rarely needed — `add` and `search` create-on-reference), and `DataSource.contentHash(bytes)` to check for an existing document before adding.
 
 `removeWhere({ metadata?, filename?, documentIds?, externalIdPrefix? })` removes every matching document (vectors, row and bytes) in pages of a thousand and returns `{ deleted, remaining }`. Chunk-level filter fields, `entities` and an empty selector are refused; deleting a whole source is an owner operation (`remy-admin datasources delete`), not something app code does.
 
 Keeping a corpus in sync with an S3 bucket: the owner connects the bucket once with `remy-admin datasources connect --source <slug> --bucket <b> --region <r> --prefix <p> --access-key-secret <NAME> --secret-key-secret <NAME>` (the two names are app secrets set with `remy-admin secrets set`; credentials are never written in code or pasted in chat). From then on `await Source.sync()` lists the bucket, diffs it by ETag against what was ingested, and runs a job over what changed — returning `{ job }` immediately; poll `Source.job(id)` or list `Source.jobs()`. A nightly sync is an ordinary cron interface job whose method calls `sync()`; nothing is spent when nothing changed. The job auto-approves under the connector's per-sync budget; a plan over it waits in `planned` for the owner to `remy-admin datasources jobs approve <id>`.
 
-When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, entities, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, entities?, replaces? }])`, `passthrough({ metadata?, entities? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several) and throws `mapper_skipped` for an object the mapper refused. A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
+When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, entities, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, entities?, replaces? }])`, `passthrough({ metadata?, entities? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. `documents()` takes an array or any iterable or async iterable with no cap on its length, so a generator that yields a document per row streams an export of any size; a document that fails validation is left out and named, the rest go in. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several, with `invalid` naming any that failed) and throws `mapper_skipped` for an object the mapper refused; an object that maps to more than a thousand documents registers the first thousand and returns `job`, which takes the rest (`documentCount` is the total). A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
 
-To write what each document says into the app's own tables (the equipment a passage names, the relations it states, the people it quotes), give the source a READER: `datasources/<slug>.reader.ts` exporting `defineReader(Source, { read, removed, cleared? })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "reader": { "path": "dist/datasources/<slug>.reader.ts", "timeoutMs"?: 60000, "concurrency"?: 16 } }]` (beside or instead of a mapper). The platform runs it over every document the source builds, however it arrived (`add()`, a sync or a job), with its own batching, retries, budgets and pauses: `read(document)` gets `{ id, sourceId, filename, contentType, externalId, metadata, entities, pageCount, chunkCount, createdAt, chunks() }`, where `chunks()` returns the passages as indexed, each with its entities; `removed(documentId)` runs when a document leaves the source; `cleared(sourceId)` once when the source is deleted. A document can be read more than once, so `read` replaces that document's rows as a set, keyed by its id, and `removed` deletes them. A throw fails that document and it is retried. The reader always runs against the LIVE release (its code and its database) and starts reading once the app publishes it; it may call models and `fetch`, and every call is spend per document. A reader may not add to, remove from or push a dictionary to its own source. The owner tests one with `remy-admin datasources reader test --source <slug> --dev`, sees its deliveries with `datasources reader status`, and reads what it hasn't with `datasources reread --source <slug> --unread`, a job priced before it runs. See the dataSources skill for the authoring loop.
+To write what each document says into the app's own tables (the equipment a passage names, the relations it states, the people it quotes), give the source a READER: `datasources/<slug>.reader.ts` exporting `defineReader(Source, { read, removed, cleared? })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "reader": { "path": "dist/datasources/<slug>.reader.ts", "timeoutMs"?: 60000, "concurrency"?: 16 } }]` (beside or instead of a mapper). The platform runs it over every document the source builds, however it arrived (`add()`, a sync or a job), with its own batching, retries, budgets and pauses: `read(document)` gets `{ id, sourceId, filename, contentType, externalId, metadata, entities, pageCount, chunkCount, createdAt, url, chunks(), bytes() }`, where `chunks()` returns the passages as indexed, each with its entities, and `bytes()` (or the short-lived `url`) the file the document came from, for a reader that needs the drawing itself; `removed(documentId)` runs when a document leaves the source; `cleared(sourceId)` once when the source is deleted; the optional `relinked(document)` when a new dictionary re-tagged a document already read (the same document, with its new entities; for rows derived from the tags, not a reason to extract again). A document can be read more than once, so `read` replaces that document's rows as a set, keyed by its id, and `removed` deletes them. A throw fails that document and it is retried. The reader always runs against the LIVE release (its code and its database) and starts reading once the app publishes it; it may call models and `fetch`, and every call is spend per document. A reader may not add to, remove from or push a dictionary to its own source. The owner tests one with `remy-admin datasources reader test --source <slug> --dev`, sees its deliveries with `datasources reader status`, and reads what it hasn't with `datasources reread --source <slug> --unread`, a job priced before it runs. See the dataSources skill for the authoring loop.
 
 Supported: pdf, docx, pptx, xlsx, odt, rtf, epub, images, txt, md, json, csv, tsv, log, html.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/agent",
-  "version": "0.1.139",
+  "version": "0.1.141",
   "description": "TypeScript SDK for MindStudio direct step execution",
   "type": "module",
   "main": "./dist/index.js",