@mindstudio-ai/agent 0.1.140 → 0.1.142

package/llms.txt

@@ -1844,7 +1844,7 @@ Other methods: `stats()`, `documents()` (the first thousand, or `documents({ ids
 
 Keeping a corpus in sync with an S3 bucket: the owner connects the bucket once with `remy-admin datasources connect --source <slug> --bucket <b> --region <r> --prefix <p> --access-key-secret <NAME> --secret-key-secret <NAME>` (the two names are app secrets set with `remy-admin secrets set`; credentials are never written in code or pasted in chat). From then on `await Source.sync()` lists the bucket, diffs it by ETag against what was ingested, and runs a job over what changed — returning `{ job }` immediately; poll `Source.job(id)` or list `Source.jobs()`. A nightly sync is an ordinary cron interface job whose method calls `sync()`; nothing is spent when nothing changed. The job auto-approves under the connector's per-sync budget; a plan over it waits in `planned` for the owner to `remy-admin datasources jobs approve <id>`.
 
-When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, entities, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, entities?, replaces? }])`, `passthrough({ metadata?, entities? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several) and throws `mapper_skipped` for an object the mapper refused. A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
+When the file is not the document — a JSON record that should become markdown plus metadata, a JSONL object holding many articles, a kill notice that means "remove this story" — give the source a MAPPER: `datasources/<slug>.mapper.ts` exporting `defineMapper(Source, { map })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "mapper": { "path": "dist/datasources/<slug>.mapper.ts" } }]`. `map(object)` gets `{ key, size, contentType, etag, lastModified, metadata, entities, bytes(), text(), json() }` and returns one of `documents([{ externalId, title, markdown, metadata?, entities?, replaces? }])`, `passthrough({ metadata?, entities? })` (ingest the raw object as-is), `skip(reason)`, or `deletes([externalId, ...])`; throwing is the object's error. `documents()` takes an array or any iterable or async iterable with no cap on its length, so a generator that yields a document per row streams an export of any size; a document that fails validation is left out and named, the rest go in. Everything entering a mapped source is mapped — jobs, syncs and `add()` alike — so `add()` on a mapped source returns `documents` (possibly several, with `invalid` naming any that failed) and throws `mapper_skipped` for an object the mapper refused; an object that maps to more than a thousand documents registers the first thousand and returns `job`, which takes the rest (`documentCount` is the total). A mapper may call models (`runTask`) or `fetch`; every call is spend per object. The owner tests one with `remy-admin datasources map test --source <slug> --dev` before deploying, reads what a job skipped with `datasources jobs quarantine <id>`, and re-applies a fixed mapper with `datasources remap --source <slug>`. See the dataSources skill for the authoring loop.
 
 To write what each document says into the app's own tables (the equipment a passage names, the relations it states, the people it quotes), give the source a READER: `datasources/<slug>.reader.ts` exporting `defineReader(Source, { read, removed, cleared? })` from `@mindstudio-ai/agent`, declared in mindstudio.json as `"dataSources": [{ "slug": "<slug>", "reader": { "path": "dist/datasources/<slug>.reader.ts", "timeoutMs"?: 60000, "concurrency"?: 16 } }]` (beside or instead of a mapper). The platform runs it over every document the source builds, however it arrived (`add()`, a sync or a job), with its own batching, retries, budgets and pauses: `read(document)` gets `{ id, sourceId, filename, contentType, externalId, metadata, entities, pageCount, chunkCount, createdAt, url, chunks(), bytes() }`, where `chunks()` returns the passages as indexed, each with its entities, and `bytes()` (or the short-lived `url`) the file the document came from, for a reader that needs the drawing itself; `removed(documentId)` runs when a document leaves the source; `cleared(sourceId)` once when the source is deleted; the optional `relinked(document)` when a new dictionary re-tagged a document already read (the same document, with its new entities; for rows derived from the tags, not a reason to extract again). A document can be read more than once, so `read` replaces that document's rows as a set, keyed by its id, and `removed` deletes them. A throw fails that document and it is retried. The reader always runs against the LIVE release (its code and its database) and starts reading once the app publishes it; it may call models and `fetch`, and every call is spend per document. A reader may not add to, remove from or push a dictionary to its own source. The owner tests one with `remy-admin datasources reader test --source <slug> --dev`, sees its deliveries with `datasources reader status`, and reads what it hasn't with `datasources reread --source <slug> --unread`, a job priced before it runs. See the dataSources skill for the authoring loop.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/agent",
-  "version": "0.1.140",
+  "version": "0.1.142",
   "description": "TypeScript SDK for MindStudio direct step execution",
   "type": "module",
   "main": "./dist/index.js",