@mindstudio-ai/remy 0.1.326 → 0.1.328

package/dist/prompt/skills/dataSources.md

@@ -46,7 +46,7 @@ Search is deterministic for a fixed corpus and configuration, so eval sets and r
 
 **Debugging retrieval.** Two opt-in options, neither of which changes the results or their order: `explain: true` adds `explain.{dense, lexical, matchedVia}` (which half of hybrid found each hit; costs two extra round trips), and `expand: 1` adds `neighbors.{before, after}` for surrounding context. When a document never comes back at all, `Policies.stats()` reports the config actually in effect and `Policies.chunks(documentId)` shows exactly how it was split.
 
-**A cold index (shared capacity).** By default a data source lives on shared retrieval capacity: its vectors sit in their own isolated partition of a pool many apps share, and the pool keeps only a working set resident. A source nobody has searched for a while is unloaded to make room and reloaded from durable storage on the next search. A small corpus reloads inside that search and nobody notices; a large one (hundreds of thousands of chunks) reloads in the background for a minute or two, and `search()` throws `index_warming` (HTTP 503) until it lands. That means *loading*, never *empty*: catch it, tell the user the knowledge base is warming up, and retry shortly. Before a demo, `remy-admin datasources hydrate --source <slug> --wait` reloads it ahead of time. The way out of the cycle is dedicated capacity (below): a source on its own provisioned retrieval is never unloaded and never warms. It has one pause of its own instead: while a bulk load runs, the index is not built behind every write, it is built once when the load finishes, and until then `search()` throws `index_building` (HTTP 503) with "N of M vectors indexed" in the message. Same handling as `index_warming`: the knowledge base is being built, never empty. `remy-admin datasources list` shows the index as `deferred`, `building` or `ready`.
+**A cold index (shared capacity).** By default a data source lives on shared retrieval capacity: its vectors sit in their own isolated partition of a pool many apps share, and the pool keeps only a working set resident. A source nobody has searched for a while is unloaded to make room and reloaded from durable storage on the next search. A small corpus reloads inside that search and nobody notices; a large one (hundreds of thousands of chunks) reloads in the background for a minute or two, and `search()` throws `index_warming` (HTTP 503) until it lands. That means *loading*, never *empty*: catch it, tell the user the knowledge base is warming up, and retry shortly. Before a demo, `remy-admin datasources hydrate --source <slug> --wait` reloads it ahead of time. The way out of the cycle is dedicated capacity (below): a source on its own provisioned retrieval is never unloaded. It has one pause of its own instead: after a large load the index catches up behind the writes, and until it has, `search()` throws `index_building` (HTTP 503) with "N of M vectors indexed" and the time left in the message. Same handling as `index_warming`: the knowledge base is being built, never empty. `remy-admin datasources list` shows the index as `building` or `ready`, with its rate and time left while it builds. A resource that resumes from hibernation, or whose instance the platform replaced, rebuilds every source placed on it from durable storage in the background; searches on those sources answer `index_warming` with the rebuild's progress until it lands.
 
 **Configuration is not declared in code** — chunking and embedding settings live on the corpus and are set with the CLI, so code and reality can't drift.
 
@@ -134,6 +134,7 @@ remy-admin secrets set ARCHIVE_S3_SECRET --prod <value>
 remy-admin datasources connect --source archive --bucket acme-docs --region us-east-1 --prefix contracts/ --access-key-secret ARCHIVE_S3_KEY --secret-key-secret ARCHIVE_S3_SECRET --budget-per-sync 5
 remy-admin datasources sync --source archive --wait     # first sync: plans the whole bucket, stops for approval if over the budget
 remy-admin datasources sync --source archive --limit 20000 --concurrency 64   # a slice, run hard: how a big backfill is measured before it is approved
+remy-admin datasources sync --source archive --concurrency 256 --priority    # the real backfill when the clock matters: the whole fleet, priority-tier embedding at 1.5x the embedding price
 remy-admin datasources connector --source archive       # what it follows, last sync, object counts
 ```
 
@@ -205,7 +206,7 @@ remy-admin datasources remap --source archive --wait
 
 **A mapper is a pure transform: one object in, documents out, nothing else.** `remap` and `jobs replay` run it again over the raw copies, and a frame runs in the context of the release that compiled it, so anything a mapper writes on the side is written twice and possibly into the wrong data plane. The per-document facts an app needs later belong in `metadata`; an app that wants its own view of a big corpus (a timeline, counts by year, a table of ids) builds it after ingest by walking `Source.allDocuments()` in a background task, and keeps it current from what each sync adds.
 
-A mapper runs on the platform, so the platform has to build it. Any push builds it, and a branch push is a private preview build, which is all a mapper needs. `map deploy` then makes that build's mapper the source's active one: jobs, syncs and `add()` run it from then on, whether or not the app has ever been published. Publishing to main activates the mapper main declares as well, so merge the branch before you publish. `jobs start` refuses with `mapper_not_deployed` while the dev session declares a mapper that is not yet active, because the job would otherwise load the raw records as documents.
+A mapper runs on the platform, so the platform has to build it. Any push builds it, and a branch push is a private preview build, which is all a mapper needs. `map deploy` then makes that build's mapper the source's active one: jobs, syncs and `add()` run it from then on, whether or not the app has ever been published. Publishing activates the mapper the live release declares — which is the one you deployed, since publishing fast-forwards the default branch to your branch. So there is nothing extra to do at publish time, and nothing to merge by hand: publishing is the merge (see the publishing skill). `jobs start` refuses with `mapper_not_deployed` while the dev session declares a mapper that is not yet active, because the job would otherwise load the raw records as documents.
 
 `map test --dev` needs the dev session running (`npx mindstudio dev`); it runs the mapper from local source through the tunnel and prints every outcome with markdown previews. The plan of a mapped job records the mapper's outcome mix on its sample; a run whose skip share climbs past twice that pauses with `pauseReason: 'skips'` for a look at the quarantine. `remap` reads the platform's own raw copies — no origin traffic — skips unchanged markdown by hash, and supersedes changed documents, so a metadata tweak on a million-document source costs frames and little else. `externalId` is the identity everything replaces by; choose it deliberately (the record's stable id, never the key of a file that gets rewritten in place).
 
@@ -213,6 +214,10 @@ A mapper runs on the platform, so the platform has to build it. Any push builds
 
 The shared pool holds a source up to a per-source cap of chunks. A corpus beyond it — a plan that answers `plan_requires_dedicated` — runs on dedicated retrieval capacity the workspace provisions and pays for hourly: `remy-admin infra list` shows the offering catalog with prices and any resources the app has, `infra provision --offering <id> --name <n> --wait` creates one, and `datasources move --source archive --to <resource-id> --wait` puts the source on it with its data intact (`create --placement <resource-id>` starts a new source there). Provisioning bills the workspace; **never provision without the user's explicit confirmation**, and show them the offering's price first. `infra --help` covers hibernate, resume, resize and destroy.
 
+A resource's capacity is bytes, not a chunk count: the catalog quotes each size as chunks at each embedding dimension on offer (a 2560-dimension chunk takes two and a half times a 1024-dimension one), and a plan or move that does not fit says how much room it would need against what the size holds. `infra get <id>` shows what the placed sources take against that, measured from the instance itself once it is up.
+
+Nothing long-running here is silent. Every `--wait` (`datasources move`, `hydrate`, `jobs approve`, `infra provision|resume|resize`) prints the phase, what is done of what, the measured rate and the time left as it goes, and keeps waiting as long as progress advances; the dashboard shows the same numbers. `infra get <id>` carries a resource's progress and `infra logs <id> --follow` streams the platform's narration of what it is doing. Hibernating releases the instance and stops billing; resuming brings a new one up and rebuilds every placed source's index from durable storage in the background, with its progress on the resource and `index_warming` from searches until it lands. Nothing is re-embedded.
+
 ## Answering from results
 
 Retrieve → join passages as context → have a model answer *from that context* → render citations. Never paste raw chunks at the user; they're fragments. For agentic flows, give the model `search` as a tool so it can query repeatedly and refine, rather than retrieving once up front.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/remy",
-  "version": "0.1.326",
+  "version": "0.1.328",
   "description": "Remy coding agent",
   "repository": {
     "type": "git",