@mindstudio-ai/remy 0.1.334 → 0.1.336

package/dist/prompt/compiled/dev-and-deploy.md

@@ -19,7 +19,7 @@ The dev session gets its own database — a snapshot of the live database at ses
 - **Truncate** — keep the schema, delete all row data (used by scenarios for a clean canvas)
 - **Schema sync** — add a field to a table interface and it's immediately available in dev
 
-The dev database is disposable. Experiment freely — there's no risk of breaking anything. Just be considerate that the user may have created their own data (user rows or other data) while testing, and it might be frustrating for them to have it wiped.
+The dev database's data is disposable — reset or truncate it whenever that helps. Just be considerate that the user may have created their own data (user rows or other data) while testing, and it might be frustrating for them to have it wiped. If a schema sync fails, the error names the table and what is out of step between the platform's schema record and the table itself; fix exactly that.
 
 ### Debugging
 

package/dist/prompt/compiled/tables.md

@@ -389,20 +389,14 @@ const [_, newOrder, pending] = await db.batch(
 
 ## Migrations
 
-No migration files. Migrations are automatic:
-- **New tables** — `CREATE TABLE` applied automatically
-- **New columns** — `ALTER TABLE ADD COLUMN` applied automatically
-- **Dropped columns** — `ALTER TABLE DROP COLUMN` applied automatically when a column is removed from the interface
-- **Dropped tables** — `DROP TABLE` applied automatically when a table file is removed from the manifest
-- **Type changes and renames** — not supported in the automatic migration path
-
-On deploy, the platform:
-1. Parses your table definition files (TypeScript AST — the interface IS the schema)
-2. Diffs against the current live database schema
-3. Generates DDL (`CREATE TABLE`, `ALTER TABLE ADD COLUMN`, `ALTER TABLE DROP COLUMN`, `DROP TABLE`)
-4. Applies to a staging copy of the database
-5. Promotes the staging copy to live
-
-The TypeScript interface is the single source of truth for the schema. Add a field to the interface, push, and the column exists. No migration files, no CLI commands.
-
-**In development**, schema changes are synced automatically to the dev database. The dev database is a disposable snapshot — it can be reset to a fresh copy of production data or truncated to empty tables at any time. There's no risk of breaking anything by experimenting with schema changes in dev.
+No migration files. The table's TypeScript interface is the schema, and the platform brings the database to match it:
+- **New tables** — created
+- **New, dropped, or retyped columns, and changed `unique` constraints** — the table is rebuilt: a table with the declared shape is created, every row is copied across, and it replaces the old one in one transaction. New columns arrive empty; dropped columns and their data are gone; a retyped column's values are carried across, converted where SQLite can.
+- **Dropped tables** — dropped when the table file is removed from the manifest
+- **Renames** — not detected; a renamed column or table is a drop plus an add, and its data does not carry over
+
+On deploy, the platform parses the table files, diffs against the live schema, applies the changes to a clone of the live database, and promotes the clone. If any change fails, the release fails and the live database is untouched.
+
+Add a field to the interface, push, and the column exists. No migration files, no CLI commands.
+
+**In development**, schema changes are applied to the dev database automatically, by the same rules. Its data is disposable — it can be reset to a fresh copy of production or truncated to empty tables at any time. If a schema sync fails, the error names the table and what is out of step between the platform's schema record and the table itself; fix exactly that.

package/dist/prompt/skills/dataSources.md

@@ -20,6 +20,7 @@ Don't use one when: the data is structured (`db`), you only need to store files
 - **Ingest is async.** `add()` returns once queued; poll `documents()`, or use `--wait` from the CLI.
 - **Reprocessing costs real money**, so changing how a corpus is built is always explicit.
 - **Limits apply**: 25 data sources per app, 10,000 chunks per document, 300 searches/minute, and 5,000 documents per source when documents are added one at a time (bulk jobs and connectors, below, are how a corpus grows past that). Well clear of normal use — but **source names must be fixed, not computed per user or per request**, since referencing one creates it. Partition inside a source with document metadata instead: tag at add time (`add(bytes, { filename, metadata: { userId } })`), narrow at search time (`search(q, { filter: { metadata: { userId } } })`).
+- **Whole documents, not excerpts.** Nothing needs trimming before it is added: a document becomes as many chunks as it needs, up to the 10,000 per document above (about 20M characters). Truncating to keep chunks manageable throws away retrievable text, and is never the fix for an ingest problem.
 - **Credentials never appear in code or in chat.** A bucket's keys are app secrets (`remy-admin secrets set NAME --prod <value>`, or the dashboard); everything else refers to them by NAME. If the user pastes a key into the conversation, set it as a secret for them to use moving forward.
 
 ## Defining and searching
@@ -46,7 +47,7 @@ Created on first use, so searching a source the build hasn't populated returns n
 
 Search is deterministic for a fixed corpus and configuration, so eval sets and regression checks are meaningful — key them on `(documentId, chunkIndex)` rather than on chunk text.
 
-**Debugging retrieval.** Two opt-in options, neither of which changes the results or their order: `explain: true` adds `explain.{dense, lexical, matchedVia}` (which half of hybrid found each hit; costs two extra round trips), and `expand: 1` adds `neighbors.{before, after}` for surrounding context. When a document never comes back at all, `Policies.stats()` reports the config actually in effect and `Policies.chunks(documentId)` shows exactly how it was split.
+**Debugging retrieval.** Two opt-in options, neither of which changes the results or their order: `explain: true` adds `explain.{dense, lexical, matchedVia}` (which half of hybrid found each hit; costs two extra round trips), and `expand: 1` adds `neighbors.{before, after}` for surrounding context. **When a document never comes back, or its chunk count looks wrong, read the split before theorising about the file.** `Policies.chunks(documentId)` shows exactly how that document was divided and `Policies.stats()` reports the config actually in effect; from the CLI, `datasources status` gives per-document state with its ingest error. One chunk for a long document, or a chunk much larger than the configured `--max-chars`, is a fact about the split, not about the file.
 
 **A cold index (shared capacity).** By default a data source lives on shared retrieval capacity: its vectors sit in their own isolated partition of a pool many apps share, and the pool keeps only a working set resident. A source nobody has searched for a while is unloaded to make room and reloaded from durable storage on the next search. A small corpus reloads inside that search and nobody notices; a large one (hundreds of thousands of chunks) reloads in the background for a minute or two, and `search()` throws `index_warming` (HTTP 503) until it lands. That means *loading*, never *empty*: catch it, tell the user the knowledge base is warming up, and retry shortly. Before a demo, `remy-admin datasources hydrate --source <slug> --wait` reloads it ahead of time. The way out of the cycle is dedicated capacity (below): a source on its own provisioned retrieval is never unloaded. It has one pause of its own instead: after a large load the index catches up behind the writes, and until it has, `search()` throws `index_building` (HTTP 503) with "N of M vectors indexed" and the time left in the message. Same handling as `index_warming`: the knowledge base is being built, never empty. `remy-admin datasources list` shows the index as `building` or `ready`, with its rate and time left while it builds. A resource that resumes from hibernation, or whose instance the platform replaced, rebuilds every source placed on it from durable storage in the background; searches on those sources answer `index_warming` with the rebuild's progress until it lands.
 
@@ -118,7 +119,7 @@ for await (const doc of Policies.allDocuments()) { /* ... */ }    // walk a corp
 await Policies.remove(documentId);
 ```
 
-Formats: pdf, docx, pptx, xlsx, odt, rtf, epub, images, txt, md, json, csv, tsv, log, html. When the file is not the document (a JSON record, a JSONL bundle of articles, a kill notice), the source needs a mapper — see below.
+Formats: pdf, docx, pptx, xlsx, odt, rtf, epub, images, txt, md, json, csv, tsv, log, html. **Add the source bytes, not your own rendering of them** — built-in extraction reads every one of these, HTML included, so fetching a page and converting it to markdown yourself swaps a tested extractor for an untested one and makes its quirks the corpus's quirks. When the file is not the document (a JSON record, a JSONL bundle of articles, a kill notice), the source needs a mapper — see below.
 
 Removing many documents at once: `Policies.removeWhere({ metadata: { year: 2019 } })` or `{ externalIdPrefix: 'archive/2019/' }` (the key a job or connector recorded) removes every match, vectors and bytes included, in pages of a thousand. From the CLI, `datasources rm --source policies --filter year=2019`. An empty filter is refused; deleting a whole source is `datasources delete`, never something app code does.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mindstudio-ai/remy",
-  "version": "0.1.334",
+  "version": "0.1.336",
   "description": "Remy coding agent",
   "repository": {
     "type": "git",