@mindstudio-ai/remy 0.1.328 → 0.1.329
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -42,7 +42,7 @@ result.$billingCost; // cost in credits (if applicable)
|
|
|
42
42
|
|--------|-------------|-----------|------------|
|
|
43
43
|
| `analyzeImage` | Vision model analysis | `prompt`, `imageUrl` | `analysis` |
|
|
44
44
|
| `analyzeVideo` | Video analysis | `prompt`, `videoUrl` | `analysis` |
|
|
45
|
-
| `transcribeAudio` | Audio to text | `audioUrl` | `
|
|
45
|
+
| `transcribeAudio` | Audio to text | `audioUrl` | `text`, `segments` |
|
|
46
46
|
| `extractText` | Extract text from documents/images | `url` | `text` |
|
|
47
47
|
| `detectPII` | Find personal data | `text` | `entities` |
|
|
48
48
|
|
|
@@ -40,6 +40,8 @@ Created on first use, so searching a source the build hasn't populated returns n
|
|
|
40
40
|
|
|
41
41
|
**Filtering** narrows a search before ranking, and every condition only narrows: `filter: { metadata: { department: 'legal', year: [2025, 2026], signedAt: { gte: 20250101 } }, filename, documentIds, pages: { min?, max? }, contains: 'all these words', phrase: 'exact adjacent sequence' }`. Metadata matches per key: scalar = equals, array = any-of, `{ gte?, lte? }` = numeric range — ranges are numeric only, so store dates as sortable integers at add time (YYYYMMDD or epoch seconds) to range on them. Metadata is tagged at add time (scalars only, ≤16 keys); re-adding the same bytes with different metadata updates the tags in place, free. Filters are the right tool for scoping retrieval (per-user, per-category, a date window); they are NOT a substitute for a `db` query over structured data.
|
|
42
42
|
|
|
43
|
+
**Counting matches.** `Policies.count(filter)` returns `{ chunks }`: how many passages match a filter, exactly, over the whole corpus, in the same grammar as `search`'s filter. `count({ contains: query })` is the honest number to show beside a search's hits: "8 of 3,891 passages mention these words". It is not a relevance count (none exists; similarity is a continuous score over every chunk) and it is not the set `search` returns (semantic hits need not contain the words, lexical hits may contain only some), so never label it "relevant results" or "matches for your query". Chunks, not documents. Counts share the 300/minute search limit; a count beside every search spends two. Same `index_warming` / `index_building` handling as search.
|
|
44
|
+
|
|
43
45
|
**Modes**: `mode: 'hybrid'` (default) fuses semantic and keyword retrieval; `'semantic'` is the embedding alone; `'lexical'` is keyword-only with **no query embedding** — cheapest and fastest, right when the query is an identifier (an error code, a SKU, a name) rather than a meaning. `maxPerDocument: 2` stops one document monopolizing the results when the answer should draw on several. `highlight: true` adds `matches` (`{start, end}` offsets into `text`) for rendering highlighted excerpts.
|
|
44
46
|
|
|
45
47
|
Search is deterministic for a fixed corpus and configuration, so eval sets and regression checks are meaningful — key them on `(documentId, chunkIndex)` rather than on chunk text.
|
|
@@ -206,6 +208,8 @@ remy-admin datasources remap --source archive --wait
|
|
|
206
208
|
|
|
207
209
|
**A mapper is a pure transform: one object in, documents out, nothing else.** `remap` and `jobs replay` run it again over the raw copies, and a frame runs in the context of the release that compiled it, so anything a mapper writes on the side is written twice and possibly into the wrong data plane. The per-document facts an app needs later belong in `metadata`; an app that wants its own view of a big corpus (a timeline, counts by year, a table of ids) builds it after ingest by walking `Source.allDocuments()` in a background task, and keeps it current from what each sync adds.
|
|
208
210
|
|
|
211
|
+
On a job, mapping is its own stage. The mapper turns each object into documents; the platform then ingests those documents in parallel batches of fifty across its workers, whatever one object became. So the size of an object does not set the pace, and a bundle of a thousand records is fine; only the number of objects sets how wide the mapping stage itself runs (three huge files map on three workers, the plan says so as a warning). `jobs status` reads "mapping N of M objects" until that stage is through, then counts documents.
|
|
212
|
+
|
|
209
213
|
A mapper runs on the platform, so the platform has to build it. Any push builds it, and a branch push is a private preview build, which is all a mapper needs. `map deploy` then makes that build's mapper the source's active one: jobs, syncs and `add()` run it from then on, whether or not the app has ever been published. Publishing activates the mapper the live release declares — which is the one you deployed, since publishing fast-forwards the default branch to your branch. So there is nothing extra to do at publish time, and nothing to merge by hand: publishing is the merge (see the publishing skill). `jobs start` refuses with `mapper_not_deployed` while the dev session declares a mapper that is not yet active, because the job would otherwise load the raw records as documents.
|
|
210
214
|
|
|
211
215
|
`map test --dev` needs the dev session running (`npx mindstudio dev`); it runs the mapper from local source through the tunnel and prints every outcome with markdown previews. The plan of a mapped job records the mapper's outcome mix on its sample; a run whose skip share climbs past twice that pauses with `pauseReason: 'skips'` for a look at the quarantine. `remap` reads the platform's own raw copies — no origin traffic — skips unchanged markdown by hash, and supersedes changed documents, so a metadata tweak on a million-document source costs frames and little else. `externalId` is the identity everything replaces by; choose it deliberately (the record's stable id, never the key of a file that gets rewritten in place).
|