@valbuild/shared 0.122.0 → 0.123.0

package/CHANGELOG.md

@@ -1,5 +1,107 @@
 # @valbuild/shared
 
+## 0.123.0
+
+### Minor Changes
+
+- [#613](https://github.com/valbuild/val/pull/613) [`c5e7dfd`](https://github.com/valbuild/val/commit/c5e7dfd12aa1371195be642e1c2fb72b6f3e3ce2) Thanks [@freekh](https://github.com/freekh)! - Val's MCP endpoint moves to its own package, and can now upload images
+
+  Everything that serves Val's content tools over MCP — the tool registry, the
+  write path behind it, the request guards and the access-token verification —
+  now lives in **`@valbuild/mcp`** instead of being split between
+  `@valbuild/server` and `@valbuild/next`.
+
+  **Nothing changes for an app that already mounts it.** `initValMcp` is still
+  exported from `@valbuild/next/server` and behaves exactly as before; it is now a
+  ten-line binding over `@valbuild/mcp`, supplying the Next version that
+  `initHandlerOptions` asks for. A host that is not Next can call
+  `initValMcp` from `@valbuild/mcp` directly.
+
+  If you built your own host on `createValTools`, import it from `@valbuild/mcp`
+  rather than `@valbuild/server`; the tool types moved with it.
+
+  ## Image uploads
+
+  An agent can now add an image, with a new `upload_image` tool. It takes a path
+  to a file on the machine your app runs on, or the image inline as base64, and
+  puts it in an `s.image()` field or an `s.images()` gallery.
+
+  Uploads are converted **only where the Studio would convert them**: `encode` is
+  off unless the schema asks for it (`s.image({ encode: { type: "webp" } })`), and
+  when it does, which images are converted, how far they are scaled and when the
+  original wins are the same decisions the browser makes — the same code, now
+  shared. An upload to a schema without `encode` is stored exactly as it arrived,
+  whatever its size.
+
+  One thing the tool does that the Studio does not: it refuses an image the
+  schema's `accept` does not cover, checked on the bytes that would actually be
+  stored. The Studio does not need to — its file picker carries `accept` — and
+  validation reports a mismatch as server-repairable, so nothing downstream would
+  stop it. An agent has no picker. Note the ordering:
+  `s.image({ accept: "image/webp", encode: { type: "webp" } })` still takes a PNG,
+  because the conversion happens first and it is the result that is checked.
+
+  It is the one tool you construct yourself, because it needs an image library
+  and `sharp` ships a compiled binary per platform. Val does not put one in every
+  project that installs it, so you decide:
+
+  ```sh
+  npm install sharp
+  ```
+
+  ```ts
+  import sharp from "sharp";
+  import { createValImageTools } from "@valbuild/mcp";
+  import { sharpImageProcessor } from "@valbuild/mcp/sharp";
+
+  const { valMcpAuthorize, valMcpTools } = initValMcp(valModules, config, {
+    extraTools: createValImageTools(sharpImageProcessor(sharp)),
+  });
+  ```
+
+  Leave `extraTools` out and everything else works as before — the agent can read,
+  validate and edit content, it just cannot add an image. `sharp` is passed in
+  rather than imported, so you can supply another encoder: `ValImageProcessor` is
+  two functions, `read` and `encode`.
+
+  Remotely stored images work too — `s.image().remote()` and
+  `s.images({ remote: true })` — and they need nothing extra from the MCP client.
+  Adding one does not upload anything to Val's content host: the bytes go into the
+  patch store like any other unpublished change, and the push to
+  `remote.val.build` happens when you publish, exactly as it does for an image
+  added through the Studio. All the tool has to do first is ask the project which
+  bucket to name in the ref, and the credential for that is the one your app
+  already has — its API key when it has one, and in local development the
+  `val login` token in your project, the same one `val validate --fix` uses. If
+  you have not logged in, it says so and writes nothing.
+
+  ## `npm create @valbuild` asks
+
+  The starter template now ships the MCP endpoint, and `npm create @valbuild`
+  asks whether you want it — and, if you do, whether agents should be able to
+  upload images, saying that this adds `sharp`. Both default to yes, and both can
+  be answered up front for a scripted setup:
+
+  ```sh
+  pnpm create @valbuild my-app --mcp --no-image-uploads
+  ```
+
+- [#620](https://github.com/valbuild/val/pull/620) [`53f670c`](https://github.com/valbuild/val/commit/53f670c0cf2d7a03a6d068c78b7874ce77652c2a) Thanks [@freekh](https://github.com/freekh)! - MCP: `search_content` — full-text search across the project's content, unpublished changes included.
+
+  The same index the Studio's search box uses, moved into `@valbuild/shared` so both run it, and built fresh on every call. That was the part that looked expensive and is not: indexing `valbuild/web`, a real production site of 20 modules and 206 KB of source, takes 162 ms, and it scales linearly from there — the 10 s default deadline is not reached until roughly 13 MB of content. Loading the modules costs more than indexing them, and every tool call already pays that.
+
+  That ratio between building the index and querying it — 162 ms against 0.2 ms — is why `queries` is a list. Everything expensive happens before the first query runs, so up to 20 of them are answered from a single pass, separately, so a caller can tell which of its guesses found the thing. A bare string still works as one query.
+
+  It returns source paths, so `get_source` reads what it finds. The rest of the arguments narrow or bound the work:
+
+  - `include` / `exclude` — module file path globs, e.g. `["/content/blogs/**"]`. `exclude` is applied after `include`.
+  - `limit` — per query, defaulting to 100. A model filters a long list more cheaply than it asks again.
+  - `timeoutMs` — stop indexing and answer with what has been indexed so far. The result then carries `timedOut`, the paths of the modules that were not reached, and a hint pointing at `include`.
+
+  Every answer says what it actually searched (`searched: { modules, of }`), so each query's `total` can be read as a count over those modules rather than over the project. Modules you excluded are not reported as omissions — an omission always means the deadline, never your filter.
+
+  `performSearch` now counts all the matches rather than the page it returns, which the Studio's result count gets too: FlexSearch stops as soon as it has the ids it was asked for, so a total taken from a page-sized search was only ever the page size again.
+
 ## 0.122.0
 
 ### Minor Changes

package/dist/declarations/src/internal/index.d.ts

@@ -25,3 +25,8 @@ export * from "./aiTools/aiImageToolPatches.js";
 export * from "./aiTools/aiSourceToolPatches.js";
 export * from "./validation/partitionValidationErrors.js";
 export * from "./validation/blockingValidationErrors.js";
+export * from "./media/encodeImageDecisions.js";
+export * from "./search/searchIndex.js";
+export * from "./search/traverseSchemaSource.js";
+export * from "./search/getFilenameFromRef.js";
+export * from "./search/sourcePath.js";

package/dist/declarations/src/internal/media/encodeImageDecisions.d.ts

@@ -0,0 +1,89 @@
+import { type ImageEncodeOption } from "@valbuild/core";
+/**
+ * Every decision `encode` makes, with none of the pixels.
+ *
+ * The re-encode itself happens in two places that share nothing: the Studio
+ * runs it on a `<canvas>` before the upload leaves the browser
+ * (`packages/ui/spa/utils/encodeImage.ts`), and the MCP image tool runs it on
+ * `sharp` in the server process (`@valbuild/mcp`). What must NOT differ between
+ * them is what `s.image({ encode })` means — which images are converted, how
+ * far they are scaled, and when the original wins anyway. So the decisions live
+ * here, once, and each side supplies only its own encoder.
+ *
+ * See `architecture/media.md`.
+ */
+export type EncodeSettings = {
+    type: "webp";
+    quality: number;
+    maxWidth: number;
+    maxHeight: number;
+};
+export declare const ENCODE_DEFAULTS: Omit<EncodeSettings, "type">;
+export declare const ENCODE_MIME_TYPE_OF: Record<EncodeSettings["type"], string>;
+/**
+ * Formats an encoder cannot round-trip without losing something: SVG is vector,
+ * an animated GIF keeps only its first frame, and AVIF is already smaller than
+ * anything we would produce.
+ */
+export declare const NEVER_ENCODED: string[];
+/**
+ * The schema's own option first, then the gallery it references, then off.
+ *
+ * Mirrors how `ImageField` resolves `accept` and `directory`. A gallery-backed
+ * field (`s.image(galleryVal)`) serializes with EMPTY options, so without the
+ * gallery fallback it could never inherit what the gallery asked for — and
+ * `s.image(galleryVal, { encode: false })` is how such a field opts back out.
+ *
+ * Nonsense numbers fall back to the defaults rather than through. `fitWithin`
+ * reads a bound of 0 (or NaN, or a negative) as "everything already fits", so
+ * an unusable `maxWidth` would otherwise SILENTLY disable the downscale it was
+ * asking for, and a `quality` outside 0–1 is ignored by `canvas.toBlob` just as
+ * quietly. Falling back is deliberately not throwing: a schema is evaluated by
+ * the CLI's `node:vm` sandbox and by the Next server, so a typo'd quality would
+ * take a whole project down over an upload-time optimisation.
+ */
+export declare function resolveEncodeSettings(fieldOption: ImageEncodeOption | undefined, galleryOption: ImageEncodeOption | undefined): EncodeSettings | null;
+/**
+ * Dimensions scaled to fit inside the box, or null when the image already does.
+ *
+ * Null is the signal for "no downscale was needed", which the size comparison
+ * in {@link chooseEncoded} turns on.
+ */
+export declare function fitWithin(width: number, height: number, maxWidth: number, maxHeight: number): {
+    width: number;
+    height: number;
+} | null;
+/**
+ * Whether a schema that declares `accept` would take this mime type.
+ *
+ * The same rules as `ImageSchema.executeValidate`, which validates the STORED
+ * mimeType against `accept` — so converting to a type `accept` forbids would
+ * upload a file the schema reports as invalid the moment it lands. Delegated to
+ * `@valbuild/core` rather than re-spelled, because a second reading of one
+ * `accept` string is how a converted image passes here and fails there.
+ */
+export declare function isMimeTypeAccepted(mimeType: string, accept: string | undefined): boolean;
+/** Formats left alone: the lossy list, plus the target type when it already fits. */
+export declare function isSkippedSource(sourceMimeType: string, targetMimeType: string, needsDownscale: boolean): boolean;
+/**
+ * Whether to upload what the encoder produced, or the file that was picked.
+ *
+ * Two ways the encoded bytes lose. They are not actually the type we asked for
+ * — `canvas.toBlob` answers an unsupported type with a PNG rather than with
+ * null, so "not null" is not the same question as "is a webp". Or they are
+ * simply bigger, which happens for small and flat images: measured, an 8x8
+ * solid PNG of 74 bytes becomes a 548 byte webp. A downscale overrides that,
+ * since the original is then the wrong size whatever it weighs.
+ */
+export declare function chooseEncoded(input: {
+    originalSize: number;
+    encodedSize: number | null;
+    encodedType: string | null;
+    targetMimeType: string;
+    needsDownscale: boolean;
+}): boolean;
+/**
+ * `photo.png` -> `photo.webp`. The extension is re-derived downstream too, but
+ * the name should not lie.
+ */
+export declare function withExtension(filename: string, ext: string): string;

package/dist/declarations/src/internal/search/getFilenameFromRef.d.ts

@@ -0,0 +1,17 @@
+/**
+ * Extract a human-readable filename from a file ref (local path or remote URL).
+ * Handles both remote refs (via splitRemoteRef) and plain `/public/...` paths.
+ */
+export declare function getFilenameFromRef(ref: string): string;
+/**
+ * Parse a file ref into its constituent parts: a clean path, filename, and
+ * folder (the path relative to `/public`).
+ *
+ * The `folder` strips the standard `/public` prefix so it shows a
+ * concise location like `/images` instead of `/public/images`.
+ */
+export declare function getRefParts(ref: string): {
+    cleanPath: string;
+    filename: string;
+    folder: string;
+};

package/dist/declarations/src/internal/search/searchIndex.d.ts

@@ -0,0 +1,76 @@
+import { Json, ModuleFilePath, SerializedSchema, SourcePath } from "@valbuild/core";
+import { Index } from "flexsearch";
+/**
+ * The search index and the labels to render for its hits.
+ *
+ * Kept as a plain module rather than living inside a worker so it can be tested:
+ * a worker entry runs `self.onmessage` on import, which no test environment here
+ * provides.
+ *
+ * In `@valbuild/shared` rather than the Studio because there are now two
+ * realms searching the same content: the Studio's `SearchStore`, which keeps an
+ * index alive in a worker and re-indexes a module at a time, and the MCP
+ * `search_content` tool, which builds one per call and throws it away. What
+ * counts as a document, and what a hit is labelled, must not differ between
+ * them — an agent and an editor searching the same project should find the same
+ * things.
+ */
+export type SearchIndex = {
+    index: Index;
+    pathToLabel: Map<string, string>;
+    /**
+     * Which document ids belong to which module, so one module can be re-indexed
+     * without touching the others.
+     *
+     * The ids are the `SourcePath`s themselves — `index.add(path, …)` below — so
+     * this could be recovered by scanning `pathToLabel` for a module-file-path
+     * prefix. It is kept explicitly anyway: the scan would be O(every document in
+     * the project) per module re-indexed, which is the whole-project cost that
+     * re-indexing one module exists to avoid.
+     */
+    docsByModule: Map<ModuleFilePath, Set<string>>;
+};
+/** An index with nothing in it. Fill it with {@link indexModule}. */
+export declare function createSearchIndex(): SearchIndex;
+/**
+ * Index one module, replacing whatever was previously indexed for it.
+ *
+ * Remove-then-add rather than `update`, because the set of paths in a module is
+ * not stable: deleting an array item or a record entry means documents that must
+ * disappear, and `update` only revises ids it is given.
+ *
+ * `.jsonValues()` entries are indexed only once their content is loaded — an
+ * un-loaded entry is an opaque marker that {@link traverseSchemaSource} skips, so
+ * the index is naturally PARTIAL and grows as batches land. That is why the search
+ * UI loads the entries on the first query and re-indexes (debounced) as they
+ * arrive, rather than pretending an empty result set is an answer.
+ */
+export declare function indexModule(searchIndex: SearchIndex, moduleFilePath: ModuleFilePath, source: Json, schema: SerializedSchema): void;
+/**
+ * Drop everything indexed for one module.
+ *
+ * Also used for a module that has gone away entirely — otherwise its documents
+ * stay searchable and a hit navigates to a path that no longer exists.
+ */
+export declare function removeModule(searchIndex: SearchIndex, moduleFilePath: ModuleFilePath): void;
+/**
+ * Index every module, from scratch.
+ *
+ * Now a loop over {@link indexModule} rather than its own walk, so the full
+ * build and the incremental one cannot drift in what they consider a document.
+ */
+export declare function buildSearchIndex(modules: Record<ModuleFilePath, {
+    source: Json;
+    schema: SerializedSchema;
+}>): SearchIndex;
+export type SearchResults = {
+    results: Array<{
+        path: SourcePath;
+        label: string;
+    }>;
+    /** Matches for the query, not just on this page. */
+    total: number;
+    /** `total` hit {@link MAX_COUNTED_RESULTS}, so it means "at least this many". */
+    totalIsLowerBound: boolean;
+};
+export declare function performSearch(searchIndex: SearchIndex | null, query: string, limit?: number, offset?: number): SearchResults;

package/dist/declarations/src/internal/search/sourcePath.d.ts

@@ -0,0 +1,30 @@
+import { ModuleFilePath, ModulePath, SourcePath } from "@valbuild/core";
+/**
+ * The one way to build a child's source path.
+ *
+ * There were five hand-rolled versions of this, three of them subtly different
+ * and two of them wrong, and the wrong ones were wrong in a way that hid: a
+ * module path segment has to be JSON-quoted if it is a string and bare if it is
+ * an array index, and both broken copies appended the raw key at the module
+ * root. An unquoted key parses fine as long as it contains no `.` — so every
+ * object field and every ordinary record key still resolved, and only a key with
+ * a dot in it broke. That is exactly what a media gallery is keyed by, so
+ * clicking a file in the Media panel navigated to
+ * `?p=/public/val/red-8x8_bfbd0.png`, which `splitModulePath` reads as the two
+ * segments `/public/val/red-8x8_bfbd0` and `png`, and the studio said the module
+ * was not found.
+ *
+ * So: one function, and never a template string. `Internal.createValPathOfItem`
+ * is the rule itself — `JSON.stringify` quotes a string and leaves a number
+ * bare, which is precisely the module-path grammar — and everything here defers
+ * to it rather than restating it.
+ */
+export declare function sourcePathOfChild(parent: SourcePath | ModuleFilePath, key: string | number): SourcePath;
+/**
+ * The same thing, for callers holding the module file path and module path apart.
+ *
+ * `ModulePath` is empty at the module root, which is the case both broken copies
+ * got wrong: there is no separator in the path yet, so the key is the first
+ * segment and still has to be quoted.
+ */
+export declare function concatModulePath(moduleFilePath: ModuleFilePath, modulePath: ModulePath, key: string | number): SourcePath;

package/dist/declarations/src/internal/search/traverseSchemaSource.d.ts

@@ -0,0 +1,20 @@
+import { Source, SerializedSchema, SourcePath } from "@valbuild/core";
+/**
+ * Traverses a schema and source pair, calling a callback for each leaf node.
+ * Handles all schema types including primitives, objects, arrays, records, unions, richtext, files, images, etc.
+ *
+ * @param source - The JSON source data
+ * @param schema - The serialized schema
+ * @param path - The current source path
+ * @param callback - Function called for each traversable node with { source, schema, path }
+ */
+export declare function traverseSchemaSource(source: Source, schema: SerializedSchema, path: SourcePath, callback: (opts: {
+    source: Source;
+    schema: SerializedSchema;
+    path: SourcePath;
+}) => void): void;
+/**
+ * Flattens richtext source to a plain string, extracting only text content.
+ * Used for indexing richtext content in search.
+ */
+export declare function flattenRichText(source: Source): string;