@matchbox-ai/train 0.1.0

package/docs/parser-api.md

@@ -0,0 +1,86 @@
+# Parser task API
+
+`defineParser` describes a constrained parsing task. It validates inputs and expected outputs and exports training metadata. It does not train a model or infer a result from natural language.
+
+```ts
+import { defineParser, type InferOutput } from "@matchbox-ai/core";
+import { z } from "zod";
+
+const task = defineParser({
+  input: z.string().min(1).max(200),
+  output: z.strictObject({
+    country: z.enum(["SE", "DE"]),
+    minimum: z.number().nonnegative(),
+    owner: z.string().nullable().optional(),
+  }),
+  fields: {
+    minimum: { type: "money", aliases: ["ARR"] },
+  },
+});
+
+type Output = InferOutput<typeof task>;
+// { country: "SE" | "DE"; minimum: number; owner?: string | null | undefined }
+
+const input = task.validateInput("Swedish customers above 50k");
+const expected = task.validateOutput({ country: "SE", minimum: 50000 });
+if (expected.success) {
+  const value: Output = expected.data;
+  console.log(value);
+} else {
+  console.log(expected.issues); // Each issue has code, path, and message.
+}
+
+const metadata = task.toJSON();
+const serialized = JSON.stringify(task); // Uses the same metadata representation.
+```
+
+Validation returns a discriminated `ValidationResult<T>`. Invalid task definitions throw `TypeError` at definition time with a schema path. Invalid example values return issues rather than being silently normalized. There is no `parse(text)` inference method yet.
+
+## Supported schemas
+
+The input is explicitly `z.string()`, optionally with supported constraints. The output root must be a strict object, array, or union of objects/arrays.
+
+Within outputs, v0 supports:
+
+- Strict objects with declared properties, including nested objects.
+- Arrays and ordinary or discriminated unions.
+- Strings, finite numbers, booleans, null, JSON literals, and enums.
+- Nullable values and optional object properties. Put `.optional()` outermost and omit absent properties; explicit `undefined` is rejected.
+- Finite numeric bounds, safe integers, string/array lengths, and regular expressions without flags.
+
+Use `z.strictObject()` or `.strict()`. Ordinary `z.object()` strips unknown keys, while the serialized output contract disallows them. Requiring strict objects keeps validation behavior explicit.
+
+Unsupported schemas fail at definition time. This includes transforms, overwrites such as `.trim()`, coercion, non-boolean defaults, catches, custom/async refinements, conditional checks, arbitrary unknown/any values, dates, bigint, records, maps, sets, tuples, intersections, lazy/recursive schemas, readonly wrappers, and string formats other than flagless regexes. Add support only when both runtime and metadata behavior are tested.
+
+Both schema and data traversal are limited to 64 nested containers. Validation accepts plain JSON data, not class instances, accessors, sparse arrays, symbol properties, or circular references. `__proto__` is reserved because Zod omits it while constructing parsed objects.
+
+## Training metadata
+
+The serialized definition contains:
+
+```ts
+{
+  formatVersion: 1,
+  kind: "parser",
+  input: { /* JSON Schema draft 2020-12 */ },
+  output: { /* JSON Schema draft 2020-12 */ },
+  fields: { minimum: { type: "money", aliases: ["ARR"] } },
+}
+```
+
+Metadata is captured at definition time. Every `toJSON()` call returns a detached copy. Treat the supplied Zod schemas as immutable after definition; use Zod's schema-building methods to create a new task when its contract changes.
+
+The format version identifies Matchbox's metadata representation, not a model or dataset version. A future loader must reject unsupported versions. This ticket provides serialization only; it does not reconstruct executable validators from JSON.
+
+Zod's global metadata registry is intentionally excluded. `.meta()` must not override structural keywords or inject values that are not JSON. Descriptions, aliases, and semantic types belong in the optional `fields` object for now. Those hints are inert, copied JSON. Matchbox does not infer a field-to-output mapping or normalize money/countries from their names.
+
+## Dependency boundary
+
+Zod 4.6.3 is an explicit, pinned core dependency and is external to the Rolldown bundle. The API uses a small isolated portion of Zod Core's schema/check definitions to reject unsupported behavior before JSON conversion. Zod upgrades must run the contract suite, including negative cases.
+
+Validation uses Zod's non-JIT path. The browser smoke check exercises definition, validation, and metadata serialization through the built package. This is the task-authoring API; its dependency size is not a claim about the future generated inference runtime.
+
+## References
+
+- [Zod JSON Schema conversion](https://zod.dev/json-schema) documents representable schemas and conversion controls.
+- [Zod Core](https://zod.dev/packages/core) documents schema and check introspection.

package/docs/pipelines.md

@@ -0,0 +1,30 @@
+# Explicit pipelines
+
+The parser defines valid input and output. The pipeline defines how the model learns. Matchbox requires pipeline.ts for conventional tasks; the scaffold writes it visibly.
+
+```ts
+import { definePipeline, wordTokens, fieldClassifier } from "@matchbox-ai/train";
+export default definePipeline({
+  input: wordTokens(),
+  prediction: fieldClassifier(),
+  acceptance: { minAccuracy: 0.95, maxBytes: 64000 },
+});
+```
+
+This preset trains the existing bag-of-words MLP. Each output field classifies values observed in training. It cannot emit an unseen numeric value, ignores word order, and may be confidently wrong on new combinations of familiar words. Unknown vocabulary causes abstention. The CLI reports this limitation.
+
+An explicit token pipeline uses application-owned supervision and decoding:
+
+```ts
+import { definePipeline, tokenClassifier } from "@matchbox-ai/train";
+export default definePipeline({
+  prediction: tokenClassifier({ recipe: "./lib/recipe.ts", decode: "./lib/decode.ts" }),
+  acceptance: { minAccuracy: 0.85, maxBytes: 24000 },
+});
+```
+
+Paths resolve relative to the task. The recipe supplies tokenizer, labels, readout, and annotation alignment. The decoder receives labeled spans and returns a candidate output or null. Matchbox verifies training annotations decode to the supplied training outputs. Recipe code runs during training; decoder code ships with the browser artifact and must remain browser-safe.
+
+These are two existing, evaluated presets. Decimal codecs, arbitrary graphs, automatic architecture search, and hidden domain normalizers are not implemented. New primitives should demonstrate their limitations and held-out behavior before becoming defaults.
+
+Programmatic training uses await train("money", { onProgress }) from @matchbox-ai/train and follows the same validation and packaging path as the CLI. Importing the authoring helpers does not initialize native TensorFlow; training loads it when invoked.

package/docs/primitives/README.md

@@ -0,0 +1,19 @@
+# Authoring primitives
+
+Import these from @matchbox-ai/train:
+
+| Primitive       | Contract                                                                       |
+| --------------- | ------------------------------------------------------------------------------ |
+| definePipeline  | Validates an explicit pipeline declaration and acceptance thresholds.          |
+| wordTokens      | Selects literal word features for field classification.                        |
+| fieldClassifier | Selects independent categorical output heads over training values.             |
+| tokenClassifier | Selects the sequence trainer with explicit recipe and decoder module paths.    |
+| tokenize        | Shares the portable tokenizer with annotation generators.                      |
+| train           | Runs project discovery, native training, validation, and packaging.            |
+| evaluate        | Evaluates a parser against examples and a supplied output-validation function. |
+| SequenceRecipe  | Types application-owned sequence supervision.                                  |
+| OutputDecoder   | Types a browser-safe sequence-to-output decoder.                               |
+
+The field classifier derives its field names and value domains from training outputs. The output schema still independently validates predictions. The sequence recipe owns tokenization, so it cannot also specify a separate input encoder.
+
+Tensor shapes, exported weight readers, and architecture-specific predictors belong to @matchbox-ai/core/internal. That subpath supports coordinated framework packages and research tests; application code should use the documented task, pipeline, and runtime interfaces.

package/docs/project-structure.md

@@ -0,0 +1,41 @@
+# Project structure
+
+A task is a directory under matchbox/. Each task has its own schema, explicit learning pipeline, training data, and independent evaluation data.
+
+```text
+my-app/
+  src/
+  scripts/generate-data.ts     # Optional project tooling.
+  matchbox/
+    money/
+      parser.ts
+      pipeline.ts
+      data/train.jsonl
+      evals/validation.jsonl
+      evals/test.jsonl
+      evals/baseline.ts        # Optional.
+      lib/                    # Optional application-owned helpers.
+  .matchbox/money/
+    model.matchbox
+    model.ts
+    model.d.matchbox.ts
+    report.json
+```
+
+The CLI discovers these paths. Multiple tasks require an explicit name; it never silently chooses the first one. Commands can also target a project directory, task directory, or explicit matchbox.config.ts. A task-local config may override paths; ordinary examples need only parser.ts and pipeline.ts.
+
+Generated artifacts are ignored. Validation gates packaging; test data does not select the model. Data generators live in the project-level scripts/ directory and write only matchbox/<task>/data/, preserving evals/ independently.
+
+## Optional task configuration
+
+Use `matchbox/<task>/matchbox.config.ts` when your files live outside the conventional layout. Paths resolve relative to that task directory:
+
+```ts
+export default {
+  train: "../../datasets/money-training.jsonl",
+  validation: "../../datasets/money-validation.jsonl",
+  eval: "../../datasets/money-test.jsonl",
+};
+```
+
+The config overrides paths and acceptance settings. Keep the encoder, supervision, prediction strategy, and decoder explicit in `pipeline.ts`. The current loader supports TypeScript configuration; there is no separate JSON config format.

package/docs/react.md

@@ -0,0 +1,16 @@
+# @matchbox-ai/core/react
+
+Load typed Matchbox parsers in React with `useMatchbox`.
+
+```tsx
+import { useMatchbox } from "@matchbox-ai/core/react";
+
+const loadFilters = () => import("./generated/filters.matchbox");
+
+function Search() {
+  const parser = useMatchbox(loadFilters);
+  return <input onChange={async (event) => console.log(await parser.parse(event.target.value))} />;
+}
+```
+
+The hook exposes loading, ready, and error state. Keep the loader outside the component. React is a peer dependency of this package; core and training do not depend on React. See the [end-to-end guide](end-to-end.md).

package/docs/releases.md

@@ -0,0 +1,50 @@
+# Releases
+
+Changesets versions and publishes three packages together:
+
+| Package              | Purpose                                      |
+| -------------------- | -------------------------------------------- |
+| `@matchbox-ai/core`  | Task contracts and browser runtime.          |
+| `@matchbox-ai/train` | Pipeline authoring and TensorFlow training.  |
+| `matchbox-ai`        | The CLI, scaffolding, and browser workbench. |
+
+The repository root, apps, and examples remain private. Packages use the MIT license. Initial versions are 0.0.0; the initial changeset proposes 0.1.0.
+
+## Normal workflow
+
+1. Run `bun run changeset` alongside a package change and commit its Markdown file.
+2. Merge the feature PR. GitHub Actions creates or updates a Version Packages PR with versions, changelogs, dependency ranges, and the Bun lockfile.
+3. Merge the version PR when the release is ready. The release workflow runs the full checks and browser tests, packs the packages, and publishes the verified archives to npm. Changesets creates package tags and GitHub releases.
+4. Retry a failed publication with Run workflow on `release.yml` from main. Changesets checks the registry and skips already published versions.
+
+The packages use a fixed version group while their APIs evolve. Internal dependency ranges are updated by Changesets. Run `bun run changeset status` to inspect the next release. Run `bun run release:pack` after building to inspect publishable archives without publishing.
+
+## npm and GitHub setup
+
+Ensure you control `matchbox-ai` and the `@matchbox-ai` scope. Enable **Allow GitHub Actions to create and approve pull requests** in the repository's Actions settings.
+
+For each npm package, configure a GitHub trusted publisher with:
+
+- Owner: `alexpatow`.
+- Repository: `matchbox`.
+- Workflow filename: `release.yml`.
+- Environment: leave blank.
+- Allowed action: enable direct `npm publish`.
+
+The workflow uses Node 24 and npm OIDC authentication. No npm token is stored in GitHub. Only the publish job receives `id-token: write`; build and validation run separately. The repository is currently private, so provenance is disabled for it. The workflow enables provenance if the repository becomes public.
+
+Trusted publisher settings are configured per existing npm package. If the names have not been published yet, bootstrap the first version from the merged Version Packages commit using an authenticated npm account with access to the names:
+
+```sh
+bun install --frozen-lockfile
+bun run check
+bun run test:browser
+npm login
+bun run release
+```
+
+Complete npm's authentication prompts, then configure the three trusted publishers for subsequent releases. Do not run `release` from a feature branch or publish version 0.0.0. Setting up this workflow does not itself claim npm names or configure their trusted publishers.
+
+The CLI declares its training and runtime dependencies so `bunx matchbox-ai init` can work before the app has any Matchbox packages installed. Scaffolding writes registry versions from the running release, preserving dependencies already present in the app.
+
+This follows [Changesets' automation guide](https://changesets.dev/guide/automating) and [npm's trusted publishing setup](https://docs.npmjs.com/trusted-publishers/).

package/docs/repository-design.md

@@ -0,0 +1,11 @@
+# Repository design
+
+Bun workspaces contain three framework packages. packages/core owns parser definitions, datasets, TensorFlow model loading, React, and Vite integration. packages/train owns explicit pipeline primitives, native training, evaluation, and packaging. packages/cli owns the command-line workflow, scaffolding templates, and browser workbench. It calls the train and evaluate APIs from @matchbox-ai/train and project loading from @matchbox-ai/train/project. Cross-package imports use exports. Core/internal is reserved for framework implementation, while runtime exports stay application-facing.
+
+Each examples/<example>/matchbox/<task> is independently trainable through the same CLI conventions. apps/playground hosts the React/Vite demo and imports generated artifacts from examples. apps/benchmarks measures browser inference separately. No training dependencies enter the playground browser graph.
+
+Core source is organized under parser/, runtime/tensorflow/, react/, vite/, dataset/, and internal/. Train source separates pipeline/, encoders/, models/, codecs/, evaluation/, packaging/, and project/. CLI source separates commands/ and workbench/, with the browser UI in packages/cli/workbench/. Public index.ts files only re-export. Existing token codecs remain application-owned.
+
+Rolldown builds ESM and TypeScript emits declarations. Public docs are copied into each package during builds for coding-agent access; root docs/ remains the source of truth. The skill lives in skills/matchbox/SKILL.md. Generated weights and copied package docs remain ignored.
+
+Run bun run check and bun run test:browser before proposing a PR. Do not merge without instruction. Preserve independent evaluation fixtures during migrations and training-data regeneration.

package/docs/runtime-backends.md

@@ -0,0 +1,18 @@
+# Browser runtime
+
+```ts
+import { matchbox } from "@matchbox-ai/core/vite";
+export default { plugins: [matchbox()] };
+```
+
+Matchbox uses TensorFlow.js CPU for browser inference. There is no runtime selector or handwritten JS evaluator. Artifacts carry TensorFlow model topology and named weights; the runtime loads them with loadLayersModel and executes model.predict. Matchbox adapts input features and output predictions, validates results, and supports abstention.
+
+Native TensorFlow trains the model. Before export completes, the toolchain captures native predictions and compares them with the serialized model loaded on TensorFlow.js CPU. This checks export fidelity without maintaining a second inference engine.
+
+TensorFlow loads lazily on the first parse. A parser retains its loaded model and exposes dispose(). Shared generated modules live for the application lifetime; the Vite plugin disposes them on HMR. Explicitly created instances should be disposed by their owner. React hooks do not dispose shared model modules when one component unmounts.
+
+Direct creation is available through @matchbox-ai/core/runtime. Inference selects TensorFlow's process-wide CPU backend. Browser apps that separately use TensorFlow should isolate unrelated backend use in another worker. Native training runs through the training toolchain and is kept out of browser entry points.
+
+Artifact format 2 includes the serialized TensorFlow model. Regenerate artifacts produced by earlier versions with matchbox-ai train. apps/benchmarks measures loading and warm inference of the shipped runtime. Cold timing includes lazy TensorFlow loading after the benchmark page itself has loaded. Mobile browser-test profiles emulate device settings on desktop hardware, not physical-phone performance.
+
+Call `await parser.load()` to initialize before the first parse, for example before going offline. The React hook waits for this initialization before reporting `ready`.

package/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@matchbox-ai/train",
+  "version": "0.1.0",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/alexpatow/matchbox.git",
+    "directory": "packages/train"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "docs",
+    "LICENSE",
+    "THIRD-PARTY-NOTICES.md",
+    "CHANGELOG.md"
+  ],
+  "type": "module",
+  "sideEffects": false,
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./project": {
+      "types": "./dist/project/index.d.ts",
+      "import": "./dist/project/index.js"
+    }
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "bun ../../scripts/package-docs.ts train && rolldown --config && tsc -p tsconfig.build.json",
+    "dev": "bun run --parallel \"dev:*\"",
+    "typecheck": "tsc --noEmit && tsc -p tsconfig.tools.json",
+    "dev:js": "rolldown --config --watch",
+    "dev:types": "tsc -p tsconfig.build.json --watch"
+  },
+  "dependencies": {
+    "@tensorflow/tfjs-node": "4.22.0",
+    "zod": "4.6.3"
+  },
+  "devDependencies": {
+    "@matchbox-ai/core": "0.1.0",
+    "@types/node": "22.20.2",
+    "rolldown": "1.2.8",
+    "typescript": "7.0.2"
+  },
+  "peerDependencies": {
+    "@matchbox-ai/core": "0.1.0"
+  },
+  "engines": {
+    "bun": ">=1.4.2",
+    "node": ">=24"
+  }
+}