pi-nebius 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,49 @@
+# Changelog
+
+Versions use Semantic Versioning. Entries describe changes included in the named version;
+a version is released only when its matching Git tag and GitHub release are published.
+
+## [0.3.0](https://github.com/PeterHdd/pi-nebius/compare/v0.2.1...v0.3.0) (2026-09-17)
+
+
+### Features
+
+* add persistent per-model Nebius settings ([d1b3114](https://github.com/PeterHdd/pi-nebius/commit/d1b31146abf17d54aa43d57f5c9080072cea78e6))
+* add persistent per-model Nebius settings ([b31dce1](https://github.com/PeterHdd/pi-nebius/commit/b31dce1087e41e41482a56634f3dc5f57213bae4))
+
+## [0.2.1](https://github.com/PeterHdd/pi-nebius/compare/v0.2.0...v0.2.1) (2026-09-17)
+
+
+### Bug Fixes
+
+* clarify auth benchmark requirements and expand visible test ([557271b](https://github.com/PeterHdd/pi-nebius/commit/557271b74b30e457cdeca7b3e3d491e84903eca4))
+* clarify auth benchmark requirements and expand visible test ([3d86156](https://github.com/PeterHdd/pi-nebius/commit/3d86156f41677cdd2672fe6b5196e44d2830b5c6))
+
+## [0.2.0](https://github.com/PeterHdd/pi-nebius/compare/v0.1.0...v0.2.0) (2026-09-17)
+
+
+### Features
+
+* add in-Pi benchmarks for custom prompts and bundled tasks ([615c800](https://github.com/PeterHdd/pi-nebius/commit/615c800975288242530a362ee7fd954e1ae08502))
+* add Nebius provider and coding benchmarks for Pi ([a770213](https://github.com/PeterHdd/pi-nebius/commit/a7702139910c11afec35235fea785f1b805ad478))
+* improve benchmark reporting and automate releases ([fbbca35](https://github.com/PeterHdd/pi-nebius/commit/fbbca35ecf46156616b74a4707d09f2fd85dbe8b))
+
+
+### Bug Fixes
+
+* **ci:** stabilize benchmark tests and target main branch only ([7ffc127](https://github.com/PeterHdd/pi-nebius/commit/7ffc127e90defa063fb6c66352385b2c9a3cfff1))
+* support Pi Git installs without dev dependencies ([18ec13b](https://github.com/PeterHdd/pi-nebius/commit/18ec13b6afba14d3948409e89a358aa4eae0eab4))
+
+## [0.1.0]
+
+Development baseline before automated GitHub releases.
+
+- Native Pi provider with authenticated Nebius model discovery and a private metadata cache.
+- Agentic benchmark CLI with deterministic validation, isolated runs, request/tool traces,
+  cumulative usage and four coding fixtures.
+- Credential-free integration tests using real Pi sessions and mocked provider responses.
+- GitHub CI, dependency updates, secret scanning, workflow validation, and automated changelogs and GitHub releases.
+- Filesystem checks reject overlapping validator paths and symlinked fixture roots/output aliases.
+
+Live Nebius acceptance remains outstanding. Supported baseline: Pi 0.85.1, Node 22.19+,
+macOS/Linux; benchmark execution is sequential.

package/CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing
+
+Use Node 22.19+ on macOS or Linux. Pi development dependencies are locked to the
+tested release; retain the lockfile and update both Pi packages together.
+
+```bash
+npm ci
+npm run check
+npm run version:check
+npm run security:check
+npm run package:check
+```
+
+`check` runs TypeScript, Biome, the build, and credential-free tests. Security checks
+download checksum-pinned official tools; package checks install a temporary tarball
+with the tested Pi peers. Both require network access. Tests need loopback sockets
+and permission to launch/stop subprocesses. Do not provide live API keys to CI.
+
+Keep changes focused and add regression tests for behavior changes. Use a descriptive
+commit message; no special commit syntax or automatic version inference is required.
+Update the changelog for user-visible changes. Report whether a test used mocks or
+live Nebius; a scripted tool loop is not a hosted-model benchmark.
+
+Never commit credentials, `.env` files, private fixtures, `node_modules`, or generated
+benchmark results. The secret scanner scans Git-visible files and history, so a
+credential committed and later deleted must still be revoked and removed from history.
+
+Release Please generates changelog/version updates in a release PR; merging it publishes
+the GitHub release after CI passes. Changes to workflow permissions,
+tool checksum pins, dependency lockfiles, and authentication deserve particular review.
+
+PR descriptions are maintained by the **PR description** workflow. Write meaningful
+commit subjects and optional bullet points in commit bodies: these populate Changes.
+Validation reports actual CI job/step outcomes for the latest PR commit and updates
+when CI runs. Add context or live-testing evidence under Notes, outside the
+`auto-pr` markers. The workflow preserves that text and leaves Release Please PRs
+alone. It uses the built-in GitHub token; no additional secret or AI service is needed.
+It becomes active after the workflow reaches the default branch. For existing PRs
+without markers, it appends its generated section without deleting your description.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 pi-nebius contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,239 @@
+# pi-nebius
+
+Nebius Token Factory models in Pi's normal model picker, discovered using your API key.
+
+This package also includes **benchmarks inside Pi**: run `/nebius-benchmark` to enter your own task prompt and compare models on copies of your current project. Results appear in Pi. A standalone CLI also supports scripted tasks and deterministic validation. See [Benchmarking](docs/benchmarking.md) for the CLI, result schema, metrics, and four ready-to-run fixtures.
+
+```bash
+npm ci
+npm run build
+npm run benchmark -- --benchmark benchmarks/fix-auth-bug \
+  --models 'EXACT_MODEL_ID_A,EXACT_MODEL_ID_B' --runs 3 \
+  --output benchmark-results/auth-comparison
+```
+
+Results appear directly in the terminal: each run prints its pass/fail status, followed by a comparison table of success counts, mean input/output tokens, median task duration, observed time to first token (TTFT), end-to-end output throughput, turns, and tool calls. Detailed JSON reports, traces, and workspaces are also saved under `benchmark-results/`; `--output` chooses the directory.
+
+Set `NEBIUS_API_KEY` first. `npm run benchmark:demo` exercises the real Pi runner with scripted model responses, without credentials.
+
+```text
+Pi → pi-nebius → https://api.tokenfactory.nebius.com/v1 → model
+```
+
+Requires **Node.js 22.19+ and Pi 0.85.1** (the tested version). This targets the current `@earendil-works` Pi packages, not older `@mariozechner` releases. Pi's package instructions require wildcard peer dependencies; that is not a claim that every Pi version is compatible.
+
+## Installation
+
+With Pi already installed:
+
+```bash
+pi install npm:pi-nebius
+export NEBIUS_API_KEY="your-api-key"
+pi
+```
+
+Run `/model`, search for `nebius`, and select a model.
+
+From this checkout:
+
+```bash
+cd /path/to/pi-nebius
+npm ci
+pi install /absolute/path/to/pi-nebius
+```
+
+Or try it without adding it to Pi's settings:
+
+```bash
+pi -e /absolute/path/to/pi-nebius
+```
+
+The provider uses Pi's `pi.extensions` manifest and loads TypeScript directly. Pi installs Git packages with `npm install --omit=dev`; the extension needs no build step. To use the benchmark CLI from a source checkout, run `npm ci` and `npm run build` to generate `dist/`. YAML is the benchmark definition parser; neither component needs a separate server.
+
+GitHub installation is also supported:
+
+```bash
+pi install git:github.com/PeterHdd/pi-nebius
+```
+
+Git installs require no compiler or development dependencies. Benchmark development uses a separate source checkout with development dependencies installed. Append a published tag (for example, `@v0.3.0`) to pin a Git installation; install a newer tag explicitly to upgrade.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md), [CHANGELOG.md](CHANGELOG.md), and [SECURITY.md](SECURITY.md).
+
+## Authentication and usage
+
+```bash
+export NEBIUS_API_KEY="your-key"
+pi
+```
+
+Inside Pi, run `/model` and search for `nebius`. Choose an exact model from the discovered list. Provider ID: `nebius`; display name: `Nebius Token Factory`. Nebius IDs such as `vendor/model` remain unchanged; Pi identifies the pair as `nebius/vendor/model`.
+
+You can also inspect the catalog and select a model explicitly:
+
+```bash
+pi --list-models nebius
+pi --provider nebius --model 'EXACT_ID_FROM_THE_LIST'
+```
+
+`/nebius-refresh` forces discovery again and updates Pi's registered models. If a selected model was removed, use `/model` to choose a current one.
+
+The extension reads `NEBIUS_API_KEY` at startup for discovery and resolves it through Pi's native authentication interface for inference. It has no `/login` flow and never writes the key to a file. Set the variable in the shell that launches Pi; changing a parent shell's environment cannot change an already-running process. Restart Pi after changing it. Pi's explicit CLI/runtime authentication overrides remain Pi features, but discovery specifically requires `NEBIUS_API_KEY`.
+
+## Benchmark inside Pi
+
+After installing the extension, run this inside Pi:
+
+```text
+/nebius-benchmark
+```
+
+An editor asks for your own task prompt. The benchmark uses your selected Nebius model and a copy of the project directory where Pi is running. To compare models, supply their exact IDs:
+
+```text
+/nebius-benchmark --models zai-org/GLM-5.3,moonshotai/Kimi-K2.6 --runs 3
+```
+
+This runs the same prompt three times per model, sequentially, starting from the same project snapshot each time. It shows progress and a comparison of input/output tokens, task duration, observed TTFT, output throughput, turns, and tools directly in Pi. Custom prompts report **correctness not checked**; completion does not prove the task was solved.
+
+The snapshot includes current working files, including uncommitted changes, and respects Git ignores. It excludes `.git`, `.pi`, dependency/build folders, prior benchmark results, `.env` files, and `.pem`/`.key` files. Dependencies are not preinstalled; include setup instructions in your task if needed. Links and special files are rejected. Snapshots are limited to 10,000 files / 50 MiB. Run Pi from the project directory you want to benchmark.
+
+Runs use paid inference. Tools have normal host permissions: file copies are not an OS sandbox. Reports and each run's resulting files are retained under `benchmark-results/` in your project.
+
+```text
+/nebius-benchmark cancel
+/nebius-benchmark help
+```
+
+For an optional bundled coding task with automatic correctness tests:
+
+```text
+/nebius-benchmark --task fix-auth-bug --runs 3
+```
+
+Other bundled tasks: `add-api-endpoint`, `refactor-module`, and `multi-file-feature`. No build step or separate terminal is needed for these Pi commands.
+
+## Dynamic model discovery
+
+The async extension factory calls authenticated `GET /v1/models?verbose=true` before Pi finishes startup. This is the documented verbose variant of `/v1/models`, supplying metadata in addition to IDs. There is **no bundled model catalog**.
+
+| Nebius field | Pi mapping |
+| --- | --- |
+| `id` | Exact, case-sensitive `model.id` |
+| `name` | Display name; defaults to ID |
+| `context_length` | `contextWindow`; defaults to 32,768 when absent/invalid |
+| `architecture.modality` | Text/image inputs; known non-text-output models are excluded |
+| `status` | Only `active` or unspecified models are included |
+| `supported_sampling_parameters` contains `reasoning_effort` | Enables reasoning and Pi's standard reasoning-effort control |
+| `supported_features` contains `reasoning` | Marks reasoning support without assuming an effort control |
+| No documented maximum output length | `maxTokens`: 4,096, capped at a quarter of the context window |
+
+Basic responses containing only IDs are supported. Missing capabilities default to text-only with no explicit reasoning control. Without modality metadata, the endpoint may include models unsuitable for chat; discovery cannot prove tool support. Pick a tool-capable model and run the integration test. Duplicate IDs are deduplicated; malformed lists are rejected rather than silently replacing a good cache with partial data.
+
+### Cache and offline fallback
+
+The cache is under `getAgentDir()/cache/pi-nebius/` (normally `~/.pi/agent/cache/pi-nebius/`). Pi's `PI_CODING_AGENT_DIR` override is respected. Files contain normalized metadata and a timestamp, never the API key. Names use a SHA-256 fingerprint of the endpoint and key to avoid sharing one account's catalog with another key.
+
+- Fresh metadata is used for 24 hours without a discovery request.
+- Stale/missing metadata triggers one request, with an 8-second timeout and a 4 MiB response limit.
+- Transient failures use stale metadata and display a warning. No cache means no discovered models; Pi continues running.
+- A 401/403 invalidates this key's cache. A fresh cache does not validate credentials; inference can still reject a revoked key.
+- Successful refresh replaces the catalog, including an authoritative empty list. Failed refresh never updates the cache timestamp.
+- Cache writes are atomic and best-effort, with private directory/file permissions. Concurrent Pi processes may each perform discovery; there is no background service or cross-process lock.
+
+Pi now has a shared native model store, but its general provider-level cache is not scoped by the discovery key. This small extension-owned cache makes key isolation explicit and guarantees discovery before model selection. It does not participate in Pi's general remote-catalog refresh; use `/nebius-refresh` or restart Pi.
+
+### Per-model settings
+
+Run `/nebius-model` to choose a discovered model, or `/nebius-model MODEL_ID` to open it directly.
+The menu saves temperature, reasoning effort, and maximum output tokens separately for each model.
+Blank numeric values or **Inherit** restore defaults; **Reset all overrides** clears that model's settings.
+Advanced settings let you override the context window and reasoning capability metadata. These do not
+change the server's capabilities. Use verified limits; the default output limit remains 4,096 tokens
+when the catalog does not provide a documented output limit.
+
+Temperature and reasoning-effort overrides are sent only when discovery advertises support.
+An advertised parameter does not guarantee every value is supported by every model; use its documented values.
+Saved request values take precedence over Pi's generated values (including `/thinking` for reasoning effort).
+Changes apply to new requests immediately and survive restarts in
+`~/.pi/agent/pi-nebius/model-settings.json` (respecting `PI_CODING_AGENT_DIR`).
+`/nebius-refresh` preserves saved overrides. External file edits require `/reload`.
+
+Both benchmark entry points use saved settings. In-Pi benchmarks snapshot them when the run starts,
+so changing the menu does not change an active comparison. Each run's `effectiveSettings` includes
+saved overrides, model limits, and `requestParameters`: the outgoing temperature, reasoning effort,
+output limit, and top-p where present, without prompt content. Omitted values mean the request did not
+specify them; server defaults are unknown. These parameters are in `results.json`, not the summary table.
+
+### Metadata overrides and user-defined fallback models
+
+Merge the `nebius` provider entry from [examples/models.json](examples/models.json) into `~/.pi/agent/models.json`, replacing the placeholder ID and limits with verified values. Do not overwrite unrelated providers. The extension must remain installed.
+
+Use `modelOverrides` to adjust discovered models and `models` to add explicit models that remain available when discovery is offline. Native providers are composed beneath these settings by Pi. No `apiKey` field is needed in this configuration. Explicitly configured models remain until you remove them, even if the server no longer lists them.
+
+
+For reasoning models, verify the model's Nebius-specific behavior before overriding `reasoning`, `thinkingLevelMap`, or `compat`. Pi supports model-specific template controls, but this extension does not guess them from model names. Always-on reasoning may still appear in Pi even when explicit effort selection is unavailable.
+
+## Architecture and compatibility
+
+`src/index.ts` is Pi's async extension factory. It registers a native provider with `pi.registerProvider(provider)`, adds `/nebius-refresh`, and supplies provider-specific error guidance through `message_end`.
+
+`src/provider.ts` uses Pi's `createProvider()` and its exported `openAICompletionsApi()` adapter. Pi owns request/message serialization, SSE parsing, multi-turn conversation state, function calls/results, usage accounting, abort handling, retries, and the tool loop. A small fetch decorator retains retry/rate-limit/request-ID headers on HTTP errors; successful response streams pass through unchanged.
+
+Compatibility settings select `system` messages and `max_tokens`, request streamed usage, and disable unsupported assumptions about `store`, strict tools, grammar tools, and reasoning effort. These are model-level properties and can be overridden through Pi.
+
+
+## Testing
+
+```bash
+npm ci
+npm run typecheck
+npm run lint
+npm test
+```
+
+Tests cover model parsing/mapping, discovery errors, missing credentials, cache behavior, native authentication, and Pi's actual extension loader. Adapter tests use Pi itself with mocked HTTP responses. The CLI integration test starts a local HTTP server, loads this package in the real Pi CLI, checks `--list-models`, executes a streamed `write` call, checks the generated file, and verifies that the next request contains the tool result. It also checks cache reuse and `models.json` composition. These tests do not require or use a real Nebius key.
+
+### Manual live integration test
+
+In a scratch directory, start Pi, select a tool-capable Nebius model with `/model`, then ask:
+
+> Create a file called nebius-test.txt containing 'Hello from Nebius'.
+
+Verify that Pi shows a successful write-tool execution, the file contains the expected text, and the model responds after receiving the tool result. Check Token Factory's usage view for the request. Ordinary text claiming the file was created is not sufficient.
+
+An opt-in automated equivalent runs a real Pi CLI in a temporary directory, limits its tools to `write`, requires an actual tool event and final response, checks exact file contents, and cleans up afterward:
+
+```bash
+export NEBIUS_API_KEY="your-key"
+export NEBIUS_MODEL="EXACT_TOOL_CAPABLE_MODEL_ID"
+npm run test:live
+```
+
+This makes paid inference requests and has a three-minute timeout. It is not part of `npm test`. See [docs/validation.md](docs/validation.md) for what has actually been run.
+
+## Troubleshooting
+
+| Problem | Action |
+| --- | --- |
+| Missing API key | Run `export NEBIUS_API_KEY="..."` in the launching shell and restart Pi. |
+| 401/403 | Token Factory rejected the key or project access. Check the key, permissions, and project; then restart Pi. |
+| Model unavailable / 404 | Run `/nebius-refresh`, then `/model`; remove obsolete explicit entries from `models.json`. |
+| 429 | Respect the preserved `retry-after` / rate-limit details. Pi handles its normal retry policy; discovery itself does not retry automatically. |
+| 5xx | Provider service failure. Retry later; discovery can use stale cached metadata. |
+| Connectivity / timeout | Check access to `api.tokenfactory.nebius.com` and Pi's network/proxy settings. Discovery times out after eight seconds. |
+| No models without a cache | Add an explicit verified model via the example configuration, or retry discovery once connectivity returns. |
+| Model talks but never uses tools | Verify that the chosen model supports functions on Nebius; run the live test or select another model. Discovery alone cannot certify this. |
+| Context/output limit errors | Set verified `contextWindow` and `maxTokens` in `modelOverrides`. Defaults are conservative assumptions, not guaranteed endpoint limits. |
+
+## Current limitations
+
+- Live Nebius inference has not been verified in the development environment because no API key was available. Mocked Pi integration tests prove adapter wiring and the tool loop, not every hosted model's capabilities.
+- Metadata availability and capability vocabulary vary. Reasoning/vision detection is conservative; model-specific thinking templates require explicit configuration and testing.
+- Maximum output limits are defaults; the verbose schema does not expose a documented equivalent of Pi's `maxTokens`.
+- Pi's adapter handles `stop`, `length`, and `tool_calls` finish reasons. Its generic stream options do not expose custom stop sequences; advanced extensions can use Pi's `onPayload` hook if needed. No new stop-sequence API is invented here.
+- The normal picker/listing integration is tested through Pi's CLI and registry. An interactive terminal screenshot/UI test was not performed.
+
+## License
+
+MIT; see [LICENSE](LICENSE).

package/SECURITY.md

@@ -0,0 +1,47 @@
+# Security policy
+
+This project accepts security fixes for the latest tagged 0.x release and
+the default branch. Older versions receive no separate backports.
+
+## Reporting
+
+When this repository is hosted on GitHub, use **Security → Report a vulnerability**
+if private vulnerability reporting is enabled. Do not put API keys, private code,
+or exploitable details into a public issue. If private reporting is unavailable,
+open an issue requesting a private contact without disclosing the vulnerability.
+There is no dedicated security mailbox or response-time guarantee yet.
+
+## Trust boundaries
+
+- Pi extensions and built-in tools execute with your host user's access. The benchmark's
+  file copies and separate processes are **not an OS sandbox**. Run untrusted tasks in
+  a disposable environment. Agents may access paths outside the workspace.
+- Benchmark definitions, setup commands, and validators are executable code: review them.
+  Restored validators prevent ordinary accidental replacement, not malicious host access.
+- The provider reads `NEBIUS_API_KEY`; its cache stores model metadata and a key fingerprint,
+  not the key. The normal interactive Pi process inherits your shell environment.
+- The benchmark sends its key over private parent/child IPC. Tool and validation subprocesses
+  receive an allowlisted environment without that key. Traces redact the known key and bearer
+  strings; arbitrary secrets in user fixtures or generated files cannot be reliably identified.
+- Network inference sends task context and tool results to Nebius. Follow your own data policy.
+- Cache/filesystem checks assume the same host user is trusted. They do not defend against
+  an attacker concurrently replacing filesystem entries or controlling the parent process.
+
+## Repository controls
+
+Test/security CI uses read-only tokens and SHA-pinned Actions, no live credentials, and ordinary
+`pull_request` triggers. After successful main-branch CI, Release Please updates a
+release PR or publishes the merged release. Its dedicated repository-scoped token
+is confined to the release workflow, which checks out no code and executes no
+repository build scripts. npm publication is disabled with `private: true`.
+
+`npm run security:check` verifies pinned official Gitleaks/actionlint binary checksums,
+scans Git-visible first-party files and available commit history with redacted output,
+and validates workflows. It downloads executable tools from GitHub. `npm audit` checks
+known dependency advisories separately. Neither scan proves the absence of vulnerabilities.
+
+See [the review record](docs/security-review.md) for findings and residual limitations.
+
+The PR-description workflow uses `pull_request_target` and `workflow_run` only to
+read GitHub metadata and update the PR body with a scoped write token. It never
+checks out PR code, installs dependencies, or consumes workflow artifacts.

package/benchmarks/add-api-endpoint/benchmark.yaml

@@ -0,0 +1,13 @@
+schemaVersion: 1
+name: add-api-endpoint
+task: |
+  Implement GET /api/sum in app.mjs, preserving GET /health. Accept query parameters
+  a and b as finite numbers and return JSON {sum: a+b} with status 200. Missing,
+  blank, nonnumeric, infinite inputs or non-finite sums must return 400. Other
+  methods on /api/sum return 405. Unknown routes return 404. Keep the existing
+  Fetch Request/Response interface and do not change tests.
+validation:
+  - command: node
+    args: [--test, "{validation}/check.test.mjs"]
+timeout: 120
+validationTimeout: 15

package/benchmarks/add-api-endpoint/fixture/app.mjs

@@ -0,0 +1,6 @@
+export function handle(request) {
+  if (new URL(request.url).pathname === "/health" && request.method === "GET") {
+    return Response.json({ ok: true });
+  }
+  return Response.json({ error: "Not found" }, { status: 404 });
+}

package/benchmarks/add-api-endpoint/fixture/app.test.mjs

@@ -0,0 +1,8 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { handle } from "./app.mjs";
+
+test("health", async () =>
+  assert.deepEqual(await handle(new Request("http://local/health")).json(), { ok: true }));
+test("sum endpoint", async () =>
+  assert.deepEqual(await handle(new Request("http://local/api/sum?a=2&b=3")).json(), { sum: 5 }));

package/benchmarks/add-api-endpoint/fixture/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "api-fixture",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test"
+  }
+}

package/benchmarks/add-api-endpoint/validation/check.test.mjs

@@ -0,0 +1,37 @@
+import assert from "node:assert/strict";
+import { resolve } from "node:path";
+import { test } from "node:test";
+import { pathToFileURL } from "node:url";
+
+const { handle } = await import(pathToFileURL(resolve("app.mjs")));
+test("HTTP endpoint contract", async () => {
+  assert.deepEqual(await handle(new Request("http://local/health")).json(), { ok: true });
+  for (const [a, b] of [
+    [2, 3],
+    [-4, 1.5],
+    [0, 0],
+    [100, -100],
+  ]) {
+    const response = await handle(new Request(`http://local/api/sum?a=${a}&b=${b}`));
+    assert.equal(response.status, 200);
+    assert.match(response.headers.get("content-type"), /application\/json/);
+    assert.deepEqual(await response.json(), { sum: a + b });
+  }
+  for (const query of [
+    "",
+    "a=1",
+    "b=2",
+    "a=&b=2",
+    "a=%20&b=2",
+    "a=no&b=2",
+    "a=Infinity&b=1",
+    "a=1e308&b=1e308",
+  ]) {
+    assert.equal((await handle(new Request(`http://local/api/sum?${query}`))).status, 400);
+  }
+  assert.equal(
+    (await handle(new Request("http://local/api/sum?a=1&b=2", { method: "POST" }))).status,
+    405,
+  );
+  assert.equal((await handle(new Request("http://local/missing"))).status, 404);
+});

package/benchmarks/fix-auth-bug/benchmark.yaml

@@ -0,0 +1,14 @@
+schemaVersion: 1
+name: fix-auth-bug
+task: |
+  Fix refresh-token expiry handling in auth.mjs. A token is valid strictly before
+  its expiresAt timestamp (Unix seconds); nowMs is in milliseconds. Both expiresAt
+  and nowMs must be finite numbers: reject NaN, Infinity, -Infinity, missing values,
+  and strings without coercing them to numbers. Reject revoked and missing tokens.
+  Run the existing tests and ensure both timestamp inputs meet these requirements.
+  Do not change the tests.
+validation:
+  - command: node
+    args: [--test, "{validation}/check.test.mjs"]
+timeout: 120
+validationTimeout: 15

package/benchmarks/fix-auth-bug/fixture/auth.mjs

@@ -0,0 +1,4 @@
+export function canRefresh(token, nowMs) {
+  if (!token || token.revoked) return false;
+  return token.expiresAt > nowMs;
+}

package/benchmarks/fix-auth-bug/fixture/auth.test.mjs

@@ -0,0 +1,26 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { canRefresh } from "./auth.mjs";
+
+test("unexpired refresh token", () => {
+  assert.equal(canRefresh({ expiresAt: 2000, revoked: false }, 1_999_000), true);
+});
+test("expiry boundary", () => {
+  assert.equal(canRefresh({ expiresAt: 2000, revoked: false }, 2_000_000), false);
+});
+
+test("expired, revoked, and missing tokens", () => {
+  assert.equal(canRefresh({ expiresAt: 2000 }, 2_000_001), false);
+  assert.equal(canRefresh({ expiresAt: 2000, revoked: true }, 1000), false);
+  assert.equal(canRefresh(null, 1000), false);
+});
+
+for (const invalid of [NaN, Infinity, -Infinity, undefined, "2000"]) {
+  test(`rejects invalid expiresAt: ${String(invalid)}`, () => {
+    assert.equal(canRefresh({ expiresAt: invalid }, 1000), false);
+  });
+
+  test(`rejects invalid nowMs: ${String(invalid)}`, () => {
+    assert.equal(canRefresh({ expiresAt: 2000 }, invalid), false);
+  });
+}

package/benchmarks/fix-auth-bug/fixture/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "auth-fixture",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test"
+  }
+}

package/benchmarks/fix-auth-bug/validation/check.test.mjs

@@ -0,0 +1,25 @@
+import assert from "node:assert/strict";
+import { resolve } from "node:path";
+import { test } from "node:test";
+import { pathToFileURL } from "node:url";
+
+const { canRefresh } = await import(pathToFileURL(resolve("auth.mjs")));
+test("refresh contract across boundaries and invalid inputs", () => {
+  assert.equal(canRefresh({ expiresAt: 2000 }, 1_999_999), true);
+  assert.equal(canRefresh({ expiresAt: 2000 }, 2_000_000), false);
+  assert.equal(canRefresh({ expiresAt: 2000 }, 2_000_001), false);
+  assert.equal(canRefresh({ expiresAt: 2000, revoked: true }, 1000), false);
+  assert.equal(canRefresh(null, 1000), false);
+  for (const bad of [NaN, Infinity, -Infinity, undefined, "2000"]) {
+    assert.equal(
+      canRefresh({ expiresAt: bad }, 1000),
+      false,
+      `expiresAt must be a finite number; received ${String(bad)}`,
+    );
+    assert.equal(
+      canRefresh({ expiresAt: 2000 }, bad),
+      false,
+      `nowMs must be a finite number; received ${String(bad)}`,
+    );
+  }
+});

package/benchmarks/multi-file-feature/benchmark.yaml

@@ -0,0 +1,15 @@
+schemaVersion: 1
+name: multi-file-feature
+task: |
+  Add task completion across store.mjs, routes.mjs and serialize.mjs. New tasks
+  start with completed:false. Export setCompleted(id, completed) from store.mjs;
+  update the stored task and return it, or return null for an unknown ID.
+  Expose PATCH /tasks/:id accepting JSON {completed:boolean}, returning the
+  updated serialized task with status 200, 400 for invalid JSON/nonboolean values,
+  or 404 for missing tasks. Serialization must include completed. Preserve GET /tasks
+  and POST /tasks. Completion must persist into subsequent GET results.
+validation:
+  - command: node
+    args: [--test, "{validation}/check.test.mjs"]
+timeout: 180
+validationTimeout: 15

package/benchmarks/multi-file-feature/fixture/package.json

@@ -0,0 +1,8 @@
+{
+  "name": "multi-file-fixture",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test"
+  }
+}

package/benchmarks/multi-file-feature/fixture/routes.mjs

@@ -0,0 +1,11 @@
+import { serialize } from "./serialize.mjs";
+import { create, list } from "./store.mjs";
+export async function handle(request) {
+  const path = new URL(request.url).pathname;
+  if (path === "/tasks" && request.method === "GET") return Response.json(list().map(serialize));
+  if (path === "/tasks" && request.method === "POST") {
+    const { title } = await request.json();
+    return Response.json(serialize(create(title)), { status: 201 });
+  }
+  return Response.json({ error: "Not found" }, { status: 404 });
+}

package/benchmarks/multi-file-feature/fixture/routes.test.mjs

@@ -0,0 +1,14 @@
+import assert from "node:assert/strict";
+import { test } from "node:test";
+import { handle } from "./routes.mjs";
+
+test("created task starts incomplete", async () => {
+  const response = await handle(
+    new Request("http://local/tasks", {
+      method: "POST",
+      body: JSON.stringify({ title: "A task" }),
+    }),
+  );
+  assert.equal(response.status, 201);
+  assert.equal((await response.json()).completed, false);
+});

package/benchmarks/multi-file-feature/fixture/serialize.mjs

@@ -0,0 +1,3 @@
+export function serialize(task) {
+  return { id: task.id, title: task.title };
+}

package/benchmarks/multi-file-feature/fixture/store.mjs

@@ -0,0 +1,10 @@
+const tasks = [];
+let nextId = 1;
+export function create(title) {
+  const task = { id: String(nextId++), title };
+  tasks.push(task);
+  return task;
+}
+export function list() {
+  return tasks;
+}