pi-nebius 0.3.0

package/docs/benchmarking.md

@@ -0,0 +1,253 @@
+# Agentic coding benchmarks with Pi + Nebius
+
+This tool measures how a model completes a coding task **through Pi's full agent/tool loop**. It observes Pi; it does not implement an agent, alter model payloads, compress context, rewrite tool results, optimize prompts, or route between models.
+
+A traditional inference benchmark measures `prompt → model → response`. Here the unit is:
+
+```text
+task → Pi → model → tool → model → tool → … → deterministic validation
+```
+
+Tokens/second alone cannot describe task efficiency. A slower model that solves a task in five turns can finish sooner than a faster model that needs twenty turns. Results therefore expose success, time, cumulative usage, and tools separately. There is no composite score or LLM judge.
+
+## Run inside Pi
+
+Use `/nebius-benchmark --models ID_A,ID_B --runs 3` to enter your own prompt in an editor. Without `--models`, it uses the currently selected Nebius model. Each run starts from a snapshot of the current project. Results appear in Pi; `/nebius-benchmark cancel` stops the run. See [the README](../README.md#benchmark-inside-pi) for snapshot exclusions and limits.
+
+Custom prompts have `definition.validationMode: "none"`, `validation.checked: false`, and no correctness verdict. `success` remains false because no validator established success; aggregate `successRate` is null. A null `failure` means execution finished without an error, not that the task was solved. The report labels this FINISHED and explicitly says correctness was not checked. Existing validated task reports keep their success semantics.
+
+Use `--task fix-auth-bug` (or another bundled task name) to run a task with deterministic validators instead. Both paths run on the installed host Pi SDK without a compiler or development dependencies.
+
+## Standalone CLI: install and run
+
+Requirements: **macOS or Linux**, **Node 22.19+**, and the tested **Pi 0.85.1** packages. The source checkout pins development versions; retain `package-lock.json` for reproducibility.
+
+```bash
+cd /path/to/pi-nebius
+npm ci
+npm run build
+export NEBIUS_API_KEY="your-key"
+
+npm run benchmark -- \
+  --benchmark benchmarks/fix-auth-bug \
+  --models 'EXACT_NEBIUS_MODEL_A,EXACT_NEBIUS_MODEL_B' \
+  --runs 3 \
+  --timeout 120 \
+  --output benchmark-results/auth-experiment
+```
+
+Alternatively, after `npm install --global .` from this checkout, use `pi-nebius benchmark ...`. The package is not published to npm. The compiled command can also be invoked with `node dist/benchmark/cli.js benchmark ...`.
+
+Model IDs are exact Nebius IDs such as `vendor/model`, without an extra `nebius/` prefix. The CLI performs one fresh authenticated discovery before the experiment and freezes the selected model definitions. It does not auto-select a model. Discovery and model availability are outside the measured runs. Models absent from discovery fail preflight explicitly.
+
+| Option | Meaning |
+| --- | --- |
+| `--benchmark` | Directory containing `benchmark.yaml`, or a YAML/JSON file |
+| `--models` | Comma-separated exact IDs, each used for every repetition |
+| `--runs` | Repetitions per model; default 1 |
+| `--timeout` | Agent-process deadline in seconds; overrides the definition |
+| `--output` | New directory; existing directories are rejected |
+| `--concurrency` | **1 only in v1**, for identical unmodified Pi system prompts |
+
+Runs are scheduled round-robin: A1, B1, A2, B2. Ctrl-C stops the active session, retains its partial measurements, writes a cancelled result set, and does not start pending runs. Exit codes: 0 when all runs pass, 1 when any run fails, 2 for configuration/preflight errors, 130 for cancellation. Input/configuration errors before a run begins do not create a complete result set.
+
+### Why v1 is sequential
+
+Pi embeds an absolute working directory in its system prompt. Different simultaneous workspace paths therefore produce different prompts even with identical SDK settings. Rather than modify that prompt, this runner creates each fresh copy at **the same active path**, archives the finished workspace, removes the active copy, and only then starts the next run. The complete prompt hash is recorded and tested for equality within an experiment.
+
+Parallel runs would need an additional isolation layer exposing the same absolute path in each process namespace. That is not implemented in v1. Even with that layer, concurrency can distort latency through provider load, rate limits, shared serving infrastructure, and local CPU contention. Sequential execution is the baseline for latency comparisons. Absolute paths—and therefore prompt hashes—can differ between separate experiments; compare the recorded metadata.
+
+## Architecture
+
+```text
+CLI / runner
+  ├─ parse benchmark definitions
+  ├─ freeze fixture + trusted validators
+  └─ for each model/repetition:
+       fresh active workspace + private Pi config
+         → separate Node worker
+         → createAgentSession() + existing Nebius provider
+         → normal Pi agent/tools
+         → stop worker and tool descendants
+         → deterministic validators
+         → archived files + trace + run.json
+  → results.json + aggregate terminal report
+```
+
+Pi's public `createAgentSession`, `DefaultResourceLoader`, `SettingsManager.inMemory`, `ModelRuntime`, and `SessionManager.inMemory` establish each session. A native provider is registered through an inline extension, using the existing `nebiusProvider` adapter. `session.prompt`, `session.subscribe`, and `session.abort` drive and observe it. `before_agent_start` records the system-prompt hash and returns nothing. Instrumentation never returns a modified event/payload.
+
+The provider's supported `fetch` injection observes every Chat Completions HTTP attempt, including adapter retries and Pi's own compaction requests. It forwards request arguments and response bytes unchanged. The observer reads only usage and small response metadata fields from the streaming protocol; Pi still parses content, constructs tools, executes them, and manages the conversation.
+
+No user-installed extensions, skills, themes, prompt templates, context files, saved sessions, or `models.json` settings enter a run. This defines the clean benchmark configuration, identically for all models. Pi's own default retries and automatic compaction remain **unchanged**, are recorded, and their requests count toward token usage. There is no benchmark-added compression/pruning. Pi can clamp its default thinking level to a model's capabilities; the effective setting and model definition are recorded rather than forced into unsupported behavior.
+
+## Definitions and fixtures
+
+```text
+benchmarks/my-task/
+  benchmark.yaml
+  fixture/             # the agent's starting files
+  validation/          # trusted checks, outside its workspace
+```
+
+```yaml
+schemaVersion: 1
+name: my-task
+task: |
+  Find and fix the bug. Preserve the public API.
+fixture: fixture
+validationDirectory: validation
+tools: [read, bash, edit, write]
+timeout: 600
+validationTimeout: 60
+validation:
+  - command: node
+    args: [--test, "{validation}/check.test.mjs"]
+```
+
+The parser rejects unknown fields, duplicate YAML keys, aliases, invalid deadlines, and paths escaping the definition directory. `systemPrompt` is optional; when omitted the normal Pi prompt is used. Supplying it is an explicit benchmark configuration applied equally to all runs—not a per-model optimization.
+
+Validation accepts command/args objects (recommended), or strings such as `npm test` executed through `/bin/sh -c`. `{validation}` expands to the restored trusted-validator directory. Avoid shell-string interpolation for paths with spaces; use argument arrays. Validation runs with the agent workspace as cwd. All commands must exit zero. Output, exit code, signal, elapsed time, timeout, and truncation state are recorded; output is capped at 256 KiB per command.
+
+Optional `setup` commands run on each fresh copy before Pi starts, with `validationTimeout` as their individual deadline. For dependency installs, supply lockfiles and use reproducible commands such as `npm ci`. The included fixtures have **no external dependencies or setup**. Setup duration is included in run wall time, not agent wall time. The agent deadline excludes validation; each validator has its own deadline.
+
+The fixture snapshot excludes `.git` metadata, rejects symlinks/special files and `.env` credential files, preserves executable file bits and empty directories, and is SHA-256 hashed. Files are copied, never hard-linked. `node_modules` is not silently excluded: prefer setup commands to committing dependency trees. Generated files with links/special entries may prevent archival/hashing; any failure is recorded explicitly.
+
+Trusted validators are copied from the original snapshot after the agent exits, and their hash is checked before and after validation. Agents may run the fixture's visible tests while working; modifying those tests does not replace the trusted checks. Generic `npm test` commands still depend on the workspace's scripts/tests, so use external validators when test tampering would matter.
+
+**Filesystem isolation is not an OS security sandbox.** Pi's ordinary tools retain host access. Copies prevent ordinary run-to-run contamination; they do not constrain a deliberately escaping shell command. Use a disposable machine/container for untrusted tasks. v1 does not implement a container orchestrator. The API key goes to the worker over IPC, remains in memory, and is absent from tool/validation environments, arguments, and configuration files.
+
+### Included suite
+
+| Benchmark | Task and trusted checks |
+| --- | --- |
+| `fix-auth-bug` | Correct seconds/milliseconds expiry handling, boundary behavior, revocation, and invalid timestamps |
+| `add-api-endpoint` | Add `/api/sum` with numeric validation, method handling, JSON responses, and preserved health route |
+| `refactor-module` | Extract a shared subtotal helper while preserving calculations and avoiding duplicated reductions |
+| `multi-file-feature` | Persist task completion across store, serializer, and PATCH/GET routes |
+
+Tests verify all four original fixtures fail trusted validation and reference solutions pass. Refactor checks include a narrow structural assertion in addition to behavior. These are small deterministic acceptance tests, not a claim to detect arbitrary adversarial solutions or prove general correctness.
+
+## Usage and cumulative tokens
+
+Nebius usage is authoritative. The runner records each response's `prompt_tokens`, `completion_tokens`, optional `prompt_tokens_details.cached_tokens`, and optional `completion_tokens_details.reasoning_tokens`. It does not run a local tokenizer. Repeated cumulative snapshots within one stream replace the request's snapshot; they are not added together.
+
+```text
+Request       Input     Output
+1             4,200        800
+2             9,100      1,200
+3            17,300      1,100
+4            29,400      2,000
+TOTAL        60,000      5,100
+```
+
+The final submitted prompt had 29,400 tokens, but the agent consumed **60,000 input tokens** across requests. `prompt_tokens` includes cached input; adding cached tokens again would double-count it. Likewise, reasoning tokens are a subset of completion tokens and must not be added again.
+
+`tokens.cumulativeInputTokens` and `tokens.cumulativeOutputTokens` sum every physical request with complete reported usage, including internal summarization/compaction. If any request lacks the required usage fields, the corresponding total is `null`. `observedInputTokens`, `observedOutputTokens`, `requestsWithUsage`, and `usageComplete` preserve the known partial measurement. Failed requests, disconnected streams, and timeouts are not assumed free. Unreported cached/reasoning fields remain `null`, not zero.
+
+### Amplification and context size
+
+`lastRequestInputTokens` is the provider-reported input size of the last **agent** request, excluding compaction requests. It is not the final conversation size after the model's last output. Exact final context size is unavailable without another provider measurement or local estimation, so `finalContextSizeTokens` and the requested final-context-based `inputAmplification` are explicitly `null`.
+
+The separately named descriptive ratio is:
+
+```text
+inputAmplificationVsLastRequest = cumulativeInputTokens / lastRequestInputTokens
+```
+
+It describes total input sent relative to the last submitted agent prompt, only when the numerator and denominator are known and the denominator is positive. For the example: `60,000 / 29,400 ≈ 2.04×`. Compaction, caching, model tokenizers, and retries affect this ratio; it is not a measure of wasted tokens or a score.
+
+## Metric boundaries
+
+| Metric | Measurement |
+| --- | --- |
+| `success` | All configured validation passes, with no timeout/cancellation or unrecovered agent/API error |
+| `modelRequests` | Observed physical Chat Completions attempts, including retries |
+| `agentTurns` | Pi `turn_start` events; can differ from request count |
+| `toolCalls` | Tool calls declared in completed assistant messages |
+| `toolErrors` | Tool results marked `isError`; a recovered tool error does not by itself fail a run |
+| Individual tools | Name/ID, execution start/end, error flag and redacted failure details; no arguments or successful outputs |
+| `wallTimeMs` | From fresh-run setup through Pi, validation and archival; excludes initial discovery/snapshot and final report persistence |
+| `agentWallTimeMs` | Worker spawn through shutdown, including Pi initialization and cancellation cleanup |
+| `modelRequestWallTimeMs` | Union of completed client-observed request intervals; null when an observed request has no endpoint |
+| `toolExecutionTimeMs` | Union of completed tool-execution intervals; null when an execution is still open |
+| `timeToFirstContentMs` | First observed text/reasoning/tool delta relative to agent start; a client-visible latency, not exact first-token timing |
+| Per-request timing | Monotonic worker-relative start, first content, and completion; differences provide request latencies |
+| `modelGenerationTimeMs` | Null: server generation cannot be separated from queuing, networking, and stream consumption |
+| `agentOverheadTimeMs` | Null: subtracting potentially overlapping timings would not isolate pure agent overhead |
+
+Instrumentation itself has overhead, including IPC and trace writes. The HTTP observer is backpressure-aware and byte-preserving, but its timing is not server telemetry. Interrupted tool executions may lack complete duration/error information. Counts always describe observed events, not unseen activity before an unresponsive worker was killed.
+
+Failure categories include `validation_failed`, `timeout`, `model_api_error`, `rate_limit`, `tool_error`, `agent_error`, `context_limit`, `cancelled`, and `unknown`. Timeout/cancellation take priority. `tool_error` means failed validation accompanied by tool errors; it does not prove the errors caused the failure. Original redacted details and individual HTTP statuses remain available. Recovered attempt failures remain in the trace even when the task ultimately succeeds.
+
+## Results and reproducibility
+
+```text
+OUTPUT/
+  results.json                    # versioned experiment + all runs + aggregates
+  snapshot/fixture/               # frozen inputs
+  snapshot/validation/            # frozen trusted checks
+  runs/001-MODEL_HASH/
+    run.json                      # individual result, including request/tool arrays
+    trace.jsonl                   # incremental observation snapshots; survives worker failure
+    workspace/                    # archived agent files
+    validation/                   # exact checks executed
+    pi/                           # isolated Pi config, without credentials or saved conversation
+```
+
+`schemaVersion: 2` applies to result documents and journal entries. Each run includes:
+
+```json
+{
+  "schemaVersion": 2,
+  "benchmark": "fix-auth-bug",
+  "model": "vendor/model",
+  "modelRevision": null,
+  "run": 1,
+  "success": true,
+  "failure": null,
+  "modelRequests": 2,
+  "agentTurns": 2,
+  "toolCalls": 1,
+  "toolErrors": 0,
+  "tokens": {
+    "cumulativeInputTokens": 320,
+    "cumulativeOutputTokens": 60,
+    "lastRequestInputTokens": 220,
+    "finalContextSizeTokens": null
+  },
+  "observation": {
+    "requests": [
+      {"request": 1, "usage": {"inputTokens": 100, "outputTokens": 40}},
+      {"request": 2, "usage": {"inputTokens": 220, "outputTokens": 20}}
+    ]
+  }
+}
+```
+
+This is an abridged illustrative schema, not a live model result. The complete TypeScript contract is [types.ts](../src/benchmark/types.ts).
+
+Metadata includes timestamp, Pi/package/Node versions, OS/architecture, benchmark definition/hash, fixture hash (including paths/content/executable bits), validator hash, frozen model definitions, effective Pi settings, system-prompt hashes. `.git` history is not required: the fixture content hash identifies the actual inputs. Response `model`, request ID, and `system_fingerprint` are captured when present. A fingerprint is **not** asserted to be a model revision; `modelRevision` remains null because Nebius's inspected catalog has no documented exact revision identifier.
+
+The terminal report places metrics down rows and models across columns. It shows success counts, median task duration, mean input/output tokens, turns and tools, observed TTFT, end-to-end output throughput, and failure diagnostics. Observed TTFT is the first agent request’s first content timestamp minus that request’s start timestamp, aggregated as the median across runs with known timing. It includes network latency but excludes worker startup; the first nonempty text, reasoning, or tool-function delta counts, while role-only and empty tool headers do not. Streaming chunks may contain multiple tokens, so this is a client-observed approximation of TTFT, not a server token-generation timestamp. Throughput is the sum of output tokens divided by the sum of complete task durations, including network, tools, and validation. It includes failed runs and is unknown if any run lacks output usage or a positive duration. This is not pure model decoding speed. Existing JSON request traces contain the timestamps needed to calculate observed TTFT. JSON also contains means, medians, min/max, population standard deviations, and known/missing counts. Failed runs remain in aggregates; unknown values are excluded from arithmetic, never converted to zero. Every individual result remains available. `results.json` is atomically updated after each completed run and on handled cancellation. An abrupt kill of the runner itself may leave status `running`; journals preserve observations already received, but resumable execution is not implemented.
+
+Traces omit prompts, model text, tool arguments, and successful tool output. Error details and validator logs are retained with the known API key redacted. This cannot identify arbitrary secrets embedded in your own fixtures/logs; use credential-free fixtures. Workspace archives intentionally contain the task's output files.
+
+## Verification and live acceptance
+
+```bash
+npm ci
+npm run typecheck
+npm run lint
+npm test
+npm run benchmark:demo
+```
+
+The demo runs real Pi SDK sessions and file tools, with **scripted mock responses**. It exercises two mock models × two repetitions and leaves inspectable results in `benchmark-results/mock-demo-*`. It is not evidence about any Nebius-hosted model's performance.
+
+The regression suite covers parsing, isolation, event instrumentation, fragmented SSE observation, cumulative usage, aggregation, validation, JSON round trips, API errors, graceful cancellation, hard timeout, CLI input handling, all four fixture validators, and real Pi tool loops. Provider-extension tests also remain in the suite.
+
+No live benchmark has been run in this environment: `NEBIUS_API_KEY` was unavailable. To complete that check, choose two tool-capable IDs from your discovered catalog and run the first command above with `--runs 1`. Tests certify the measurement wiring, not the capabilities, stability, or billing behavior of every hosted model.
+
+See [benchmark-research.md](benchmark-research.md) for inspected source APIs and Nebius methodology.
+
+Result schema version 2 removes monetary fields from version 1 (per-run estimates, pricing snapshots, aggregate costs, and the pricing hash). Existing saved results are not rewritten. Task definitions still use schema version 1.

package/docs/security-review.md

@@ -0,0 +1,49 @@
+# Local security and release review
+
+Reviewed 2026-09-16 on macOS, Node 22.22.3, Pi 0.85.1. Scope: first-party provider,
+discovery/cache, benchmark runner/worker, process execution, parsing, filesystem
+isolation, credential handling, dependencies, packaging, and workflows. This targeted
+engineering review is not an independent penetration test or audit certification.
+
+## Findings addressed
+
+| Finding | Resolution | Evidence |
+| --- | --- | --- |
+| Different fixture/validator path strings could alias or nest | Compare resolved directories and reject overlap | Tests for relative aliases, symlink aliases and nesting |
+| Output parent symlinks could bypass lexical containment checks | Resolve existing ancestors before creating directories | Test for fixture alias with nonexistent output descendants |
+| Fixture root could itself be a symlink | Require a real directory at the copy/hash root | Root-link regression test |
+| `.env` variants and common key files were not ignored | Ignore `.env.*`, `*.pem`, `*.key`, logs and OS metadata | Git-visible source inspection |
+| No repository CI/release controls | SHA-pinned read-only CI, dependency updates, security scans, version consistency, release PRs and automated releases | Local workflow/version validation |
+| Accidental npm publication was possible | `private: true`, enforced by version checker; no publish workflow | Package metadata check |
+
+These path fixes protect against static aliases/configuration mistakes, not a malicious
+process concurrently changing files or using Pi's shell tools outside the workspace.
+
+## Executed verification
+
+- `npm audit --json`: zero known vulnerabilities, including development dependencies.
+- Gitleaks 8.30.1: no findings in Git-visible first-party files, including untracked files.
+  There are no commits, so history scanning was explicitly skipped. CI fetches full history.
+- actionlint 1.7.12: workflow structure and expressions passed. Optional ShellCheck/Pyflakes
+  integration is disabled consistently; those separate tools were not run.
+- Security-tool binaries were downloaded from official releases and SHA-256 verified.
+- Type checking, Biome lint, build, and 35 tests passed; none skipped.
+- A clean tarball install with Pi 0.85.1 peers passed compiled extension import and CLI checks.
+- Manifest/lockfile/changelog agreement and matching/mismatched tag checks passed.
+- New-file whitespace checks reported no errors.
+
+## Remaining boundaries
+
+Pi tools, setup commands and validators retain host access. Workspace copies are not an
+OS sandbox; use a disposable environment for untrusted tasks. A same-user attacker can
+escape copies or change trusted files. See [SECURITY.md](../SECURITY.md).
+
+Known credentials are redacted from benchmark traces, but arbitrary secrets in fixtures,
+errors, validation logs and archived files cannot be identified reliably. Normal interactive
+Pi inherits its shell environment. Dependency install scripts execute during source installs.
+
+Scanners only cover known patterns/advisories; clean output is not proof of security.
+No live Nebius acceptance was possible without credentials. GitHub Actions have not run
+on GitHub; the configured OS/Node matrix is not claimed as remotely verified. No repository,
+commit, tag, release or npm publication was created. Branch/tag rules, push protection and
+private reporting must be enabled after repository creation.

package/docs/validation.md

@@ -0,0 +1,51 @@
+# Validation record
+
+Environment: macOS, Node.js **22.22.3**, published Pi / pi-ai **0.85.1**. Date: **2026-09-16**.
+
+## Executed checks
+
+- `npm install`: dependencies installed; npm audit reported zero vulnerabilities.
+- `npm run typecheck`: passed against the published Pi types.
+- `npm run lint`: passed (Biome recommended rules and formatting).
+- `npm test`: **35 tests passed**, no skipped tests (19 provider tests and 16 benchmark tests). Requires permission to bind a local loopback socket.
+- `npm pack --dry-run --json`: passed; package includes compiled CLI, extension source, four benchmark fixtures and validators, README, documentation, example configuration, and MIT license; excludes test transport, node_modules, and credentials.
+- New-file diffs and package contents reviewed. This is a new standalone repository; no release was published.
+
+## What the tests establish
+
+The production provider delegates to Pi's actual adapter. Mocked HTTP adapter tests cover streaming text, system messages, temperature, maximum output tokens, usage totals, reasoning effort/content/replay, finish reasons, HTTP 401/404/429/500/503 errors, retry-header preservation, and AbortSignal propagation.
+
+Discovery tests cover basic/rich model lists, mapping, duplicate and non-text model filtering, invalid JSON/schema, missing credentials, authenticated requests, cache TTL and permissions, key isolation, forced/empty refresh, stale fallback, 401/403 invalidation, corrupt/unwritable caches, bounded responses, and cancellation.
+
+The **real bundled Pi CLI** is run in an isolated temporary directory with this package loaded by `-e`. A test-only transport routes its requests to a local HTTP server; no genuine API key is used. The test verifies:
+
+1. The async extension loads discovered models before `--list-models`.
+2. A subsequent Pi invocation selects the discovered model and reuses its cache.
+3. The HTTP response streams a fragmented `write` function call.
+4. Pi actually writes `nebius-test.txt` with `Hello from Nebius`.
+5. Pi sends an assistant tool call and matching tool-result message in the next HTTP request.
+6. The model's mocked final response completes the turn.
+7. A disappeared model yields actionable diagnostics while preserving the server error and retry information.
+8. `/nebius-refresh` bypasses a fresh cache.
+9. Standard `models.json` can add offline models alongside the native catalog.
+10. Missing credentials produce setup guidance without crashing Pi or requesting discovery.
+
+These establish provider integration and agent/tool transport. They do **not** establish a live model's willingness or ability to call tools.
+
+## Benchmark verification
+
+The final type check, Biome lint, build, and complete 35-test regression suite passed. Benchmark tests cover definition parsing, copy isolation, event instrumentation, byte-preserving streaming observation, cumulative tokens, repeated-run aggregation, validation, failure classification, JSON serialization, cancellation, and hard worker timeouts. All four original fixtures fail their trusted checks; all four reference solutions pass.
+
+`npm run benchmark:demo` passed with **two mock models × two runs** through real Pi SDK sessions. All four runs passed deterministic validation and shared one system-prompt hash. Each recorded two HTTP requests, two turns, one executed write tool, 320 cumulative input tokens, 60 output tokens, 50 cached input tokens, and seven reasoning tokens. These are scripted measurements, not hosted-model performance.
+
+The retained local result is `benchmark-results/mock-demo-1789550582003/results.json`, with individual runs, traces, and archived workspaces alongside it. Result files are excluded from the distributable package. The demo initially encountered a sandbox IPC permission error and passed after running with the required permission; package inspection similarly required access to npm's cache.
+
+Source changes and package contents were reviewed. v1 supports sequential runs only to preserve Pi's unmodified absolute-path system prompt; exact final context size, server generation time, pure agent overhead, and exact model revisions remain unavailable. See [benchmarking.md](benchmarking.md) for measurement boundaries and live acceptance instructions.
+
+## Not executed
+
+- **Authenticated Nebius discovery and live inference**: no `NEBIUS_API_KEY` was available. The unauthenticated model endpoint returned the expected credentials error; that is not a live inference test.
+- **Interactive TUI visual testing**: normal model availability was exercised through Pi's real CLI/registry, not a screenshot of the terminal picker.
+- **Every hosted model**, reasoning template, or vision input: requires model-specific live validation. No such claim is made.
+
+To complete the live acceptance check, set `NEBIUS_API_KEY` and `NEBIUS_MODEL` locally and run `npm run test:live` from the source checkout. That script uses the genuine Pi CLI against Token Factory, validates a successful write-tool event, a final model response, and exact file contents, then removes its scratch directory. It may incur inference charges.

package/examples/models.json

@@ -0,0 +1,31 @@
+{
+  "providers": {
+    "nebius": {
+      "models": [
+        {
+          "id": "REPLACE_WITH_EXACT_NEBIUS_MODEL_ID",
+          "name": "My verified Nebius model",
+          "reasoning": false,
+          "input": ["text"],
+          "contextWindow": 32768,
+          "maxTokens": 4096,
+          "compat": {
+            "supportsStore": false,
+            "supportsDeveloperRole": false,
+            "supportsStrictMode": false,
+            "supportsOpenAIGrammarTools": false,
+            "supportsReasoningEffort": false,
+            "supportsUsageInStreaming": true,
+            "maxTokensField": "max_tokens"
+          }
+        }
+      ],
+      "modelOverrides": {
+        "REPLACE_WITH_EXACT_NEBIUS_MODEL_ID": {
+          "contextWindow": 32768,
+          "maxTokens": 4096
+        }
+      }
+    }
+  }
+}

package/package.json

@@ -0,0 +1,74 @@
+{
+  "name": "pi-nebius",
+  "version": "0.3.0",
+  "description": "Nebius Token Factory provider and agentic coding benchmarks for Pi",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/PeterHdd/pi-nebius.git"
+  },
+  "homepage": "https://github.com/PeterHdd/pi-nebius#readme",
+  "bugs": {
+    "url": "https://github.com/PeterHdd/pi-nebius/issues"
+  },
+  "bin": {
+    "pi-nebius": "dist/benchmark/cli.js"
+  },
+  "type": "module",
+  "license": "MIT",
+  "keywords": [
+    "pi-package",
+    "nebius",
+    "token-factory"
+  ],
+  "files": [
+    "src",
+    "dist",
+    "benchmarks",
+    "examples",
+    "docs",
+    "README.md",
+    "CHANGELOG.md",
+    "CONTRIBUTING.md",
+    "SECURITY.md",
+    "LICENSE"
+  ],
+  "pi": {
+    "extensions": [
+      "./src/index.ts"
+    ]
+  },
+  "engines": {
+    "node": ">=22.19.0"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-ai": "*",
+    "@earendil-works/pi-coding-agent": "*"
+  },
+  "devDependencies": {
+    "@earendil-works/pi-ai": "0.85.1",
+    "@earendil-works/pi-coding-agent": "0.85.1",
+    "@types/node": "^22.0.0",
+    "@biomejs/biome": "^2.2.0",
+    "tsx": "^4.20.0",
+    "typescript": "^5.9.0"
+  },
+  "dependencies": {
+    "yaml": "^2.8.1"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.build.json",
+    "version:check": "node scripts/check-version.mjs",
+    "security:check": "node scripts/security-check.mjs",
+    "package:check": "npm run install:check && npm run build && node scripts/check-package.mjs",
+    "install:check": "node scripts/check-install.mjs",
+    "prepack": "npm run build",
+    "pretest": "npm run build",
+    "typecheck": "tsc --noEmit",
+    "lint": "biome check .",
+    "test": "node --import tsx --test tests/*.test.ts tests/benchmark/*.test.ts",
+    "benchmark": "node dist/benchmark/cli.js benchmark",
+    "benchmark:demo": "npm run build && tsx tests/benchmark/demo.ts",
+    "test:live": "node --import tsx tests/live.ts",
+    "check": "npm run typecheck && npm run lint && npm test"
+  }
+}

package/src/benchmark/cli.ts

@@ -0,0 +1,118 @@
+#!/usr/bin/env node
+import { randomUUID } from "node:crypto";
+import { mkdtemp, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+import { parseArgs } from "node:util";
+import { getAgentDir } from "@earendil-works/pi-coding-agent";
+import { discoverModels, MISSING_KEY } from "../discovery.ts";
+import { applyModelSettings, loadSettings, settingsPath } from "../model-settings.ts";
+import { loadDefinition, positive } from "./definition.ts";
+import { redactor } from "./instrumentation.ts";
+import { terminalReport } from "./report.ts";
+import { runBenchmark } from "./runner.ts";
+
+const HELP = `Usage: pi-nebius benchmark --benchmark DIRECTORY --models ID,ID [options]
+
+  --benchmark PATH   Directory with benchmark.yaml, or YAML/JSON definition file
+  --models IDS       Comma-separated, exact Nebius model IDs (no automatic selection)
+  --runs N           Repetitions per model (default: 1)
+  --timeout SECONDS  Per-agent deadline, including worker/session startup
+  --output PATH      New result directory (must not already exist)
+  --concurrency N    v1 supports 1 only: identical Pi prompts without cwd rewriting
+  --help             Show this help
+
+Each run receives a fresh fixture copy and an isolated Pi configuration.
+Pi tools have normal host access: filesystem copies are not a security sandbox.
+Use a disposable machine/container for untrusted fixtures or model-generated shell commands.
+No API key or prompt/response/tool content is retained in request traces.
+`;
+
+async function main() {
+  const args = process.argv.slice(2);
+  if (args[0] === "--help" || args[0] === "-h" || args.length === 0) {
+    console.log(HELP);
+    return;
+  }
+  if (args.shift() !== "benchmark")
+    throw new Error("Expected the benchmark subcommand. Use --help.");
+  const { values } = parseArgs({
+    args,
+    strict: true,
+    options: {
+      benchmark: { type: "string" },
+      models: { type: "string" },
+      runs: { type: "string" },
+      timeout: { type: "string" },
+      output: { type: "string" },
+      concurrency: { type: "string" },
+      help: { type: "boolean", short: "h" },
+    },
+  });
+  if (values.help) {
+    console.log(HELP);
+    return;
+  }
+  if (!values.benchmark || !values.models) throw new Error("--benchmark and --models are required");
+  const runs = positive(Number(values.runs ?? 1), "runs", 1000);
+  const concurrency = positive(Number(values.concurrency ?? 1), "concurrency", 1);
+  const ids = values.models.split(",").map((id) => id.trim());
+  if (ids.some((id) => !id) || new Set(ids).size !== ids.length)
+    throw new Error("--models must contain nonempty, unique exact IDs");
+  if (runs * ids.length > 10000) throw new Error("At most 10,000 runs per invocation");
+  const { definition, directory } = await loadDefinition(values.benchmark);
+  if (values.timeout) definition.timeout = positive(Number(values.timeout), "timeout");
+  const apiKey = process.env.NEBIUS_API_KEY?.trim();
+  if (!apiKey) throw new Error(MISSING_KEY);
+  const cache = await mkdtemp(join(tmpdir(), "pi-nebius-benchmark-discovery-"));
+  const discovered = await discoverModels({ apiKey, agentDir: cache, force: true }).finally(() =>
+    rm(cache, { recursive: true, force: true }),
+  );
+  if (discovered.warning) process.stderr.write(`${discovered.warning}\n`);
+  const modelSettings = await loadSettings(settingsPath(getAgentDir()));
+  const models = ids.map((id) => {
+    const model = discovered.models.find((candidate) => candidate.id === id);
+    if (!model)
+      throw new Error(
+        `Requested model was not discovered: ${id}. Check its exact ID and your access.`,
+      );
+    return applyModelSettings(model, modelSettings[id]);
+  });
+  const output = resolve(
+    values.output ??
+      join(
+        "benchmark-results",
+        `${new Date().toISOString().replaceAll(":", "-")}-${randomUUID().slice(0, 8)}`,
+      ),
+  );
+  const controller = new AbortController();
+  const abort = () => controller.abort();
+  process.once("SIGINT", abort);
+  process.once("SIGTERM", abort);
+  try {
+    const results = await runBenchmark({
+      definition,
+      directory,
+      models,
+      modelSettings,
+      runs,
+      concurrency,
+      output,
+      apiKey,
+      signal: controller.signal,
+      onRun: (run) =>
+        process.stderr.write(`${run.model} #${run.run}: ${run.success ? "PASS" : run.failure}\n`),
+    });
+    console.log(terminalReport(results));
+    console.log(`Results: ${join(output, "results.json")}`);
+    process.exitCode =
+      results.status === "cancelled" ? 130 : results.runs.every((run) => run.success) ? 0 : 1;
+  } finally {
+    process.removeListener("SIGINT", abort);
+    process.removeListener("SIGTERM", abort);
+  }
+}
+void main().catch((error) => {
+  console.error(redactor([process.env.NEBIUS_API_KEY ?? ""])(String(error)));
+  process.exitCode = 2;
+});