@pipeworx/mcp-ai-model-experiments 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mojibake Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,139 @@
+# @pipeworx/ai-model-experiments
+
+**Model Lab** — run the same prompt(s) across many AI models at once (Anthropic,
+OpenAI, Google, Meta, Mistral, DeepSeek, Qwen and more via OpenRouter) and
+compare their outputs, cost, and latency side by side, with an optional
+AI-written comparison summary when the run completes.
+
+Part of [Pipeworx](https://pipeworx.io) — an MCP gateway connecting AI agents to 1558+ live data sources.
+
+## Tools
+
+- `experiment_models(search?, min_context?, max_price_per_mtok?, limit?)` — browse ~300 available model ids with context window and our billed per-token price (provider cost × 1.5). Use the returned ids in `experiment_create`.
+- `experiment_estimate(prompts[], models[], reps?, params?, summary?)` — free dry-run: cell count + estimated billed cost range for a spec, before creating it.
+- `experiment_create(name?, prompts[], models[], reps?, params?, summary?, max_spend_usd)` — creates and starts an experiment (prompts × models × reps). **Async**: returns `experiment_id` immediately; execution happens on a separate cron worker within ~1 minute. Requires prepaid balance ≥ `max_spend_usd`.
+- `experiment_status(experiment_id)` — cell counts by state, spend vs cap, whether complete. Poll this after create.
+- `experiment_results(experiment_id, include_outputs?)` — per-model aggregates (latency, tokens, cost, error rate), per-cell outputs, and the AI-written comparison summary.
+- `experiment_list(limit?)` — caller's experiments, newest first.
+- `experiment_cancel(experiment_id)` — skips pending cells (unbilled); in-flight cells finish and bill.
+- `experiment_topup(amount_usd?)` — current balance + the x402 top-up flow.
+
+## Auth
+
+**Prepaid only — no free tier, no BYO-key mode.** Every call to `experiment_create`
+requires a Pipeworx platform credit balance ≥ `max_spend_usd` (1 credit = $0.0001).
+Top up via x402: `POST https://gateway.pipeworx.io/credits/topup?amount_usd=N`
+returns an HTTP 402 payment challenge (USDC on Base); retry with a
+`PAYMENT-SIGNATURE` header to settle, and credits land instantly. See
+`experiment_topup` for the exact flow and current balance.
+
+Billing is **provider cost × 1.5**, floored at $0.10/experiment. Model prices
+returned by `experiment_models` are already the billed (marked-up) price, never
+the raw provider cost.
+
+Execution is handled by a separate cron worker (`workers/experiment-runner`,
+fires every minute) — a created experiment is not synchronous. Poll
+`experiment_status` rather than expecting `experiment_create` to block until
+done.
+
+## Data sources
+
+- <https://openrouter.ai/api/v1/models> — the model catalog (id, context window, provider pricing) that `experiment_models` wraps and re-prices.
+- Inference for each cell is executed against OpenRouter's chat completions API by `workers/experiment-runner`, which also reads OpenRouter's `/generation` endpoint for the actual per-call cost so the 1.5× markup is billed on real cost, not an estimate.
+
+Full build plan, architecture, and current live-vs-planned status:
+`docs/model-lab-plan.md`.
+
+## Quick Start
+
+Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):
+
+```json
+{
+  "mcpServers": {
+    "ai-model-experiments": {
+      "url": "https://gateway.pipeworx.io/ai-model-experiments/mcp"
+    }
+  }
+}
+```
+
+### What this endpoint actually serves
+
+`tools/list` at `https://gateway.pipeworx.io/ai-model-experiments/mcp` returns the tools in the table
+above **plus the shared Pipeworx meta-tools** — `ask_pipeworx`,
+`discover_tools`, `search_within`, `remember`/`recall` and the rest of the
+gateway-wide set. So the tool count you see is larger than this table: a
+single-pack endpoint currently lists roughly 30 shared tools alongside the
+pack's own. The connection's `initialize` response states its exact scope, and
+is the authoritative answer for a given day.
+
+This is deliberate, not multiplexing by accident. The meta-tools are what let a
+scoped connection answer a question this pack does not cover — via
+`ask_pipeworx`, which routes across the whole catalog — without you adding a
+second MCP server. There is currently no way to mount a pack endpoint without
+them; if the extra schemas cost you more context than the routing is worth,
+connect to the full gateway once rather than to several pack endpoints.
+
+Or connect to the full Pipeworx gateway to get every pack's tools listed
+directly, instead of just this one's:
+
+```json
+{
+  "mcpServers": {
+    "pipeworx": {
+      "url": "https://gateway.pipeworx.io/mcp"
+    }
+  }
+}
+```
+
+Both URLs reach the same gateway and the same 1558+ data sources. The
+only difference is which pack's tools are listed **directly**; `ask_pipeworx`
+reaches all of them from either one.
+
+## Standalone (no gateway account)
+
+This package also runs as a local stdio MCP server — no Pipeworx account, no
+gateway round-trip:
+
+```json
+{
+  "mcpServers": {
+    "ai-model-experiments": {
+      "command": "npx",
+      "args": ["-y", "@pipeworx/mcp-ai-model-experiments"]
+    }
+  }
+}
+```
+
+Or run it directly to confirm it starts:
+
+```bash
+npx -y @pipeworx/mcp-ai-model-experiments
+```
+
+It speaks MCP over stdin/stdout and answers `initialize`/`tools/list`/`tools/call`
+for **only** this pack's tools — none of the shared meta-tools the gateway
+connection above adds. Same source, same tools, no ask_pipeworx routing.
+
+## Using with ask_pipeworx
+
+Instead of calling tools directly, you can ask questions in plain English —
+this works on the pack endpoint above as well as on the full gateway:
+
+```
+ask_pipeworx({ question: "your question about Ai Model Experiments data" })
+```
+
+The gateway picks the right tool and fills the arguments automatically.
+
+## More
+
+- [Docs and guides](https://pipeworx.io/docs)
+- [pipeworx.io](https://pipeworx.io)
+
+## License
+
+MIT

package/bin/cli.js

@@ -0,0 +1,17 @@
+#!/usr/bin/env node
+//
+// Entry point for `npx @pipeworx/mcp-<slug>`.
+//
+// Packs ship as raw TypeScript (no build step — see publish-pack.sh for why:
+// tsx sidesteps every extensionless-import / bare-JSON-import edge case a
+// per-pack tsc build would have to solve one pack at a time). This file
+// registers tsx's ESM loader programmatically, then hands off to src/server.ts,
+// which wraps the pack's {tools, callTool} export in a stdio MCP server.
+//
+// Copied verbatim into every published pack repo by scripts/publish-pack.sh —
+// edit this file, not a per-pack copy.
+import { register } from 'tsx/esm/api';
+
+register();
+
+await import('../src/server.ts');

package/package.json

@@ -0,0 +1,27 @@
+{
+  "name": "@pipeworx/mcp-ai-model-experiments",
+  "version": "0.1.0",
+  "description": "AI Model Experiments MCP ('Model Lab') — run the same prompts across many",
+  "type": "module",
+  "main": "src/index.ts",
+  "types": "src/index.ts",
+  "bin": {
+    "mcp-ai-model-experiments": "bin/cli.js"
+  },
+  "keywords": ["mcp", "mcp-server", "model-context-protocol", "pipeworx", "ai-model-experiments"],
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/pipeworx-io/mcp-ai-model-experiments.git"
+  },
+  "scripts": {
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.30.0",
+    "tsx": "^4.19.0"
+  },
+  "devDependencies": {
+    "typescript": "^5.7.0"
+  }
+}

package/server.json

@@ -0,0 +1,18 @@
+{
+  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
+  "name": "io.github.pipeworx-io/ai-model-experiments",
+  "title": "Ai Model Experiments",
+  "description": "AI Model Experiments MCP ('Model Lab') — run the same prompts across many",
+  "version": "0.1.0",
+  "websiteUrl": "https://pipeworx.io/packs/ai-model-experiments",
+  "repository": {
+    "url": "https://github.com/pipeworx-io/mcp-ai-model-experiments",
+    "source": "github"
+  },
+  "remotes": [
+    {
+      "type": "streamable-http",
+      "url": "https://gateway.pipeworx.io/ai-model-experiments/mcp"
+    }
+  ]
+}