proofswe 0.1.3

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 proofswe contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,116 @@
+# proofswe
+
+**SWE-bench tells you how models score in the lab. This is proof from real work.**
+
+proofswe is an open benchmark for coding agents, built from real developer
+sessions instead of synthetic tasks. The question it answers is the one no
+existing benchmark can:
+
+> Oracle benchmarks measure whether a model can close tasks that *have answers*.
+> proofswe measures whether a model's work *survives* in tasks that don't.
+
+Status: **design exploration.** No collector, no binary yet. We are nailing the
+hard parts on paper before writing a line of code:
+
+- [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) — how raw real-world sessions
+  become a statistically meaningful benchmark (survival analysis, IRT,
+  hierarchical Bradley–Terry, learning the metric weights from the OSS merge
+  oracle). This is what decides whether proofswe is a benchmark or just telemetry
+  with a leaderboard attached.
+- [`docs/CAPTURE.md`](docs/CAPTURE.md) — the data-capture pipeline architecture:
+  a Go binary that captures Claude Code / Codex / (later) Cursor sessions
+  through a harness-agnostic narrow waist, backed by how the best Go CLIs
+  (gh, hugo, fzf, GoReleaser, go-git) actually build this.
+
+## Why this category is empty
+
+Every serious coding benchmark today is **oracle-based**: it needs a
+machine-checkable definition of success *before* the task runs.
+
+| Benchmark | The question it actually answers |
+|---|---|
+| SWE-bench | Can the model close historical issues that happen to have test oracles? |
+| Terminal-Bench / Aider | Can the model complete synthetic tasks with verifiable end states? |
+| Arena-style (Copilot Arena, LMArena) | Which output do strangers prefer at a glance? |
+| **proofswe** | **Whose work do informed owners keep, in tasks that have no oracle?** |
+
+The first three are *pre-task* benchmarks. Success must be definable before work
+starts, which structurally forbids ambiguity. But ambiguity is the defining
+property of real software engineering: "refactor this," "build me a dashboard,"
+"make it faster," "figure out why staging is weird." None of those can ever
+enter an oracle benchmark, no matter how good the benchmark gets.
+
+proofswe is a *post-task* benchmark. Success is defined by what happened
+*afterward*: did the person who owns the codebase and knows the real
+requirements **keep** the work, **commit** it, **build on top of it** — or
+revert it. That is the most informed judgment available for an unverifiable
+task, and it is expressed as a costly action rather than a cheap rating.
+
+## Three properties no existing benchmark has at once
+
+- **Contamination-proof by construction.** Every datapoint is a fresh task in a
+  private codebase that did not exist at training time.
+- **Ungameable in the usual way.** A lab cannot fine-tune against a metric that
+  lives downstream of thousands of uncontrolled real environments. (One real
+  attack vector exists — astroturfing the public pool — and the methodology
+  addresses it head-on.)
+- **The real task distribution.** Including the boring, ambiguous, underspecified
+  majority of work that defines what "good at software engineering" means.
+
+## The collection ladder
+
+Ordered by how much the data is worth to research:
+
+1. **Observational outcomes** — privacy-safe, line-hash telemetry. Scales widest,
+   weakest for causal claims.
+2. **Paired replay** — the same real task attempted by two models, judged by the
+   same informed user. One paired comparison is worth 10–100 observational ones.
+   The feature users want most is also the data researchers need most.
+3. **Open-source transcripts** — when the repo is public, the privacy cost of
+   donating the full transcript collapses, and PR merge status becomes an
+   external oracle. This is the successor dataset to SWE-bench.
+
+The leaderboard is marketing for the dataset. The dataset — real multi-turn
+trajectories with survival-based reward signals, openly licensed and auditable —
+is the research contribution the labs sit on privately and nobody else has.
+
+## Current Pipeline
+
+Install the CLI through npm:
+
+```sh
+npx proofswe version
+```
+
+Submit a real Codex or Claude Code transcript for server-side judging:
+
+```sh
+proofswe submit
+```
+
+`submit` auto-detects the latest supported Codex or Claude Code transcript for
+the current git repo, builds the same scrubbed reproducible task as
+`contribute`, sends it to the proofswe API, and prints the server scorecard. The
+contributor does not need an OpenAI or Anthropic key; the official judge runs on
+the proofswe server. Use `proofswe submit <path>` for an explicit transcript,
+`--no-wait` for automation, and `PROOFSWE_API_URL` or `--endpoint` to point at a
+staging server.
+
+Agent chat helpers are explicit opt-in:
+
+- Codex: `/prompts:benchmark` plus `$proofswe-benchmark`
+- Claude Code: `$proofswe-benchmark`
+
+```sh
+proofswe agent install
+```
+
+Run the staging judge endpoint with server-side credentials:
+
+```sh
+OPENAI_API_KEY=... proofswe serve --addr=:8080 --judge-provider=openai
+```
+
+## License
+
+MIT. See [`LICENSE`](LICENSE).

package/npm/bin/proofswe.js

@@ -0,0 +1,87 @@
+#!/usr/bin/env node
+
+const { spawnSync } = require("node:child_process");
+const fs = require("node:fs");
+const path = require("node:path");
+
+function packageName() {
+  const platform = currentPlatform();
+  const arch = currentArch();
+  const supported = new Set([
+    "darwin:arm64",
+    "darwin:x64",
+    "linux:arm64",
+    "linux:x64",
+    "win32:arm64",
+    "win32:x64",
+  ]);
+  const key = `${platform}:${arch}`;
+  if (!supported.has(key)) {
+    throw new Error(`unsupported platform ${platform}/${arch}`);
+  }
+  return `proofswe-${packagePlatform(platform)}-${arch}`;
+}
+
+function devOverridesEnabled() {
+  return process.env.PROOFSWE_ENABLE_DEV_OVERRIDES === "1";
+}
+
+function currentPlatform() {
+  if (devOverridesEnabled() && process.env.PROOFSWE_TEST_PLATFORM) {
+    return process.env.PROOFSWE_TEST_PLATFORM;
+  }
+  return process.platform;
+}
+
+function currentArch() {
+  if (devOverridesEnabled() && process.env.PROOFSWE_TEST_ARCH) {
+    return process.env.PROOFSWE_TEST_ARCH;
+  }
+  return process.arch;
+}
+
+function packagePlatform(platform) {
+  return platform === "win32" ? "windows" : platform;
+}
+
+function binaryPath() {
+  if (devOverridesEnabled() && process.env.PROOFSWE_BINARY_PATH) {
+    return process.env.PROOFSWE_BINARY_PATH;
+  }
+
+  const platform = currentPlatform();
+  const suffix = platform === "win32" ? ".exe" : "";
+  const pkg = packageName();
+  if (devOverridesEnabled() && process.env.PROOFSWE_PACKAGE_ROOT) {
+    const candidate = path.join(process.env.PROOFSWE_PACKAGE_ROOT, "node_modules", pkg, "bin", `proofswe${suffix}`);
+    if (fs.existsSync(candidate)) {
+      return candidate;
+    }
+  }
+  try {
+    return require.resolve(`${pkg}/bin/proofswe${suffix}`);
+  } catch (err) {
+    const local = path.resolve(__dirname, "..", "..", "dist", `proofswe${suffix}`);
+    if (fs.existsSync(local)) {
+      return local;
+    }
+    throw new Error(
+      `could not find native proofswe binary package ${pkg}; reinstall with optional dependencies enabled`
+    );
+  }
+}
+
+let bin;
+try {
+  bin = binaryPath();
+} catch (err) {
+  console.error(`proofswe: ${err.message}`);
+  process.exit(1);
+}
+
+const result = spawnSync(bin, process.argv.slice(2), { stdio: "inherit" });
+if (result.error) {
+  console.error(`proofswe: ${result.error.message}`);
+  process.exit(1);
+}
+process.exit(result.status === null ? 1 : result.status);

package/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "proofswe",
+  "version": "0.1.3",
+  "description": "Benchmark coding agents from real developer sessions.",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/Atharva-Kanherkar/proofswe"
+  },
+  "bin": {
+    "proofswe": "npm/bin/proofswe.js"
+  },
+  "files": [
+    "npm/bin/proofswe.js",
+    "README.md",
+    "LICENSE"
+  ],
+  "optionalDependencies": {
+    "proofswe-darwin-arm64": "0.1.3",
+    "proofswe-darwin-x64": "0.1.3",
+    "proofswe-linux-arm64": "0.1.3",
+    "proofswe-linux-x64": "0.1.3",
+    "proofswe-windows-arm64": "0.1.3",
+    "proofswe-windows-x64": "0.1.3"
+  },
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "smoke": "node npm/bin/proofswe.js version",
+    "test": "node npm/test-wrapper.mjs"
+  }
+}