jevqa 0.1.0__tar.gz

jevqa-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,10 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv build && uv run --with dist/*.whl python -c "import jevqa.cli, jevqa.tester, jevqa.report; print('ok')"
+      - run: uv run --with dist/*.whl jevqa run --help

jevqa-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,20 @@
+name: release
+on:
+  push:
+    tags: ["v*"]
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write      # PyPI trusted publishing, no token in secrets
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+      - uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          generate_release_notes: true

jevqa-0.1.0/.gitignore

@@ -0,0 +1,5 @@
+.venv/
+jevqa-runs/
+__pycache__/
+*.egg-info/
+dist/

jevqa-0.1.0/BENCHMARK.md

@@ -0,0 +1,54 @@
+# Benchmark: autonomous bug finding on WebTestBench
+
+No vendor in this category publishes recall. This page does, for jevqa and for every tester anyone cares to submit.
+
+## Setup
+
+- **Dataset**: [WebTestBench](https://github.com/friedrichor/WebTestBench) — 100 AI-generated web apps, each with a human-verified checklist of spec items and which of them fail. We use apps 1–10 (58 failing items) as the development split and apps 11–20 (49 failing items) as the **holdout**, never touched while tuning.
+- **Budget**: 50 actions per app. One run per app per row unless stated.
+- **Scoring**: an Opus 5 majority-vote matcher maps every report to a checklist item or to nothing. Report mapped to a *failing* item = **true positive** (counted once per item). Mapped to a *passing* item = **false positive**. Mapped to nothing = **unmatched**: noise, or a real defect the checklist does not list. Unmatched is not counted as a false positive.
+- **Cost**: model tokens only (Jev at $0.042/MTok input, Claude at list price). Matcher cost excluded for every row.
+- **Noise**: the same code run twice differs by up to 4 bugs out of 58; the same reports rescored differ by up to 4. **Any single-run difference under 4 bugs is noise.** Treat 1-bug deltas as ties.
+
+## Leaderboard
+
+| tester | split | bugs found | recall | precision | $/app | min/app | source |
+|---|---|---|---|---|---|---|---|
+| **jevqa v15** (Jev + Claude checklist) | apps 1–10 | 14/58 | 24% | 0.33 | 0.37 | 5.8 | this repo |
+| Claude Opus 5 + Playwright MCP, free exploration | apps 1–10 | 10/58 | 17% | 0.33 | 3.20 | 5.3 | `bench/run.py --tool claude` |
+| jevqa v14 (per-bug tuned) | apps 1–10 | 11–13/58 | 19–22% | 0.39–0.52 | 0.20–0.26 | 4.1 | 3 seeds |
+| Claude Sonnet 5 + Playwright MCP | apps 1–10 | 6/58 | 10% | 0.22 | 1.16 | 3.6 | |
+| gremlins.js (random monkey, console errors only) | apps 1–10 | 0/58 | 0% | — | 0.00 | 0.1 | |
+| **jevqa v15** | **holdout 11–20** | 9/49, 10/49 | 18–20% | 0.16–0.18 | 0.29–0.41 | 5.0–5.6 | 2 seeds |
+| jevqa v9 (no absence path) | holdout 11–20 | 8/49 | 16% | 0.29 | 0.34 | 5.0 | |
+| jevqa v14 | holdout 11–20 | 6/49 | 12% | 0.21–0.25 | 0.30–0.34 | 4.2 | 2 seeds |
+
+Opus and Sonnet agents were not run on the holdout split (cost). Submissions welcome.
+
+Combined over 20 apps, jevqa v15 finds 23–24 of 107 known bugs (22%) at $0.29–0.41 per app. About 1 in 8 of its reports maps to a confirmed defect. We publish that number because a tool that hides it is lying to you.
+
+## What we learned tuning it (so you do not repeat it)
+
+- 15 versions, ~80 benchmark runs. Only two changes moved recall on the holdout: structured before/after diffs fed to the judge, and the **absence path** (a spec-named control that appears on no screen becomes a finding).
+- Everything tuned per bug on apps 1–10 (8 targeted oracles and probes, +3 to +5 bugs there) gave **0** on the holdout. Judge every mechanism on a split it was not tuned on.
+- Oracle-prompt tuning, risk triage, form auto-completion, widget drivers, report rewording, 3× sampling: no measurable gain. The recall ceiling (~20%) is on the action side, shared with the Opus agent, and is not the judge model.
+
+## Submit a tester
+
+Open a pull request adding a row. Include:
+1. The exact command and version, and the split (1–10, 11–20 or both).
+2. Raw reports per app (one JSON list of strings per app) so we can rescore them with the same matcher.
+3. Token or dollar cost per app and wall-clock time.
+
+We rescore every submission with the same matcher and add the row with a link to your raw reports. Rows with fewer than 10 apps are listed but marked as partial.
+
+## By bug class (jevqa v15, apps 1–10 + holdout seed b, 24 bugs)
+
+| WebTestBench class | found | of |
+|---|---|---|
+| functionality (missing or broken feature) | 16 | 56 |
+| constraint (validation, business rule) | 5 | 29 |
+| content (wrong or missing content) | 1 | 7 |
+| interaction (UI behaviour, widgets) | 2 | 15 |
+
+Missing or unreachable features are half of everything it finds. Interaction quality is where it is weakest.

jevqa-0.1.0/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2026 Dmytro Tolok
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

jevqa-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.5
+Name: jevqa
+Version: 0.1.0
+Summary: Autonomous pre-QA bug hunter for web apps: Jev (TypeSafe) picks actions and judges screens, Claude reads the spec once.
+Author: Dmytro Tolok
+License-Expression: MIT
+License-File: LICENSE
+Keywords: jev,monkey-testing,playwright,qa,testing,typesafe
+Requires-Python: >=3.12
+Requires-Dist: anthropic>=0.40
+Requires-Dist: playwright>=1.63
+Requires-Dist: requests>=2.32
+Description-Content-Type: text/markdown
+
+# jevqa
+
+**Five minutes and 35 cents before you show anyone your app.**
+
+Point it at a deployed web app and the README it was built from. It explores the app on its own, then writes `report.md`: the missing delete button, the date field that accepts 1823, the Save that saves nothing, each with a screenshot.
+
+```bash
+uvx jevqa run http://localhost:3000 --spec README.md
+```
+
+![jevqa run: checklist, exploration, flags, report.md](docs/demo.gif)
+
+No selectors. No test scripts. Nothing to maintain. The first run asks for two API keys and remembers them.
+
+→ **[See a real report](examples/blog-app/report.md)** from a blog app: 7 findings in 269 seconds, two of them confirmed bugs (no way to edit posts, no way to publish).
+
+## Why it is cheap
+
+Every decision on every screen is a typed judgment from [Jev](https://typesafe.ai), TypeSafe's System One model: a probability or a choice, not generated text, at $0.042 per million tokens. Jev picks the next action and answers a fixed set of oracle questions about each screen. Claude reads your spec once to write the checklist. A full run is 5–6 minutes and $0.29–0.41.
+
+The same job done by a Claude Opus 5 agent with Playwright MCP: $3.20 per app, and it found fewer known bugs. [Benchmark →](BENCHMARK.md)
+
+## What it finds
+
+Measured on 20 [WebTestBench](https://github.com/friedrichor/WebTestBench) apps with 107 known bugs: 22% of them, at 5–6 minutes and $0.29–0.41 per app. In order of how often it finds them:
+
+1. **Missing or unreachable features** — no edit/delete/search/sort control, a role that can only view. Half of everything it finds, and the most reliable half.
+2. **Validation gaps** — past dates accepted, bad phone numbers, duplicates, start after end.
+3. **Dead controls** — a click that does nothing, "Save" with no effect, a list that does not update after submit.
+4. **Data lost after reload.**
+5. **Broken first render** — charts of zeros, page errors on entry.
+
+## What it will NOT find
+
+- **Interaction quality**: sliders, drag and drop, animation, keyboard flows. 2 of 15 known interaction bugs.
+- **Wrong numbers and wrong content**: a total that is off by one, a stat that lies. It sees text, not truth.
+- **Permissions and ownership**: user A editing user B's data. It rarely switches roles deep enough.
+- **Long multi-step flows**: checkout, onboarding, anything past 5–6 dependent steps in a 50-action budget.
+- **Anything your spec does not mention.** The checklist comes from the spec; a thin README gives a thin run.
+
+**About 1 in 8 findings is a confirmed defect.** The rest is noise or defects the spec does not name. We publish that number because a tool that hides it is lying to you. Read the report top to bottom: missing features come first and are the most reliable; skim the rest in ninety seconds. This is a pre-QA smoke bot. It clears the obvious defects before QA starts; it does not replace QA on UX or business logic.
+
+## GitHub Action
+
+Run it on every preview deployment:
+
+```yaml
+- uses: Todmy/jevqa@main
+  with:
+    url: ${{ steps.deploy.outputs.preview_url }}
+    spec: docs/PRD.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+The report lands in the job summary and as a workflow artifact. Outputs: `report` (path), `findings` (count).
+
+## Setup
+
+Two keys, asked for on first run and stored in `~/.config/jevqa/config.json` (env vars override, which is what CI uses):
+
+- **TypeSafe** — [console.typesafe.ai](https://console.typesafe.ai). Jev does the exploring and judging: ~$0.05–0.10 per app.
+- **Claude** — an [Anthropic API key](https://console.anthropic.com), or pick "use the Claude Code CLI" if you have it logged in. Three calls per app: ~$0.25.
+
+Chromium for Playwright is downloaded automatically on the first run (~150 MB, once per machine). `jevqa config` shows or resets the keys.
+
+| flag | default | |
+|---|---|---|
+| `--spec FILE` | required | spec, PRD or README the app was built from |
+| `--steps N` | 50 | action budget; time and cost scale linearly |
+| `--out DIR` | `jevqa-runs` | where runs are written |
+| `--headed` | off | watch the browser |
+
+Models: Claude Sonnet 5 for the checklist, Haiku 4.5 for form payloads; override with `JEVQA_SONNET` / `JEVQA_HAIKU`.
+
+## How it works
+
+1. Claude turns the spec into 15–30 expectations, each with concrete UI steps.
+2. A bounded crawl visits nav, detail and role states. Spec-quoted controls that appear on no screen become **missing feature** findings.
+3. The action loop follows scenario steps where Jev can match them to a real element, otherwise explores, and submits boundary payloads (negative numbers, past dates, reversed ranges, duplicates) on every form.
+4. After each action Jev answers the oracle questions on the before/after text. Flags are verified by replay; non-reproducible ones are dropped.
+5. Each finding becomes one deterministic sentence with Expected / Actual and a screenshot.
+
+## Hosted version?
+
+Today jevqa is a local CLI and an Action, bring your own keys. If you would pay ~$29/month for a GitHub App that comments on pull requests, keeps run history and needs no keys, **[👍 this issue](../../issues/1)**. We build it when enough people do.
+
+## Telemetry
+
+Anonymous counts only (version, step budget, number of findings, seconds, cost estimate). Never the URL, the spec or the report. Off with `JEVQA_TELEMETRY=0`; off by default in CI.
+
+## Add your tester to the benchmark
+
+[BENCHMARK.md](BENCHMARK.md) has the methodology, the leaderboard and how to submit. Nobody else in this category publishes recall. Prove us wrong.
+
+## License
+
+MIT

jevqa-0.1.0/README.md

@@ -0,0 +1,99 @@
+# jevqa
+
+**Five minutes and 35 cents before you show anyone your app.**
+
+Point it at a deployed web app and the README it was built from. It explores the app on its own, then writes `report.md`: the missing delete button, the date field that accepts 1823, the Save that saves nothing, each with a screenshot.
+
+```bash
+uvx jevqa run http://localhost:3000 --spec README.md
+```
+
+![jevqa run: checklist, exploration, flags, report.md](docs/demo.gif)
+
+No selectors. No test scripts. Nothing to maintain. The first run asks for two API keys and remembers them.
+
+→ **[See a real report](examples/blog-app/report.md)** from a blog app: 7 findings in 269 seconds, two of them confirmed bugs (no way to edit posts, no way to publish).
+
+## Why it is cheap
+
+Every decision on every screen is a typed judgment from [Jev](https://typesafe.ai), TypeSafe's System One model: a probability or a choice, not generated text, at $0.042 per million tokens. Jev picks the next action and answers a fixed set of oracle questions about each screen. Claude reads your spec once to write the checklist. A full run is 5–6 minutes and $0.29–0.41.
+
+The same job done by a Claude Opus 5 agent with Playwright MCP: $3.20 per app, and it found fewer known bugs. [Benchmark →](BENCHMARK.md)
+
+## What it finds
+
+Measured on 20 [WebTestBench](https://github.com/friedrichor/WebTestBench) apps with 107 known bugs: 22% of them, at 5–6 minutes and $0.29–0.41 per app. In order of how often it finds them:
+
+1. **Missing or unreachable features** — no edit/delete/search/sort control, a role that can only view. Half of everything it finds, and the most reliable half.
+2. **Validation gaps** — past dates accepted, bad phone numbers, duplicates, start after end.
+3. **Dead controls** — a click that does nothing, "Save" with no effect, a list that does not update after submit.
+4. **Data lost after reload.**
+5. **Broken first render** — charts of zeros, page errors on entry.
+
+## What it will NOT find
+
+- **Interaction quality**: sliders, drag and drop, animation, keyboard flows. 2 of 15 known interaction bugs.
+- **Wrong numbers and wrong content**: a total that is off by one, a stat that lies. It sees text, not truth.
+- **Permissions and ownership**: user A editing user B's data. It rarely switches roles deep enough.
+- **Long multi-step flows**: checkout, onboarding, anything past 5–6 dependent steps in a 50-action budget.
+- **Anything your spec does not mention.** The checklist comes from the spec; a thin README gives a thin run.
+
+**About 1 in 8 findings is a confirmed defect.** The rest is noise or defects the spec does not name. We publish that number because a tool that hides it is lying to you. Read the report top to bottom: missing features come first and are the most reliable; skim the rest in ninety seconds. This is a pre-QA smoke bot. It clears the obvious defects before QA starts; it does not replace QA on UX or business logic.
+
+## GitHub Action
+
+Run it on every preview deployment:
+
+```yaml
+- uses: Todmy/jevqa@main
+  with:
+    url: ${{ steps.deploy.outputs.preview_url }}
+    spec: docs/PRD.md
+  env:
+    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+The report lands in the job summary and as a workflow artifact. Outputs: `report` (path), `findings` (count).
+
+## Setup
+
+Two keys, asked for on first run and stored in `~/.config/jevqa/config.json` (env vars override, which is what CI uses):
+
+- **TypeSafe** — [console.typesafe.ai](https://console.typesafe.ai). Jev does the exploring and judging: ~$0.05–0.10 per app.
+- **Claude** — an [Anthropic API key](https://console.anthropic.com), or pick "use the Claude Code CLI" if you have it logged in. Three calls per app: ~$0.25.
+
+Chromium for Playwright is downloaded automatically on the first run (~150 MB, once per machine). `jevqa config` shows or resets the keys.
+
+| flag | default | |
+|---|---|---|
+| `--spec FILE` | required | spec, PRD or README the app was built from |
+| `--steps N` | 50 | action budget; time and cost scale linearly |
+| `--out DIR` | `jevqa-runs` | where runs are written |
+| `--headed` | off | watch the browser |
+
+Models: Claude Sonnet 5 for the checklist, Haiku 4.5 for form payloads; override with `JEVQA_SONNET` / `JEVQA_HAIKU`.
+
+## How it works
+
+1. Claude turns the spec into 15–30 expectations, each with concrete UI steps.
+2. A bounded crawl visits nav, detail and role states. Spec-quoted controls that appear on no screen become **missing feature** findings.
+3. The action loop follows scenario steps where Jev can match them to a real element, otherwise explores, and submits boundary payloads (negative numbers, past dates, reversed ranges, duplicates) on every form.
+4. After each action Jev answers the oracle questions on the before/after text. Flags are verified by replay; non-reproducible ones are dropped.
+5. Each finding becomes one deterministic sentence with Expected / Actual and a screenshot.
+
+## Hosted version?
+
+Today jevqa is a local CLI and an Action, bring your own keys. If you would pay ~$29/month for a GitHub App that comments on pull requests, keeps run history and needs no keys, **[👍 this issue](../../issues/1)**. We build it when enough people do.
+
+## Telemetry
+
+Anonymous counts only (version, step budget, number of findings, seconds, cost estimate). Never the URL, the spec or the report. Off with `JEVQA_TELEMETRY=0`; off by default in CI.
+
+## Add your tester to the benchmark
+
+[BENCHMARK.md](BENCHMARK.md) has the methodology, the leaderboard and how to submit. Nobody else in this category publishes recall. Prove us wrong.
+
+## License
+
+MIT

jevqa-0.1.0/action.yml

@@ -0,0 +1,42 @@
+name: jevqa
+description: Autonomous pre-QA bug hunt of a deployed web app (Jev + Claude), report.md with screenshots as a workflow artifact
+inputs:
+  url:
+    description: URL of the deployed app to test
+    required: true
+  spec:
+    description: Path to the spec / PRD / README the app was built from
+    required: true
+  steps:
+    description: Action budget
+    default: "50"
+  python-version:
+    default: "3.12"
+outputs:
+  report:
+    description: Path to report.md
+    value: ${{ steps.run.outputs.report }}
+  findings:
+    description: Number of findings
+    value: ${{ steps.run.outputs.findings }}
+runs:
+  using: composite
+  steps:
+    - uses: actions/setup-python@v5
+      with:
+        python-version: ${{ inputs.python-version }}
+    - shell: bash
+      run: |
+        pip install -q "jevqa @ git+https://github.com/${{ github.action_repository || 'Todmy/jevqa' }}@${{ github.action_ref || 'main' }}"
+        python -m playwright install --with-deps chromium
+    - id: run
+      shell: bash
+      run: jevqa run "${{ inputs.url }}" --spec "${{ inputs.spec }}" --steps "${{ inputs.steps }}" --out jevqa-runs
+    - uses: actions/upload-artifact@v4
+      with:
+        name: jevqa-report
+        path: jevqa-runs
+    - shell: bash
+      run: |
+        echo "## jevqa: ${{ steps.run.outputs.findings }} findings" >> "$GITHUB_STEP_SUMMARY"
+        sed 's/^!\[.*//' "${{ steps.run.outputs.report }}" >> "$GITHUB_STEP_SUMMARY"

jevqa-0.1.0/docs/launch/devto-post.md

@@ -0,0 +1,50 @@
+# We ran an autonomous tester on 20 vibe-coded apps. Here is every bug it found.
+
+*(dev.to, same morning as Show HN. Tags: testing, ai, playwright, showdev)*
+
+About 1 in 8 of its findings was a confirmed bug. I am going to lead with that number because nobody else in this category publishes one, and because if you skip the rest of this post that is the thing to remember.
+
+## The setup
+
+WebTestBench is a dataset of 100 web apps generated by coding agents from a written spec, each with a human-verified checklist of what works and what does not. I took 20 of them: 107 known bugs. Budget: 50 UI actions per app.
+
+The tester is called jevqa. `uvx jevqa run <url> --spec README.md`. Claude reads the spec once and writes a checklist. From there every decision is made by Jev, a model from TypeSafe that answers questions with a probability instead of a paragraph, at $0.042 per million tokens. Which element to act on next. Does this screen look broken after that action. Is there any screen in the app where the "Delete" control the spec promises actually exists.
+
+## The numbers
+
+| | bugs found | precision | $/app | min/app |
+|---|---|---|---|---|
+| jevqa | 24/107 (22%) | 0.16–0.33 | 0.29–0.41 | 5–6 |
+| Claude Opus 5 + Playwright MCP (apps 1–10 only) | 10/58 (17%) | 0.33 | 3.20 | 5 |
+
+Full methodology, the noise protocol (two identical runs differ by up to 4 bugs, so treat small deltas as ties) and every version we tried: [BENCHMARK.md](https://github.com/Todmy/jevqa/blob/main/BENCHMARK.md).
+
+## What it actually found
+
+*(paste 8–10 findings from examples/ and the benchmark runs, one screenshot each, grouped:)*
+
+**Missing features (half of everything).** No way to edit a post. No way to delete a published item. A "series" page the spec promises that appears on no screen. A reviewer role that can only view.
+
+**Validation that is not there.** A wedding date in 1823. A phone number of "abc". The same workout submitted twice. A booking whose end time is before its start time.
+
+**Buttons that do nothing.** "Save" with no effect. A button that should redirect and does not. A reservation list that does not update after submit.
+
+**Data that vanishes.** Availability profiles gone after reload.
+
+**Broken on arrival.** A dashboard chart where every value on the axis is 0.
+
+## What it did not find, and will not
+
+Slider behaviour. Wrong totals. User A editing user B's data. Anything more than five dependent steps deep. Anything the spec does not mention. 2 of 15 known interaction bugs; that is the weak spot and I do not expect it to improve without a different action strategy.
+
+## What I learned tuning it (15 versions)
+
+Only two changes moved recall on a holdout split: feeding structured before/after diffs to the judge, and treating "a spec-named control appears on no screen" as a finding. Eight mechanisms tuned bug-by-bug on the first ten apps gained +3 to +5 there and exactly 0 on the other ten. If you build one of these: keep a holdout split and never look at it while tuning.
+
+Oracle prompt tuning, risk triage, form auto-fill, widget drivers, report rewording, sampling the judge three times: nothing. The judge is near-deterministic. The ceiling is on the action side and the Opus agent hits the same one.
+
+## Try it
+
+`uvx jevqa run http://localhost:3000 --spec README.md`, or the GitHub Action on preview deploys. MIT. Two keys, asked for on first run.
+
+If you have a tester of your own, the leaderboard takes PRs. I would genuinely like to be beaten on it.

jevqa-0.1.0/docs/launch/show-hn.md

@@ -0,0 +1,31 @@
+# Show HN
+
+Post Tue–Thu, 08:00 US Eastern (14:00 Kyiv). Benchmark page must be live. Ping the warm list at post time: 30–50 upvotes in the first hour is the whole game.
+
+**Title (78 chars max):**
+Show HN: jevqa – a $0.35 monkey tester that found 22% of known bugs in 20 apps
+
+**URL:** https://github.com/Todmy/jevqa
+
+**First comment (post immediately after submitting):**
+
+Hi HN, I built this after ~80 benchmark runs trying to make an autonomous web tester that is cheap enough to run on every preview deploy.
+
+What it does: `uvx jevqa run <url> --spec README.md`. It reads your spec once (Claude), then explores the app on its own. Every decision on every screen – which element to act on, does this screen look broken – is a typed judgment from Jev, TypeSafe's "System One" model, which returns a probability instead of text at $0.042/MTok. A full 50-action run is 5–6 minutes and $0.29–0.41. Output is report.md with a sentence and a screenshot per finding.
+
+Numbers, because nobody in this category publishes any: on WebTestBench (20 AI-generated apps, 107 human-verified bugs) it finds 22% of the known bugs. A Claude Opus 5 + Playwright MCP agent on the same apps: 17% at $3.20/app. About 1 in 8 of jevqa's findings maps to a confirmed bug; the rest is noise or bugs the checklist does not list. Methodology, noise protocol and per-version table: https://github.com/Todmy/jevqa/blob/main/BENCHMARK.md
+
+What it is good at: missing or unreachable features (no edit/delete/search, a role that can only view) – half of everything it finds. Validation gaps. Dead buttons. Data lost on reload.
+
+What it is bad at: sliders and drag, wrong numbers, permissions, long flows. It is a pre-QA smoke bot, not a QA replacement.
+
+Things I would love to hear: whether the 1-in-8 is tolerable for a 5-minute report you skim, and whether anyone has a tester I can put on the leaderboard – submissions are a PR.
+
+MIT, Python, bring your own keys (TypeSafe + Anthropic, or your Claude Code CLI).
+
+**Answers to have ready:**
+- "Why not just use Playwright MCP + Claude?" → we did, it is the $3.20 row. Same recall band, 9× the cost, and it needs a frontier model in the loop for every click.
+- "1 in 8 is terrible" → for a tool that pages you, yes. For a 90-second skim before QA, it is the trade. Findings are ordered: missing features first, those are the reliable ones. Ranking by Jev's calibrated confidence is next.
+- "Is Jev just a small LLM?" → it does not generate text; it returns typed answers with calibrated probabilities. Near-deterministic in our runs (spread ≤0.13 across repeats), which is why 3× sampling bought nothing.
+- "Recall ceiling 20%?" → yes, and the Opus agent hits the same wall. The bottleneck is action coverage (role switching, deep flows), not the judge.
+- "Will TypeSafe raise prices?" → unknown; the judgment interface is swappable and I say so in the README.

jevqa-0.1.0/docs/launch/vibe-coder-outreach.md

@@ -0,0 +1,23 @@
+# 10 unsolicited reports (channel test 1)
+
+Goal: reciprocity first. Run jevqa on 10 public vibe-coded apps, send each builder their report as a gist. No ask in the first message.
+
+**Where to find targets:** Lovable "Launched" gallery, Bolt community showcase, r/vibecoding "I built…" posts, X search `"built with lovable"` / `"built with bolt.new"` last 7 days, Product Hunt launches tagged no-code/AI-built. Pick apps with a public URL and a README or landing copy that can serve as the spec.
+
+**Run:** `jevqa run <url> --spec spec.md --out runs/<name>`; paste the landing page / About text into spec.md if there is no README. Read the report, delete obviously-wrong findings by hand before sending (this is a first impression, not a benchmark).
+
+**Message (DM or reply, ≤ 80 words):**
+
+> Ran an autonomous tester I'm building against <app> for 5 minutes, thought you'd want the output before anyone else finds it: <gist link>. 7 findings, I'd guess 2–3 are real (the missing edit on <page>, the date field that takes past dates). Ignore the rest. No ask, just figured it's more useful in your hands than mine.
+
+**If they reply:** then, and only then: "It's `uvx jevqa run <url> --spec README.md` if you want to rerun after fixing. Would love to know which findings were real – that's the number I'm trying to improve."
+
+**Signal:** ≥6 replies, ≥3 self-installs, ≥1 public post of their own report within 14 days.
+**Kill:** <3 replies, or replies of the form "I knew all of that".
+
+**Tracker:**
+
+| # | app | builder | platform | sent | replied | installed | posted |
+|---|---|---|---|---|---|---|---|
+| 1 | | | | | | | |
+| … | | | | | | | |

jevqa-0.1.0/docs-typesafe-email.md

@@ -0,0 +1,20 @@
+To: hello@typesafe.ai
+Subject: jevqa — an open-source bug hunter built on Jev, with a published benchmark
+
+Hi,
+
+Your launch post asked where Jev works and where it falls short. Here is one answer with numbers.
+
+jevqa is an MIT-licensed autonomous tester for web apps. Jev makes every decision: which element to act on next (choice), whether the screen after an action looks broken (noul oracles), how likely each spec expectation is to fail (score). Claude reads the spec once for a checklist; everything else is Jev, about 150 calls per app.
+
+Measured on WebTestBench (20 AI-generated apps, 107 known bugs): it finds 22% of them at $0.29–0.41 and 5–6 minutes per app. A Claude Opus 5 + Playwright MCP agent on the same apps: 17% at $3.20. Nobody else in the QA-tool category publishes recall at all, so the benchmark page is the launch: <repo>/BENCHMARK.md
+
+Where Jev fell short, honestly: recall plateaus around 20% and the ceiling is on the action side, not the judge. Jev's answers were near-deterministic (spread ≤0.13 across repeats), so 3× sampling bought nothing; oracle-prompt tuning bought nothing either. The judge is not the bottleneck, which is a good problem for you to have.
+
+Two asks:
+1. Is the $0.042/MTok price durable past early access? The whole cost story rests on it.
+2. If a benchmarked, open-source Jev application is useful to you as a showcase, I would like to be it. Launching on Hacker News in two weeks.
+
+Repo: <repo>   Report example: <repo>/examples/blog-app/report.md
+
+Dmytro Tolok