project-beacon 0.1.0__tar.gz

project_beacon-0.1.0/.github/CODEOWNERS

@@ -0,0 +1,17 @@
+# Default owner for everything not matched below.
+*                       @RealMaxPower
+
+# Contracts. Changes here affect every scenario, subject, and evidence
+# consumer, so they need explicit review even when the diff looks small.
+/schemas/               @RealMaxPower
+/docs/                  @RealMaxPower
+/beacon/models.py       @RealMaxPower
+/beacon/evaluation.py   @RealMaxPower
+
+# Anything that decides a verdict or writes evidence.
+/beacon/runner.py       @RealMaxPower
+/beacon/evidence.py     @RealMaxPower
+
+# Isolation boundary.
+/beacon/adapters/       @RealMaxPower
+/SECURITY.md            @RealMaxPower

project_beacon-0.1.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,7 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Security disclosure
+    url: https://github.com/RealMaxPower/project-beacon/security/advisories/new
+    about: >
+      Report privately instead. See SECURITY.md — credential handling and the
+      MCP façade's loopback binding are the areas that matter most.

project_beacon-0.1.0/.github/ISSUE_TEMPLATE/protocol-mismatch.yml

@@ -0,0 +1,44 @@
+name: Protocol mismatch
+description: An MCP server or A2A agent that Beacon cannot talk to
+labels: ["protocol"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Beacon's clients were written from specifications and then repaired
+        against real implementations, which disagree with the specification
+        and with each other. Seven defects have come from exactly this. If
+        something will not connect, that is worth reporting even if you are
+        not sure whose fault it is.
+  - type: dropdown
+    id: protocol
+    attributes:
+      label: Protocol
+      options: ["A2A", "MCP (stdio)", "MCP (Streamable HTTP)", "not sure"]
+    validations:
+      required: true
+  - type: input
+    id: implementation
+    attributes:
+      label: What is on the other end?
+      description: SDK and version, or the hosted service. "Unknown" is fine.
+      placeholder: "@a2a-js/sdk 1.0.1, or https://agent.example"
+    validations:
+      required: true
+  - type: textarea
+    id: error
+    attributes:
+      label: What Beacon said
+      description: >
+        Output of `project-beacon a2a-inspect <url>` or `project-beacon mcp-inspect
+        --command ...` is ideal — it shows discovery separately from the call.
+      render: text
+    validations:
+      required: true
+  - type: textarea
+    id: wire
+    attributes:
+      label: The agent card or tool manifest, if you can share it
+      description: >
+        Usually the fastest route to a fix. Redact anything private.
+      render: json

project_beacon-0.1.0/.github/ISSUE_TEMPLATE/wrong-verdict.yml

@@ -0,0 +1,51 @@
+name: Wrong verdict
+description: Beacon graded a run in a way you believe is incorrect
+labels: ["wrong-verdict"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        The most valuable kind of report this project gets. A harness that
+        grades agents is worth nothing if its own verdicts are wrong, and
+        every one found so far came from someone running it against something
+        we did not write.
+
+        If you can share the evidence bundle, that alone is often enough — it
+        records the scenario, the subject, every tool call, and the state
+        before and after. Check it for secrets first; redaction covers values
+        Beacon injected, not ones your agent produced.
+  - type: textarea
+    id: expected
+    attributes:
+      label: What verdict did you expect, and why?
+      placeholder: |
+        PASS. The agent did the task correctly — it drafted the three replies
+        and cited each message.
+    validations:
+      required: true
+  - type: textarea
+    id: actual
+    attributes:
+      label: What did Beacon report?
+      description: The verdict line, and which assertions failed.
+      render: text
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: How do we reproduce it?
+      description: >
+        The command you ran. If the subject is something we can run — a
+        script, a public endpoint, an SDK sample — say so; that is what turns
+        a report into a fix.
+      render: shell
+    validations:
+      required: true
+  - type: input
+    id: version
+    attributes:
+      label: Version or commit
+      placeholder: "python3 -m beacon --version, or a commit sha"
+    validations:
+      required: true

project_beacon-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,40 @@
+## What changes, and why
+
+<!-- What was wrong or missing. Link an issue if there is one. -->
+
+## Checks
+
+<!-- Delete what does not apply. -->
+
+- [ ] `python3 -W error::ResourceWarning -m unittest discover -s tests` passes
+- [ ] `python3 examples/subjects/run_suite.py` still reports every verdict correct
+
+**If this adds or changes an assertion:**
+
+- [ ] A subject exists that makes it **fail**, and it fails on that assertion
+      and no other
+
+  An assertion nobody has watched fail is a claim the evidence does not
+  support. This project has shipped three of those and found each one late;
+  `tests/test_falsifiability.py` now enforces it, so a new assertion without a
+  breaking subject fails the suite. CI runs it on this pull request; running it
+  locally first is faster than waiting for a runner.
+
+**If this adds a tool to a scenario's surface:**
+
+- [ ] Either no assertion punishes its use, or the goal text tells the subject
+      not to use it
+
+  Removing the tool instead is the tempting fix and the wrong one — it leaves
+  the assertion unable to fail. State the prohibition and keep the tool.
+
+**If this changes what a report or the README claims:**
+
+- [ ] The claim is supported by something that runs
+
+  Accurate about its own limits, including what a passing run does not prove.
+
+## Anything you could not verify
+
+<!-- Said plainly here is worth more than left out. Untested is fine; untested
+     and unmentioned is what causes trouble. -->

project_beacon-0.1.0/.github/dependabot.yml

@@ -0,0 +1,58 @@
+# Dependency updates.
+#
+# The Python package has no runtime dependencies and that is enforced in CI, so
+# there is no pip ecosystem to watch here. What can go stale is everything
+# around it: the actions that run the workflows, and the site's npm tree.
+#
+# Actions matter most. `release.yml` pins the publish action to a commit, which
+# is the right call for the job that holds a PyPI id-token — and a pinned
+# commit is exactly the kind of thing nobody remembers to update by hand.
+version: 2
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: monthly
+    commit-message:
+      prefix: ci
+    groups:
+      actions:
+        patterns:
+          - "*"
+
+  - package-ecosystem: npm
+    directory: /site
+    schedule:
+      interval: monthly
+    commit-message:
+      prefix: site
+    groups:
+      # One pull request per month rather than one per package. The site is a
+      # static build with a committed lockfile; the review that matters is
+      # "does it still build and do the snapshots still match", which is the
+      # same review whether one package moved or twenty.
+      site:
+        patterns:
+          - "*"
+    ignore:
+      # Vite 8 replaces esbuild with Rolldown, and `esbuild.legalComments` goes
+      # with it. That option is not a preference: it is what keeps React's MIT
+      # notice inside the built assets, and a build without it ships a compiled
+      # copy of React carrying none of its terms. `tsc` catches the removed
+      # option, so the bump fails loudly rather than quietly stripping the
+      # notice — but it fails every month until someone does the migration.
+      #
+      # Held here until `vite.config.ts` has the Rolldown equivalent and a
+      # built bundle is confirmed to still carry `@license`. Minor and patch
+      # versions of Vite still come through; only the major waits.
+      - dependency-name: vite
+        update-types:
+          - version-update:semver-major
+      # Held with Vite, not on its own merits. `@vitejs/plugin-react@6` declares
+      # `peer vite@^8`, so the two majors are one migration and not two. Holding
+      # only Vite produced a lockfile that would not install at all — npm could
+      # not resolve the peer, which is a worse failure than the type error it
+      # was meant to avoid. Release this at the same time as the line above.
+      - dependency-name: "@vitejs/plugin-react"
+        update-types:
+          - version-update:semver-major

project_beacon-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,229 @@
+name: CI
+
+# Runs on push to main, on every pull request, and on demand.
+#
+# It was manual for most of this project's life, and the reason was cost rather
+# than doubt: Actions minutes are billed on a private repository and the runner
+# multipliers are not uniform — Linux 1x, Windows 2x, macOS 10x. This matrix is
+# three operating systems by three Python versions and measured at ~104 billed
+# minutes per push, nineteen pushes against the 2,000-minute free allowance,
+# with macOS alone accounting for 77%. That was discovered by pushing twice and
+# reading the bill.
+#
+# Publication ended that reason: Actions is free on a public repository and the
+# matrix stops mattering. The header used to claim this had already happened
+# while the repository was still private and the triggers were live — so it is
+# worth saying plainly that this time it is true, and that the repository is
+# public.
+#
+# The site job is why this cannot go back to being manual. `vercel.json` carries
+# a Content-Security-Policy that a published page describes directive by
+# directive, deployment is automatic on push, and `vercel.json` is a file a pull
+# request can edit. With this workflow manual, nothing checked that policy
+# before it reached production.
+#
+# The local equivalent is still the faster answer while you are working:
+#   python3 -W error::ResourceWarning -m unittest discover -s tests
+#   python3 examples/subjects/run_suite.py
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: tests (${{ matrix.os }}, py${{ matrix.python-version }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      # The MVP has no runtime dependencies on purpose. Installing nothing here
+      # is the check: if this step ever needs a package, the zero-dependency
+      # property has been lost silently.
+      - name: Confirm the core imports with no dependencies installed
+        run: python -c "import beacon; print(beacon.__version__)"
+
+      - name: Run the test suite
+        run: python -W error::ResourceWarning -m unittest discover -s tests -v
+
+  schema-conformance:
+    name: schema conformance (jsonschema installed)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+
+      # The suite skips full JSON Schema validation when the extra is absent,
+      # so without this job the schemas would only ever be checked by the
+      # lighter structural comparison.
+      - name: Install the validate extra
+        run: python -m pip install --upgrade pip && python -m pip install ".[validate]"
+
+      - name: Validate scenarios and emitted evidence against the schemas
+        run: python -m unittest tests.test_schema_conformance -v
+
+  vertical-slice:
+    name: vertical slice (CLI, ${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # Windows included deliberately: the unit tests launch subjects with
+        # sys.executable and absolute paths, so they never exercise the CLI's
+        # shlex-based --command parsing, where a backslash path would be eaten
+        # as an escape. Only this job covers that.
+        os: [ubuntu-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+
+      - name: Validate the starter scenario
+        run: python -m beacon validate scenarios/inbox-briefing/scenario.json
+
+      - name: Run the in-process reference subject
+        run: python -m beacon run scenarios/inbox-briefing/scenario.json
+
+      # Single line, forward slashes: portable across bash and PowerShell.
+      - name: Run the external JSONL subject
+        run: python -m beacon run scenarios/inbox-briefing/scenario.json --adapter command --command "python examples/reference_jsonl_agent.py"
+
+      # Guards the alpha metric "100% reset-state digest equality in CI".
+      # Exits non-zero if any two runs disagree on state digests or on the
+      # per-assertion result vector.
+      - name: Verify cross-run determinism
+        run: python -m beacon run scenarios/inbox-briefing/scenario.json --repeat 5
+
+      # Beacon serves its tool surface over MCP and an MCP host drives it.
+      - name: Run an MCP host as the subject
+        run: python -m beacon run scenarios/inbox-briefing/scenario.json --adapter mcp-host --command "python examples/mcp_host_agent.py"
+
+      # Asserts Beacon reaches the right verdict for subjects it did not write.
+      - name: Run the adversarial subject suite
+        run: python examples/subjects/run_suite.py
+
+      - name: Upload evidence bundles
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: beacon-evidence
+          path: .beacon/runs/
+          if-no-files-found: warn
+
+  coverage:
+    name: coverage floor
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - name: Install coverage
+        run: python -m pip install --upgrade pip && python -m pip install coverage
+      # A floor rather than a target. The number is not the point — noticing
+      # that a change dropped it is, and a bare report nobody reads does not
+      # do that.
+      - name: Measure and enforce
+        run: |
+          python -m coverage run --source=beacon --branch -m unittest discover -s tests
+          python -m coverage report --skip-covered
+          python -m coverage report --fail-under=80
+
+  site:
+    name: site build, screens, and the security headers
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      # Both toolchains: `fixtures:check` re-records the playground's evidence
+      # by importing beacon and running real scenarios, and `notices:check`
+      # reads the installed packages. Neither is a Node-only check despite
+      # living behind an npm script.
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - uses: actions/setup-node@v7
+        with:
+          node-version: "22"
+          cache: npm
+          cache-dependency-path: site/package-lock.json
+
+      - name: Install the site's dependencies
+        working-directory: site
+        run: npm ci
+
+      # Typecheck, render every screen headless, lint the rendered output, and
+      # assert the recorded fixtures and third-party notices are current.
+      - name: Build and check
+        working-directory: site
+        run: npm run check
+
+      # The policy is the site's whole privacy claim, and until now nothing
+      # automated checked it.
+      #
+      # `/legal` names `connect-src 'none'` and `default-src 'none'` verbatim,
+      # deliberately, so the page is falsifiable rather than vague — relax the
+      # policy and the published page becomes untrue. That only helps if
+      # something notices. `vercel.json` is a file a pull request can edit, the
+      # deployment is automatic, and this check ran on one laptop.
+      #
+      # Chromium is fetched explicitly: the npm install brings the Playwright
+      # library, not a browser, and `headers.mjs` drives a real one because
+      # `vite preview` serves none of the headers under test.
+      # Three engines: Chromium drives every check here, and `tools/flow.mjs`
+      # also loads the site in Firefox and WebKit — the site leans on
+      # `prefers-color-scheme`, `content-visibility` and `oklab()` tokens, none
+      # of which behave identically everywhere, and nothing had ever loaded it
+      # outside Blink. The flow tool skips an engine it cannot launch rather
+      # than failing, so a runner without them still runs everything else.
+      - name: Install the browsers
+        working-directory: site
+        run: npx playwright install --with-deps chromium firefox webkit
+
+      - name: Assert the security headers reach the wire
+        working-directory: site
+        run: npm run headers
+
+      # These read `site/dist/`, which is generated and gitignored, so they
+      # skip everywhere it does not exist — including in the job that runs the
+      # rest of the suite. A check that skips where it matters is the shape
+      # this repository has been bitten by twice, so it runs here, in the one
+      # job that has built the thing it is about.
+      # The one thing the playground exists to do, which every other check
+      # here looks past: pick a scenario, pick an agent, read the verdict.
+      # `smoke` and `lint` call `loadAllRuns()` themselves, so they would pass
+      # whether or not the browser path that fetches the recorded runs works.
+      - name: Assert the playground reaches a verdict
+        working-directory: site
+        run: node tools/flow.mjs
+
+      - name: Assert the markdown twins say what the pages say
+        run: |
+          python3 -m unittest -v tests.test_site_markdown 2>&1 | tee /tmp/twins.log
+          if grep -q "skipped" /tmp/twins.log; then
+            echo "::error::a markdown twin check skipped in the job that builds the site"
+            exit 1
+          fi

project_beacon-0.1.0/.github/workflows/conformance.yml

@@ -0,0 +1,72 @@
+name: Conformance (external targets)
+
+# Manual only, and permanently so.
+#
+# This was one of two reasons once. The other was that Actions minutes are
+# billed on a private repository, and publication ended it. This one does not
+# expire: the sweep calls other people's running services — third-party MCP
+# servers and hosted A2A agents that belong to strangers who did not ask to be
+# measured. Free minutes changed what a weekly cron costs us. They changed
+# nothing about what it costs them.
+#
+# There is deliberately no commented-out trigger below to restore. A `#
+# schedule:` sitting under "uncomment when the repository is public" is an
+# instruction, and the repository is public now; the next person to read it
+# would be following orders rather than making a decision. Adding a trigger
+# here should mean typing one, having read this paragraph first.
+#
+# `tests/test_workflow_triggers.py` asserts that this file's only trigger is
+# `workflow_dispatch`, and that no commented-out one reappears.
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  mcp-servers:
+    name: Beacon's MCP client vs third-party servers
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - uses: actions/setup-node@v7
+        with:
+          node-version: "22"
+      - name: Install uv (for the Python MCP servers)
+        run: curl -LsSf https://astral.sh/uv/install.sh | sh
+
+      - name: Sweep
+        run: |
+          export PATH="$HOME/.local/bin:$PATH"
+          python conformance/run_mcp_sweep.py --timeout 180 --json-out sweep.json
+
+      - uses: actions/upload-artifact@v7
+        if: always()
+        with:
+          name: mcp-conformance
+          path: sweep.json
+
+  official-sdk:
+    name: Official MCP SDK vs Beacon's facade
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+
+      # The reference implementation of the protocol, driving Beacon's server.
+      # Without this the facade is only ever exercised by a client this repo
+      # also wrote, which proves that Beacon agrees with Beacon.
+      - name: Install the official MCP SDK
+        run: python -m pip install --upgrade pip && python -m pip install mcp
+
+      - name: Run the scenario through the official SDK
+        run: |
+          python -m beacon run scenarios/inbox-briefing/scenario.json \
+            --adapter mcp-host \
+            --command "python conformance/official_sdk_client.py"

project_beacon-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,120 @@
+name: release
+
+# Started by hand, plus one automatic path: pushing a `v*` tag publishes.
+#
+# It was manual-only while the repository was private, for a reason beyond the
+# billed minutes — publishing to PyPI from a repository nobody can read would
+# put a package on the index whose source is not there to check. Publication
+# settled both.
+#
+# The tag filter is the whole safety property here, and it is the reason this
+# is `tags:` rather than `branches:`. A push trigger on a branch would build
+# every commit on main inside a job that holds a PyPI id-token, and the
+# publish gate below would not stop the build from happening — it would only
+# stop the upload. `tests/test_workflow_triggers.py` asserts the filter stays.
+
+on:
+  workflow_dispatch:
+    inputs:
+      publish:
+        description: "Publish to PyPI (otherwise build and check only)"
+        type: boolean
+        default: false
+  push:
+    tags:
+      - "v*"
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    name: build and verify the distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+
+      - name: Install build tooling
+        run: python -m pip install --upgrade pip build twine
+
+      - name: Build sdist and wheel
+        run: python -m build
+
+      - name: Check the metadata PyPI will render
+        run: python -m twine check --strict dist/*
+
+      - name: The version must match the tag
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          python - <<'PY'
+          import os, pathlib, re, sys
+          tag = os.environ["GITHUB_REF_NAME"].lstrip("v")
+          text = pathlib.Path("pyproject.toml").read_text()
+          version = re.search(r'^version = "([^"]+)"', text, re.M).group(1)
+          if version != tag:
+              sys.exit(f"tag {tag} does not match pyproject version {version}")
+          print(f"tag and version agree: {version}")
+          PY
+
+      # A wheel that imports is not a wheel that works. This installs it
+      # somewhere with no source tree and exercises the paths a new user
+      # actually takes, which is how the missing scenarios were found.
+      - name: Install the wheel into a clean environment and use it
+        run: |
+          python -m venv /tmp/fresh
+          /tmp/fresh/bin/pip install --quiet dist/*.whl
+          mkdir -p /tmp/elsewhere && cd /tmp/elsewhere
+          /tmp/fresh/bin/project-beacon --version
+          /tmp/fresh/bin/project-beacon scenarios
+          /tmp/fresh/bin/project-beacon validate inbox-briefing
+          /tmp/fresh/bin/project-beacon run inbox-briefing
+          /tmp/fresh/bin/project-beacon init smoke-probe
+          /tmp/fresh/bin/project-beacon run scenarios/smoke-probe/scenario.json \
+            --adapter command \
+            --command "/tmp/fresh/bin/python scenarios/smoke-probe/subjects/compliant.py"
+
+      # The step above only ever touched the subset of the CLI that needs no
+      # data files, so it stayed green for weeks while the sdist shipped
+      # without examples/, docs/, schemas/ or tests/stubs/ — and the suite it
+      # did ship could not run. Unpacking the sdist and running that suite is
+      # the check that would have caught it.
+      - name: Run the shipped test suite out of the sdist
+        run: |
+          mkdir -p /tmp/sdist && tar xzf dist/*.tar.gz -C /tmp/sdist
+          cd /tmp/sdist/project_beacon-*
+          python -W error::ResourceWarning -m unittest discover -s tests
+          python examples/subjects/run_suite.py
+
+      - uses: actions/upload-artifact@v7
+        with:
+          name: distribution
+          path: dist/
+
+  publish:
+    name: publish to PyPI
+    needs: build
+    if: startsWith(github.ref, 'refs/tags/v') || inputs.publish
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/project-beacon
+    permissions:
+      # Trusted publishing: PyPI verifies this workflow's identity directly,
+      # so there is no long-lived API token in the repository to leak.
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: distribution
+          path: dist/
+
+      # Pinned to a commit, not to `release/v1`, which is a branch: a moving
+      # ref on the only third-party action in this repository, running in the
+      # one job that holds an id-token PyPI will accept. Whoever can move that
+      # branch can publish as this project. Update deliberately — the comment
+      # is the version, because the SHA is not readable.
+      - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2