actionsbench 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- actionsbench-0.1.1/.gitignore +26 -0
- actionsbench-0.1.1/CITATION.cff +23 -0
- actionsbench-0.1.1/CONTRIBUTING.md +113 -0
- actionsbench-0.1.1/LICENSE +21 -0
- actionsbench-0.1.1/PKG-INFO +124 -0
- actionsbench-0.1.1/README.md +96 -0
- actionsbench-0.1.1/SECURITY.md +35 -0
- actionsbench-0.1.1/corpus/CHANGELOG.md +59 -0
- actionsbench-0.1.1/corpus/LICENSE +396 -0
- actionsbench-0.1.1/corpus/README.md +77 -0
- actionsbench-0.1.1/corpus/cases/AB-ART-0001/.github/workflows/artipacked.yml +19 -0
- actionsbench-0.1.1/corpus/cases/AB-ART-0001/case.yaml +39 -0
- actionsbench-0.1.1/corpus/cases/AB-ART-0002/.github/workflows/unverified-download.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-ART-0002/case.yaml +31 -0
- actionsbench-0.1.1/corpus/cases/AB-CTL-0001/.github/workflows/constant-condition.yml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-CTL-0001/case.yaml +24 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0001/.github/workflows/format-wrapping.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0001/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0002/.github/workflows/contains-and-value.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0002/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0003/.github/workflows/tojson.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0003/case.yaml +23 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0004/.github/workflows/ref-name.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0004/case.yaml +27 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0005/.github/workflows/commit-author.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0005/case.yaml +24 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0006/.github/workflows/github-script.yml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0006/case.yaml +27 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0007/.github/actions/greet/action.yml +11 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0007/.github/workflows/caller.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0007/case.yaml +29 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0008/.github/workflows/multiline-script.yml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0008/case.yaml +23 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0009/.github/workflows/issue-title.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0009/case.yaml +20 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0010/.github/workflows/review-comment.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-INJ-0010/case.yaml +24 -0
- actionsbench-0.1.1/corpus/cases/AB-KVC-0001/.github/workflows/vulnerable-download.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-KVC-0001/case.yaml +26 -0
- actionsbench-0.1.1/corpus/cases/AB-KVC-0002/.github/workflows/vulnerable-setup-php.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-KVC-0002/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0001/.github/workflows/env-indirection.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0001/case.yaml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0002/.github/workflows/boolean-contains.yml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0002/case.yaml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0003/.github/workflows/pinned.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0003/case.yaml +17 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0004/.github/workflows/read-only.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0004/case.yaml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0005/.github/workflows/pull-request-head.yml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0005/case.yaml +17 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0006/.github/workflows/explicit-secret.yml +12 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0006/case.yaml +17 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0007/.github/workflows/patched-download.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0007/case.yaml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0008/.github/workflows/no-persist.yml +21 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0008/case.yaml +19 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0009/.github/workflows/real-condition.yml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0009/case.yaml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0010/.github/workflows/trigger-only.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0010/case.yaml +21 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0011/.github/workflows/multiline-safe.yml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0011/case.yaml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0012/.github/workflows/safe-contexts.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0012/case.yaml +22 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0013/.github/workflows/reusable-pinned.yml +10 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0013/case.yaml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0014/.github/workflows/secret-credentials.yml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0014/case.yaml +17 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0015/.github/workflows/named-secret.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0015/case.yaml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0016/.github/workflows/patched-setup-php.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0016/case.yaml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0017/.github/workflows/current-macos.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0017/case.yaml +14 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0018/.github/workflows/verified-download.yml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-NEG-0018/case.yaml +16 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0001/.github/workflows/tag-ref.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0001/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0002/.github/workflows/branch-ref.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0002/case.yaml +24 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0003/.github/workflows/reusable-tag.yml +10 -0
- actionsbench-0.1.1/corpus/cases/AB-PIN-0003/case.yaml +28 -0
- actionsbench-0.1.1/corpus/cases/AB-PRM-0001/.github/workflows/write-all.yml +12 -0
- actionsbench-0.1.1/corpus/cases/AB-PRM-0001/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-PRM-0002/.github/workflows/workflow-write.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-PRM-0002/case.yaml +26 -0
- actionsbench-0.1.1/corpus/cases/AB-RUN-0001/.github/workflows/retired-runner.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-RUN-0001/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-RUN-0002/.github/workflows/retired-macos.yml +13 -0
- actionsbench-0.1.1/corpus/cases/AB-RUN-0002/case.yaml +25 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0001/.github/workflows/inherit.yml +11 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0001/case.yaml +27 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0002/.github/workflows/hardcoded-credentials.yml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0002/case.yaml +24 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0003/.github/workflows/all-secrets.yml +15 -0
- actionsbench-0.1.1/corpus/cases/AB-SEC-0003/case.yaml +22 -0
- actionsbench-0.1.1/corpus/cases/AB-TRG-0001/.github/workflows/pwn-request.yml +18 -0
- actionsbench-0.1.1/corpus/cases/AB-TRG-0001/case.yaml +29 -0
- actionsbench-0.1.1/docs/PRD.md +313 -0
- actionsbench-0.1.1/docs/adr/0001-python-and-minimal-dependencies.md +53 -0
- actionsbench-0.1.1/docs/adr/0002-corpus-format.md +58 -0
- actionsbench-0.1.1/docs/adr/0003-scanner-scope-and-scoring.md +103 -0
- actionsbench-0.1.1/docs/adr/0004-branching-model.md +60 -0
- actionsbench-0.1.1/docs/adr/README.md +11 -0
- actionsbench-0.1.1/docs/evidence-audit.md +141 -0
- actionsbench-0.1.1/docs/methodology.md +219 -0
- actionsbench-0.1.1/docs/publishing.md +84 -0
- actionsbench-0.1.1/docs/rule-mappings.md +103 -0
- actionsbench-0.1.1/pyproject.toml +98 -0
- actionsbench-0.1.1/scripts/check_wheel.py +116 -0
- actionsbench-0.1.1/scripts/install-actionlint.sh +73 -0
- actionsbench-0.1.1/src/actionsbench/__init__.py +3 -0
- actionsbench-0.1.1/src/actionsbench/__main__.py +3 -0
- actionsbench-0.1.1/src/actionsbench/adapters/__init__.py +27 -0
- actionsbench-0.1.1/src/actionsbench/adapters/actionlint.py +169 -0
- actionsbench-0.1.1/src/actionsbench/adapters/base.py +83 -0
- actionsbench-0.1.1/src/actionsbench/adapters/zizmor.py +118 -0
- actionsbench-0.1.1/src/actionsbench/cli.py +285 -0
- actionsbench-0.1.1/src/actionsbench/corpus.py +227 -0
- actionsbench-0.1.1/src/actionsbench/models.py +80 -0
- actionsbench-0.1.1/src/actionsbench/py.typed +0 -0
- actionsbench-0.1.1/src/actionsbench/regions.py +127 -0
- actionsbench-0.1.1/src/actionsbench/report.py +87 -0
- actionsbench-0.1.1/src/actionsbench/runner.py +190 -0
- actionsbench-0.1.1/src/actionsbench/sarif.py +92 -0
- actionsbench-0.1.1/src/actionsbench/schemas/case.schema.json +67 -0
- actionsbench-0.1.1/src/actionsbench/schemas/findings.schema.json +70 -0
- actionsbench-0.1.1/src/actionsbench/scoring.py +218 -0
- actionsbench-0.1.1/src/actionsbench/taxonomy.py +43 -0
- actionsbench-0.1.1/tests/__init__.py +0 -0
- actionsbench-0.1.1/tests/conftest.py +60 -0
- actionsbench-0.1.1/tests/fakes.py +125 -0
- actionsbench-0.1.1/tests/test_adapter_actionlint.py +216 -0
- actionsbench-0.1.1/tests/test_adapter_zizmor.py +196 -0
- actionsbench-0.1.1/tests/test_cli.py +330 -0
- actionsbench-0.1.1/tests/test_corpus.py +199 -0
- actionsbench-0.1.1/tests/test_integration_actionlint.py +63 -0
- actionsbench-0.1.1/tests/test_integration_zizmor.py +97 -0
- actionsbench-0.1.1/tests/test_regions.py +162 -0
- actionsbench-0.1.1/tests/test_runner.py +187 -0
- actionsbench-0.1.1/tests/test_sarif.py +136 -0
- actionsbench-0.1.1/tests/test_scoring.py +354 -0
- actionsbench-0.1.1/tests/test_taxonomy_schema.py +40 -0
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
.venv/
|
|
5
|
+
*.egg-info/
|
|
6
|
+
build/
|
|
7
|
+
dist/
|
|
8
|
+
|
|
9
|
+
# Tooling caches
|
|
10
|
+
.mypy_cache/
|
|
11
|
+
.pytest_cache/
|
|
12
|
+
.ruff_cache/
|
|
13
|
+
|
|
14
|
+
# Local scanner run output (reports are published deliberately, never committed by accident)
|
|
15
|
+
results/
|
|
16
|
+
|
|
17
|
+
# OS / editor
|
|
18
|
+
.DS_Store
|
|
19
|
+
.idea/
|
|
20
|
+
.vscode/
|
|
21
|
+
|
|
22
|
+
# Pinned scanner binaries installed by scripts/
|
|
23
|
+
.tools/
|
|
24
|
+
|
|
25
|
+
# Node tooling under tools/
|
|
26
|
+
node_modules/
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use ActionsBench, please cite it as below and state the corpus version you used."
|
|
3
|
+
title: "ActionsBench: a labeled benchmark for GitHub Actions workflow security scanners"
|
|
4
|
+
type: dataset
|
|
5
|
+
authors:
|
|
6
|
+
- family-names: Tijani
|
|
7
|
+
given-names: Aliu
|
|
8
|
+
version: 0.1.1
|
|
9
|
+
date-released: 2026-10-06
|
|
10
|
+
license: CC-BY-4.0
|
|
11
|
+
repository-code: "https://github.com/NeesimEra/ActionsBench"
|
|
12
|
+
url: "https://github.com/NeesimEra/ActionsBench"
|
|
13
|
+
abstract: >-
|
|
14
|
+
A labeled corpus of minimized GitHub Actions workflows, each tagged with the weakness it contains or marked
|
|
15
|
+
clean, and a harness that scores workflow security scanners against it per weakness class. The corpus is
|
|
16
|
+
licensed CC BY 4.0, which is the license named here, and the harness code is licensed MIT; both licenses
|
|
17
|
+
ship with the package. Version 0.1.1 is an initial release (pre-1.0) of 44 hand-built cases: all labels are
|
|
18
|
+
proposed by one author and audited against primary sources, but not independently reviewed.
|
|
19
|
+
keywords:
|
|
20
|
+
- github-actions
|
|
21
|
+
- security
|
|
22
|
+
- benchmark
|
|
23
|
+
- static-analysis
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Labels are claims, and the project improves when they are challenged. There are three
|
|
4
|
+
ways to help: add a case, challenge a label, or improve the harness.
|
|
5
|
+
|
|
6
|
+
## Development setup
|
|
7
|
+
|
|
8
|
+
Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
uv sync
|
|
12
|
+
uv run ruff check . && uv run ruff format --check .
|
|
13
|
+
uv run mypy
|
|
14
|
+
uv run pytest
|
|
15
|
+
uv run actionsbench validate
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
CI runs the same commands on Python 3.11 and 3.13.
|
|
19
|
+
|
|
20
|
+
Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/)
|
|
21
|
+
(`feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, with a scope when useful, for
|
|
22
|
+
example `feat(corpus): add AB-INJ-0008`). Keep unrelated changes in separate commits.
|
|
23
|
+
|
|
24
|
+
## Branches and pull requests
|
|
25
|
+
|
|
26
|
+
- **`dev` is the default branch and the only target for pull requests.** Branch from `dev`
|
|
27
|
+
(for example `feat/...`, `fix/...`, `docs/...`) and open the pull request against `dev`. It
|
|
28
|
+
is squash-merged, and the pull request title becomes the commit title, so write it as a
|
|
29
|
+
Conventional Commit.
|
|
30
|
+
- **`main` is the released state.** It changes only through a release pull request from `dev`,
|
|
31
|
+
merged with a merge commit by a maintainer. Release tags (`v*`) are created on `main` and are
|
|
32
|
+
immutable.
|
|
33
|
+
- The required check **PR target** fails any pull request whose base is not `dev`, except the
|
|
34
|
+
release pull request from `dev` into `main`. If it fails, change the base branch of your pull
|
|
35
|
+
request to `dev`; the check re-runs on its own.
|
|
36
|
+
- Direct pushes to `dev` and `main` are blocked. Both CI checks must pass.
|
|
37
|
+
|
|
38
|
+
Reasoning and trade-offs: [ADR 0004](docs/adr/0004-branching-model.md).
|
|
39
|
+
|
|
40
|
+
### Releasing (maintainers)
|
|
41
|
+
|
|
42
|
+
1. Update `corpus/CHANGELOG.md` on `dev` through a normal pull request.
|
|
43
|
+
2. Open a pull request from `dev` to `main` titled `release: vX.Y.Z` and merge it with a merge
|
|
44
|
+
commit.
|
|
45
|
+
3. Create the GitHub release for `vX.Y.Z` targeting that merge commit; this creates the tag. Tags cannot be
|
|
46
|
+
moved or deleted. Publishing to PyPI and archiving on Zenodo follow from the release; the one-time setup and
|
|
47
|
+
the full sequence are in [docs/publishing.md](docs/publishing.md).
|
|
48
|
+
|
|
49
|
+
## Adding a case
|
|
50
|
+
|
|
51
|
+
Read [docs/methodology.md](docs/methodology.md) first. In short:
|
|
52
|
+
|
|
53
|
+
1. Create `corpus/cases/<id>/` containing the workflow files under `.github/` and a
|
|
54
|
+
`case.yaml`. Use the next free ID. The ID prefix is the case's primary weakness class
|
|
55
|
+
(`INJ`, `PIN`, ...) or `NEG` for a clean case that looks risky but is safe.
|
|
56
|
+
2. Keep the case minimal and single-purpose. Pin every `uses:` to a full commit SHA and
|
|
57
|
+
set `permissions:` unless the case is about exactly that, so that only the intended
|
|
58
|
+
weakness is labeled.
|
|
59
|
+
3. Anchor each expected finding at the **sink**: the line where untrusted data reaches the
|
|
60
|
+
interpreter. Put a short `evidence` substring that appears on that line.
|
|
61
|
+
4. Write a `rationale` a reviewer can check, and add `references`. If you reasoned the
|
|
62
|
+
label from first principles and found no independent source, say so in the rationale.
|
|
63
|
+
5. Leave `status: proposed` and `reviewer: null`. An independent reviewer moves a case to
|
|
64
|
+
`agreed`. You cannot review your own case.
|
|
65
|
+
6. Run `uv run actionsbench validate`.
|
|
66
|
+
|
|
67
|
+
Never add a workflow from a real repository that still has a live vulnerability, real
|
|
68
|
+
secrets, or anything that works as a real exploit. See [SECURITY.md](SECURITY.md).
|
|
69
|
+
|
|
70
|
+
## Challenging a label
|
|
71
|
+
|
|
72
|
+
Open an issue with the **Challenge a label** template. Include evidence: documentation,
|
|
73
|
+
an advisory, or a minimal reproduction. A challenge that cannot be resolved marks the case
|
|
74
|
+
`disputed`; it is not deleted. The outcome is recorded in the case's `notes`.
|
|
75
|
+
|
|
76
|
+
## Reviewing a case
|
|
77
|
+
|
|
78
|
+
Read the workflow, decide independently whether the label is right, then set `status:
|
|
79
|
+
agreed` and your handle as `reviewer` in a pull request, or comment on the case's pull
|
|
80
|
+
request or issue if you disagree.
|
|
81
|
+
|
|
82
|
+
## Adding a scanner adapter
|
|
83
|
+
|
|
84
|
+
Two adapters exist so far (zizmor and actionlint). Add another only after checking the tool's
|
|
85
|
+
real output:
|
|
86
|
+
|
|
87
|
+
1. Run the tool by hand on an isolated copy of a case's `.github/` and read its raw output.
|
|
88
|
+
Record the version and the exact flags that make the run deterministic.
|
|
89
|
+
2. Add `src/actionsbench/adapters/<tool>.py` implementing the contract in `adapters/base.py`:
|
|
90
|
+
the command, accepted exit codes, a parser, a pinned version, and a rule map. Use `prepare`
|
|
91
|
+
if the tool needs something in the isolated copy (actionlint needs a project marker) and
|
|
92
|
+
`probe_version` if the tool does not print its version in its output. If the tool is a
|
|
93
|
+
binary, add a pinned installer under `scripts/` that verifies a checksum.
|
|
94
|
+
If any setting changes the tool's verdict (zizmor's persona is one), expose it through
|
|
95
|
+
`from_options`, pass it to the tool explicitly, and report it in `configuration`; the
|
|
96
|
+
baseline is the tool's default made explicit.
|
|
97
|
+
3. Map only rules whose meaning you have reviewed against a benchmark class. Everything else
|
|
98
|
+
stays unmapped and is reported by the runner. Record each decision, mapped or deliberately not
|
|
99
|
+
mapped, in [docs/rule-mappings.md](docs/rule-mappings.md), including where you disagree with
|
|
100
|
+
the source study. Never map a rule just because it fired on a labeled case. Set `scope` to the classes that have a
|
|
101
|
+
reviewed mapping, not to everything the tool claims.
|
|
102
|
+
4. Add unit tests with a trimmed real output sample, and an opt-in integration test
|
|
103
|
+
(`pytest -m integration`) that runs the real tool.
|
|
104
|
+
5. Register it in `adapters/__init__.py` and say in the pull request which rules you mapped and
|
|
105
|
+
why.
|
|
106
|
+
|
|
107
|
+
Any scanner can also be scored without an adapter by producing a findings file that matches
|
|
108
|
+
[findings.schema.json](src/actionsbench/schemas/findings.schema.json).
|
|
109
|
+
|
|
110
|
+
## License
|
|
111
|
+
|
|
112
|
+
By contributing you agree that your contribution is licensed under the license of the part
|
|
113
|
+
of the repository it touches: MIT for code, CC BY 4.0 for the corpus under `corpus/`.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Aliu Tijani
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: actionsbench
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
|
|
5
|
+
Project-URL: Homepage, https://github.com/NeesimEra/ActionsBench
|
|
6
|
+
Project-URL: Repository, https://github.com/NeesimEra/ActionsBench
|
|
7
|
+
Project-URL: Issues, https://github.com/NeesimEra/ActionsBench/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/NeesimEra/ActionsBench/blob/main/corpus/CHANGELOG.md
|
|
9
|
+
Author: Aliu Tijani
|
|
10
|
+
License-Expression: MIT AND CC-BY-4.0
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
License-File: corpus/LICENSE
|
|
13
|
+
Keywords: benchmark,ci,github-actions,security,static-analysis
|
|
14
|
+
Classifier: Development Status :: 3 - Alpha
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Intended Audience :: Information Technology
|
|
17
|
+
Classifier: Operating System :: OS Independent
|
|
18
|
+
Classifier: Programming Language :: Python :: 3
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
+
Classifier: Topic :: Security
|
|
23
|
+
Classifier: Typing :: Typed
|
|
24
|
+
Requires-Python: >=3.11
|
|
25
|
+
Requires-Dist: jsonschema>=4.21
|
|
26
|
+
Requires-Dist: pyyaml>=6.0
|
|
27
|
+
Description-Content-Type: text/markdown
|
|
28
|
+
|
|
29
|
+
# ActionsBench
|
|
30
|
+
|
|
31
|
+
A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
|
|
32
|
+
|
|
33
|
+
> **Status: initial release (0.1.0, pre-1.0).** The corpus is a small hand-built set (44 cases) and every label is still
|
|
34
|
+
> `proposed` (one author). The facts behind them were audited against primary sources
|
|
35
|
+
> ([docs/evidence-audit.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/evidence-audit.md)), but no label has been independently reviewed.
|
|
36
|
+
> Nothing here is a result yet. It is public early so the
|
|
37
|
+
> labels can be challenged, not because it is finished.
|
|
38
|
+
|
|
39
|
+
## Why this exists
|
|
40
|
+
|
|
41
|
+
Several open-source scanners analyze GitHub Actions workflows for security weaknesses,
|
|
42
|
+
and they disagree widely. In a 2026 study of 9 scanners on 2,722 real workflows, one tool
|
|
43
|
+
reported 107 injection findings where another reported 1,778 on the same files. The same
|
|
44
|
+
study found no verified, labeled dataset of workflows with known weaknesses, so precision
|
|
45
|
+
and recall could not be computed ([arXiv 2601.14455](https://arxiv.org/html/2601.14455v2)).
|
|
46
|
+
|
|
47
|
+
ActionsBench provides the missing piece:
|
|
48
|
+
|
|
49
|
+
1. A **labeled corpus** of minimized workflows, each tagged with the weakness it contains,
|
|
50
|
+
or marked clean.
|
|
51
|
+
2. A **harness** that scores any scanner against the corpus, per weakness class.
|
|
52
|
+
3. **Versioned, reproducible releases** so results stay comparable over time.
|
|
53
|
+
|
|
54
|
+
It is not a scanner, and it does not rank tools. It reports per-class evidence.
|
|
55
|
+
|
|
56
|
+
Full motivation, scope and risks: [docs/PRD.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/PRD.md).
|
|
57
|
+
|
|
58
|
+
## Quickstart
|
|
59
|
+
|
|
60
|
+
Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
uv sync
|
|
64
|
+
uv run actionsbench validate # check every case in corpus/cases
|
|
65
|
+
uv run actionsbench stats # cases per weakness class
|
|
66
|
+
uv run actionsbench run --tool zizmor # needs uvx; add --config persona=auditor for another setting
|
|
67
|
+
sh scripts/install-actionlint.sh # pinned, checksum-verified download into .tools/
|
|
68
|
+
uv run actionsbench run --tool actionlint
|
|
69
|
+
uv run actionsbench score results/zizmor-1.30.1-regular/findings.json
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
`run` executes a scanner that has a reviewed adapter (so far zizmor and actionlint, each scored
|
|
73
|
+
on the injection class only) and writes a findings file; raw output goes to `results/` (gitignored).
|
|
74
|
+
`score` takes any normalized findings file (schema:
|
|
75
|
+
[findings.schema.json](https://github.com/NeesimEra/ActionsBench/blob/main/src/actionsbench/schemas/findings.schema.json)), so a scanner
|
|
76
|
+
without an adapter can still be scored by producing that file. More adapters are planned
|
|
77
|
+
(see [plan.md](https://github.com/NeesimEra/ActionsBench/blob/main/plan.md)).
|
|
78
|
+
|
|
79
|
+
Run the commands from the repository root, or pass `--corpus` explicitly.
|
|
80
|
+
|
|
81
|
+
## How scoring works
|
|
82
|
+
|
|
83
|
+
- A tool is judged only on the weakness classes it declares in `scope`. Anything outside
|
|
84
|
+
that scope is reported as a coverage gap, never as a miss or a false alarm.
|
|
85
|
+
- Cases the tool did not run on are excluded, not counted as clean.
|
|
86
|
+
- Matching is one-to-one (class, file, and the same step, job or top-level key; use
|
|
87
|
+
`--match line` for exact lines), so duplicate findings do
|
|
88
|
+
not inflate recall.
|
|
89
|
+
- Results are per class. There is no overall score on purpose.
|
|
90
|
+
|
|
91
|
+
Details: [docs/methodology.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/methodology.md).
|
|
92
|
+
|
|
93
|
+
## Repository layout
|
|
94
|
+
|
|
95
|
+
| Path | Contents |
|
|
96
|
+
|---|---|
|
|
97
|
+
| `corpus/cases/` | The labeled cases. Each case directory is a fake repository root plus a `case.yaml`. |
|
|
98
|
+
| `src/actionsbench/` | Corpus loader and validator, SARIF parser, scorer, CLI. |
|
|
99
|
+
| `src/actionsbench/schemas/` | JSON Schemas for `case.yaml` and the findings file (the public contracts). |
|
|
100
|
+
| `docs/` | PRD, methodology, rule mappings (the review record), architecture decision records. |
|
|
101
|
+
| `plan.md`, `status.md` | The work plan with human gates, and an append-only progress log. |
|
|
102
|
+
|
|
103
|
+
## A note on the corpus
|
|
104
|
+
|
|
105
|
+
The cases contain deliberately vulnerable workflows. They are inert: GitHub only runs
|
|
106
|
+
workflows under the repository root's `.github/workflows/`, and these live inside
|
|
107
|
+
`corpus/cases/`. Payloads are benign placeholders. Never copy them into a real repository.
|
|
108
|
+
See [SECURITY.md](https://github.com/NeesimEra/ActionsBench/blob/main/SECURITY.md).
|
|
109
|
+
|
|
110
|
+
## Contributing
|
|
111
|
+
|
|
112
|
+
Labels are claims, and they are meant to be challenged. See
|
|
113
|
+
[CONTRIBUTING.md](https://github.com/NeesimEra/ActionsBench/blob/main/CONTRIBUTING.md) for adding cases, challenging a label, and the
|
|
114
|
+
development setup.
|
|
115
|
+
|
|
116
|
+
All pull requests target the `dev` branch. `main` and the `v*` tags are the released
|
|
117
|
+
state (see [ADR 0004](https://github.com/NeesimEra/ActionsBench/blob/main/docs/adr/0004-branching-model.md)).
|
|
118
|
+
|
|
119
|
+
## License
|
|
120
|
+
|
|
121
|
+
- **Code** (`src/`, `tests/`, tooling): [MIT](https://github.com/NeesimEra/ActionsBench/blob/main/LICENSE).
|
|
122
|
+
- **Corpus** (`corpus/`, the labeled cases and their labels):
|
|
123
|
+
[CC BY 4.0](https://github.com/NeesimEra/ActionsBench/blob/main/corpus/LICENSE). Reuse is welcome with attribution; please cite ActionsBench
|
|
124
|
+
and the corpus version you used.
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# ActionsBench
|
|
2
|
+
|
|
3
|
+
A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
|
|
4
|
+
|
|
5
|
+
> **Status: initial release (0.1.0, pre-1.0).** The corpus is a small hand-built set (44 cases) and every label is still
|
|
6
|
+
> `proposed` (one author). The facts behind them were audited against primary sources
|
|
7
|
+
> ([docs/evidence-audit.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/evidence-audit.md)), but no label has been independently reviewed.
|
|
8
|
+
> Nothing here is a result yet. It is public early so the
|
|
9
|
+
> labels can be challenged, not because it is finished.
|
|
10
|
+
|
|
11
|
+
## Why this exists
|
|
12
|
+
|
|
13
|
+
Several open-source scanners analyze GitHub Actions workflows for security weaknesses,
|
|
14
|
+
and they disagree widely. In a 2026 study of 9 scanners on 2,722 real workflows, one tool
|
|
15
|
+
reported 107 injection findings where another reported 1,778 on the same files. The same
|
|
16
|
+
study found no verified, labeled dataset of workflows with known weaknesses, so precision
|
|
17
|
+
and recall could not be computed ([arXiv 2601.14455](https://arxiv.org/html/2601.14455v2)).
|
|
18
|
+
|
|
19
|
+
ActionsBench provides the missing piece:
|
|
20
|
+
|
|
21
|
+
1. A **labeled corpus** of minimized workflows, each tagged with the weakness it contains,
|
|
22
|
+
or marked clean.
|
|
23
|
+
2. A **harness** that scores any scanner against the corpus, per weakness class.
|
|
24
|
+
3. **Versioned, reproducible releases** so results stay comparable over time.
|
|
25
|
+
|
|
26
|
+
It is not a scanner, and it does not rank tools. It reports per-class evidence.
|
|
27
|
+
|
|
28
|
+
Full motivation, scope and risks: [docs/PRD.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/PRD.md).
|
|
29
|
+
|
|
30
|
+
## Quickstart
|
|
31
|
+
|
|
32
|
+
Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
uv sync
|
|
36
|
+
uv run actionsbench validate # check every case in corpus/cases
|
|
37
|
+
uv run actionsbench stats # cases per weakness class
|
|
38
|
+
uv run actionsbench run --tool zizmor # needs uvx; add --config persona=auditor for another setting
|
|
39
|
+
sh scripts/install-actionlint.sh # pinned, checksum-verified download into .tools/
|
|
40
|
+
uv run actionsbench run --tool actionlint
|
|
41
|
+
uv run actionsbench score results/zizmor-1.30.1-regular/findings.json
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
`run` executes a scanner that has a reviewed adapter (so far zizmor and actionlint, each scored
|
|
45
|
+
on the injection class only) and writes a findings file; raw output goes to `results/` (gitignored).
|
|
46
|
+
`score` takes any normalized findings file (schema:
|
|
47
|
+
[findings.schema.json](https://github.com/NeesimEra/ActionsBench/blob/main/src/actionsbench/schemas/findings.schema.json)), so a scanner
|
|
48
|
+
without an adapter can still be scored by producing that file. More adapters are planned
|
|
49
|
+
(see [plan.md](https://github.com/NeesimEra/ActionsBench/blob/main/plan.md)).
|
|
50
|
+
|
|
51
|
+
Run the commands from the repository root, or pass `--corpus` explicitly.
|
|
52
|
+
|
|
53
|
+
## How scoring works
|
|
54
|
+
|
|
55
|
+
- A tool is judged only on the weakness classes it declares in `scope`. Anything outside
|
|
56
|
+
that scope is reported as a coverage gap, never as a miss or a false alarm.
|
|
57
|
+
- Cases the tool did not run on are excluded, not counted as clean.
|
|
58
|
+
- Matching is one-to-one (class, file, and the same step, job or top-level key; use
|
|
59
|
+
`--match line` for exact lines), so duplicate findings do
|
|
60
|
+
not inflate recall.
|
|
61
|
+
- Results are per class. There is no overall score on purpose.
|
|
62
|
+
|
|
63
|
+
Details: [docs/methodology.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/methodology.md).
|
|
64
|
+
|
|
65
|
+
## Repository layout
|
|
66
|
+
|
|
67
|
+
| Path | Contents |
|
|
68
|
+
|---|---|
|
|
69
|
+
| `corpus/cases/` | The labeled cases. Each case directory is a fake repository root plus a `case.yaml`. |
|
|
70
|
+
| `src/actionsbench/` | Corpus loader and validator, SARIF parser, scorer, CLI. |
|
|
71
|
+
| `src/actionsbench/schemas/` | JSON Schemas for `case.yaml` and the findings file (the public contracts). |
|
|
72
|
+
| `docs/` | PRD, methodology, rule mappings (the review record), architecture decision records. |
|
|
73
|
+
| `plan.md`, `status.md` | The work plan with human gates, and an append-only progress log. |
|
|
74
|
+
|
|
75
|
+
## A note on the corpus
|
|
76
|
+
|
|
77
|
+
The cases contain deliberately vulnerable workflows. They are inert: GitHub only runs
|
|
78
|
+
workflows under the repository root's `.github/workflows/`, and these live inside
|
|
79
|
+
`corpus/cases/`. Payloads are benign placeholders. Never copy them into a real repository.
|
|
80
|
+
See [SECURITY.md](https://github.com/NeesimEra/ActionsBench/blob/main/SECURITY.md).
|
|
81
|
+
|
|
82
|
+
## Contributing
|
|
83
|
+
|
|
84
|
+
Labels are claims, and they are meant to be challenged. See
|
|
85
|
+
[CONTRIBUTING.md](https://github.com/NeesimEra/ActionsBench/blob/main/CONTRIBUTING.md) for adding cases, challenging a label, and the
|
|
86
|
+
development setup.
|
|
87
|
+
|
|
88
|
+
All pull requests target the `dev` branch. `main` and the `v*` tags are the released
|
|
89
|
+
state (see [ADR 0004](https://github.com/NeesimEra/ActionsBench/blob/main/docs/adr/0004-branching-model.md)).
|
|
90
|
+
|
|
91
|
+
## License
|
|
92
|
+
|
|
93
|
+
- **Code** (`src/`, `tests/`, tooling): [MIT](https://github.com/NeesimEra/ActionsBench/blob/main/LICENSE).
|
|
94
|
+
- **Corpus** (`corpus/`, the labeled cases and their labels):
|
|
95
|
+
[CC BY 4.0](https://github.com/NeesimEra/ActionsBench/blob/main/corpus/LICENSE). Reuse is welcome with attribution; please cite ActionsBench
|
|
96
|
+
and the corpus version you used.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## What is in this repository
|
|
4
|
+
|
|
5
|
+
ActionsBench is a benchmark for security scanners, so `corpus/cases/` deliberately contains
|
|
6
|
+
vulnerable GitHub Actions workflows. They are inert:
|
|
7
|
+
|
|
8
|
+
- GitHub only runs workflows under the repository root's `.github/workflows/`. The corpus
|
|
9
|
+
files live deeper, inside `corpus/cases/<id>/.github/`, and are never executed.
|
|
10
|
+
- Payloads are benign placeholders. There is no working exploit code and no real secret.
|
|
11
|
+
- Do not copy corpus workflows into a real repository.
|
|
12
|
+
|
|
13
|
+
## What must never be added
|
|
14
|
+
|
|
15
|
+
- A workflow copied from a real repository that still contains a live, unfixed
|
|
16
|
+
vulnerability. Publishing it would hand out an attack path.
|
|
17
|
+
- Real credentials, tokens, or private keys, even revoked ones.
|
|
18
|
+
- Cases derived from a real repository without attribution and a license check
|
|
19
|
+
(see plan.md, gate D).
|
|
20
|
+
|
|
21
|
+
Because the corpus is licensed CC BY 4.0, anything added to it becomes public and
|
|
22
|
+
redistributable. Treat a contribution as permanent.
|
|
23
|
+
|
|
24
|
+
## Reporting a problem
|
|
25
|
+
|
|
26
|
+
If you find a case that exposes a live issue in a real project, or a vulnerability in the
|
|
27
|
+
harness itself (for example, a way a crafted `case.yaml` could read files outside its
|
|
28
|
+
directory), please report it privately rather than opening a public issue.
|
|
29
|
+
|
|
30
|
+
Use GitHub's private vulnerability reporting: open the repository's **Security** tab and
|
|
31
|
+
choose **Report a vulnerability**. Only the maintainers can see the report.
|
|
32
|
+
|
|
33
|
+
## Supported versions
|
|
34
|
+
|
|
35
|
+
Pre-alpha. There are no supported releases yet.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Corpus changelog
|
|
2
|
+
|
|
3
|
+
Corpus versions are immutable once released. A new version adds or amends cases; it never
|
|
4
|
+
silently rewrites old ones. Each case records the version that introduced it in `added_in`.
|
|
5
|
+
|
|
6
|
+
## 0.1.1 (2026-10-06)
|
|
7
|
+
|
|
8
|
+
The corpus is unchanged from 0.1.0: the same 44 cases with the same labels. This release changes how it is
|
|
9
|
+
packaged. The wheel now bundles the corpus and both licenses, so a `pip install` is usable on its own, and the
|
|
10
|
+
CLI falls back to the bundled corpus when there is no `corpus/cases` in the working directory. The version of
|
|
11
|
+
0.1.0's wheel did not contain the corpus.
|
|
12
|
+
|
|
13
|
+
## 0.1.0 (2026-10-04)
|
|
14
|
+
|
|
15
|
+
Seed set of 44 cases (26 positive, 18 negative), all `proposed`.
|
|
16
|
+
|
|
17
|
+
Injection (first batch):
|
|
18
|
+
|
|
19
|
+
- AB-INJ-0001: untrusted value wrapped in `format()`
|
|
20
|
+
- AB-INJ-0002: `contains(...) && untrusted value`
|
|
21
|
+
- AB-INJ-0003: untrusted value passed through `toJSON()`
|
|
22
|
+
- AB-INJ-0004: branch name (`github.ref_name`)
|
|
23
|
+
- AB-INJ-0005: commit author name (`head_commit.author.name`)
|
|
24
|
+
- AB-INJ-0006: untrusted value in an `actions/github-script` script
|
|
25
|
+
- AB-INJ-0007: composite action sink with a tainted input and shallow indentation
|
|
26
|
+
- AB-NEG-0001: untrusted value passed through an environment variable
|
|
27
|
+
- AB-NEG-0002: `contains()` used as a boolean condition only
|
|
28
|
+
|
|
29
|
+
Other classes (second batch, each positive has a clean twin):
|
|
30
|
+
|
|
31
|
+
- AB-PIN-0001, AB-PIN-0002: action referenced by a tag, by a branch
|
|
32
|
+
- AB-PRM-0001: `permissions: write-all`
|
|
33
|
+
- AB-TRG-0001: `pull_request_target` checking out and building the pull request head
|
|
34
|
+
- AB-SEC-0001: `secrets: inherit` to a reusable workflow
|
|
35
|
+
- AB-KVC-0001: action pinned inside a published advisory's vulnerable range
|
|
36
|
+
- AB-RUN-0001: retired `ubuntu-18.04` runner image
|
|
37
|
+
- AB-CTL-0001: constant `if` condition
|
|
38
|
+
- AB-ART-0001: checkout credentials uploaded in an artifact
|
|
39
|
+
- AB-NEG-0003 to 0009: clean twins for the above
|
|
40
|
+
|
|
41
|
+
Third batch (harder variants and more than one positive per class, each with a clean twin):
|
|
42
|
+
|
|
43
|
+
- AB-INJ-0008: untrusted value on a deep line of a multi-line script
|
|
44
|
+
- AB-INJ-0009, AB-INJ-0010: issue title, review comment body
|
|
45
|
+
- AB-PIN-0003: reusable workflow referenced by a tag
|
|
46
|
+
- AB-PRM-0002: workflow-level `contents: write`
|
|
47
|
+
- AB-SEC-0002: hardcoded container registry password
|
|
48
|
+
- AB-SEC-0003: `toJSON(secrets)` in a step's environment
|
|
49
|
+
- AB-KVC-0002: `shivammathur/setup-php` 2.37.0, inside GHSA-pqwm-q9pv-ph8r's vulnerable range
|
|
50
|
+
- AB-RUN-0002: retired `macos-10.15` runner image
|
|
51
|
+
- AB-ART-0002: archive downloaded and unpacked without an integrity check
|
|
52
|
+
- AB-NEG-0010 to 0018: clean twins and probes (a `pull_request_target` trigger with nothing untrusted,
|
|
53
|
+
a multi-line script via the environment, safe contexts, a pinned reusable workflow, credentials
|
|
54
|
+
from secrets, one named secret, patched setup-php, `macos-latest`, a verified download)
|
|
55
|
+
|
|
56
|
+
Before this release an evidence audit checked the facts behind every label (docs/evidence-audit.md). It
|
|
57
|
+
corrected AB-ART-0001 and AB-NEG-0008, whose first draft pinned a checkout version where the described
|
|
58
|
+
weakness does not exist (they now pin 4.2.2), and it demonstrated the injection cases with GitHub's own
|
|
59
|
+
expression engine. No label has been independently reviewed: every case is still `proposed`.
|