actionsbench 0.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (144) hide show
  1. actionsbench-0.1.1/.gitignore +26 -0
  2. actionsbench-0.1.1/CITATION.cff +23 -0
  3. actionsbench-0.1.1/CONTRIBUTING.md +113 -0
  4. actionsbench-0.1.1/LICENSE +21 -0
  5. actionsbench-0.1.1/PKG-INFO +124 -0
  6. actionsbench-0.1.1/README.md +96 -0
  7. actionsbench-0.1.1/SECURITY.md +35 -0
  8. actionsbench-0.1.1/corpus/CHANGELOG.md +59 -0
  9. actionsbench-0.1.1/corpus/LICENSE +396 -0
  10. actionsbench-0.1.1/corpus/README.md +77 -0
  11. actionsbench-0.1.1/corpus/cases/AB-ART-0001/.github/workflows/artipacked.yml +19 -0
  12. actionsbench-0.1.1/corpus/cases/AB-ART-0001/case.yaml +39 -0
  13. actionsbench-0.1.1/corpus/cases/AB-ART-0002/.github/workflows/unverified-download.yml +15 -0
  14. actionsbench-0.1.1/corpus/cases/AB-ART-0002/case.yaml +31 -0
  15. actionsbench-0.1.1/corpus/cases/AB-CTL-0001/.github/workflows/constant-condition.yml +14 -0
  16. actionsbench-0.1.1/corpus/cases/AB-CTL-0001/case.yaml +24 -0
  17. actionsbench-0.1.1/corpus/cases/AB-INJ-0001/.github/workflows/format-wrapping.yml +13 -0
  18. actionsbench-0.1.1/corpus/cases/AB-INJ-0001/case.yaml +25 -0
  19. actionsbench-0.1.1/corpus/cases/AB-INJ-0002/.github/workflows/contains-and-value.yml +13 -0
  20. actionsbench-0.1.1/corpus/cases/AB-INJ-0002/case.yaml +25 -0
  21. actionsbench-0.1.1/corpus/cases/AB-INJ-0003/.github/workflows/tojson.yml +13 -0
  22. actionsbench-0.1.1/corpus/cases/AB-INJ-0003/case.yaml +23 -0
  23. actionsbench-0.1.1/corpus/cases/AB-INJ-0004/.github/workflows/ref-name.yml +13 -0
  24. actionsbench-0.1.1/corpus/cases/AB-INJ-0004/case.yaml +27 -0
  25. actionsbench-0.1.1/corpus/cases/AB-INJ-0005/.github/workflows/commit-author.yml +13 -0
  26. actionsbench-0.1.1/corpus/cases/AB-INJ-0005/case.yaml +24 -0
  27. actionsbench-0.1.1/corpus/cases/AB-INJ-0006/.github/workflows/github-script.yml +16 -0
  28. actionsbench-0.1.1/corpus/cases/AB-INJ-0006/case.yaml +27 -0
  29. actionsbench-0.1.1/corpus/cases/AB-INJ-0007/.github/actions/greet/action.yml +11 -0
  30. actionsbench-0.1.1/corpus/cases/AB-INJ-0007/.github/workflows/caller.yml +15 -0
  31. actionsbench-0.1.1/corpus/cases/AB-INJ-0007/case.yaml +29 -0
  32. actionsbench-0.1.1/corpus/cases/AB-INJ-0008/.github/workflows/multiline-script.yml +16 -0
  33. actionsbench-0.1.1/corpus/cases/AB-INJ-0008/case.yaml +23 -0
  34. actionsbench-0.1.1/corpus/cases/AB-INJ-0009/.github/workflows/issue-title.yml +13 -0
  35. actionsbench-0.1.1/corpus/cases/AB-INJ-0009/case.yaml +20 -0
  36. actionsbench-0.1.1/corpus/cases/AB-INJ-0010/.github/workflows/review-comment.yml +13 -0
  37. actionsbench-0.1.1/corpus/cases/AB-INJ-0010/case.yaml +24 -0
  38. actionsbench-0.1.1/corpus/cases/AB-KVC-0001/.github/workflows/vulnerable-download.yml +15 -0
  39. actionsbench-0.1.1/corpus/cases/AB-KVC-0001/case.yaml +26 -0
  40. actionsbench-0.1.1/corpus/cases/AB-KVC-0002/.github/workflows/vulnerable-setup-php.yml +15 -0
  41. actionsbench-0.1.1/corpus/cases/AB-KVC-0002/case.yaml +25 -0
  42. actionsbench-0.1.1/corpus/cases/AB-NEG-0001/.github/workflows/env-indirection.yml +15 -0
  43. actionsbench-0.1.1/corpus/cases/AB-NEG-0001/case.yaml +16 -0
  44. actionsbench-0.1.1/corpus/cases/AB-NEG-0002/.github/workflows/boolean-contains.yml +14 -0
  45. actionsbench-0.1.1/corpus/cases/AB-NEG-0002/case.yaml +15 -0
  46. actionsbench-0.1.1/corpus/cases/AB-NEG-0003/.github/workflows/pinned.yml +15 -0
  47. actionsbench-0.1.1/corpus/cases/AB-NEG-0003/case.yaml +17 -0
  48. actionsbench-0.1.1/corpus/cases/AB-NEG-0004/.github/workflows/read-only.yml +13 -0
  49. actionsbench-0.1.1/corpus/cases/AB-NEG-0004/case.yaml +16 -0
  50. actionsbench-0.1.1/corpus/cases/AB-NEG-0005/.github/workflows/pull-request-head.yml +18 -0
  51. actionsbench-0.1.1/corpus/cases/AB-NEG-0005/case.yaml +17 -0
  52. actionsbench-0.1.1/corpus/cases/AB-NEG-0006/.github/workflows/explicit-secret.yml +12 -0
  53. actionsbench-0.1.1/corpus/cases/AB-NEG-0006/case.yaml +17 -0
  54. actionsbench-0.1.1/corpus/cases/AB-NEG-0007/.github/workflows/patched-download.yml +15 -0
  55. actionsbench-0.1.1/corpus/cases/AB-NEG-0007/case.yaml +15 -0
  56. actionsbench-0.1.1/corpus/cases/AB-NEG-0008/.github/workflows/no-persist.yml +21 -0
  57. actionsbench-0.1.1/corpus/cases/AB-NEG-0008/case.yaml +19 -0
  58. actionsbench-0.1.1/corpus/cases/AB-NEG-0009/.github/workflows/real-condition.yml +14 -0
  59. actionsbench-0.1.1/corpus/cases/AB-NEG-0009/case.yaml +14 -0
  60. actionsbench-0.1.1/corpus/cases/AB-NEG-0010/.github/workflows/trigger-only.yml +13 -0
  61. actionsbench-0.1.1/corpus/cases/AB-NEG-0010/case.yaml +21 -0
  62. actionsbench-0.1.1/corpus/cases/AB-NEG-0011/.github/workflows/multiline-safe.yml +18 -0
  63. actionsbench-0.1.1/corpus/cases/AB-NEG-0011/case.yaml +16 -0
  64. actionsbench-0.1.1/corpus/cases/AB-NEG-0012/.github/workflows/safe-contexts.yml +13 -0
  65. actionsbench-0.1.1/corpus/cases/AB-NEG-0012/case.yaml +22 -0
  66. actionsbench-0.1.1/corpus/cases/AB-NEG-0013/.github/workflows/reusable-pinned.yml +10 -0
  67. actionsbench-0.1.1/corpus/cases/AB-NEG-0013/case.yaml +18 -0
  68. actionsbench-0.1.1/corpus/cases/AB-NEG-0014/.github/workflows/secret-credentials.yml +18 -0
  69. actionsbench-0.1.1/corpus/cases/AB-NEG-0014/case.yaml +17 -0
  70. actionsbench-0.1.1/corpus/cases/AB-NEG-0015/.github/workflows/named-secret.yml +15 -0
  71. actionsbench-0.1.1/corpus/cases/AB-NEG-0015/case.yaml +14 -0
  72. actionsbench-0.1.1/corpus/cases/AB-NEG-0016/.github/workflows/patched-setup-php.yml +15 -0
  73. actionsbench-0.1.1/corpus/cases/AB-NEG-0016/case.yaml +14 -0
  74. actionsbench-0.1.1/corpus/cases/AB-NEG-0017/.github/workflows/current-macos.yml +13 -0
  75. actionsbench-0.1.1/corpus/cases/AB-NEG-0017/case.yaml +14 -0
  76. actionsbench-0.1.1/corpus/cases/AB-NEG-0018/.github/workflows/verified-download.yml +16 -0
  77. actionsbench-0.1.1/corpus/cases/AB-NEG-0018/case.yaml +16 -0
  78. actionsbench-0.1.1/corpus/cases/AB-PIN-0001/.github/workflows/tag-ref.yml +15 -0
  79. actionsbench-0.1.1/corpus/cases/AB-PIN-0001/case.yaml +25 -0
  80. actionsbench-0.1.1/corpus/cases/AB-PIN-0002/.github/workflows/branch-ref.yml +15 -0
  81. actionsbench-0.1.1/corpus/cases/AB-PIN-0002/case.yaml +24 -0
  82. actionsbench-0.1.1/corpus/cases/AB-PIN-0003/.github/workflows/reusable-tag.yml +10 -0
  83. actionsbench-0.1.1/corpus/cases/AB-PIN-0003/case.yaml +28 -0
  84. actionsbench-0.1.1/corpus/cases/AB-PRM-0001/.github/workflows/write-all.yml +12 -0
  85. actionsbench-0.1.1/corpus/cases/AB-PRM-0001/case.yaml +25 -0
  86. actionsbench-0.1.1/corpus/cases/AB-PRM-0002/.github/workflows/workflow-write.yml +13 -0
  87. actionsbench-0.1.1/corpus/cases/AB-PRM-0002/case.yaml +26 -0
  88. actionsbench-0.1.1/corpus/cases/AB-RUN-0001/.github/workflows/retired-runner.yml +13 -0
  89. actionsbench-0.1.1/corpus/cases/AB-RUN-0001/case.yaml +25 -0
  90. actionsbench-0.1.1/corpus/cases/AB-RUN-0002/.github/workflows/retired-macos.yml +13 -0
  91. actionsbench-0.1.1/corpus/cases/AB-RUN-0002/case.yaml +25 -0
  92. actionsbench-0.1.1/corpus/cases/AB-SEC-0001/.github/workflows/inherit.yml +11 -0
  93. actionsbench-0.1.1/corpus/cases/AB-SEC-0001/case.yaml +27 -0
  94. actionsbench-0.1.1/corpus/cases/AB-SEC-0002/.github/workflows/hardcoded-credentials.yml +18 -0
  95. actionsbench-0.1.1/corpus/cases/AB-SEC-0002/case.yaml +24 -0
  96. actionsbench-0.1.1/corpus/cases/AB-SEC-0003/.github/workflows/all-secrets.yml +15 -0
  97. actionsbench-0.1.1/corpus/cases/AB-SEC-0003/case.yaml +22 -0
  98. actionsbench-0.1.1/corpus/cases/AB-TRG-0001/.github/workflows/pwn-request.yml +18 -0
  99. actionsbench-0.1.1/corpus/cases/AB-TRG-0001/case.yaml +29 -0
  100. actionsbench-0.1.1/docs/PRD.md +313 -0
  101. actionsbench-0.1.1/docs/adr/0001-python-and-minimal-dependencies.md +53 -0
  102. actionsbench-0.1.1/docs/adr/0002-corpus-format.md +58 -0
  103. actionsbench-0.1.1/docs/adr/0003-scanner-scope-and-scoring.md +103 -0
  104. actionsbench-0.1.1/docs/adr/0004-branching-model.md +60 -0
  105. actionsbench-0.1.1/docs/adr/README.md +11 -0
  106. actionsbench-0.1.1/docs/evidence-audit.md +141 -0
  107. actionsbench-0.1.1/docs/methodology.md +219 -0
  108. actionsbench-0.1.1/docs/publishing.md +84 -0
  109. actionsbench-0.1.1/docs/rule-mappings.md +103 -0
  110. actionsbench-0.1.1/pyproject.toml +98 -0
  111. actionsbench-0.1.1/scripts/check_wheel.py +116 -0
  112. actionsbench-0.1.1/scripts/install-actionlint.sh +73 -0
  113. actionsbench-0.1.1/src/actionsbench/__init__.py +3 -0
  114. actionsbench-0.1.1/src/actionsbench/__main__.py +3 -0
  115. actionsbench-0.1.1/src/actionsbench/adapters/__init__.py +27 -0
  116. actionsbench-0.1.1/src/actionsbench/adapters/actionlint.py +169 -0
  117. actionsbench-0.1.1/src/actionsbench/adapters/base.py +83 -0
  118. actionsbench-0.1.1/src/actionsbench/adapters/zizmor.py +118 -0
  119. actionsbench-0.1.1/src/actionsbench/cli.py +285 -0
  120. actionsbench-0.1.1/src/actionsbench/corpus.py +227 -0
  121. actionsbench-0.1.1/src/actionsbench/models.py +80 -0
  122. actionsbench-0.1.1/src/actionsbench/py.typed +0 -0
  123. actionsbench-0.1.1/src/actionsbench/regions.py +127 -0
  124. actionsbench-0.1.1/src/actionsbench/report.py +87 -0
  125. actionsbench-0.1.1/src/actionsbench/runner.py +190 -0
  126. actionsbench-0.1.1/src/actionsbench/sarif.py +92 -0
  127. actionsbench-0.1.1/src/actionsbench/schemas/case.schema.json +67 -0
  128. actionsbench-0.1.1/src/actionsbench/schemas/findings.schema.json +70 -0
  129. actionsbench-0.1.1/src/actionsbench/scoring.py +218 -0
  130. actionsbench-0.1.1/src/actionsbench/taxonomy.py +43 -0
  131. actionsbench-0.1.1/tests/__init__.py +0 -0
  132. actionsbench-0.1.1/tests/conftest.py +60 -0
  133. actionsbench-0.1.1/tests/fakes.py +125 -0
  134. actionsbench-0.1.1/tests/test_adapter_actionlint.py +216 -0
  135. actionsbench-0.1.1/tests/test_adapter_zizmor.py +196 -0
  136. actionsbench-0.1.1/tests/test_cli.py +330 -0
  137. actionsbench-0.1.1/tests/test_corpus.py +199 -0
  138. actionsbench-0.1.1/tests/test_integration_actionlint.py +63 -0
  139. actionsbench-0.1.1/tests/test_integration_zizmor.py +97 -0
  140. actionsbench-0.1.1/tests/test_regions.py +162 -0
  141. actionsbench-0.1.1/tests/test_runner.py +187 -0
  142. actionsbench-0.1.1/tests/test_sarif.py +136 -0
  143. actionsbench-0.1.1/tests/test_scoring.py +354 -0
  144. actionsbench-0.1.1/tests/test_taxonomy_schema.py +40 -0
@@ -0,0 +1,26 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ .venv/
5
+ *.egg-info/
6
+ build/
7
+ dist/
8
+
9
+ # Tooling caches
10
+ .mypy_cache/
11
+ .pytest_cache/
12
+ .ruff_cache/
13
+
14
+ # Local scanner run output (reports are published deliberately, never committed by accident)
15
+ results/
16
+
17
+ # OS / editor
18
+ .DS_Store
19
+ .idea/
20
+ .vscode/
21
+
22
+ # Pinned scanner binaries installed by scripts/
23
+ .tools/
24
+
25
+ # Node tooling under tools/
26
+ node_modules/
@@ -0,0 +1,23 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use ActionsBench, please cite it as below and state the corpus version you used."
3
+ title: "ActionsBench: a labeled benchmark for GitHub Actions workflow security scanners"
4
+ type: dataset
5
+ authors:
6
+ - family-names: Tijani
7
+ given-names: Aliu
8
+ version: 0.1.1
9
+ date-released: 2026-10-06
10
+ license: CC-BY-4.0
11
+ repository-code: "https://github.com/NeesimEra/ActionsBench"
12
+ url: "https://github.com/NeesimEra/ActionsBench"
13
+ abstract: >-
14
+ A labeled corpus of minimized GitHub Actions workflows, each tagged with the weakness it contains or marked
15
+ clean, and a harness that scores workflow security scanners against it per weakness class. The corpus is
16
+ licensed CC BY 4.0, which is the license named here, and the harness code is licensed MIT; both licenses
17
+ ship with the package. Version 0.1.1 is an initial release (pre-1.0) of 44 hand-built cases: all labels are
18
+ proposed by one author and audited against primary sources, but not independently reviewed.
19
+ keywords:
20
+ - github-actions
21
+ - security
22
+ - benchmark
23
+ - static-analysis
@@ -0,0 +1,113 @@
1
+ # Contributing
2
+
3
+ Labels are claims, and the project improves when they are challenged. There are three
4
+ ways to help: add a case, challenge a label, or improve the harness.
5
+
6
+ ## Development setup
7
+
8
+ Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
9
+
10
+ ```bash
11
+ uv sync
12
+ uv run ruff check . && uv run ruff format --check .
13
+ uv run mypy
14
+ uv run pytest
15
+ uv run actionsbench validate
16
+ ```
17
+
18
+ CI runs the same commands on Python 3.11 and 3.13.
19
+
20
+ Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/)
21
+ (`feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`, with a scope when useful, for
22
+ example `feat(corpus): add AB-INJ-0008`). Keep unrelated changes in separate commits.
23
+
24
+ ## Branches and pull requests
25
+
26
+ - **`dev` is the default branch and the only target for pull requests.** Branch from `dev`
27
+ (for example `feat/...`, `fix/...`, `docs/...`) and open the pull request against `dev`. It
28
+ is squash-merged, and the pull request title becomes the commit title, so write it as a
29
+ Conventional Commit.
30
+ - **`main` is the released state.** It changes only through a release pull request from `dev`,
31
+ merged with a merge commit by a maintainer. Release tags (`v*`) are created on `main` and are
32
+ immutable.
33
+ - The required check **PR target** fails any pull request whose base is not `dev`, except the
34
+ release pull request from `dev` into `main`. If it fails, change the base branch of your pull
35
+ request to `dev`; the check re-runs on its own.
36
+ - Direct pushes to `dev` and `main` are blocked. Both CI checks must pass.
37
+
38
+ Reasoning and trade-offs: [ADR 0004](docs/adr/0004-branching-model.md).
39
+
40
+ ### Releasing (maintainers)
41
+
42
+ 1. Update `corpus/CHANGELOG.md` on `dev` through a normal pull request.
43
+ 2. Open a pull request from `dev` to `main` titled `release: vX.Y.Z` and merge it with a merge
44
+ commit.
45
+ 3. Create the GitHub release for `vX.Y.Z` targeting that merge commit; this creates the tag. Tags cannot be
46
+ moved or deleted. Publishing to PyPI and archiving on Zenodo follow from the release; the one-time setup and
47
+ the full sequence are in [docs/publishing.md](docs/publishing.md).
48
+
49
+ ## Adding a case
50
+
51
+ Read [docs/methodology.md](docs/methodology.md) first. In short:
52
+
53
+ 1. Create `corpus/cases/<id>/` containing the workflow files under `.github/` and a
54
+ `case.yaml`. Use the next free ID. The ID prefix is the case's primary weakness class
55
+ (`INJ`, `PIN`, ...) or `NEG` for a clean case that looks risky but is safe.
56
+ 2. Keep the case minimal and single-purpose. Pin every `uses:` to a full commit SHA and
57
+ set `permissions:` unless the case is about exactly that, so that only the intended
58
+ weakness is labeled.
59
+ 3. Anchor each expected finding at the **sink**: the line where untrusted data reaches the
60
+ interpreter. Put a short `evidence` substring that appears on that line.
61
+ 4. Write a `rationale` a reviewer can check, and add `references`. If you reasoned the
62
+ label from first principles and found no independent source, say so in the rationale.
63
+ 5. Leave `status: proposed` and `reviewer: null`. An independent reviewer moves a case to
64
+ `agreed`. You cannot review your own case.
65
+ 6. Run `uv run actionsbench validate`.
66
+
67
+ Never add a workflow from a real repository that still has a live vulnerability, real
68
+ secrets, or anything that works as a real exploit. See [SECURITY.md](SECURITY.md).
69
+
70
+ ## Challenging a label
71
+
72
+ Open an issue with the **Challenge a label** template. Include evidence: documentation,
73
+ an advisory, or a minimal reproduction. A challenge that cannot be resolved marks the case
74
+ `disputed`; it is not deleted. The outcome is recorded in the case's `notes`.
75
+
76
+ ## Reviewing a case
77
+
78
+ Read the workflow, decide independently whether the label is right, then set `status:
79
+ agreed` and your handle as `reviewer` in a pull request, or comment on the case's pull
80
+ request or issue if you disagree.
81
+
82
+ ## Adding a scanner adapter
83
+
84
+ Two adapters exist so far (zizmor and actionlint). Add another only after checking the tool's
85
+ real output:
86
+
87
+ 1. Run the tool by hand on an isolated copy of a case's `.github/` and read its raw output.
88
+ Record the version and the exact flags that make the run deterministic.
89
+ 2. Add `src/actionsbench/adapters/<tool>.py` implementing the contract in `adapters/base.py`:
90
+ the command, accepted exit codes, a parser, a pinned version, and a rule map. Use `prepare`
91
+ if the tool needs something in the isolated copy (actionlint needs a project marker) and
92
+ `probe_version` if the tool does not print its version in its output. If the tool is a
93
+ binary, add a pinned installer under `scripts/` that verifies a checksum.
94
+ If any setting changes the tool's verdict (zizmor's persona is one), expose it through
95
+ `from_options`, pass it to the tool explicitly, and report it in `configuration`; the
96
+ baseline is the tool's default made explicit.
97
+ 3. Map only rules whose meaning you have reviewed against a benchmark class. Everything else
98
+ stays unmapped and is reported by the runner. Record each decision, mapped or deliberately not
99
+ mapped, in [docs/rule-mappings.md](docs/rule-mappings.md), including where you disagree with
100
+ the source study. Never map a rule just because it fired on a labeled case. Set `scope` to the classes that have a
101
+ reviewed mapping, not to everything the tool claims.
102
+ 4. Add unit tests with a trimmed real output sample, and an opt-in integration test
103
+ (`pytest -m integration`) that runs the real tool.
104
+ 5. Register it in `adapters/__init__.py` and say in the pull request which rules you mapped and
105
+ why.
106
+
107
+ Any scanner can also be scored without an adapter by producing a findings file that matches
108
+ [findings.schema.json](src/actionsbench/schemas/findings.schema.json).
109
+
110
+ ## License
111
+
112
+ By contributing you agree that your contribution is licensed under the license of the part
113
+ of the repository it touches: MIT for code, CC BY 4.0 for the corpus under `corpus/`.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Aliu Tijani
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,124 @@
1
+ Metadata-Version: 2.5
2
+ Name: actionsbench
3
+ Version: 0.1.1
4
+ Summary: A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
5
+ Project-URL: Homepage, https://github.com/NeesimEra/ActionsBench
6
+ Project-URL: Repository, https://github.com/NeesimEra/ActionsBench
7
+ Project-URL: Issues, https://github.com/NeesimEra/ActionsBench/issues
8
+ Project-URL: Changelog, https://github.com/NeesimEra/ActionsBench/blob/main/corpus/CHANGELOG.md
9
+ Author: Aliu Tijani
10
+ License-Expression: MIT AND CC-BY-4.0
11
+ License-File: LICENSE
12
+ License-File: corpus/LICENSE
13
+ Keywords: benchmark,ci,github-actions,security,static-analysis
14
+ Classifier: Development Status :: 3 - Alpha
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Intended Audience :: Information Technology
17
+ Classifier: Operating System :: OS Independent
18
+ Classifier: Programming Language :: Python :: 3
19
+ Classifier: Programming Language :: Python :: 3.11
20
+ Classifier: Programming Language :: Python :: 3.12
21
+ Classifier: Programming Language :: Python :: 3.13
22
+ Classifier: Topic :: Security
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.11
25
+ Requires-Dist: jsonschema>=4.21
26
+ Requires-Dist: pyyaml>=6.0
27
+ Description-Content-Type: text/markdown
28
+
29
+ # ActionsBench
30
+
31
+ A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
32
+
33
+ > **Status: initial release (0.1.0, pre-1.0).** The corpus is a small hand-built set (44 cases) and every label is still
34
+ > `proposed` (one author). The facts behind them were audited against primary sources
35
+ > ([docs/evidence-audit.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/evidence-audit.md)), but no label has been independently reviewed.
36
+ > Nothing here is a result yet. It is public early so the
37
+ > labels can be challenged, not because it is finished.
38
+
39
+ ## Why this exists
40
+
41
+ Several open-source scanners analyze GitHub Actions workflows for security weaknesses,
42
+ and they disagree widely. In a 2026 study of 9 scanners on 2,722 real workflows, one tool
43
+ reported 107 injection findings where another reported 1,778 on the same files. The same
44
+ study found no verified, labeled dataset of workflows with known weaknesses, so precision
45
+ and recall could not be computed ([arXiv 2601.14455](https://arxiv.org/html/2601.14455v2)).
46
+
47
+ ActionsBench provides the missing piece:
48
+
49
+ 1. A **labeled corpus** of minimized workflows, each tagged with the weakness it contains,
50
+ or marked clean.
51
+ 2. A **harness** that scores any scanner against the corpus, per weakness class.
52
+ 3. **Versioned, reproducible releases** so results stay comparable over time.
53
+
54
+ It is not a scanner, and it does not rank tools. It reports per-class evidence.
55
+
56
+ Full motivation, scope and risks: [docs/PRD.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/PRD.md).
57
+
58
+ ## Quickstart
59
+
60
+ Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
61
+
62
+ ```bash
63
+ uv sync
64
+ uv run actionsbench validate # check every case in corpus/cases
65
+ uv run actionsbench stats # cases per weakness class
66
+ uv run actionsbench run --tool zizmor # needs uvx; add --config persona=auditor for another setting
67
+ sh scripts/install-actionlint.sh # pinned, checksum-verified download into .tools/
68
+ uv run actionsbench run --tool actionlint
69
+ uv run actionsbench score results/zizmor-1.30.1-regular/findings.json
70
+ ```
71
+
72
+ `run` executes a scanner that has a reviewed adapter (so far zizmor and actionlint, each scored
73
+ on the injection class only) and writes a findings file; raw output goes to `results/` (gitignored).
74
+ `score` takes any normalized findings file (schema:
75
+ [findings.schema.json](https://github.com/NeesimEra/ActionsBench/blob/main/src/actionsbench/schemas/findings.schema.json)), so a scanner
76
+ without an adapter can still be scored by producing that file. More adapters are planned
77
+ (see [plan.md](https://github.com/NeesimEra/ActionsBench/blob/main/plan.md)).
78
+
79
+ Run the commands from the repository root, or pass `--corpus` explicitly.
80
+
81
+ ## How scoring works
82
+
83
+ - A tool is judged only on the weakness classes it declares in `scope`. Anything outside
84
+ that scope is reported as a coverage gap, never as a miss or a false alarm.
85
+ - Cases the tool did not run on are excluded, not counted as clean.
86
+ - Matching is one-to-one (class, file, and the same step, job or top-level key; use
87
+ `--match line` for exact lines), so duplicate findings do
88
+ not inflate recall.
89
+ - Results are per class. There is no overall score on purpose.
90
+
91
+ Details: [docs/methodology.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/methodology.md).
92
+
93
+ ## Repository layout
94
+
95
+ | Path | Contents |
96
+ |---|---|
97
+ | `corpus/cases/` | The labeled cases. Each case directory is a fake repository root plus a `case.yaml`. |
98
+ | `src/actionsbench/` | Corpus loader and validator, SARIF parser, scorer, CLI. |
99
+ | `src/actionsbench/schemas/` | JSON Schemas for `case.yaml` and the findings file (the public contracts). |
100
+ | `docs/` | PRD, methodology, rule mappings (the review record), architecture decision records. |
101
+ | `plan.md`, `status.md` | The work plan with human gates, and an append-only progress log. |
102
+
103
+ ## A note on the corpus
104
+
105
+ The cases contain deliberately vulnerable workflows. They are inert: GitHub only runs
106
+ workflows under the repository root's `.github/workflows/`, and these live inside
107
+ `corpus/cases/`. Payloads are benign placeholders. Never copy them into a real repository.
108
+ See [SECURITY.md](https://github.com/NeesimEra/ActionsBench/blob/main/SECURITY.md).
109
+
110
+ ## Contributing
111
+
112
+ Labels are claims, and they are meant to be challenged. See
113
+ [CONTRIBUTING.md](https://github.com/NeesimEra/ActionsBench/blob/main/CONTRIBUTING.md) for adding cases, challenging a label, and the
114
+ development setup.
115
+
116
+ All pull requests target the `dev` branch. `main` and the `v*` tags are the released
117
+ state (see [ADR 0004](https://github.com/NeesimEra/ActionsBench/blob/main/docs/adr/0004-branching-model.md)).
118
+
119
+ ## License
120
+
121
+ - **Code** (`src/`, `tests/`, tooling): [MIT](https://github.com/NeesimEra/ActionsBench/blob/main/LICENSE).
122
+ - **Corpus** (`corpus/`, the labeled cases and their labels):
123
+ [CC BY 4.0](https://github.com/NeesimEra/ActionsBench/blob/main/corpus/LICENSE). Reuse is welcome with attribution; please cite ActionsBench
124
+ and the corpus version you used.
@@ -0,0 +1,96 @@
1
+ # ActionsBench
2
+
3
+ A labeled benchmark and evaluation harness for GitHub Actions workflow security scanners.
4
+
5
+ > **Status: initial release (0.1.0, pre-1.0).** The corpus is a small hand-built set (44 cases) and every label is still
6
+ > `proposed` (one author). The facts behind them were audited against primary sources
7
+ > ([docs/evidence-audit.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/evidence-audit.md)), but no label has been independently reviewed.
8
+ > Nothing here is a result yet. It is public early so the
9
+ > labels can be challenged, not because it is finished.
10
+
11
+ ## Why this exists
12
+
13
+ Several open-source scanners analyze GitHub Actions workflows for security weaknesses,
14
+ and they disagree widely. In a 2026 study of 9 scanners on 2,722 real workflows, one tool
15
+ reported 107 injection findings where another reported 1,778 on the same files. The same
16
+ study found no verified, labeled dataset of workflows with known weaknesses, so precision
17
+ and recall could not be computed ([arXiv 2601.14455](https://arxiv.org/html/2601.14455v2)).
18
+
19
+ ActionsBench provides the missing piece:
20
+
21
+ 1. A **labeled corpus** of minimized workflows, each tagged with the weakness it contains,
22
+ or marked clean.
23
+ 2. A **harness** that scores any scanner against the corpus, per weakness class.
24
+ 3. **Versioned, reproducible releases** so results stay comparable over time.
25
+
26
+ It is not a scanner, and it does not rank tools. It reports per-class evidence.
27
+
28
+ Full motivation, scope and risks: [docs/PRD.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/PRD.md).
29
+
30
+ ## Quickstart
31
+
32
+ Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
33
+
34
+ ```bash
35
+ uv sync
36
+ uv run actionsbench validate # check every case in corpus/cases
37
+ uv run actionsbench stats # cases per weakness class
38
+ uv run actionsbench run --tool zizmor # needs uvx; add --config persona=auditor for another setting
39
+ sh scripts/install-actionlint.sh # pinned, checksum-verified download into .tools/
40
+ uv run actionsbench run --tool actionlint
41
+ uv run actionsbench score results/zizmor-1.30.1-regular/findings.json
42
+ ```
43
+
44
+ `run` executes a scanner that has a reviewed adapter (so far zizmor and actionlint, each scored
45
+ on the injection class only) and writes a findings file; raw output goes to `results/` (gitignored).
46
+ `score` takes any normalized findings file (schema:
47
+ [findings.schema.json](https://github.com/NeesimEra/ActionsBench/blob/main/src/actionsbench/schemas/findings.schema.json)), so a scanner
48
+ without an adapter can still be scored by producing that file. More adapters are planned
49
+ (see [plan.md](https://github.com/NeesimEra/ActionsBench/blob/main/plan.md)).
50
+
51
+ Run the commands from the repository root, or pass `--corpus` explicitly.
52
+
53
+ ## How scoring works
54
+
55
+ - A tool is judged only on the weakness classes it declares in `scope`. Anything outside
56
+ that scope is reported as a coverage gap, never as a miss or a false alarm.
57
+ - Cases the tool did not run on are excluded, not counted as clean.
58
+ - Matching is one-to-one (class, file, and the same step, job or top-level key; use
59
+ `--match line` for exact lines), so duplicate findings do
60
+ not inflate recall.
61
+ - Results are per class. There is no overall score on purpose.
62
+
63
+ Details: [docs/methodology.md](https://github.com/NeesimEra/ActionsBench/blob/main/docs/methodology.md).
64
+
65
+ ## Repository layout
66
+
67
+ | Path | Contents |
68
+ |---|---|
69
+ | `corpus/cases/` | The labeled cases. Each case directory is a fake repository root plus a `case.yaml`. |
70
+ | `src/actionsbench/` | Corpus loader and validator, SARIF parser, scorer, CLI. |
71
+ | `src/actionsbench/schemas/` | JSON Schemas for `case.yaml` and the findings file (the public contracts). |
72
+ | `docs/` | PRD, methodology, rule mappings (the review record), architecture decision records. |
73
+ | `plan.md`, `status.md` | The work plan with human gates, and an append-only progress log. |
74
+
75
+ ## A note on the corpus
76
+
77
+ The cases contain deliberately vulnerable workflows. They are inert: GitHub only runs
78
+ workflows under the repository root's `.github/workflows/`, and these live inside
79
+ `corpus/cases/`. Payloads are benign placeholders. Never copy them into a real repository.
80
+ See [SECURITY.md](https://github.com/NeesimEra/ActionsBench/blob/main/SECURITY.md).
81
+
82
+ ## Contributing
83
+
84
+ Labels are claims, and they are meant to be challenged. See
85
+ [CONTRIBUTING.md](https://github.com/NeesimEra/ActionsBench/blob/main/CONTRIBUTING.md) for adding cases, challenging a label, and the
86
+ development setup.
87
+
88
+ All pull requests target the `dev` branch. `main` and the `v*` tags are the released
89
+ state (see [ADR 0004](https://github.com/NeesimEra/ActionsBench/blob/main/docs/adr/0004-branching-model.md)).
90
+
91
+ ## License
92
+
93
+ - **Code** (`src/`, `tests/`, tooling): [MIT](https://github.com/NeesimEra/ActionsBench/blob/main/LICENSE).
94
+ - **Corpus** (`corpus/`, the labeled cases and their labels):
95
+ [CC BY 4.0](https://github.com/NeesimEra/ActionsBench/blob/main/corpus/LICENSE). Reuse is welcome with attribution; please cite ActionsBench
96
+ and the corpus version you used.
@@ -0,0 +1,35 @@
1
+ # Security policy
2
+
3
+ ## What is in this repository
4
+
5
+ ActionsBench is a benchmark for security scanners, so `corpus/cases/` deliberately contains
6
+ vulnerable GitHub Actions workflows. They are inert:
7
+
8
+ - GitHub only runs workflows under the repository root's `.github/workflows/`. The corpus
9
+ files live deeper, inside `corpus/cases/<id>/.github/`, and are never executed.
10
+ - Payloads are benign placeholders. There is no working exploit code and no real secret.
11
+ - Do not copy corpus workflows into a real repository.
12
+
13
+ ## What must never be added
14
+
15
+ - A workflow copied from a real repository that still contains a live, unfixed
16
+ vulnerability. Publishing it would hand out an attack path.
17
+ - Real credentials, tokens, or private keys, even revoked ones.
18
+ - Cases derived from a real repository without attribution and a license check
19
+ (see plan.md, gate D).
20
+
21
+ Because the corpus is licensed CC BY 4.0, anything added to it becomes public and
22
+ redistributable. Treat a contribution as permanent.
23
+
24
+ ## Reporting a problem
25
+
26
+ If you find a case that exposes a live issue in a real project, or a vulnerability in the
27
+ harness itself (for example, a way a crafted `case.yaml` could read files outside its
28
+ directory), please report it privately rather than opening a public issue.
29
+
30
+ Use GitHub's private vulnerability reporting: open the repository's **Security** tab and
31
+ choose **Report a vulnerability**. Only the maintainers can see the report.
32
+
33
+ ## Supported versions
34
+
35
+ Pre-alpha. There are no supported releases yet.
@@ -0,0 +1,59 @@
1
+ # Corpus changelog
2
+
3
+ Corpus versions are immutable once released. A new version adds or amends cases; it never
4
+ silently rewrites old ones. Each case records the version that introduced it in `added_in`.
5
+
6
+ ## 0.1.1 (2026-10-06)
7
+
8
+ The corpus is unchanged from 0.1.0: the same 44 cases with the same labels. This release changes how it is
9
+ packaged. The wheel now bundles the corpus and both licenses, so a `pip install` is usable on its own, and the
10
+ CLI falls back to the bundled corpus when there is no `corpus/cases` in the working directory. The version of
11
+ 0.1.0's wheel did not contain the corpus.
12
+
13
+ ## 0.1.0 (2026-10-04)
14
+
15
+ Seed set of 44 cases (26 positive, 18 negative), all `proposed`.
16
+
17
+ Injection (first batch):
18
+
19
+ - AB-INJ-0001: untrusted value wrapped in `format()`
20
+ - AB-INJ-0002: `contains(...) && untrusted value`
21
+ - AB-INJ-0003: untrusted value passed through `toJSON()`
22
+ - AB-INJ-0004: branch name (`github.ref_name`)
23
+ - AB-INJ-0005: commit author name (`head_commit.author.name`)
24
+ - AB-INJ-0006: untrusted value in an `actions/github-script` script
25
+ - AB-INJ-0007: composite action sink with a tainted input and shallow indentation
26
+ - AB-NEG-0001: untrusted value passed through an environment variable
27
+ - AB-NEG-0002: `contains()` used as a boolean condition only
28
+
29
+ Other classes (second batch, each positive has a clean twin):
30
+
31
+ - AB-PIN-0001, AB-PIN-0002: action referenced by a tag, by a branch
32
+ - AB-PRM-0001: `permissions: write-all`
33
+ - AB-TRG-0001: `pull_request_target` checking out and building the pull request head
34
+ - AB-SEC-0001: `secrets: inherit` to a reusable workflow
35
+ - AB-KVC-0001: action pinned inside a published advisory's vulnerable range
36
+ - AB-RUN-0001: retired `ubuntu-18.04` runner image
37
+ - AB-CTL-0001: constant `if` condition
38
+ - AB-ART-0001: checkout credentials uploaded in an artifact
39
+ - AB-NEG-0003 to 0009: clean twins for the above
40
+
41
+ Third batch (harder variants and more than one positive per class, each with a clean twin):
42
+
43
+ - AB-INJ-0008: untrusted value on a deep line of a multi-line script
44
+ - AB-INJ-0009, AB-INJ-0010: issue title, review comment body
45
+ - AB-PIN-0003: reusable workflow referenced by a tag
46
+ - AB-PRM-0002: workflow-level `contents: write`
47
+ - AB-SEC-0002: hardcoded container registry password
48
+ - AB-SEC-0003: `toJSON(secrets)` in a step's environment
49
+ - AB-KVC-0002: `shivammathur/setup-php` 2.37.0, inside GHSA-pqwm-q9pv-ph8r's vulnerable range
50
+ - AB-RUN-0002: retired `macos-10.15` runner image
51
+ - AB-ART-0002: archive downloaded and unpacked without an integrity check
52
+ - AB-NEG-0010 to 0018: clean twins and probes (a `pull_request_target` trigger with nothing untrusted,
53
+ a multi-line script via the environment, safe contexts, a pinned reusable workflow, credentials
54
+ from secrets, one named secret, patched setup-php, `macos-latest`, a verified download)
55
+
56
+ Before this release an evidence audit checked the facts behind every label (docs/evidence-audit.md). It
57
+ corrected AB-ART-0001 and AB-NEG-0008, whose first draft pinned a checkout version where the described
58
+ weakness does not exist (they now pin 4.2.2), and it demonstrated the injection cases with GitHub's own
59
+ expression engine. No label has been independently reviewed: every case is still `proposed`.