crewscore 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- crewscore-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md +22 -0
- crewscore-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md +14 -0
- crewscore-0.1.0/.github/ISSUE_TEMPLATE/scoring_feedback.md +37 -0
- crewscore-0.1.0/.github/workflows/crewscore-selftest.yml +118 -0
- crewscore-0.1.0/.github/workflows/example-ci.yml +68 -0
- crewscore-0.1.0/.github/workflows/pytest.yml +37 -0
- crewscore-0.1.0/.github/workflows/release.yml +186 -0
- crewscore-0.1.0/.gitignore +28 -0
- crewscore-0.1.0/AGENTS.md +143 -0
- crewscore-0.1.0/CHANGELOG.md +132 -0
- crewscore-0.1.0/CNAME +1 -0
- crewscore-0.1.0/CONTRIBUTING.md +56 -0
- crewscore-0.1.0/LICENSE +21 -0
- crewscore-0.1.0/PKG-INFO +615 -0
- crewscore-0.1.0/README.md +589 -0
- crewscore-0.1.0/action.yml +212 -0
- crewscore-0.1.0/crewscore/__init__.py +8 -0
- crewscore-0.1.0/crewscore/cli.py +1245 -0
- crewscore-0.1.0/crewscore/export_eval.py +100 -0
- crewscore-0.1.0/crewscore/metrics.py +81 -0
- crewscore-0.1.0/crewscore/profiles.py +104 -0
- crewscore-0.1.0/crewscore/report.py +348 -0
- crewscore-0.1.0/crewscore/rules_catalog.py +248 -0
- crewscore-0.1.0/crewscore/scan.py +222 -0
- crewscore-0.1.0/crewscore/scorers/__init__.py +1 -0
- crewscore-0.1.0/crewscore/scorers/fix_patterns.py +232 -0
- crewscore-0.1.0/crewscore/scorers/structural_analysis.py +508 -0
- crewscore-0.1.0/crewscore/scoring.py +166 -0
- crewscore-0.1.0/crewscore/smells.py +323 -0
- crewscore-0.1.0/crewscore/summary.py +235 -0
- crewscore-0.1.0/crewscore/vendor_scorecard.py +409 -0
- crewscore-0.1.0/crewscore/web_export.py +556 -0
- crewscore-0.1.0/docs/demo.svg +52 -0
- crewscore-0.1.0/docs/hero-demo.gif +0 -0
- crewscore-0.1.0/docs/hero-demo.mp4 +0 -0
- crewscore-0.1.0/docs/next-steps-eval.md +72 -0
- crewscore-0.1.0/docs/social-card.png +0 -0
- crewscore-0.1.0/docs/validation.md +149 -0
- crewscore-0.1.0/examples/corpus/LEADERBOARD.md +39 -0
- crewscore-0.1.0/examples/corpus/README.md +27 -0
- crewscore-0.1.0/examples/corpus/prompts/01-bare-assistant.md +1 -0
- crewscore-0.1.0/examples/corpus/prompts/02-demo-agent.md +11 -0
- crewscore-0.1.0/examples/corpus/prompts/04-partial-hygiene.md +14 -0
- crewscore-0.1.0/examples/corpus/prompts/05-hardened-ops.md +65 -0
- crewscore-0.1.0/examples/corpus/repo-config/AGENTS.md +12 -0
- crewscore-0.1.0/examples/sample-prompt.md +1 -0
- crewscore-0.1.0/favicon.ico +0 -0
- crewscore-0.1.0/favicon.svg +5 -0
- crewscore-0.1.0/index.html +1580 -0
- crewscore-0.1.0/pyproject.toml +69 -0
- crewscore-0.1.0/score-engine.js +939 -0
- crewscore-0.1.0/scripts/export_web_engine.py +23 -0
- crewscore-0.1.0/scripts/make_social_card.py +115 -0
- crewscore-0.1.0/scripts/score_corpus.py +121 -0
- crewscore-0.1.0/tests/test_action_manifest.py +570 -0
- crewscore-0.1.0/tests/test_cli.py +977 -0
- crewscore-0.1.0/tests/test_corpus.py +73 -0
- crewscore-0.1.0/tests/test_explain.py +147 -0
- crewscore-0.1.0/tests/test_export_eval.py +38 -0
- crewscore-0.1.0/tests/test_metrics.py +56 -0
- crewscore-0.1.0/tests/test_profiles.py +87 -0
- crewscore-0.1.0/tests/test_report.py +311 -0
- crewscore-0.1.0/tests/test_rules_catalog.py +242 -0
- crewscore-0.1.0/tests/test_scan.py +599 -0
- crewscore-0.1.0/tests/test_smells.py +282 -0
- crewscore-0.1.0/tests/test_structural_analysis.py +310 -0
- crewscore-0.1.0/tests/test_summary.py +180 -0
- crewscore-0.1.0/tests/test_vendor.py +190 -0
- crewscore-0.1.0/tests/test_web_engine.py +506 -0
- crewscore-0.1.0/tests/test_web_ux.py +523 -0
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Bug report
|
|
3
|
+
about: Something broken in CLI, Action, or web scorecard
|
|
4
|
+
title: "[bug] "
|
|
5
|
+
labels: bug
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## What happened
|
|
9
|
+
|
|
10
|
+
## Expected
|
|
11
|
+
|
|
12
|
+
## Repro
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
crewscore test --prompt "..."
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Environment
|
|
19
|
+
|
|
20
|
+
- OS:
|
|
21
|
+
- `crewscore --version`:
|
|
22
|
+
- Install method: pip / source
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Feature request
|
|
3
|
+
about: New dimension, fix template, or CI surface
|
|
4
|
+
title: "[feat] "
|
|
5
|
+
labels: enhancement
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Problem
|
|
9
|
+
|
|
10
|
+
## Proposed solution
|
|
11
|
+
|
|
12
|
+
## Why not a runtime / adversarial tool instead?
|
|
13
|
+
|
|
14
|
+
CrewScore stays offline structural by default. Say if this must stay structural.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Scoring false positive/negative
|
|
3
|
+
about: Pattern matched wrong or missed a real guardrail
|
|
4
|
+
title: "[scoring] "
|
|
5
|
+
labels: scoring
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
> **Check [`docs/validation.md`](../../docs/validation.md) first.** Some of this
|
|
9
|
+
> is already known and disclosed. In particular, `cost`, `compliance`, and
|
|
10
|
+
> `audit` ship with known-poor construct validity: `cost` and `audit` hold only
|
|
11
|
+
> five patterns each, and `compliance` is keyword detection over regime names,
|
|
12
|
+
> graded `author-intuition` in the catalog. A report on those three confirms
|
|
13
|
+
> what we already published; a **concrete pattern that would fix it** is what we
|
|
14
|
+
> need. Precision reports on the other five dimensions are very welcome.
|
|
15
|
+
>
|
|
16
|
+
> Note also that the total is coverage, not quality — stating all eight controls
|
|
17
|
+
> clearly, once each, scores 28/100 — so "my good prompt scored low" is expected
|
|
18
|
+
> behavior, not a false negative. Report the specific rule that should have
|
|
19
|
+
> matched instead.
|
|
20
|
+
|
|
21
|
+
## Dimension
|
|
22
|
+
|
|
23
|
+
injection / hallucination / citation / cost / human_gate / safe_stop / audit / compliance
|
|
24
|
+
|
|
25
|
+
## Prompt excerpt (redact secrets)
|
|
26
|
+
|
|
27
|
+
```
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## What CrewScore did
|
|
31
|
+
|
|
32
|
+
- Score:
|
|
33
|
+
- Matched / missing (from `--explain` if possible):
|
|
34
|
+
|
|
35
|
+
## What you expected
|
|
36
|
+
|
|
37
|
+
## Suggested pattern (optional)
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# Self-test for the composite CrewScore action in this repo.
|
|
2
|
+
# Installs from github.action_path so it works before PyPI publish.
|
|
3
|
+
name: CrewScore Self-Test
|
|
4
|
+
|
|
5
|
+
on:
|
|
6
|
+
push:
|
|
7
|
+
pull_request:
|
|
8
|
+
|
|
9
|
+
jobs:
|
|
10
|
+
selftest:
|
|
11
|
+
runs-on: ubuntu-latest
|
|
12
|
+
permissions:
|
|
13
|
+
contents: read
|
|
14
|
+
pull-requests: write
|
|
15
|
+
steps:
|
|
16
|
+
- name: Checkout
|
|
17
|
+
uses: actions/checkout@v4
|
|
18
|
+
|
|
19
|
+
- name: Score sample prompt (smoke, threshold 0)
|
|
20
|
+
id: score
|
|
21
|
+
uses: ./
|
|
22
|
+
with:
|
|
23
|
+
prompt-file: examples/sample-prompt.md
|
|
24
|
+
threshold: "0"
|
|
25
|
+
|
|
26
|
+
- name: Assert score and tier outputs parse
|
|
27
|
+
run: |
|
|
28
|
+
echo "score=${{ steps.score.outputs.score }}"
|
|
29
|
+
echo "tier=${{ steps.score.outputs.tier }}"
|
|
30
|
+
python -c "
|
|
31
|
+
import sys
|
|
32
|
+
score = int(sys.argv[1])
|
|
33
|
+
tier = sys.argv[2]
|
|
34
|
+
assert 0 <= score <= 100, score
|
|
35
|
+
assert tier, 'tier output empty'
|
|
36
|
+
print(f'OK score={score} tier={tier!r}')
|
|
37
|
+
" "${{ steps.score.outputs.score }}" "${{ steps.score.outputs.tier }}"
|
|
38
|
+
|
|
39
|
+
- name: Threshold gate fails when score is below threshold
|
|
40
|
+
run: |
|
|
41
|
+
set +e
|
|
42
|
+
crewscore test --prompt-file examples/sample-prompt.md --json --threshold 50
|
|
43
|
+
EXIT=$?
|
|
44
|
+
set -e
|
|
45
|
+
if [ "$EXIT" -ne 2 ]; then
|
|
46
|
+
echo "Expected exit code 2 for bare sample prompt under threshold 50, got $EXIT"
|
|
47
|
+
exit 1
|
|
48
|
+
fi
|
|
49
|
+
echo "Threshold failure behaved as expected (exit 2)"
|
|
50
|
+
|
|
51
|
+
- name: Open rules catalog is non-empty
|
|
52
|
+
run: |
|
|
53
|
+
crewscore rules --json > /tmp/rules.json
|
|
54
|
+
python -c "
|
|
55
|
+
import json
|
|
56
|
+
data = json.load(open('/tmp/rules.json'))
|
|
57
|
+
assert data['rule_count'] > 0
|
|
58
|
+
assert data['method']['llm_calls'] is False
|
|
59
|
+
assert 'structural_analysis' in data['method']['source_of_truth']
|
|
60
|
+
print('OK rules', data['rule_count'], data['ruleset'])
|
|
61
|
+
"
|
|
62
|
+
|
|
63
|
+
- name: Repo scan scores corpus prompts (threshold 0)
|
|
64
|
+
id: scan
|
|
65
|
+
uses: ./
|
|
66
|
+
with:
|
|
67
|
+
scan-path: "."
|
|
68
|
+
threshold: "0"
|
|
69
|
+
|
|
70
|
+
- name: Assert scan outputs
|
|
71
|
+
run: |
|
|
72
|
+
echo "scan_score=${{ steps.scan.outputs.score }}"
|
|
73
|
+
echo "scan_tier=${{ steps.scan.outputs.tier }}"
|
|
74
|
+
python -c "
|
|
75
|
+
import sys
|
|
76
|
+
score = int(sys.argv[1])
|
|
77
|
+
assert 0 <= score <= 100, score
|
|
78
|
+
print('OK scan score', score)
|
|
79
|
+
" "${{ steps.scan.outputs.score }}"
|
|
80
|
+
|
|
81
|
+
- name: Markdown summary writer
|
|
82
|
+
run: |
|
|
83
|
+
crewscore test --prompt-file examples/sample-prompt.md --summary /tmp/cs-summary.md --no-explain
|
|
84
|
+
test -s /tmp/cs-summary.md
|
|
85
|
+
grep -q 'crewscore-hygiene@' /tmp/cs-summary.md
|
|
86
|
+
grep -q 'rules' /tmp/cs-summary.md
|
|
87
|
+
echo "OK summary markdown"
|
|
88
|
+
|
|
89
|
+
- name: Scan summary + corpus gradient
|
|
90
|
+
run: |
|
|
91
|
+
crewscore scan examples/corpus --summary /tmp/cs-scan.md --json > /tmp/cs-scan.json
|
|
92
|
+
test -s /tmp/cs-scan.md
|
|
93
|
+
grep -q 'Worst score' /tmp/cs-scan.md || grep -q 'Path' /tmp/cs-scan.md
|
|
94
|
+
python -c "
|
|
95
|
+
import json
|
|
96
|
+
from pathlib import PurePosixPath, PureWindowsPath
|
|
97
|
+
rows = json.load(open('/tmp/cs-scan.json'))
|
|
98
|
+
by = {}
|
|
99
|
+
config = {}
|
|
100
|
+
for r in rows:
|
|
101
|
+
p = r['path'].replace('\\\\', '/')
|
|
102
|
+
name = PurePosixPath(p).name
|
|
103
|
+
# Coding-agent config carries no governance grade at all - not a
|
|
104
|
+
# zero, not a key. Reading 'overall' off every row is what this
|
|
105
|
+
# guard exists to stop.
|
|
106
|
+
if r.get('governance_applicable', True):
|
|
107
|
+
by[name] = int(r['overall'])
|
|
108
|
+
else:
|
|
109
|
+
config[name] = r
|
|
110
|
+
assert by['01-bare-assistant.md'] == 0
|
|
111
|
+
assert by['05-hardened-ops.md'] >= 70
|
|
112
|
+
assert by['05-hardened-ops.md'] > by['04-partial-hygiene.md']
|
|
113
|
+
assert config, 'corpus lost its coding-agent config fixture'
|
|
114
|
+
for name, r in config.items():
|
|
115
|
+
assert 'overall' not in r, name + ' was handed a governance score'
|
|
116
|
+
assert 'dimensions' not in r, name + ' was handed dimension scores'
|
|
117
|
+
print('OK corpus', by, 'ungraded config', sorted(config))
|
|
118
|
+
"
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
# Documented copy-paste example for consumers.
|
|
2
|
+
#
|
|
3
|
+
# In your own repo, prefer the published tag:
|
|
4
|
+
# uses: shmindmaster/crewscore@v1
|
|
5
|
+
#
|
|
6
|
+
# When developing this monorepo (or a fork), local path works:
|
|
7
|
+
# uses: ./
|
|
8
|
+
#
|
|
9
|
+
# Point prompt-file at your agent system prompt. The sample path below is only
|
|
10
|
+
# for illustration; examples/sample-prompt.md intentionally scores low.
|
|
11
|
+
#
|
|
12
|
+
# Prefer scan-path when the repo has multiple agent artifacts (AGENTS.md,
|
|
13
|
+
# system-prompt.md, prompts/*, etc.). The Action reports the worst (min) overall.
|
|
14
|
+
#
|
|
15
|
+
# workflow_dispatch only so this example does not fail default CI on the bare
|
|
16
|
+
# sample prompt. Use crewscore-selftest.yml for green self-tests.
|
|
17
|
+
name: CrewScore CI Example
|
|
18
|
+
|
|
19
|
+
on:
|
|
20
|
+
workflow_dispatch:
|
|
21
|
+
|
|
22
|
+
jobs:
|
|
23
|
+
score:
|
|
24
|
+
runs-on: ubuntu-latest
|
|
25
|
+
steps:
|
|
26
|
+
- name: Checkout
|
|
27
|
+
uses: actions/checkout@v4
|
|
28
|
+
|
|
29
|
+
# Published action (recommended for consumer repos):
|
|
30
|
+
# permissions: { contents: read, pull-requests: write } # for sticky PR comments
|
|
31
|
+
# - name: CrewScore
|
|
32
|
+
# uses: shmindmaster/crewscore@v1
|
|
33
|
+
# with:
|
|
34
|
+
# prompt-file: agents/system-prompt.md
|
|
35
|
+
# threshold: "50"
|
|
36
|
+
# # explain: "true"
|
|
37
|
+
# # pr-comment: "true" # default on pull_request
|
|
38
|
+
|
|
39
|
+
# Repo-wide structural scan (preferred when you have multiple agent files):
|
|
40
|
+
# - name: CrewScore scan
|
|
41
|
+
# id: crewscore
|
|
42
|
+
# uses: shmindmaster/crewscore@v1
|
|
43
|
+
# with:
|
|
44
|
+
# scan-path: "."
|
|
45
|
+
# threshold: "50"
|
|
46
|
+
|
|
47
|
+
# Local composite action (this repo / monorepo self-check) — single file:
|
|
48
|
+
- name: CrewScore
|
|
49
|
+
id: crewscore
|
|
50
|
+
uses: ./
|
|
51
|
+
with:
|
|
52
|
+
prompt-file: examples/sample-prompt.md
|
|
53
|
+
threshold: "50"
|
|
54
|
+
pr-comment: "false"
|
|
55
|
+
|
|
56
|
+
# Local scan example (uncomment when `crewscore scan` is available):
|
|
57
|
+
# - name: CrewScore scan
|
|
58
|
+
# id: crewscore-scan
|
|
59
|
+
# uses: ./
|
|
60
|
+
# with:
|
|
61
|
+
# scan-path: "."
|
|
62
|
+
# threshold: "50"
|
|
63
|
+
|
|
64
|
+
- name: Use score outputs
|
|
65
|
+
if: always()
|
|
66
|
+
run: |
|
|
67
|
+
echo "score=${{ steps.crewscore.outputs.score }}"
|
|
68
|
+
echo "tier=${{ steps.crewscore.outputs.tier }}"
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
name: pytest
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
test:
|
|
10
|
+
runs-on: ${{ matrix.os }}
|
|
11
|
+
strategy:
|
|
12
|
+
# Don't let one platform's failure hide the others — we need to know
|
|
13
|
+
# which platforms are affected, not just that something broke.
|
|
14
|
+
fail-fast: false
|
|
15
|
+
matrix:
|
|
16
|
+
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
17
|
+
python-version: ["3.11", "3.12", "3.13"]
|
|
18
|
+
exclude:
|
|
19
|
+
# One Python per non-Linux platform is enough to catch OS-specific
|
|
20
|
+
# breakage (path handling, console encoding, line endings); the
|
|
21
|
+
# version matrix is what Linux is for.
|
|
22
|
+
- {os: windows-latest, python-version: "3.11"}
|
|
23
|
+
- {os: windows-latest, python-version: "3.12"}
|
|
24
|
+
- {os: macos-latest, python-version: "3.11"}
|
|
25
|
+
- {os: macos-latest, python-version: "3.12"}
|
|
26
|
+
steps:
|
|
27
|
+
- uses: actions/checkout@v4
|
|
28
|
+
- uses: actions/setup-python@v5
|
|
29
|
+
with:
|
|
30
|
+
python-version: ${{ matrix.python-version }}
|
|
31
|
+
- uses: actions/setup-node@v4
|
|
32
|
+
with:
|
|
33
|
+
node-version: "20"
|
|
34
|
+
- name: Install
|
|
35
|
+
run: pip install -e ".[dev]"
|
|
36
|
+
- name: Test
|
|
37
|
+
run: pytest -q
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# Release pipeline for CrewScore.
|
|
2
|
+
#
|
|
3
|
+
# Publishing uses PyPI Trusted Publishing (OIDC), not an API token: there is
|
|
4
|
+
# deliberately no PYPI_TOKEN secret in this repo, because a long-lived token
|
|
5
|
+
# that can publish under this project name is the single worst secret to leak.
|
|
6
|
+
# GitHub mints a short-lived identity token per run instead.
|
|
7
|
+
#
|
|
8
|
+
# ONE-TIME SETUP (must be done by the PyPI project owner, in a browser):
|
|
9
|
+
# https://pypi.org/manage/project/crewscore/settings/publishing/
|
|
10
|
+
# Add a trusted publisher with:
|
|
11
|
+
# Owner: shmindmaster Repository: crewscore
|
|
12
|
+
# Workflow: release.yml Environment: pypi
|
|
13
|
+
# Until that exists, the publish job fails closed with an auth error and
|
|
14
|
+
# nothing is uploaded. The build and verify job still runs and still produces
|
|
15
|
+
# artifacts, so a tag is never silently half-released.
|
|
16
|
+
name: Release
|
|
17
|
+
|
|
18
|
+
on:
|
|
19
|
+
push:
|
|
20
|
+
tags: ["v*"]
|
|
21
|
+
workflow_dispatch:
|
|
22
|
+
inputs:
|
|
23
|
+
dry-run:
|
|
24
|
+
description: "Build and verify only; do not publish"
|
|
25
|
+
type: boolean
|
|
26
|
+
default: true
|
|
27
|
+
|
|
28
|
+
permissions:
|
|
29
|
+
contents: read
|
|
30
|
+
|
|
31
|
+
jobs:
|
|
32
|
+
verify:
|
|
33
|
+
name: Test, build, and verify artifacts
|
|
34
|
+
runs-on: ${{ matrix.os }}
|
|
35
|
+
strategy:
|
|
36
|
+
fail-fast: false
|
|
37
|
+
matrix:
|
|
38
|
+
os: [ubuntu-latest, windows-latest, macos-latest]
|
|
39
|
+
steps:
|
|
40
|
+
- uses: actions/checkout@v4
|
|
41
|
+
- uses: actions/setup-python@v5
|
|
42
|
+
with:
|
|
43
|
+
python-version: "3.12"
|
|
44
|
+
- uses: actions/setup-node@v4
|
|
45
|
+
with:
|
|
46
|
+
node-version: "20"
|
|
47
|
+
|
|
48
|
+
- name: Install with dev extras
|
|
49
|
+
run: pip install -e ".[dev]"
|
|
50
|
+
|
|
51
|
+
- name: Run the suite
|
|
52
|
+
run: pytest -q
|
|
53
|
+
|
|
54
|
+
# The tag is the release's only claim about which version it is. If it
|
|
55
|
+
# disagrees with the package, the wrong thing gets published under the
|
|
56
|
+
# right name and it cannot be taken back.
|
|
57
|
+
- name: Tag must match the package version
|
|
58
|
+
if: startsWith(github.ref, 'refs/tags/v') && runner.os == 'Linux'
|
|
59
|
+
shell: bash
|
|
60
|
+
env:
|
|
61
|
+
REF_NAME: ${{ github.ref_name }}
|
|
62
|
+
run: |
|
|
63
|
+
set -euo pipefail
|
|
64
|
+
PKG=$(python -c "import crewscore; print(crewscore.__version__)")
|
|
65
|
+
TAG="${REF_NAME#v}"
|
|
66
|
+
if [ "$PKG" != "$TAG" ]; then
|
|
67
|
+
echo "Tag $REF_NAME says $TAG but the package says $PKG" >&2
|
|
68
|
+
exit 1
|
|
69
|
+
fi
|
|
70
|
+
echo "OK version $PKG matches tag $REF_NAME"
|
|
71
|
+
|
|
72
|
+
# score-engine.js is generated from the Python source. A stale copy means
|
|
73
|
+
# the website and the CLI disagree, which is the failure this project can
|
|
74
|
+
# least afford to ship.
|
|
75
|
+
- name: Web engine must not be stale
|
|
76
|
+
if: runner.os == 'Linux'
|
|
77
|
+
shell: bash
|
|
78
|
+
run: |
|
|
79
|
+
set -euo pipefail
|
|
80
|
+
python scripts/export_web_engine.py
|
|
81
|
+
if ! git diff --quiet -- score-engine.js; then
|
|
82
|
+
echo "score-engine.js is stale; re-run scripts/export_web_engine.py" >&2
|
|
83
|
+
git diff --stat -- score-engine.js >&2
|
|
84
|
+
exit 1
|
|
85
|
+
fi
|
|
86
|
+
echo "OK web engine is current"
|
|
87
|
+
|
|
88
|
+
- name: Build sdist and wheel
|
|
89
|
+
if: runner.os == 'Linux'
|
|
90
|
+
run: |
|
|
91
|
+
pip install build twine
|
|
92
|
+
python -m build
|
|
93
|
+
twine check dist/*
|
|
94
|
+
|
|
95
|
+
# Install the built artifact somewhere the source tree cannot rescue it.
|
|
96
|
+
- name: Smoke-test the built wheel in a clean environment
|
|
97
|
+
if: runner.os == 'Linux'
|
|
98
|
+
shell: bash
|
|
99
|
+
run: |
|
|
100
|
+
set -euo pipefail
|
|
101
|
+
python -m venv /tmp/clean
|
|
102
|
+
/tmp/clean/bin/pip install --quiet dist/*.whl
|
|
103
|
+
cd /tmp
|
|
104
|
+
printf 'You are a helpful assistant.\n' > p.md
|
|
105
|
+
printf '# Rules\n\nUse pnpm.\n' > AGENTS.md
|
|
106
|
+
/tmp/clean/bin/crewscore --version
|
|
107
|
+
/tmp/clean/bin/crewscore test --prompt-file p.md --json > gov.json
|
|
108
|
+
/tmp/clean/bin/crewscore rules --json > rules.json
|
|
109
|
+
# The 0.1.0 contract: config carries no governance grade at all.
|
|
110
|
+
/tmp/clean/bin/crewscore test --prompt-file AGENTS.md --json > cfg.json
|
|
111
|
+
python - <<'PY'
|
|
112
|
+
import json
|
|
113
|
+
cfg = json.load(open('/tmp/cfg.json'))
|
|
114
|
+
for k in ('overall', 'dimensions', 'findings', 'transparency'):
|
|
115
|
+
assert k not in cfg, k + ' leaked into a coding-agent config payload'
|
|
116
|
+
assert cfg['governance_applicable'] is False
|
|
117
|
+
gov = json.load(open('/tmp/gov.json'))
|
|
118
|
+
assert isinstance(gov['overall'], int)
|
|
119
|
+
rules = json.load(open('/tmp/rules.json'))
|
|
120
|
+
assert rules['rule_count'] > 0
|
|
121
|
+
assert rules['method']['llm_calls'] is False
|
|
122
|
+
print('OK wheel honors the published contracts')
|
|
123
|
+
PY
|
|
124
|
+
|
|
125
|
+
- name: Upload artifacts
|
|
126
|
+
if: runner.os == 'Linux'
|
|
127
|
+
uses: actions/upload-artifact@v4
|
|
128
|
+
with:
|
|
129
|
+
name: dist
|
|
130
|
+
path: dist/
|
|
131
|
+
|
|
132
|
+
publish:
|
|
133
|
+
name: Publish to PyPI
|
|
134
|
+
needs: verify
|
|
135
|
+
if: startsWith(github.ref, 'refs/tags/v') && !inputs.dry-run
|
|
136
|
+
runs-on: ubuntu-latest
|
|
137
|
+
environment:
|
|
138
|
+
name: pypi
|
|
139
|
+
url: https://pypi.org/p/crewscore
|
|
140
|
+
permissions:
|
|
141
|
+
id-token: write # OIDC for trusted publishing — no token stored anywhere
|
|
142
|
+
steps:
|
|
143
|
+
- uses: actions/download-artifact@v4
|
|
144
|
+
with:
|
|
145
|
+
name: dist
|
|
146
|
+
path: dist/
|
|
147
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
148
|
+
with:
|
|
149
|
+
attestations: true
|
|
150
|
+
|
|
151
|
+
github-release:
|
|
152
|
+
name: Create the GitHub release
|
|
153
|
+
needs: publish
|
|
154
|
+
if: startsWith(github.ref, 'refs/tags/v')
|
|
155
|
+
runs-on: ubuntu-latest
|
|
156
|
+
permissions:
|
|
157
|
+
contents: write
|
|
158
|
+
steps:
|
|
159
|
+
- uses: actions/checkout@v4
|
|
160
|
+
- uses: actions/download-artifact@v4
|
|
161
|
+
with:
|
|
162
|
+
name: dist
|
|
163
|
+
path: dist/
|
|
164
|
+
- name: Publish release notes from the changelog
|
|
165
|
+
env:
|
|
166
|
+
GH_TOKEN: ${{ github.token }}
|
|
167
|
+
REF_NAME: ${{ github.ref_name }}
|
|
168
|
+
shell: bash
|
|
169
|
+
run: |
|
|
170
|
+
set -euo pipefail
|
|
171
|
+
VERSION="${REF_NAME#v}"
|
|
172
|
+
# Pull just this version's section out of CHANGELOG.md. Uses index()
|
|
173
|
+
# rather than a dynamic regex: the regex form silently matched
|
|
174
|
+
# nothing (awk mangles the bracket escapes), which would have shipped
|
|
175
|
+
# a release with empty notes if the -s check below did not catch it.
|
|
176
|
+
awk -v hdr="## [$VERSION]" '
|
|
177
|
+
index($0, hdr) == 1 { inside = 1; next }
|
|
178
|
+
inside && index($0, "## [") == 1 { exit }
|
|
179
|
+
inside { print }
|
|
180
|
+
' CHANGELOG.md > notes.md
|
|
181
|
+
if [ ! -s notes.md ]; then
|
|
182
|
+
echo "No CHANGELOG section for $VERSION" >&2
|
|
183
|
+
exit 1
|
|
184
|
+
fi
|
|
185
|
+
gh release create "$REF_NAME" dist/* \
|
|
186
|
+
--title "$REF_NAME" --notes-file notes.md
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.pyc
|
|
3
|
+
*.egg-info/
|
|
4
|
+
dist/
|
|
5
|
+
build/
|
|
6
|
+
.venv/
|
|
7
|
+
.env
|
|
8
|
+
*.sqlite3
|
|
9
|
+
agent-guard-report-*.html
|
|
10
|
+
.DS_Store
|
|
11
|
+
|
|
12
|
+
# Local / agent-only (never publish)
|
|
13
|
+
.superpowers/
|
|
14
|
+
.worktrees/
|
|
15
|
+
_product-experience/
|
|
16
|
+
_production/
|
|
17
|
+
docs/superpowers/
|
|
18
|
+
docs/research/
|
|
19
|
+
docs/launch/
|
|
20
|
+
docs/pmf-research-*.md
|
|
21
|
+
docs/viral-product-spec-*.md
|
|
22
|
+
docs/publish-checklist.md
|
|
23
|
+
docs/ui-*.png
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
# Marketing media pipeline (keep out of public tree)
|
|
27
|
+
scripts/record-hero-demo.mjs
|
|
28
|
+
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# CrewScore — agent instructions
|
|
2
|
+
|
|
3
|
+
## What this is
|
|
4
|
+
|
|
5
|
+
Offline CLI that classifies an agent-instruction file by name and judges it one of two ways:
|
|
6
|
+
|
|
7
|
+
- **Coding-agent config** (`AGENTS.md`, `CLAUDE.md`, `.cursorrules`, and friends — see `crewscore/profiles.py`) is scanned for **configuration smells** (Context Bloat, Init Fossilization, Lint Leakage — arXiv:2606.15828) and gets **no governance score at all**.
|
|
8
|
+
- **Agent system prompts** (everything else) are scanned for **coverage of 8 governance-signal dimensions** (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance). The number is a match count against a published pattern list, not a validated quality or maturity ranking — **read `docs/validation.md` before trusting it for anything more than "which controls are missing."** In short: at matched length, CrewScore could not show production prompts scoring higher than amateur ones (delta +0.061, 95% CI −0.05 to +0.17, p=0.36), and three of the eight dimensions (Cost, Compliance, Audit) ship with known-poor construct validity.
|
|
9
|
+
|
|
10
|
+
It also applies fix patterns (system-prompt profile only — `fix` refuses to write governance templates into coding-agent config) and optionally runs a non-technical AI vendor checklist (self-attest only).
|
|
11
|
+
|
|
12
|
+
Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
|
|
13
|
+
|
|
14
|
+
It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes. For live testing, hand off to Promptfoo / garak — see `docs/next-steps-eval.md`.
|
|
15
|
+
|
|
16
|
+
Ruleset id: **`crewscore-hygiene@0.1.0`** (`crewscore rules` prints it; `crewscore/scoring.py:RULESET_ID` is the source of truth — it may lag this doc by a patch release mid-work).
|
|
17
|
+
|
|
18
|
+
## Stack
|
|
19
|
+
|
|
20
|
+
- Python 3.11+
|
|
21
|
+
- click + rich
|
|
22
|
+
- hatchling packaging
|
|
23
|
+
- pytest
|
|
24
|
+
|
|
25
|
+
## Commands
|
|
26
|
+
|
|
27
|
+
All verified by running `py -m crewscore.cli ...` on this branch (Windows; use `py`, not `python`).
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
# install (editable)
|
|
31
|
+
pip install -e ".[dev]"
|
|
32
|
+
|
|
33
|
+
# repo scan (preferred for CI / monorepos) — classifies each file (auto), scores
|
|
34
|
+
# system prompts on governance, config files on smells; --threshold is a no-op
|
|
35
|
+
# on config and says so in `warnings`
|
|
36
|
+
crewscore scan .
|
|
37
|
+
crewscore scan . --json --threshold 50
|
|
38
|
+
crewscore scan . --max-smells 0 # gate coding-agent config only
|
|
39
|
+
crewscore scan . --profile system_prompt # force one ruleset for every file found
|
|
40
|
+
|
|
41
|
+
# score a single prompt (same auto-classification; --profile overrides it)
|
|
42
|
+
crewscore test --prompt "You are a helpful assistant..."
|
|
43
|
+
crewscore test --prompt-file ./system-prompt.md --explain
|
|
44
|
+
crewscore test --prompt-file ./system-prompt.md --json --explain
|
|
45
|
+
crewscore test --prompt-file ./system-prompt.md --json --threshold 50
|
|
46
|
+
crewscore test --prompt-file AGENTS.md --max-smells 0 # config CI gate
|
|
47
|
+
crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
|
|
48
|
+
|
|
49
|
+
# apply guardrail patterns (system-prompt profile only)
|
|
50
|
+
crewscore fix --prompt-file ./system-prompt.md --plan # dry-run: list planned dimensions, no write
|
|
51
|
+
crewscore fix --prompt-file ./system-prompt.md --dry-run # alias for --plan
|
|
52
|
+
crewscore fix --prompt-file ./system-prompt.md
|
|
53
|
+
crewscore fix --prompt-file ./system-prompt.md --apply
|
|
54
|
+
crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
|
|
55
|
+
|
|
56
|
+
# fix refuses on coding-agent config: exits 1, prints the reason, JSON has
|
|
57
|
+
# "refused": true; force it anyway with --profile system_prompt (records
|
|
58
|
+
# "forced_governance_write": true)
|
|
59
|
+
crewscore fix --prompt-file AGENTS.md --plan
|
|
60
|
+
crewscore fix --prompt-file AGENTS.md --profile system_prompt --plan --json
|
|
61
|
+
|
|
62
|
+
# vendor checklist (secondary / self-attest)
|
|
63
|
+
crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
|
|
64
|
+
|
|
65
|
+
# live-eval handoff stubs (Promptfoo config + garak notes; does not run either tool)
|
|
66
|
+
crewscore export-eval --prompt-file ./system-prompt.md --output-dir ./crewscore-eval
|
|
67
|
+
|
|
68
|
+
# list the open rule catalog (never a black box)
|
|
69
|
+
crewscore rules
|
|
70
|
+
crewscore rules --json
|
|
71
|
+
crewscore rules --dimension injection
|
|
72
|
+
|
|
73
|
+
# after pattern changes: keep web in lockstep
|
|
74
|
+
python scripts/export_web_engine.py
|
|
75
|
+
|
|
76
|
+
# tests
|
|
77
|
+
pytest
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
|
|
81
|
+
|
|
82
|
+
## Layout
|
|
83
|
+
|
|
84
|
+
```
|
|
85
|
+
crewscore/
|
|
86
|
+
cli.py # click entry (test, fix, scan, rules, export-eval, assess-vendor)
|
|
87
|
+
scoring.py # shared result model / tiers / RULESET_ID
|
|
88
|
+
profiles.py # artifact classification (filename-only) -> which ruleset applies
|
|
89
|
+
scan.py # repo-walk file discovery + per-file scoring for `scan`
|
|
90
|
+
smells.py # offline config-smell detection (arXiv:2606.15828)
|
|
91
|
+
rules_catalog.py # open rule catalog + per-dimension provenance
|
|
92
|
+
summary.py # PR/job markdown (transparent)
|
|
93
|
+
vendor_scorecard.py # assess-vendor command
|
|
94
|
+
export_eval.py # Promptfoo/garak handoff stub writer (export-eval command)
|
|
95
|
+
metrics.py # privacy-safe local metrics schema (no prompt text, no network)
|
|
96
|
+
web_export.py # builds score-engine.js payload
|
|
97
|
+
report.py # HTML report + SVG badge
|
|
98
|
+
scorers/
|
|
99
|
+
structural_analysis.py # governance-dimension regex matching
|
|
100
|
+
fix_patterns.py # FIX_TEMPLATES per dimension
|
|
101
|
+
scripts/
|
|
102
|
+
export_web_engine.py # regenerate score-engine.js after pattern changes
|
|
103
|
+
score_corpus.py # examples/corpus leaderboard
|
|
104
|
+
examples/corpus/ # synthetic bare->hardened demo fixtures
|
|
105
|
+
prompts/ # system-prompt fixtures scored by governance dimensions
|
|
106
|
+
repo-config/AGENTS.md # config fixture scored by smells
|
|
107
|
+
score-engine.js # generated — commit after pattern changes
|
|
108
|
+
index.html # builder-first site (uses score-engine.js)
|
|
109
|
+
action.yml # composite GH Action (scan/test + sticky PR comment)
|
|
110
|
+
docs/
|
|
111
|
+
validation.md # what the score does/does not measure — read before touching claims
|
|
112
|
+
next-steps-eval.md # Promptfoo / garak handoff (public user doc)
|
|
113
|
+
demo.svg, hero-demo.gif, hero-demo.mp4 # README hero assets
|
|
114
|
+
tests/
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
## Product constraints
|
|
118
|
+
|
|
119
|
+
- Prefer honest capability claims over roadmap theater.
|
|
120
|
+
- Structural scores are pattern matches on prompt text, not proof of runtime behavior.
|
|
121
|
+
- **The number is coverage, not a quality ranking.** A low score is actionable (you likely have not written a control down); a high score only means the text is present. Do not rank prompts, teams, or vendors by it, and do not treat a threshold as a safety bar — see `docs/validation.md`.
|
|
122
|
+
- **Honesty scoring:** do not claim certification, audit, or red-team results; templates can inflate scores; under-score rather than over-claim.
|
|
123
|
+
- **The published formula is the whole formula.** Score is a function of rule matches only. Never add a term (length, recency, file type) that isn't in `rules_catalog.SCORING_METHOD` and the README. A 0.2.x length bonus was removed in 0.3.0 precisely because it was undocumented and rewarded Context Bloat.
|
|
124
|
+
- **Length is never rewarded.** Every line costs the agent context on every run. Fix templates stay terse, and `fix` reports its own context cost.
|
|
125
|
+
- **Never hand a governance grade to coding-agent config.** `AGENTS.md`/`CLAUDE.md`-class files are classified by filename only (`profiles.py:classify_path`, content is never sniffed) and judged on configuration smells instead. Measured on the arXiv:2606.15828 corpus, the governance ruleset scored 100/100 real config files in the worst tier — the number is meaningless for that artifact, and publishing it is the fastest way to look broken. Any new output surface (report, badge, summary, share text, web) must branch on `governance_applicable`. `--profile` is the only override; when a caller forces governance templates onto config via `crewscore fix --profile system_prompt`, the JSON payload must carry `forced_governance_write: true` and human output must warn.
|
|
126
|
+
- **Config `--json` payloads carry no governance fields.** When `governance_applicable` is `false`, `overall`, `dimensions`, `findings`, and `transparency` are **absent** from `crewscore test --json` / `crewscore scan --json` output — not zeroed, omitted. Consumers read `tier`, `smells`, `source`, `warnings`, `profile`, `ruleset` instead. Do not reintroduce those fields for config rows.
|
|
127
|
+
- **Smells are advisory, never scored.** Folding them into the number would silently change what `--threshold N` means in someone's CI. `--max-smells N` is the separate gate for config files; changing smells into a score needs corpus evidence, not a patch release.
|
|
128
|
+
- **Rules declare their provenance.** New dimensions must be graded `evidence-backed` / `plausible` / `author-intuition` in `rules_catalog.DIMENSION_PROVENANCE`; evidence-backed requires a citation. Approximations of a published detector must say so in their output. Per `docs/validation.md`, Cost, Compliance, and Audit currently have known-poor construct validity — treat proposals to strengthen or remove them as scoring work needing evidence, not a doc fix.
|
|
129
|
+
- **Console output must be cp1252-encodable.** Windows redirects stdout through the ANSI code page; a stray arrow character once crashed `crewscore rules`. Use ASCII in printed strings.
|
|
130
|
+
- Prefer `crewscore scan .` / Action `scan-path` for repo-native hygiene; demote vendor checklist in UX and docs.
|
|
131
|
+
- Keep the package dependency-light (no LLM SDKs required for the core path).
|
|
132
|
+
- Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
|
|
133
|
+
- Breaking CLI flags are acceptable if all docs and tests update in the same change.
|
|
134
|
+
- Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
|
|
135
|
+
- **Read `docs/validation.md` before writing or editing any claim about what the governance score proves.** It documents a discrimination study against 1,368 real prompts; do not re-inflate the "production-readiness" framing this file used before 0.1.0 without new evidence to support it.
|
|
136
|
+
|
|
137
|
+
## Do not
|
|
138
|
+
|
|
139
|
+
- Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
|
|
140
|
+
- Link to non-existent report hosts or wrong GitHub/PyPI names.
|
|
141
|
+
- Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
|
|
142
|
+
- Overclaim "production safety certification," "production-readiness," or "7 regulated systems" beyond structural scanning and coverage — see `docs/validation.md`.
|
|
143
|
+
- Elevate the vendor self-attest checklist to equal primary product surface.
|