crewscore 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. crewscore-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md +22 -0
  2. crewscore-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md +14 -0
  3. crewscore-0.1.0/.github/ISSUE_TEMPLATE/scoring_feedback.md +37 -0
  4. crewscore-0.1.0/.github/workflows/crewscore-selftest.yml +118 -0
  5. crewscore-0.1.0/.github/workflows/example-ci.yml +68 -0
  6. crewscore-0.1.0/.github/workflows/pytest.yml +37 -0
  7. crewscore-0.1.0/.github/workflows/release.yml +186 -0
  8. crewscore-0.1.0/.gitignore +28 -0
  9. crewscore-0.1.0/AGENTS.md +143 -0
  10. crewscore-0.1.0/CHANGELOG.md +132 -0
  11. crewscore-0.1.0/CNAME +1 -0
  12. crewscore-0.1.0/CONTRIBUTING.md +56 -0
  13. crewscore-0.1.0/LICENSE +21 -0
  14. crewscore-0.1.0/PKG-INFO +615 -0
  15. crewscore-0.1.0/README.md +589 -0
  16. crewscore-0.1.0/action.yml +212 -0
  17. crewscore-0.1.0/crewscore/__init__.py +8 -0
  18. crewscore-0.1.0/crewscore/cli.py +1245 -0
  19. crewscore-0.1.0/crewscore/export_eval.py +100 -0
  20. crewscore-0.1.0/crewscore/metrics.py +81 -0
  21. crewscore-0.1.0/crewscore/profiles.py +104 -0
  22. crewscore-0.1.0/crewscore/report.py +348 -0
  23. crewscore-0.1.0/crewscore/rules_catalog.py +248 -0
  24. crewscore-0.1.0/crewscore/scan.py +222 -0
  25. crewscore-0.1.0/crewscore/scorers/__init__.py +1 -0
  26. crewscore-0.1.0/crewscore/scorers/fix_patterns.py +232 -0
  27. crewscore-0.1.0/crewscore/scorers/structural_analysis.py +508 -0
  28. crewscore-0.1.0/crewscore/scoring.py +166 -0
  29. crewscore-0.1.0/crewscore/smells.py +323 -0
  30. crewscore-0.1.0/crewscore/summary.py +235 -0
  31. crewscore-0.1.0/crewscore/vendor_scorecard.py +409 -0
  32. crewscore-0.1.0/crewscore/web_export.py +556 -0
  33. crewscore-0.1.0/docs/demo.svg +52 -0
  34. crewscore-0.1.0/docs/hero-demo.gif +0 -0
  35. crewscore-0.1.0/docs/hero-demo.mp4 +0 -0
  36. crewscore-0.1.0/docs/next-steps-eval.md +72 -0
  37. crewscore-0.1.0/docs/social-card.png +0 -0
  38. crewscore-0.1.0/docs/validation.md +149 -0
  39. crewscore-0.1.0/examples/corpus/LEADERBOARD.md +39 -0
  40. crewscore-0.1.0/examples/corpus/README.md +27 -0
  41. crewscore-0.1.0/examples/corpus/prompts/01-bare-assistant.md +1 -0
  42. crewscore-0.1.0/examples/corpus/prompts/02-demo-agent.md +11 -0
  43. crewscore-0.1.0/examples/corpus/prompts/04-partial-hygiene.md +14 -0
  44. crewscore-0.1.0/examples/corpus/prompts/05-hardened-ops.md +65 -0
  45. crewscore-0.1.0/examples/corpus/repo-config/AGENTS.md +12 -0
  46. crewscore-0.1.0/examples/sample-prompt.md +1 -0
  47. crewscore-0.1.0/favicon.ico +0 -0
  48. crewscore-0.1.0/favicon.svg +5 -0
  49. crewscore-0.1.0/index.html +1580 -0
  50. crewscore-0.1.0/pyproject.toml +69 -0
  51. crewscore-0.1.0/score-engine.js +939 -0
  52. crewscore-0.1.0/scripts/export_web_engine.py +23 -0
  53. crewscore-0.1.0/scripts/make_social_card.py +115 -0
  54. crewscore-0.1.0/scripts/score_corpus.py +121 -0
  55. crewscore-0.1.0/tests/test_action_manifest.py +570 -0
  56. crewscore-0.1.0/tests/test_cli.py +977 -0
  57. crewscore-0.1.0/tests/test_corpus.py +73 -0
  58. crewscore-0.1.0/tests/test_explain.py +147 -0
  59. crewscore-0.1.0/tests/test_export_eval.py +38 -0
  60. crewscore-0.1.0/tests/test_metrics.py +56 -0
  61. crewscore-0.1.0/tests/test_profiles.py +87 -0
  62. crewscore-0.1.0/tests/test_report.py +311 -0
  63. crewscore-0.1.0/tests/test_rules_catalog.py +242 -0
  64. crewscore-0.1.0/tests/test_scan.py +599 -0
  65. crewscore-0.1.0/tests/test_smells.py +282 -0
  66. crewscore-0.1.0/tests/test_structural_analysis.py +310 -0
  67. crewscore-0.1.0/tests/test_summary.py +180 -0
  68. crewscore-0.1.0/tests/test_vendor.py +190 -0
  69. crewscore-0.1.0/tests/test_web_engine.py +506 -0
  70. crewscore-0.1.0/tests/test_web_ux.py +523 -0
@@ -0,0 +1,22 @@
1
+ ---
2
+ name: Bug report
3
+ about: Something broken in CLI, Action, or web scorecard
4
+ title: "[bug] "
5
+ labels: bug
6
+ ---
7
+
8
+ ## What happened
9
+
10
+ ## Expected
11
+
12
+ ## Repro
13
+
14
+ ```bash
15
+ crewscore test --prompt "..."
16
+ ```
17
+
18
+ ## Environment
19
+
20
+ - OS:
21
+ - `crewscore --version`:
22
+ - Install method: pip / source
@@ -0,0 +1,14 @@
1
+ ---
2
+ name: Feature request
3
+ about: New dimension, fix template, or CI surface
4
+ title: "[feat] "
5
+ labels: enhancement
6
+ ---
7
+
8
+ ## Problem
9
+
10
+ ## Proposed solution
11
+
12
+ ## Why not a runtime / adversarial tool instead?
13
+
14
+ CrewScore stays offline structural by default. Say if this must stay structural.
@@ -0,0 +1,37 @@
1
+ ---
2
+ name: Scoring false positive/negative
3
+ about: Pattern matched wrong or missed a real guardrail
4
+ title: "[scoring] "
5
+ labels: scoring
6
+ ---
7
+
8
+ > **Check [`docs/validation.md`](../../docs/validation.md) first.** Some of this
9
+ > is already known and disclosed. In particular, `cost`, `compliance`, and
10
+ > `audit` ship with known-poor construct validity: `cost` and `audit` hold only
11
+ > five patterns each, and `compliance` is keyword detection over regime names,
12
+ > graded `author-intuition` in the catalog. A report on those three confirms
13
+ > what we already published; a **concrete pattern that would fix it** is what we
14
+ > need. Precision reports on the other five dimensions are very welcome.
15
+ >
16
+ > Note also that the total is coverage, not quality — stating all eight controls
17
+ > clearly, once each, scores 28/100 — so "my good prompt scored low" is expected
18
+ > behavior, not a false negative. Report the specific rule that should have
19
+ > matched instead.
20
+
21
+ ## Dimension
22
+
23
+ injection / hallucination / citation / cost / human_gate / safe_stop / audit / compliance
24
+
25
+ ## Prompt excerpt (redact secrets)
26
+
27
+ ```
28
+ ```
29
+
30
+ ## What CrewScore did
31
+
32
+ - Score:
33
+ - Matched / missing (from `--explain` if possible):
34
+
35
+ ## What you expected
36
+
37
+ ## Suggested pattern (optional)
@@ -0,0 +1,118 @@
1
+ # Self-test for the composite CrewScore action in this repo.
2
+ # Installs from github.action_path so it works before PyPI publish.
3
+ name: CrewScore Self-Test
4
+
5
+ on:
6
+ push:
7
+ pull_request:
8
+
9
+ jobs:
10
+ selftest:
11
+ runs-on: ubuntu-latest
12
+ permissions:
13
+ contents: read
14
+ pull-requests: write
15
+ steps:
16
+ - name: Checkout
17
+ uses: actions/checkout@v4
18
+
19
+ - name: Score sample prompt (smoke, threshold 0)
20
+ id: score
21
+ uses: ./
22
+ with:
23
+ prompt-file: examples/sample-prompt.md
24
+ threshold: "0"
25
+
26
+ - name: Assert score and tier outputs parse
27
+ run: |
28
+ echo "score=${{ steps.score.outputs.score }}"
29
+ echo "tier=${{ steps.score.outputs.tier }}"
30
+ python -c "
31
+ import sys
32
+ score = int(sys.argv[1])
33
+ tier = sys.argv[2]
34
+ assert 0 <= score <= 100, score
35
+ assert tier, 'tier output empty'
36
+ print(f'OK score={score} tier={tier!r}')
37
+ " "${{ steps.score.outputs.score }}" "${{ steps.score.outputs.tier }}"
38
+
39
+ - name: Threshold gate fails when score is below threshold
40
+ run: |
41
+ set +e
42
+ crewscore test --prompt-file examples/sample-prompt.md --json --threshold 50
43
+ EXIT=$?
44
+ set -e
45
+ if [ "$EXIT" -ne 2 ]; then
46
+ echo "Expected exit code 2 for bare sample prompt under threshold 50, got $EXIT"
47
+ exit 1
48
+ fi
49
+ echo "Threshold failure behaved as expected (exit 2)"
50
+
51
+ - name: Open rules catalog is non-empty
52
+ run: |
53
+ crewscore rules --json > /tmp/rules.json
54
+ python -c "
55
+ import json
56
+ data = json.load(open('/tmp/rules.json'))
57
+ assert data['rule_count'] > 0
58
+ assert data['method']['llm_calls'] is False
59
+ assert 'structural_analysis' in data['method']['source_of_truth']
60
+ print('OK rules', data['rule_count'], data['ruleset'])
61
+ "
62
+
63
+ - name: Repo scan scores corpus prompts (threshold 0)
64
+ id: scan
65
+ uses: ./
66
+ with:
67
+ scan-path: "."
68
+ threshold: "0"
69
+
70
+ - name: Assert scan outputs
71
+ run: |
72
+ echo "scan_score=${{ steps.scan.outputs.score }}"
73
+ echo "scan_tier=${{ steps.scan.outputs.tier }}"
74
+ python -c "
75
+ import sys
76
+ score = int(sys.argv[1])
77
+ assert 0 <= score <= 100, score
78
+ print('OK scan score', score)
79
+ " "${{ steps.scan.outputs.score }}"
80
+
81
+ - name: Markdown summary writer
82
+ run: |
83
+ crewscore test --prompt-file examples/sample-prompt.md --summary /tmp/cs-summary.md --no-explain
84
+ test -s /tmp/cs-summary.md
85
+ grep -q 'crewscore-hygiene@' /tmp/cs-summary.md
86
+ grep -q 'rules' /tmp/cs-summary.md
87
+ echo "OK summary markdown"
88
+
89
+ - name: Scan summary + corpus gradient
90
+ run: |
91
+ crewscore scan examples/corpus --summary /tmp/cs-scan.md --json > /tmp/cs-scan.json
92
+ test -s /tmp/cs-scan.md
93
+ grep -q 'Worst score' /tmp/cs-scan.md || grep -q 'Path' /tmp/cs-scan.md
94
+ python -c "
95
+ import json
96
+ from pathlib import PurePosixPath, PureWindowsPath
97
+ rows = json.load(open('/tmp/cs-scan.json'))
98
+ by = {}
99
+ config = {}
100
+ for r in rows:
101
+ p = r['path'].replace('\\\\', '/')
102
+ name = PurePosixPath(p).name
103
+ # Coding-agent config carries no governance grade at all - not a
104
+ # zero, not a key. Reading 'overall' off every row is what this
105
+ # guard exists to stop.
106
+ if r.get('governance_applicable', True):
107
+ by[name] = int(r['overall'])
108
+ else:
109
+ config[name] = r
110
+ assert by['01-bare-assistant.md'] == 0
111
+ assert by['05-hardened-ops.md'] >= 70
112
+ assert by['05-hardened-ops.md'] > by['04-partial-hygiene.md']
113
+ assert config, 'corpus lost its coding-agent config fixture'
114
+ for name, r in config.items():
115
+ assert 'overall' not in r, name + ' was handed a governance score'
116
+ assert 'dimensions' not in r, name + ' was handed dimension scores'
117
+ print('OK corpus', by, 'ungraded config', sorted(config))
118
+ "
@@ -0,0 +1,68 @@
1
+ # Documented copy-paste example for consumers.
2
+ #
3
+ # In your own repo, prefer the published tag:
4
+ # uses: shmindmaster/crewscore@v1
5
+ #
6
+ # When developing this monorepo (or a fork), local path works:
7
+ # uses: ./
8
+ #
9
+ # Point prompt-file at your agent system prompt. The sample path below is only
10
+ # for illustration; examples/sample-prompt.md intentionally scores low.
11
+ #
12
+ # Prefer scan-path when the repo has multiple agent artifacts (AGENTS.md,
13
+ # system-prompt.md, prompts/*, etc.). The Action reports the worst (min) overall.
14
+ #
15
+ # workflow_dispatch only so this example does not fail default CI on the bare
16
+ # sample prompt. Use crewscore-selftest.yml for green self-tests.
17
+ name: CrewScore CI Example
18
+
19
+ on:
20
+ workflow_dispatch:
21
+
22
+ jobs:
23
+ score:
24
+ runs-on: ubuntu-latest
25
+ steps:
26
+ - name: Checkout
27
+ uses: actions/checkout@v4
28
+
29
+ # Published action (recommended for consumer repos):
30
+ # permissions: { contents: read, pull-requests: write } # for sticky PR comments
31
+ # - name: CrewScore
32
+ # uses: shmindmaster/crewscore@v1
33
+ # with:
34
+ # prompt-file: agents/system-prompt.md
35
+ # threshold: "50"
36
+ # # explain: "true"
37
+ # # pr-comment: "true" # default on pull_request
38
+
39
+ # Repo-wide structural scan (preferred when you have multiple agent files):
40
+ # - name: CrewScore scan
41
+ # id: crewscore
42
+ # uses: shmindmaster/crewscore@v1
43
+ # with:
44
+ # scan-path: "."
45
+ # threshold: "50"
46
+
47
+ # Local composite action (this repo / monorepo self-check) — single file:
48
+ - name: CrewScore
49
+ id: crewscore
50
+ uses: ./
51
+ with:
52
+ prompt-file: examples/sample-prompt.md
53
+ threshold: "50"
54
+ pr-comment: "false"
55
+
56
+ # Local scan example (uncomment when `crewscore scan` is available):
57
+ # - name: CrewScore scan
58
+ # id: crewscore-scan
59
+ # uses: ./
60
+ # with:
61
+ # scan-path: "."
62
+ # threshold: "50"
63
+
64
+ - name: Use score outputs
65
+ if: always()
66
+ run: |
67
+ echo "score=${{ steps.crewscore.outputs.score }}"
68
+ echo "tier=${{ steps.crewscore.outputs.tier }}"
@@ -0,0 +1,37 @@
1
+ name: pytest
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ jobs:
9
+ test:
10
+ runs-on: ${{ matrix.os }}
11
+ strategy:
12
+ # Don't let one platform's failure hide the others — we need to know
13
+ # which platforms are affected, not just that something broke.
14
+ fail-fast: false
15
+ matrix:
16
+ os: [ubuntu-latest, windows-latest, macos-latest]
17
+ python-version: ["3.11", "3.12", "3.13"]
18
+ exclude:
19
+ # One Python per non-Linux platform is enough to catch OS-specific
20
+ # breakage (path handling, console encoding, line endings); the
21
+ # version matrix is what Linux is for.
22
+ - {os: windows-latest, python-version: "3.11"}
23
+ - {os: windows-latest, python-version: "3.12"}
24
+ - {os: macos-latest, python-version: "3.11"}
25
+ - {os: macos-latest, python-version: "3.12"}
26
+ steps:
27
+ - uses: actions/checkout@v4
28
+ - uses: actions/setup-python@v5
29
+ with:
30
+ python-version: ${{ matrix.python-version }}
31
+ - uses: actions/setup-node@v4
32
+ with:
33
+ node-version: "20"
34
+ - name: Install
35
+ run: pip install -e ".[dev]"
36
+ - name: Test
37
+ run: pytest -q
@@ -0,0 +1,186 @@
1
+ # Release pipeline for CrewScore.
2
+ #
3
+ # Publishing uses PyPI Trusted Publishing (OIDC), not an API token: there is
4
+ # deliberately no PYPI_TOKEN secret in this repo, because a long-lived token
5
+ # that can publish under this project name is the single worst secret to leak.
6
+ # GitHub mints a short-lived identity token per run instead.
7
+ #
8
+ # ONE-TIME SETUP (must be done by the PyPI project owner, in a browser):
9
+ # https://pypi.org/manage/project/crewscore/settings/publishing/
10
+ # Add a trusted publisher with:
11
+ # Owner: shmindmaster Repository: crewscore
12
+ # Workflow: release.yml Environment: pypi
13
+ # Until that exists, the publish job fails closed with an auth error and
14
+ # nothing is uploaded. The build and verify job still runs and still produces
15
+ # artifacts, so a tag is never silently half-released.
16
+ name: Release
17
+
18
+ on:
19
+ push:
20
+ tags: ["v*"]
21
+ workflow_dispatch:
22
+ inputs:
23
+ dry-run:
24
+ description: "Build and verify only; do not publish"
25
+ type: boolean
26
+ default: true
27
+
28
+ permissions:
29
+ contents: read
30
+
31
+ jobs:
32
+ verify:
33
+ name: Test, build, and verify artifacts
34
+ runs-on: ${{ matrix.os }}
35
+ strategy:
36
+ fail-fast: false
37
+ matrix:
38
+ os: [ubuntu-latest, windows-latest, macos-latest]
39
+ steps:
40
+ - uses: actions/checkout@v4
41
+ - uses: actions/setup-python@v5
42
+ with:
43
+ python-version: "3.12"
44
+ - uses: actions/setup-node@v4
45
+ with:
46
+ node-version: "20"
47
+
48
+ - name: Install with dev extras
49
+ run: pip install -e ".[dev]"
50
+
51
+ - name: Run the suite
52
+ run: pytest -q
53
+
54
+ # The tag is the release's only claim about which version it is. If it
55
+ # disagrees with the package, the wrong thing gets published under the
56
+ # right name and it cannot be taken back.
57
+ - name: Tag must match the package version
58
+ if: startsWith(github.ref, 'refs/tags/v') && runner.os == 'Linux'
59
+ shell: bash
60
+ env:
61
+ REF_NAME: ${{ github.ref_name }}
62
+ run: |
63
+ set -euo pipefail
64
+ PKG=$(python -c "import crewscore; print(crewscore.__version__)")
65
+ TAG="${REF_NAME#v}"
66
+ if [ "$PKG" != "$TAG" ]; then
67
+ echo "Tag $REF_NAME says $TAG but the package says $PKG" >&2
68
+ exit 1
69
+ fi
70
+ echo "OK version $PKG matches tag $REF_NAME"
71
+
72
+ # score-engine.js is generated from the Python source. A stale copy means
73
+ # the website and the CLI disagree, which is the failure this project can
74
+ # least afford to ship.
75
+ - name: Web engine must not be stale
76
+ if: runner.os == 'Linux'
77
+ shell: bash
78
+ run: |
79
+ set -euo pipefail
80
+ python scripts/export_web_engine.py
81
+ if ! git diff --quiet -- score-engine.js; then
82
+ echo "score-engine.js is stale; re-run scripts/export_web_engine.py" >&2
83
+ git diff --stat -- score-engine.js >&2
84
+ exit 1
85
+ fi
86
+ echo "OK web engine is current"
87
+
88
+ - name: Build sdist and wheel
89
+ if: runner.os == 'Linux'
90
+ run: |
91
+ pip install build twine
92
+ python -m build
93
+ twine check dist/*
94
+
95
+ # Install the built artifact somewhere the source tree cannot rescue it.
96
+ - name: Smoke-test the built wheel in a clean environment
97
+ if: runner.os == 'Linux'
98
+ shell: bash
99
+ run: |
100
+ set -euo pipefail
101
+ python -m venv /tmp/clean
102
+ /tmp/clean/bin/pip install --quiet dist/*.whl
103
+ cd /tmp
104
+ printf 'You are a helpful assistant.\n' > p.md
105
+ printf '# Rules\n\nUse pnpm.\n' > AGENTS.md
106
+ /tmp/clean/bin/crewscore --version
107
+ /tmp/clean/bin/crewscore test --prompt-file p.md --json > gov.json
108
+ /tmp/clean/bin/crewscore rules --json > rules.json
109
+ # The 0.1.0 contract: config carries no governance grade at all.
110
+ /tmp/clean/bin/crewscore test --prompt-file AGENTS.md --json > cfg.json
111
+ python - <<'PY'
112
+ import json
113
+ cfg = json.load(open('/tmp/cfg.json'))
114
+ for k in ('overall', 'dimensions', 'findings', 'transparency'):
115
+ assert k not in cfg, k + ' leaked into a coding-agent config payload'
116
+ assert cfg['governance_applicable'] is False
117
+ gov = json.load(open('/tmp/gov.json'))
118
+ assert isinstance(gov['overall'], int)
119
+ rules = json.load(open('/tmp/rules.json'))
120
+ assert rules['rule_count'] > 0
121
+ assert rules['method']['llm_calls'] is False
122
+ print('OK wheel honors the published contracts')
123
+ PY
124
+
125
+ - name: Upload artifacts
126
+ if: runner.os == 'Linux'
127
+ uses: actions/upload-artifact@v4
128
+ with:
129
+ name: dist
130
+ path: dist/
131
+
132
+ publish:
133
+ name: Publish to PyPI
134
+ needs: verify
135
+ if: startsWith(github.ref, 'refs/tags/v') && !inputs.dry-run
136
+ runs-on: ubuntu-latest
137
+ environment:
138
+ name: pypi
139
+ url: https://pypi.org/p/crewscore
140
+ permissions:
141
+ id-token: write # OIDC for trusted publishing — no token stored anywhere
142
+ steps:
143
+ - uses: actions/download-artifact@v4
144
+ with:
145
+ name: dist
146
+ path: dist/
147
+ - uses: pypa/gh-action-pypi-publish@release/v1
148
+ with:
149
+ attestations: true
150
+
151
+ github-release:
152
+ name: Create the GitHub release
153
+ needs: publish
154
+ if: startsWith(github.ref, 'refs/tags/v')
155
+ runs-on: ubuntu-latest
156
+ permissions:
157
+ contents: write
158
+ steps:
159
+ - uses: actions/checkout@v4
160
+ - uses: actions/download-artifact@v4
161
+ with:
162
+ name: dist
163
+ path: dist/
164
+ - name: Publish release notes from the changelog
165
+ env:
166
+ GH_TOKEN: ${{ github.token }}
167
+ REF_NAME: ${{ github.ref_name }}
168
+ shell: bash
169
+ run: |
170
+ set -euo pipefail
171
+ VERSION="${REF_NAME#v}"
172
+ # Pull just this version's section out of CHANGELOG.md. Uses index()
173
+ # rather than a dynamic regex: the regex form silently matched
174
+ # nothing (awk mangles the bracket escapes), which would have shipped
175
+ # a release with empty notes if the -s check below did not catch it.
176
+ awk -v hdr="## [$VERSION]" '
177
+ index($0, hdr) == 1 { inside = 1; next }
178
+ inside && index($0, "## [") == 1 { exit }
179
+ inside { print }
180
+ ' CHANGELOG.md > notes.md
181
+ if [ ! -s notes.md ]; then
182
+ echo "No CHANGELOG section for $VERSION" >&2
183
+ exit 1
184
+ fi
185
+ gh release create "$REF_NAME" dist/* \
186
+ --title "$REF_NAME" --notes-file notes.md
@@ -0,0 +1,28 @@
1
+ __pycache__/
2
+ *.pyc
3
+ *.egg-info/
4
+ dist/
5
+ build/
6
+ .venv/
7
+ .env
8
+ *.sqlite3
9
+ agent-guard-report-*.html
10
+ .DS_Store
11
+
12
+ # Local / agent-only (never publish)
13
+ .superpowers/
14
+ .worktrees/
15
+ _product-experience/
16
+ _production/
17
+ docs/superpowers/
18
+ docs/research/
19
+ docs/launch/
20
+ docs/pmf-research-*.md
21
+ docs/viral-product-spec-*.md
22
+ docs/publish-checklist.md
23
+ docs/ui-*.png
24
+
25
+
26
+ # Marketing media pipeline (keep out of public tree)
27
+ scripts/record-hero-demo.mjs
28
+
@@ -0,0 +1,143 @@
1
+ # CrewScore — agent instructions
2
+
3
+ ## What this is
4
+
5
+ Offline CLI that classifies an agent-instruction file by name and judges it one of two ways:
6
+
7
+ - **Coding-agent config** (`AGENTS.md`, `CLAUDE.md`, `.cursorrules`, and friends — see `crewscore/profiles.py`) is scanned for **configuration smells** (Context Bloat, Init Fossilization, Lint Leakage — arXiv:2606.15828) and gets **no governance score at all**.
8
+ - **Agent system prompts** (everything else) are scanned for **coverage of 8 governance-signal dimensions** (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance). The number is a match count against a published pattern list, not a validated quality or maturity ranking — **read `docs/validation.md` before trusting it for anything more than "which controls are missing."** In short: at matched length, CrewScore could not show production prompts scoring higher than amateur ones (delta +0.061, 95% CI −0.05 to +0.17, p=0.36), and three of the eight dimensions (Cost, Compliance, Audit) ship with known-poor construct validity.
9
+
10
+ It also applies fix patterns (system-prompt profile only — `fix` refuses to write governance templates into coding-agent config) and optionally runs a non-technical AI vendor checklist (self-attest only).
11
+
12
+ Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
13
+
14
+ It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes. For live testing, hand off to Promptfoo / garak — see `docs/next-steps-eval.md`.
15
+
16
+ Ruleset id: **`crewscore-hygiene@0.1.0`** (`crewscore rules` prints it; `crewscore/scoring.py:RULESET_ID` is the source of truth — it may lag this doc by a patch release mid-work).
17
+
18
+ ## Stack
19
+
20
+ - Python 3.11+
21
+ - click + rich
22
+ - hatchling packaging
23
+ - pytest
24
+
25
+ ## Commands
26
+
27
+ All verified by running `py -m crewscore.cli ...` on this branch (Windows; use `py`, not `python`).
28
+
29
+ ```bash
30
+ # install (editable)
31
+ pip install -e ".[dev]"
32
+
33
+ # repo scan (preferred for CI / monorepos) — classifies each file (auto), scores
34
+ # system prompts on governance, config files on smells; --threshold is a no-op
35
+ # on config and says so in `warnings`
36
+ crewscore scan .
37
+ crewscore scan . --json --threshold 50
38
+ crewscore scan . --max-smells 0 # gate coding-agent config only
39
+ crewscore scan . --profile system_prompt # force one ruleset for every file found
40
+
41
+ # score a single prompt (same auto-classification; --profile overrides it)
42
+ crewscore test --prompt "You are a helpful assistant..."
43
+ crewscore test --prompt-file ./system-prompt.md --explain
44
+ crewscore test --prompt-file ./system-prompt.md --json --explain
45
+ crewscore test --prompt-file ./system-prompt.md --json --threshold 50
46
+ crewscore test --prompt-file AGENTS.md --max-smells 0 # config CI gate
47
+ crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
48
+
49
+ # apply guardrail patterns (system-prompt profile only)
50
+ crewscore fix --prompt-file ./system-prompt.md --plan # dry-run: list planned dimensions, no write
51
+ crewscore fix --prompt-file ./system-prompt.md --dry-run # alias for --plan
52
+ crewscore fix --prompt-file ./system-prompt.md
53
+ crewscore fix --prompt-file ./system-prompt.md --apply
54
+ crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
55
+
56
+ # fix refuses on coding-agent config: exits 1, prints the reason, JSON has
57
+ # "refused": true; force it anyway with --profile system_prompt (records
58
+ # "forced_governance_write": true)
59
+ crewscore fix --prompt-file AGENTS.md --plan
60
+ crewscore fix --prompt-file AGENTS.md --profile system_prompt --plan --json
61
+
62
+ # vendor checklist (secondary / self-attest)
63
+ crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
64
+
65
+ # live-eval handoff stubs (Promptfoo config + garak notes; does not run either tool)
66
+ crewscore export-eval --prompt-file ./system-prompt.md --output-dir ./crewscore-eval
67
+
68
+ # list the open rule catalog (never a black box)
69
+ crewscore rules
70
+ crewscore rules --json
71
+ crewscore rules --dimension injection
72
+
73
+ # after pattern changes: keep web in lockstep
74
+ python scripts/export_web_engine.py
75
+
76
+ # tests
77
+ pytest
78
+ ```
79
+
80
+ Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
81
+
82
+ ## Layout
83
+
84
+ ```
85
+ crewscore/
86
+ cli.py # click entry (test, fix, scan, rules, export-eval, assess-vendor)
87
+ scoring.py # shared result model / tiers / RULESET_ID
88
+ profiles.py # artifact classification (filename-only) -> which ruleset applies
89
+ scan.py # repo-walk file discovery + per-file scoring for `scan`
90
+ smells.py # offline config-smell detection (arXiv:2606.15828)
91
+ rules_catalog.py # open rule catalog + per-dimension provenance
92
+ summary.py # PR/job markdown (transparent)
93
+ vendor_scorecard.py # assess-vendor command
94
+ export_eval.py # Promptfoo/garak handoff stub writer (export-eval command)
95
+ metrics.py # privacy-safe local metrics schema (no prompt text, no network)
96
+ web_export.py # builds score-engine.js payload
97
+ report.py # HTML report + SVG badge
98
+ scorers/
99
+ structural_analysis.py # governance-dimension regex matching
100
+ fix_patterns.py # FIX_TEMPLATES per dimension
101
+ scripts/
102
+ export_web_engine.py # regenerate score-engine.js after pattern changes
103
+ score_corpus.py # examples/corpus leaderboard
104
+ examples/corpus/ # synthetic bare->hardened demo fixtures
105
+ prompts/ # system-prompt fixtures scored by governance dimensions
106
+ repo-config/AGENTS.md # config fixture scored by smells
107
+ score-engine.js # generated — commit after pattern changes
108
+ index.html # builder-first site (uses score-engine.js)
109
+ action.yml # composite GH Action (scan/test + sticky PR comment)
110
+ docs/
111
+ validation.md # what the score does/does not measure — read before touching claims
112
+ next-steps-eval.md # Promptfoo / garak handoff (public user doc)
113
+ demo.svg, hero-demo.gif, hero-demo.mp4 # README hero assets
114
+ tests/
115
+ ```
116
+
117
+ ## Product constraints
118
+
119
+ - Prefer honest capability claims over roadmap theater.
120
+ - Structural scores are pattern matches on prompt text, not proof of runtime behavior.
121
+ - **The number is coverage, not a quality ranking.** A low score is actionable (you likely have not written a control down); a high score only means the text is present. Do not rank prompts, teams, or vendors by it, and do not treat a threshold as a safety bar — see `docs/validation.md`.
122
+ - **Honesty scoring:** do not claim certification, audit, or red-team results; templates can inflate scores; under-score rather than over-claim.
123
+ - **The published formula is the whole formula.** Score is a function of rule matches only. Never add a term (length, recency, file type) that isn't in `rules_catalog.SCORING_METHOD` and the README. A 0.2.x length bonus was removed in 0.3.0 precisely because it was undocumented and rewarded Context Bloat.
124
+ - **Length is never rewarded.** Every line costs the agent context on every run. Fix templates stay terse, and `fix` reports its own context cost.
125
+ - **Never hand a governance grade to coding-agent config.** `AGENTS.md`/`CLAUDE.md`-class files are classified by filename only (`profiles.py:classify_path`, content is never sniffed) and judged on configuration smells instead. Measured on the arXiv:2606.15828 corpus, the governance ruleset scored 100/100 real config files in the worst tier — the number is meaningless for that artifact, and publishing it is the fastest way to look broken. Any new output surface (report, badge, summary, share text, web) must branch on `governance_applicable`. `--profile` is the only override; when a caller forces governance templates onto config via `crewscore fix --profile system_prompt`, the JSON payload must carry `forced_governance_write: true` and human output must warn.
126
+ - **Config `--json` payloads carry no governance fields.** When `governance_applicable` is `false`, `overall`, `dimensions`, `findings`, and `transparency` are **absent** from `crewscore test --json` / `crewscore scan --json` output — not zeroed, omitted. Consumers read `tier`, `smells`, `source`, `warnings`, `profile`, `ruleset` instead. Do not reintroduce those fields for config rows.
127
+ - **Smells are advisory, never scored.** Folding them into the number would silently change what `--threshold N` means in someone's CI. `--max-smells N` is the separate gate for config files; changing smells into a score needs corpus evidence, not a patch release.
128
+ - **Rules declare their provenance.** New dimensions must be graded `evidence-backed` / `plausible` / `author-intuition` in `rules_catalog.DIMENSION_PROVENANCE`; evidence-backed requires a citation. Approximations of a published detector must say so in their output. Per `docs/validation.md`, Cost, Compliance, and Audit currently have known-poor construct validity — treat proposals to strengthen or remove them as scoring work needing evidence, not a doc fix.
129
+ - **Console output must be cp1252-encodable.** Windows redirects stdout through the ANSI code page; a stray arrow character once crashed `crewscore rules`. Use ASCII in printed strings.
130
+ - Prefer `crewscore scan .` / Action `scan-path` for repo-native hygiene; demote vendor checklist in UX and docs.
131
+ - Keep the package dependency-light (no LLM SDKs required for the core path).
132
+ - Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
133
+ - Breaking CLI flags are acceptable if all docs and tests update in the same change.
134
+ - Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
135
+ - **Read `docs/validation.md` before writing or editing any claim about what the governance score proves.** It documents a discrimination study against 1,368 real prompts; do not re-inflate the "production-readiness" framing this file used before 0.1.0 without new evidence to support it.
136
+
137
+ ## Do not
138
+
139
+ - Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
140
+ - Link to non-existent report hosts or wrong GitHub/PyPI names.
141
+ - Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
142
+ - Overclaim "production safety certification," "production-readiness," or "7 regulated systems" beyond structural scanning and coverage — see `docs/validation.md`.
143
+ - Elevate the vendor self-attest checklist to equal primary product surface.