crewscore 0.2.1__tar.gz → 0.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (61) hide show
  1. {crewscore-0.2.1 → crewscore-0.2.2}/.github/workflows/example-ci.yml +20 -1
  2. {crewscore-0.2.1 → crewscore-0.2.2}/AGENTS.md +90 -82
  3. {crewscore-0.2.1 → crewscore-0.2.2}/PKG-INFO +94 -28
  4. {crewscore-0.2.1 → crewscore-0.2.2}/README.md +93 -27
  5. crewscore-0.2.2/action.yml +98 -0
  6. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/__init__.py +3 -3
  7. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/cli.py +252 -29
  8. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/report.py +50 -7
  9. crewscore-0.2.2/crewscore/rules_catalog.py +127 -0
  10. crewscore-0.2.2/crewscore/scan.py +151 -0
  11. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/scorers/structural_analysis.py +201 -88
  12. crewscore-0.2.2/crewscore/scoring.py +116 -0
  13. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/web_export.py +326 -260
  14. crewscore-0.2.2/docs/next-steps-eval.md +72 -0
  15. crewscore-0.2.2/docs/research/2026-07-28-useful-product-and-virality-research.md +318 -0
  16. crewscore-0.2.2/docs/superpowers/plans/2026-07-28-useful-product-slice.md +63 -0
  17. crewscore-0.2.2/docs/superpowers/plans/2026-07-28-workstream-a-scoring-report.md +84 -0
  18. crewscore-0.2.2/docs/superpowers/plans/2026-07-28-workstream-b-scan-report.md +184 -0
  19. crewscore-0.2.2/docs/superpowers/plans/2026-07-28-workstream-cd-product-action-report.md +113 -0
  20. crewscore-0.2.2/favicon.ico +0 -0
  21. {crewscore-0.2.1 → crewscore-0.2.2}/index.html +556 -545
  22. {crewscore-0.2.1 → crewscore-0.2.2}/pyproject.toml +62 -62
  23. {crewscore-0.2.1 → crewscore-0.2.2}/score-engine.js +327 -92
  24. crewscore-0.2.2/tests/test_action_manifest.py +55 -0
  25. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_cli.py +20 -1
  26. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_explain.py +37 -0
  27. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_report.py +17 -1
  28. crewscore-0.2.2/tests/test_rules_catalog.py +105 -0
  29. crewscore-0.2.2/tests/test_scan.py +263 -0
  30. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_structural_analysis.py +130 -86
  31. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_vendor.py +25 -0
  32. crewscore-0.2.1/action.yml +0 -62
  33. crewscore-0.2.1/crewscore/scoring.py +0 -75
  34. crewscore-0.2.1/tests/test_action_manifest.py +0 -10
  35. {crewscore-0.2.1 → crewscore-0.2.2}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
  36. {crewscore-0.2.1 → crewscore-0.2.2}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
  37. {crewscore-0.2.1 → crewscore-0.2.2}/.github/ISSUE_TEMPLATE/scoring_feedback.md +0 -0
  38. {crewscore-0.2.1 → crewscore-0.2.2}/.github/workflows/crewscore-selftest.yml +0 -0
  39. {crewscore-0.2.1 → crewscore-0.2.2}/.github/workflows/pytest.yml +0 -0
  40. {crewscore-0.2.1 → crewscore-0.2.2}/.gitignore +0 -0
  41. {crewscore-0.2.1 → crewscore-0.2.2}/CNAME +0 -0
  42. {crewscore-0.2.1 → crewscore-0.2.2}/CONTRIBUTING.md +0 -0
  43. {crewscore-0.2.1 → crewscore-0.2.2}/LICENSE +0 -0
  44. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/scorers/__init__.py +0 -0
  45. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/scorers/fix_patterns.py +0 -0
  46. {crewscore-0.2.1 → crewscore-0.2.2}/crewscore/vendor_scorecard.py +0 -0
  47. {crewscore-0.2.1 → crewscore-0.2.2}/docs/demo.svg +0 -0
  48. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/README.md +0 -0
  49. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/devto.md +0 -0
  50. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/linkedin.md +0 -0
  51. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/reddit.md +0 -0
  52. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/show-hn.md +0 -0
  53. {crewscore-0.2.1 → crewscore-0.2.2}/docs/launch/x-post.md +0 -0
  54. {crewscore-0.2.1 → crewscore-0.2.2}/docs/pmf-research-2026-07-28.md +0 -0
  55. {crewscore-0.2.1 → crewscore-0.2.2}/docs/publish-checklist.md +0 -0
  56. {crewscore-0.2.1 → crewscore-0.2.2}/docs/superpowers/plans/2026-07-28-crewscore-viral-launch.md +0 -0
  57. {crewscore-0.2.1 → crewscore-0.2.2}/docs/viral-product-spec-2026-07-28.md +0 -0
  58. {crewscore-0.2.1 → crewscore-0.2.2}/examples/sample-prompt.md +0 -0
  59. {crewscore-0.2.1 → crewscore-0.2.2}/favicon.svg +0 -0
  60. {crewscore-0.2.1 → crewscore-0.2.2}/scripts/export_web_engine.py +0 -0
  61. {crewscore-0.2.1 → crewscore-0.2.2}/tests/test_web_engine.py +0 -0
{crewscore-0.2.1 → crewscore-0.2.2}/.github/workflows/example-ci.yml RENAMED
@@ -9,6 +9,9 @@
9
9
  # Point prompt-file at your agent system prompt. The sample path below is only
10
10
  # for illustration; examples/sample-prompt.md intentionally scores low.
11
11
  #
12
+ # Prefer scan-path when the repo has multiple agent artifacts (AGENTS.md,
13
+ # system-prompt.md, prompts/*, etc.). The Action reports the worst (min) overall.
14
+ #
12
15
  # workflow_dispatch only so this example does not fail default CI on the bare
13
16
  # sample prompt. Use crewscore-selftest.yml for green self-tests.
14
17
  name: CrewScore CI Example
@@ -31,7 +34,15 @@ jobs:
31
34
  # threshold: "50"
32
35
  # # explain: "true"
33
36
 
34
- # Local composite action (this repo / monorepo self-check):
37
+ # Repo-wide structural scan (preferred when you have multiple agent files):
38
+ # - name: CrewScore scan
39
+ # id: crewscore
40
+ # uses: shmindmaster/crewscore@v1
41
+ # with:
42
+ # scan-path: "."
43
+ # threshold: "50"
44
+
45
+ # Local composite action (this repo / monorepo self-check) — single file:
35
46
  - name: CrewScore
36
47
  id: crewscore
37
48
  uses: ./
@@ -39,6 +50,14 @@ jobs:
39
50
  prompt-file: examples/sample-prompt.md
40
51
  threshold: "50"
41
52
 
53
+ # Local scan example (uncomment when `crewscore scan` is available):
54
+ # - name: CrewScore scan
55
+ # id: crewscore-scan
56
+ # uses: ./
57
+ # with:
58
+ # scan-path: "."
59
+ # threshold: "50"
60
+
42
61
  - name: Use score outputs
43
62
  if: always()
44
63
  run: |
{crewscore-0.2.1 → crewscore-0.2.2}/AGENTS.md RENAMED
@@ -1,82 +1,90 @@
1
- # CrewScore — agent instructions
2
-
3
- ## What this is
4
-
5
- Offline CLI that **structurally** scores AI agent system prompts for production-readiness signals (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance), applies fix patterns, and runs a non-technical AI vendor checklist.
6
-
7
- Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
8
-
9
- It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes.
10
-
11
- ## Stack
12
-
13
- - Python 3.11+
14
- - click + rich
15
- - hatchling packaging
16
- - pytest
17
-
18
- ## Commands
19
-
20
- ```bash
21
- # install (editable)
22
- pip install -e ".[dev]"
23
-
24
- # score a prompt
25
- crewscore test --prompt "You are a helpful assistant..."
26
- crewscore test --prompt-file ./system-prompt.md --explain
27
- crewscore test --prompt-file ./system-prompt.md --json --explain
28
- crewscore test --prompt-file ./system-prompt.md --json --threshold 50
29
- crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
30
-
31
- # apply guardrail patterns
32
- crewscore fix --prompt-file ./system-prompt.md
33
- crewscore fix --prompt-file ./system-prompt.md --apply
34
- crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
35
-
36
- # vendor checklist
37
- crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
38
-
39
- # after pattern changes: keep web in lockstep
40
- python scripts/export_web_engine.py
41
-
42
- # tests
43
- pytest
44
- ```
45
-
46
- Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
47
-
48
- ## Layout
49
-
50
- ```
51
- crewscore/
52
- cli.py # click entry (test, fix)
53
- scoring.py # shared result model / tiers
54
- vendor_scorecard.py # assess-vendor command
55
- web_export.py # builds score-engine.js payload
56
- report.py # HTML report + SVG badge
57
- scorers/
58
- structural_analysis.py
59
- fix_patterns.py
60
- scripts/export_web_engine.py
61
- score-engine.js # generated — commit after pattern changes
62
- index.html # dual-tab site (uses score-engine.js)
63
- action.yml # composite GH Action
64
- docs/launch/ # launch copy kit
65
- tests/
66
- ```
67
-
68
- ## Product constraints
69
-
70
- - Prefer honest capability claims over roadmap theater.
71
- - Structural scores are pattern matches on prompt text, not proof of runtime behavior.
72
- - Keep the package dependency-light (no LLM SDKs required for the core path).
73
- - Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
74
- - Breaking CLI flags are acceptable if all docs and tests update in the same change.
75
- - Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
76
-
77
- ## Do not
78
-
79
- - Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
80
- - Link to non-existent report hosts or wrong GitHub/PyPI names.
81
- - Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
82
- - Overclaim “production safety certification” or “7 regulated systems” beyond structural scanning.
1
+ # CrewScore — agent instructions
2
+
3
+ ## What this is
4
+
5
+ Offline CLI that **structurally** scores AI agent system prompts for production-readiness signals (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance), applies fix patterns, and optionally runs a non-technical AI vendor checklist (self-attest only).
6
+
7
+ Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
8
+
9
+ It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes. For live testing, hand off to Promptfoo / garak — see `docs/next-steps-eval.md`.
10
+
11
+ ## Stack
12
+
13
+ - Python 3.11+
14
+ - click + rich
15
+ - hatchling packaging
16
+ - pytest
17
+
18
+ ## Commands
19
+
20
+ ```bash
21
+ # install (editable)
22
+ pip install -e ".[dev]"
23
+
24
+ # repo scan (preferred for CI / monorepos)
25
+ crewscore scan .
26
+ crewscore scan . --json --threshold 50
27
+
28
+ # score a single prompt
29
+ crewscore test --prompt "You are a helpful assistant..."
30
+ crewscore test --prompt-file ./system-prompt.md --explain
31
+ crewscore test --prompt-file ./system-prompt.md --json --explain
32
+ crewscore test --prompt-file ./system-prompt.md --json --threshold 50
33
+ crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
34
+
35
+ # apply guardrail patterns
36
+ crewscore fix --prompt-file ./system-prompt.md
37
+ crewscore fix --prompt-file ./system-prompt.md --apply
38
+ crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
39
+
40
+ # vendor checklist (secondary / self-attest)
41
+ crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
42
+
43
+ # after pattern changes: keep web in lockstep
44
+ python scripts/export_web_engine.py
45
+
46
+ # tests
47
+ pytest
48
+ ```
49
+
50
+ Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
51
+
52
+ ## Layout
53
+
54
+ ```
55
+ crewscore/
56
+ cli.py # click entry (test, fix, scan)
57
+ scoring.py # shared result model / tiers
58
+ vendor_scorecard.py # assess-vendor command
59
+ web_export.py # builds score-engine.js payload
60
+ report.py # HTML report + SVG badge
61
+ scorers/
62
+ structural_analysis.py
63
+ fix_patterns.py
64
+ scripts/export_web_engine.py
65
+ score-engine.js # generated — commit after pattern changes
66
+ index.html # builder-first site (uses score-engine.js)
67
+ action.yml # composite GH Action (prompt-file or scan-path)
68
+ docs/launch/ # launch copy kit
69
+ docs/next-steps-eval.md # Promptfoo / garak handoff
70
+ tests/
71
+ ```
72
+
73
+ ## Product constraints
74
+
75
+ - Prefer honest capability claims over roadmap theater.
76
+ - Structural scores are pattern matches on prompt text, not proof of runtime behavior.
77
+ - **Honesty scoring:** do not claim certification, audit, or red-team results; templates can inflate scores; under-score rather than over-claim.
78
+ - Prefer `crewscore scan .` / Action `scan-path` for repo-native hygiene; demote vendor checklist in UX and docs.
79
+ - Keep the package dependency-light (no LLM SDKs required for the core path).
80
+ - Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
81
+ - Breaking CLI flags are acceptable if all docs and tests update in the same change.
82
+ - Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
83
+
84
+ ## Do not
85
+
86
+ - Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
87
+ - Link to non-existent report hosts or wrong GitHub/PyPI names.
88
+ - Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
89
+ - Overclaim “production safety certification” or “7 regulated systems” beyond structural scanning.
90
+ - Elevate the vendor self-attest checklist to equal primary product surface.
{crewscore-0.2.1 → crewscore-0.2.2}/PKG-INFO RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: crewscore
3
- Version: 0.2.1
3
+ Version: 0.2.2
4
4
  Summary: CrewScore — offline structural production-readiness scorecard for AI agent system prompts.
5
5
  Project-URL: Homepage, https://crewscore.ai
6
6
  Project-URL: Repository, https://github.com/shmindmaster/crewscore
@@ -37,26 +37,19 @@ Description-Content-Type: text/markdown
37
37
 
38
38
  ```
39
39
  $ pip install crewscore
40
- $ crewscore test --prompt "You are a helpful assistant that..."
40
+ $ crewscore scan .
41
41
 
42
- CREWSCORE — Structural Production Readiness Report
43
- ====================================================
42
+ path overall tier
43
+ -------------------------------- ------- ---------------------------
44
+ ./AGENTS.md 42 STRUCTURAL: CRITICAL GAPS
45
+ ./agents/system-prompt.md 61 STRUCTURAL: WEAK
44
46
 
45
- Prompt Injection Resistance [----------] 0/100 MISSING
46
- Hallucination Guardrails [----------] 0/100 MISSING
47
- Source Citation Requirements [----------] 0/100 MISSING
48
- Cost Runaway Protection [----------] 0/100 MISSING
49
- Human-in-the-Loop Gates [----------] 0/100 MISSING
50
- Safe-Stop Behavior [----------] 0/100 MISSING
51
- Audit Trail & Provenance [----------] 0/100 MISSING
52
- Compliance Readiness [----------] 0/100 MISSING
53
-
54
- OVERALL SCORE: 0/100 NOT PRODUCTION READY
47
+ worst: 42 (gate with --threshold)
55
48
  ```
56
49
 
57
- Score the **text** of your agent instructions. Fix gaps. Gate CI when scores drop.
50
+ Score the **text** of your agent instructions. Fix gaps. Gate CI when scores drop. Not a red-team, not a certification.
58
51
 
59
- [Install](#install) · [Usage](#usage) · [Share](#share-your-score) · [How scoring works](#how-scoring-works) · [CI](#ci-integration) · [Limits](#what-this-is-and-is-not)
52
+ [Install](#install) · [Usage](#usage) · [Scoring charter](#scoring-charter) · [Share](#share-your-score) · [How scoring works](#how-scoring-works) · [CI](#ci-integration) · [After CrewScore](#after-crewscore) · [Limits](#what-this-is-and-is-not)
60
53
 
61
54
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
62
55
  [![Python](https://img.shields.io/badge/Python-3.11+-green.svg)](https://python.org)
@@ -91,7 +84,7 @@ Most teams never inspect those instructions systematically. **CrewScore** does a
91
84
  | Offline structural scan of system-prompt text | Live adversarial LLM red-teaming |
92
85
  | Fix mode that appends guardrail sections | Proof that the runtime will obey the text |
93
86
  | JSON output + exit threshold for CI | LangGraph / CrewAI graph execution analysis |
94
- | Vendor checklist (`assess-vendor`) for procurement diligence | Independent security certification |
87
+ | Optional vendor self-attest checklist | Independent security certification |
95
88
 
96
89
  Scores reflect **prompt-text signals**. They are a useful smoke test, not a guarantee of runtime safety.
97
90
 
@@ -99,6 +92,27 @@ Scores reflect **prompt-text signals**. They are a useful smoke test, not a guar
99
92
 
100
93
  ---
101
94
 
95
+ ## Scoring charter (not a black box)
96
+
97
+ Honest principles we ship by:
98
+
99
+ 1. CrewScore measures **presence of hygiene signals in text**, not agent behavior.
100
+ 2. Scores are **rule-pack versioned** (`crewscore-hygiene@0.2.2`) and **deterministic** — no LLM, no hidden model.
101
+ 3. **Every rule is public.** List them anytime:
102
+ ```bash
103
+ crewscore rules # human: formula + every rule_id + regex
104
+ crewscore rules --json # machine-readable full catalog
105
+ ```
106
+ 4. Findings show **open `rule_id`s**, match snippets, or explicit missing labels (default in CLI and JSON).
107
+ 5. `fix` improves **text coverage**, not runtime safety; template boilerplate triggers a warning.
108
+ 6. We never call a score a **certification**, **audit**, or **red-team result**.
109
+ 7. When in doubt, **under-score** rather than inflate.
110
+ 8. Source of truth: [`crewscore/scorers/structural_analysis.py`](crewscore/scorers/structural_analysis.py).
111
+
112
+ See also [docs/next-steps-eval.md](docs/next-steps-eval.md) for when to graduate to live eval tools.
113
+
114
+ ---
115
+
102
116
  ## Install
103
117
 
104
118
  ```bash
@@ -114,7 +128,25 @@ No API key for structural mode.
114
128
 
115
129
  ## Usage
116
130
 
117
- ### Score a system prompt
131
+ ### Scan a repo for agent prompts (recommended)
132
+
133
+ Discover and score likely agent instruction files (`AGENTS.md`, `CLAUDE.md`, `system-prompt.md`, paths under `prompts/` / `agents/`, and similar):
134
+
135
+ ```bash
136
+ crewscore scan .
137
+ crewscore scan ./agents --json
138
+ crewscore scan . --threshold 50
139
+ crewscore scan . --json --threshold 50
140
+ ```
141
+
142
+ - Prints a table of path → overall → tier (JSON with `--json`).
143
+ - `--threshold N` exits `2` if **any** file scores below `N`.
144
+ - Exit `1` if no candidate files are found.
145
+ - Skips `node_modules`, `.git`, `venv`, `dist`, `__pycache__`, `.venv`.
146
+
147
+ Use this as the default CI gate for monorepos with multiple agent artifacts.
148
+
149
+ ### Score a single system prompt
118
150
 
119
151
  ```bash
120
152
  crewscore test --prompt "You are a customer service agent for..."
@@ -163,18 +195,22 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
163
195
  crewscore fix --prompt-file ./system-prompt.md --apply --json
164
196
  ```
165
197
 
166
- ### Score an AI vendor (checklist)
198
+ These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
199
+
200
+ ### Vendor checklist (self-attest, secondary)
201
+
202
+ Optional procurement diligence checklist — not the main product path:
167
203
 
168
204
  ```bash
169
205
  crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y"
170
206
  crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
171
207
  ```
172
208
 
173
- Answers are `y` / `n` / `dk` for each of 10 diligence questions.
209
+ Answers are `y` / `n` / `dk` for each of 10 diligence questions. Self-attested only — not an audit.
174
210
 
175
211
  ### Browser demo
176
212
 
177
- Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a zero-install structural scan, one-click fix, templates, and vendor checklist. Scoring rules are exported from the same Python engine into `score-engine.js` so browser and CLI stay in lockstep.
213
+ Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a zero-install structural scan, one-click fix, and templates. Scoring rules are exported from the same Python engine into `score-engine.js` so browser and CLI stay in lockstep.
178
214
 
179
215
  ---
180
216
 
@@ -182,6 +218,13 @@ Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a ze
182
218
 
183
219
  **One engine:** Python CLI and the [crewscore.ai](https://crewscore.ai) site use the same patterns. The browser loads `score-engine.js`, generated from Python via `scripts/export_web_engine.py` (CI fails if it drifts).
184
220
 
221
+ **Formula (fully public):**
222
+
223
+ - Per dimension: count how many open rules’ regexes match (case-insensitive).
224
+ `score = 0` if no matches, else `min(100, round(15 + 85 × matches / total_rules))`.
225
+ - Overall: integer mean of the 8 dimension scores.
226
+ - Inspect any rule: `crewscore rules` / `crewscore rules --json`.
227
+
185
228
  Eight dimensions, equal weight, each 0–100:
186
229
 
187
230
  | Dimension | What the scanner looks for |
@@ -195,14 +238,16 @@ Eight dimensions, equal weight, each 0–100:
195
238
  | Audit Trail | Log decisions / immutable trail language |
196
239
  | Compliance Readiness | HIPAA/SOC2/GDPR/EU AI Act style handling language |
197
240
 
198
- ### Score tiers
241
+ ### Score tiers (structural framing)
199
242
 
200
243
  | Score | Verdict |
201
244
  |-------|---------|
202
- | 90–100 | PRODUCTION READY (structurally) |
203
- | 70–89 | SHIP WITH MONITORING |
204
- | 50–69 | NEEDS WORK |
205
- | 0–49 | NOT PRODUCTION READY |
245
+ | 90–100 | `STRUCTURAL: STRONG` |
246
+ | 70–89 | `STRUCTURAL: OK WITH GAPS` |
247
+ | 50–69 | `STRUCTURAL: WEAK` |
248
+ | 0–49 | `STRUCTURAL: CRITICAL GAPS` |
249
+
250
+ Labels describe **prompt-text coverage**, not production certification.
206
251
 
207
252
  ---
208
253
 
@@ -225,6 +270,8 @@ jobs:
225
270
  id: crewscore
226
271
  uses: shmindmaster/crewscore@v1
227
272
  with:
273
+ # Prefer repo scan when you have multiple agent artifacts:
274
+ # scan-path: "."
228
275
  prompt-file: ./agents/system-prompt.md
229
276
  threshold: "50"
230
277
  # explain: "true" # optional: matched vs missing signals
@@ -239,10 +286,13 @@ jobs:
239
286
 
240
287
  | Input | Required | Default | Description |
241
288
  |-------|----------|---------|-------------|
242
- | `prompt-file` | yes | — | Path to the system prompt file |
289
+ | `prompt-file` | one of | — | Path to a single system prompt file |
290
+ | `scan-path` | one of | — | Path for `crewscore scan` (worst score becomes the output) |
243
291
  | `threshold` | no | `50` | Fail the step (exit 2) when overall score is below this |
244
292
  | `explain` | no | `false` | Pass `true` to include matched/missing signals |
245
293
 
294
+ Provide **either** `prompt-file` **or** `scan-path` (not neither). For scan mode, outputs use the **minimum** overall across discovered files.
295
+
246
296
  **Outputs:** `score` (0–100), `tier` (label string).
247
297
 
248
298
  The composite action installs CrewScore from the action path (`pip install "${{ github.action_path }}"`), so monorepo / pre-PyPI self-tests work with `uses: ./`.
@@ -266,7 +316,9 @@ jobs:
266
316
  with:
267
317
  python-version: "3.12"
268
318
  - run: pip install crewscore
269
- - run: crewscore test --prompt-file ./agents/system-prompt.md --json --threshold 50
319
+ - run: crewscore scan . --json --threshold 50
320
+ # or single file:
321
+ # - run: crewscore test --prompt-file ./agents/system-prompt.md --json --threshold 50
270
322
  ```
271
323
 
272
324
  Or parse JSON yourself:
@@ -277,6 +329,20 @@ SCORE=$(crewscore test --prompt-file ./agents/system-prompt.md --json | jq '.ove
277
329
 
278
330
  ---
279
331
 
332
+ ## After CrewScore
333
+
334
+ CrewScore is the cheap lint / structural pre-gate. When you need **live** behavior:
335
+
336
+ | Need | Tool |
337
+ |------|------|
338
+ | Prompt eval suites, YAML scenarios, CI assertions | [Promptfoo](https://www.promptfoo.dev/) |
339
+ | LLM vulnerability / jailbreak scanning | [garak](https://github.com/NVIDIA/garak) (NVIDIA) |
340
+ | Deeper agent red-team | Promptfoo agents / PyRIT / your own harness |
341
+
342
+ Structural scores do **not** measure jailbreak resistance or multi-turn tool abuse. Use those tools after the prompt text has basic hygiene. Details: [docs/next-steps-eval.md](docs/next-steps-eval.md).
343
+
344
+ ---
345
+
280
346
  ## Fix patterns
281
347
 
282
348
  When dimensions score below threshold, `crewscore fix` appends production-style guardrail sections for:
{crewscore-0.2.1 → crewscore-0.2.2}/README.md RENAMED
@@ -10,26 +10,19 @@
10
10
 
11
11
  ```
12
12
  $ pip install crewscore
13
- $ crewscore test --prompt "You are a helpful assistant that..."
13
+ $ crewscore scan .
14
14
 
15
- CREWSCORE — Structural Production Readiness Report
16
- ====================================================
15
+ path overall tier
16
+ -------------------------------- ------- ---------------------------
17
+ ./AGENTS.md 42 STRUCTURAL: CRITICAL GAPS
18
+ ./agents/system-prompt.md 61 STRUCTURAL: WEAK
17
19
 
18
- Prompt Injection Resistance [----------] 0/100 MISSING
19
- Hallucination Guardrails [----------] 0/100 MISSING
20
- Source Citation Requirements [----------] 0/100 MISSING
21
- Cost Runaway Protection [----------] 0/100 MISSING
22
- Human-in-the-Loop Gates [----------] 0/100 MISSING
23
- Safe-Stop Behavior [----------] 0/100 MISSING
24
- Audit Trail & Provenance [----------] 0/100 MISSING
25
- Compliance Readiness [----------] 0/100 MISSING
26
-
27
- OVERALL SCORE: 0/100 NOT PRODUCTION READY
20
+ worst: 42 (gate with --threshold)
28
21
  ```
29
22
 
30
- Score the **text** of your agent instructions. Fix gaps. Gate CI when scores drop.
23
+ Score the **text** of your agent instructions. Fix gaps. Gate CI when scores drop. Not a red-team, not a certification.
31
24
 
32
- [Install](#install) · [Usage](#usage) · [Share](#share-your-score) · [How scoring works](#how-scoring-works) · [CI](#ci-integration) · [Limits](#what-this-is-and-is-not)
25
+ [Install](#install) · [Usage](#usage) · [Scoring charter](#scoring-charter) · [Share](#share-your-score) · [How scoring works](#how-scoring-works) · [CI](#ci-integration) · [After CrewScore](#after-crewscore) · [Limits](#what-this-is-and-is-not)
33
26
 
34
27
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
35
28
  [![Python](https://img.shields.io/badge/Python-3.11+-green.svg)](https://python.org)
@@ -64,7 +57,7 @@ Most teams never inspect those instructions systematically. **CrewScore** does a
64
57
  | Offline structural scan of system-prompt text | Live adversarial LLM red-teaming |
65
58
  | Fix mode that appends guardrail sections | Proof that the runtime will obey the text |
66
59
  | JSON output + exit threshold for CI | LangGraph / CrewAI graph execution analysis |
67
- | Vendor checklist (`assess-vendor`) for procurement diligence | Independent security certification |
60
+ | Optional vendor self-attest checklist | Independent security certification |
68
61
 
69
62
  Scores reflect **prompt-text signals**. They are a useful smoke test, not a guarantee of runtime safety.
70
63
 
@@ -72,6 +65,27 @@ Scores reflect **prompt-text signals**. They are a useful smoke test, not a guar
72
65
 
73
66
  ---
74
67
 
68
+ ## Scoring charter (not a black box)
69
+
70
+ Honest principles we ship by:
71
+
72
+ 1. CrewScore measures **presence of hygiene signals in text**, not agent behavior.
73
+ 2. Scores are **rule-pack versioned** (`crewscore-hygiene@0.2.2`) and **deterministic** — no LLM, no hidden model.
74
+ 3. **Every rule is public.** List them anytime:
75
+ ```bash
76
+ crewscore rules # human: formula + every rule_id + regex
77
+ crewscore rules --json # machine-readable full catalog
78
+ ```
79
+ 4. Findings show **open `rule_id`s**, match snippets, or explicit missing labels (default in CLI and JSON).
80
+ 5. `fix` improves **text coverage**, not runtime safety; template boilerplate triggers a warning.
81
+ 6. We never call a score a **certification**, **audit**, or **red-team result**.
82
+ 7. When in doubt, **under-score** rather than inflate.
83
+ 8. Source of truth: [`crewscore/scorers/structural_analysis.py`](crewscore/scorers/structural_analysis.py).
84
+
85
+ See also [docs/next-steps-eval.md](docs/next-steps-eval.md) for when to graduate to live eval tools.
86
+
87
+ ---
88
+
75
89
  ## Install
76
90
 
77
91
  ```bash
@@ -87,7 +101,25 @@ No API key for structural mode.
87
101
 
88
102
  ## Usage
89
103
 
90
- ### Score a system prompt
104
+ ### Scan a repo for agent prompts (recommended)
105
+
106
+ Discover and score likely agent instruction files (`AGENTS.md`, `CLAUDE.md`, `system-prompt.md`, paths under `prompts/` / `agents/`, and similar):
107
+
108
+ ```bash
109
+ crewscore scan .
110
+ crewscore scan ./agents --json
111
+ crewscore scan . --threshold 50
112
+ crewscore scan . --json --threshold 50
113
+ ```
114
+
115
+ - Prints a table of path → overall → tier (JSON with `--json`).
116
+ - `--threshold N` exits `2` if **any** file scores below `N`.
117
+ - Exit `1` if no candidate files are found.
118
+ - Skips `node_modules`, `.git`, `venv`, `dist`, `__pycache__`, `.venv`.
119
+
120
+ Use this as the default CI gate for monorepos with multiple agent artifacts.
121
+
122
+ ### Score a single system prompt
91
123
 
92
124
  ```bash
93
125
  crewscore test --prompt "You are a customer service agent for..."
@@ -136,18 +168,22 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
136
168
  crewscore fix --prompt-file ./system-prompt.md --apply --json
137
169
  ```
138
170
 
139
- ### Score an AI vendor (checklist)
171
+ These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
172
+
173
+ ### Vendor checklist (self-attest, secondary)
174
+
175
+ Optional procurement diligence checklist — not the main product path:
140
176
 
141
177
  ```bash
142
178
  crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y"
143
179
  crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
144
180
  ```
145
181
 
146
- Answers are `y` / `n` / `dk` for each of 10 diligence questions.
182
+ Answers are `y` / `n` / `dk` for each of 10 diligence questions. Self-attested only — not an audit.
147
183
 
148
184
  ### Browser demo
149
185
 
150
- Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a zero-install structural scan, one-click fix, templates, and vendor checklist. Scoring rules are exported from the same Python engine into `score-engine.js` so browser and CLI stay in lockstep.
186
+ Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a zero-install structural scan, one-click fix, and templates. Scoring rules are exported from the same Python engine into `score-engine.js` so browser and CLI stay in lockstep.
151
187
 
152
188
  ---
153
189
 
@@ -155,6 +191,13 @@ Open **[crewscore.ai](https://crewscore.ai)** (or `index.html` locally) for a ze
155
191
 
156
192
  **One engine:** Python CLI and the [crewscore.ai](https://crewscore.ai) site use the same patterns. The browser loads `score-engine.js`, generated from Python via `scripts/export_web_engine.py` (CI fails if it drifts).
157
193
 
194
+ **Formula (fully public):**
195
+
196
+ - Per dimension: count how many open rules’ regexes match (case-insensitive).
197
+ `score = 0` if no matches, else `min(100, round(15 + 85 × matches / total_rules))`.
198
+ - Overall: integer mean of the 8 dimension scores.
199
+ - Inspect any rule: `crewscore rules` / `crewscore rules --json`.
200
+
158
201
  Eight dimensions, equal weight, each 0–100:
159
202
 
160
203
  | Dimension | What the scanner looks for |
@@ -168,14 +211,16 @@ Eight dimensions, equal weight, each 0–100:
168
211
  | Audit Trail | Log decisions / immutable trail language |
169
212
  | Compliance Readiness | HIPAA/SOC2/GDPR/EU AI Act style handling language |
170
213
 
171
- ### Score tiers
214
+ ### Score tiers (structural framing)
172
215
 
173
216
  | Score | Verdict |
174
217
  |-------|---------|
175
- | 90–100 | PRODUCTION READY (structurally) |
176
- | 70–89 | SHIP WITH MONITORING |
177
- | 50–69 | NEEDS WORK |
178
- | 0–49 | NOT PRODUCTION READY |
218
+ | 90–100 | `STRUCTURAL: STRONG` |
219
+ | 70–89 | `STRUCTURAL: OK WITH GAPS` |
220
+ | 50–69 | `STRUCTURAL: WEAK` |
221
+ | 0–49 | `STRUCTURAL: CRITICAL GAPS` |
222
+
223
+ Labels describe **prompt-text coverage**, not production certification.
179
224
 
180
225
  ---
181
226
 
@@ -198,6 +243,8 @@ jobs:
198
243
  id: crewscore
199
244
  uses: shmindmaster/crewscore@v1
200
245
  with:
246
+ # Prefer repo scan when you have multiple agent artifacts:
247
+ # scan-path: "."
201
248
  prompt-file: ./agents/system-prompt.md
202
249
  threshold: "50"
203
250
  # explain: "true" # optional: matched vs missing signals
@@ -212,10 +259,13 @@ jobs:
212
259
 
213
260
  | Input | Required | Default | Description |
214
261
  |-------|----------|---------|-------------|
215
- | `prompt-file` | yes | — | Path to the system prompt file |
262
+ | `prompt-file` | one of | — | Path to a single system prompt file |
263
+ | `scan-path` | one of | — | Path for `crewscore scan` (worst score becomes the output) |
216
264
  | `threshold` | no | `50` | Fail the step (exit 2) when overall score is below this |
217
265
  | `explain` | no | `false` | Pass `true` to include matched/missing signals |
218
266
 
267
+ Provide **either** `prompt-file` **or** `scan-path` (not neither). For scan mode, outputs use the **minimum** overall across discovered files.
268
+
219
269
  **Outputs:** `score` (0–100), `tier` (label string).
220
270
 
221
271
  The composite action installs CrewScore from the action path (`pip install "${{ github.action_path }}"`), so monorepo / pre-PyPI self-tests work with `uses: ./`.
@@ -239,7 +289,9 @@ jobs:
239
289
  with:
240
290
  python-version: "3.12"
241
291
  - run: pip install crewscore
242
- - run: crewscore test --prompt-file ./agents/system-prompt.md --json --threshold 50
292
+ - run: crewscore scan . --json --threshold 50
293
+ # or single file:
294
+ # - run: crewscore test --prompt-file ./agents/system-prompt.md --json --threshold 50
243
295
  ```
244
296
 
245
297
  Or parse JSON yourself:
@@ -250,6 +302,20 @@ SCORE=$(crewscore test --prompt-file ./agents/system-prompt.md --json | jq '.ove
250
302
 
251
303
  ---
252
304
 
305
+ ## After CrewScore
306
+
307
+ CrewScore is the cheap lint / structural pre-gate. When you need **live** behavior:
308
+
309
+ | Need | Tool |
310
+ |------|------|
311
+ | Prompt eval suites, YAML scenarios, CI assertions | [Promptfoo](https://www.promptfoo.dev/) |
312
+ | LLM vulnerability / jailbreak scanning | [garak](https://github.com/NVIDIA/garak) (NVIDIA) |
313
+ | Deeper agent red-team | Promptfoo agents / PyRIT / your own harness |
314
+
315
+ Structural scores do **not** measure jailbreak resistance or multi-turn tool abuse. Use those tools after the prompt text has basic hygiene. Details: [docs/next-steps-eval.md](docs/next-steps-eval.md).
316
+
317
+ ---
318
+
253
319
  ## Fix patterns
254
320
 
255
321
  When dimensions score below threshold, `crewscore fix` appends production-style guardrail sections for: