crewscore 0.2.4__tar.gz → 0.2.6__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. {crewscore-0.2.4 → crewscore-0.2.6}/AGENTS.md +2 -0
  2. {crewscore-0.2.4 → crewscore-0.2.6}/PKG-INFO +10 -3
  3. {crewscore-0.2.4 → crewscore-0.2.6}/README.md +9 -2
  4. crewscore-0.2.6/_product-experience/00-discovery.md +51 -0
  5. crewscore-0.2.6/_product-experience/01-audit.md +94 -0
  6. crewscore-0.2.6/_product-experience/02-workflow-and-feature-design.md +60 -0
  7. crewscore-0.2.6/_product-experience/06-outcome-measurement-plan.md +59 -0
  8. {crewscore-0.2.4 → crewscore-0.2.6}/action.yml +1 -1
  9. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/__init__.py +1 -1
  10. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/cli.py +61 -1
  11. crewscore-0.2.6/crewscore/metrics.py +81 -0
  12. crewscore-0.2.6/docs/superpowers/plans/2026-07-28-preflight-completion-sdd.md +105 -0
  13. crewscore-0.2.6/docs/ui-after-polish.png +0 -0
  14. crewscore-0.2.6/docs/ui-preflight-workflow.png +0 -0
  15. crewscore-0.2.6/favicon.svg +5 -0
  16. crewscore-0.2.6/index.html +1025 -0
  17. {crewscore-0.2.4 → crewscore-0.2.6}/pyproject.toml +1 -1
  18. {crewscore-0.2.4 → crewscore-0.2.6}/score-engine.js +1 -1
  19. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_cli.py +83 -0
  20. crewscore-0.2.6/tests/test_metrics.py +56 -0
  21. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_vendor.py +3 -1
  22. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_web_engine.py +6 -1
  23. crewscore-0.2.6/tests/test_web_ux.py +69 -0
  24. crewscore-0.2.4/favicon.svg +0 -5
  25. crewscore-0.2.4/index.html +0 -556
  26. {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
  27. {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
  28. {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/scoring_feedback.md +0 -0
  29. {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/crewscore-selftest.yml +0 -0
  30. {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/example-ci.yml +0 -0
  31. {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/pytest.yml +0 -0
  32. {crewscore-0.2.4 → crewscore-0.2.6}/.gitignore +0 -0
  33. {crewscore-0.2.4 → crewscore-0.2.6}/CNAME +0 -0
  34. {crewscore-0.2.4 → crewscore-0.2.6}/CONTRIBUTING.md +0 -0
  35. {crewscore-0.2.4 → crewscore-0.2.6}/LICENSE +0 -0
  36. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/export_eval.py +0 -0
  37. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/report.py +0 -0
  38. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/rules_catalog.py +0 -0
  39. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scan.py +0 -0
  40. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/__init__.py +0 -0
  41. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/fix_patterns.py +0 -0
  42. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/structural_analysis.py +0 -0
  43. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scoring.py +0 -0
  44. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/summary.py +0 -0
  45. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/vendor_scorecard.py +0 -0
  46. {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/web_export.py +0 -0
  47. {crewscore-0.2.4 → crewscore-0.2.6}/docs/demo.svg +0 -0
  48. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/README.md +0 -0
  49. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/devto.md +0 -0
  50. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/linkedin.md +0 -0
  51. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/reddit.md +0 -0
  52. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/show-hn.md +0 -0
  53. {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/x-post.md +0 -0
  54. {crewscore-0.2.4 → crewscore-0.2.6}/docs/next-steps-eval.md +0 -0
  55. {crewscore-0.2.4 → crewscore-0.2.6}/docs/pmf-research-2026-07-28.md +0 -0
  56. {crewscore-0.2.4 → crewscore-0.2.6}/docs/publish-checklist.md +0 -0
  57. {crewscore-0.2.4 → crewscore-0.2.6}/docs/research/2026-07-28-useful-product-and-virality-research.md +0 -0
  58. {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-crewscore-viral-launch.md +0 -0
  59. {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-useful-product-slice.md +0 -0
  60. {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-a-scoring-report.md +0 -0
  61. {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-b-scan-report.md +0 -0
  62. {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-cd-product-action-report.md +0 -0
  63. {crewscore-0.2.4 → crewscore-0.2.6}/docs/viral-product-spec-2026-07-28.md +0 -0
  64. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/LEADERBOARD.md +0 -0
  65. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/README.md +0 -0
  66. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/01-bare-assistant.md +0 -0
  67. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/02-demo-agent.md +0 -0
  68. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/03-agents-md-weak.md +0 -0
  69. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/04-partial-hygiene.md +0 -0
  70. {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/05-hardened-ops.md +0 -0
  71. {crewscore-0.2.4 → crewscore-0.2.6}/examples/sample-prompt.md +0 -0
  72. {crewscore-0.2.4 → crewscore-0.2.6}/favicon.ico +0 -0
  73. {crewscore-0.2.4 → crewscore-0.2.6}/scripts/export_web_engine.py +0 -0
  74. {crewscore-0.2.4 → crewscore-0.2.6}/scripts/score_corpus.py +0 -0
  75. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_action_manifest.py +0 -0
  76. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_corpus.py +0 -0
  77. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_explain.py +0 -0
  78. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_export_eval.py +0 -0
  79. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_report.py +0 -0
  80. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_rules_catalog.py +0 -0
  81. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_scan.py +0 -0
  82. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_structural_analysis.py +0 -0
  83. {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_summary.py +0 -0
@@ -33,6 +33,8 @@ crewscore test --prompt-file ./system-prompt.md --json --threshold 50
33
33
  crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
34
34
 
35
35
  # apply guardrail patterns
36
+ crewscore fix --prompt-file ./system-prompt.md --plan # dry-run: list planned dimensions, no write
37
+ crewscore fix --prompt-file ./system-prompt.md --dry-run # alias for --plan
36
38
  crewscore fix --prompt-file ./system-prompt.md
37
39
  crewscore fix --prompt-file ./system-prompt.md --apply
38
40
  crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: crewscore
3
- Version: 0.2.4
3
+ Version: 0.2.6
4
4
  Summary: CrewScore — offline structural production-readiness scorecard for AI agent system prompts.
5
5
  Project-URL: Homepage, https://crewscore.ai
6
6
  Project-URL: Repository, https://github.com/shmindmaster/crewscore
@@ -189,7 +189,12 @@ Human mode also prints a one-line share blurb with your overall score and [crews
189
189
  ### Apply guardrail patterns
190
190
 
191
191
  ```bash
192
- # print enhanced prompt
192
+ # plan only — list dimensions that would be fixed (no file write)
193
+ crewscore fix --prompt-file ./system-prompt.md --plan
194
+ # alias:
195
+ crewscore fix --prompt-file ./system-prompt.md --dry-run
196
+
197
+ # print enhanced prompt (stdout only)
193
198
  crewscore fix --prompt-file ./system-prompt.md
194
199
 
195
200
  # write in place and show score delta
@@ -200,9 +205,11 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
200
205
 
201
206
  # machine-readable summary
202
207
  crewscore fix --prompt-file ./system-prompt.md --apply --json
208
+ # plan as JSON (fixes_planned, written: false)
209
+ crewscore fix --prompt-file ./system-prompt.md --plan --json
203
210
  ```
204
211
 
205
- These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
212
+ `--plan` / `--dry-run` is mutually exclusive with `--apply` and `--output`. These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
206
213
 
207
214
  ### Vendor checklist (self-attest, secondary)
208
215
 
@@ -162,7 +162,12 @@ Human mode also prints a one-line share blurb with your overall score and [crews
162
162
  ### Apply guardrail patterns
163
163
 
164
164
  ```bash
165
- # print enhanced prompt
165
+ # plan only — list dimensions that would be fixed (no file write)
166
+ crewscore fix --prompt-file ./system-prompt.md --plan
167
+ # alias:
168
+ crewscore fix --prompt-file ./system-prompt.md --dry-run
169
+
170
+ # print enhanced prompt (stdout only)
166
171
  crewscore fix --prompt-file ./system-prompt.md
167
172
 
168
173
  # write in place and show score delta
@@ -173,9 +178,11 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
173
178
 
174
179
  # machine-readable summary
175
180
  crewscore fix --prompt-file ./system-prompt.md --apply --json
181
+ # plan as JSON (fixes_planned, written: false)
182
+ crewscore fix --prompt-file ./system-prompt.md --plan --json
176
183
  ```
177
184
 
178
- These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
185
+ `--plan` / `--dry-run` is mutually exclusive with `--apply` and `--output`. These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
179
186
 
180
187
  ### Vendor checklist (self-attest, secondary)
181
188
 
@@ -0,0 +1,51 @@
1
+ # CrewScore — Discovery
2
+
3
+ **Date:** 2026-07-28
4
+ **Repo:** `shmindmaster/crewscore`
5
+ **Surfaces in scope:** Public web (crewscore.ai / `index.html`), CLI, GitHub Action, docs/launch.
6
+
7
+ ## Product promise
8
+
9
+ Offline **structural pre-gate** for AI agent system prompts: discover missing hygiene signals, explain with open rules, fix text templates, gate CI — never claim red-team or certification.
10
+
11
+ ## Users & jobs
12
+
13
+ | Persona | Job | Success looks like |
14
+ | --- | --- | --- |
15
+ | **Agent builder** (primary) | Before merge/demo, know if system prompt lacks production hygiene | Score + top gaps + fix or CI gate in <2 min |
16
+ | **OSS / HN visitor** | Understand product in 15s without install | One click demo → memorable number → trust honesty |
17
+ | **AppSec-curious** | Prefer open rules over black-box scores | Expand full rule IDs; match CLI |
18
+ | **Buyer / ops** (secondary) | Self-attest vendor diligence | Separate, demoted checklist — not the hero |
19
+
20
+ ## Domain objects
21
+
22
+ - **Prompt artifact** (text / file / URL)
23
+ - **Ruleset** (`crewscore-hygiene@x.y.z`) + **rule_id** findings
24
+ - **Score result** (8 dimensions, overall, structural tier, warnings)
25
+ - **Fix pack** (appended templates; mutates text only)
26
+ - **Scan set** (repo paths; CLI/Action)
27
+ - **Share artifact** (image / text; must not overclaim)
28
+
29
+ ## Capability map (what exists)
30
+
31
+ | Surface | Status | Notes |
32
+ | --- | --- | --- |
33
+ | Public marketing / demo site | Deep reviewed | Single-page score tool; primary viral surface |
34
+ | CLI `test` / `scan` / `fix` / `rules` / `export-eval` | Deep reviewed | Strong; truth source for web |
35
+ | GitHub Action + PR sticky summary | Shallow reviewed | Functional; not browser UX |
36
+ | Vendor self-attest | Shallow reviewed | Demoted; still competes for attention if tab-equal |
37
+ | Auth / billing / admin | Not applicable | Zero-install OSS |
38
+ | Live LLM / agent runtime | Not applicable | Explicitly out of scope |
39
+
40
+ ## Constraints
41
+
42
+ - Deterministic scoring (no model calls on critical path)
43
+ - Browser must stay offline-capable (local `score-engine.js`)
44
+ - Honesty > virality theater
45
+ - Keep HTML contract tests (builder-first hero, authenticity, escapeHtml, engine parity)
46
+
47
+ ## Evidence sources
48
+
49
+ - Runtime: live site + Playwright primary journey
50
+ - Code: `index.html`, `score-engine.js`, CLI
51
+ - Research: `docs/research/2026-07-28-useful-product-and-virality-research.md`
@@ -0,0 +1,94 @@
1
+ # CrewScore — Product experience audit
2
+
3
+ **Date:** 2026-07-28
4
+ **Method:** Discovery + live journey (Playwright) + implementation review
5
+ **Scope:** Public web primary; CLI/Action as continuity targets
6
+
7
+ ## Coverage ledger
8
+
9
+ | Surface | Status | Notes |
10
+ | --- | --- | --- |
11
+ | Public marketing / zero-install demo | Deep reviewed | Core product surface for cold traffic |
12
+ | Customer product (web score/fix) | Deep reviewed | Same page as marketing |
13
+ | Developer integrations (CLI/Action) | Shallow reviewed | Strong utility; linked in footer |
14
+ | Vendor diligence | Shallow reviewed | Secondary; over-weighted in IA if equal tab |
15
+ | Auth / billing / ops / support portals | Not applicable | |
16
+
17
+ ## Primary journey audited
18
+
19
+ **Try template → score → understand → fix → share**
20
+
21
+ Evidence (2026-07-28): Weak demo → 0/100 → Fix → 46/100 works. Full rule dump previously overloaded; polish reduced dump but **workflow still single-panel form**, not a task lifecycle.
22
+
23
+ ## Findings
24
+
25
+ ### F1 — Workflow is a form, not a preflight job
26
+ **Severity:** High · **Class:** product-fix-required
27
+ **Evidence:** One panel with score, fix, URL, vendor tab; no explicit stages, plan-before-mutate, or completion.
28
+ **Impact:** Builders don't feel "I completed a gate"; visitors don't learn the product story.
29
+ **Recommendation:** Stage the job: Prompt → Inspect → Act → Export.
30
+ **Validate:** Task completion without reading docs; time-to-first-insight < 20s.
31
+
32
+ ### F2 — "Fix" mutates without plan preview (agentic control gap)
33
+ **Severity:** High · **Class:** product-fix-required
34
+ **Evidence:** `fixAndRescore` replaces textarea immediately; no preview of sections that will be appended.
35
+ **Impact:** Surprising text mutation; reduces trust for production prompts.
36
+ **Recommendation:** Preview plan (dimensions/sections to add) → explicit Apply.
37
+ **Validate:** User can cancel fix; prompt unchanged until Apply.
38
+
39
+ ### F3 — Capability boundary is easy to misread as "AI safety product"
40
+ **Severity:** High · **Class:** product-fix-required
41
+ **Evidence:** Hero about agent prompts; scoring is regex. Honesty present but competes with marketing.
42
+ **Impact:** Trust collapse on HN; wrong mental model.
43
+ **Recommendation:** Persistent capability chip + result stamp: "Structural pre-gate · not red-team".
44
+ **Validate:** After first score, user can restate what product is/isn't.
45
+
46
+ ### F4 — Results still present as dashboard, not decisions
47
+ **Severity:** Medium · **Class:** product-fix-required
48
+ **Evidence:** 8 bars + optional full rules; top gaps improved but no "next action" hierarchy beyond Fix.
49
+ **Impact:** Cognitive load; weak link to CI/export-eval.
50
+ **Recommendation:** Decision stack: Top gaps (act) → dimension radar/checklist → optional rules → CI/export CTA.
51
+ **Validate:** First click after score is gap-relevant action ≥70% of sessions (instrument).
52
+
53
+ ### F5 — Vendor path steals IA equality
54
+ **Severity:** Medium · **Class:** product-fix-required
55
+ **Evidence:** Tab strip co-primary with agent scoring. Research demoted vendor.
56
+ **Impact:** Dilutes builder product; resurrects questionnaire theater.
57
+ **Recommendation:** Vendor as secondary link / `#vendor`, not equal tab chrome.
58
+ **Validate:** Cold open shows only agent job.
59
+
60
+ ### F6 — Visual system still "AI product default"
61
+ **Severity:** Medium · **Class:** product-fix-required
62
+ **Evidence:** Dark slate + cyan gradient + system sans; generic after 2026 AI sites.
63
+ **Impact:** Low memorability; fails distinctive frontend-design bar.
64
+ **Recommendation:** Preflight/instrument aesthetic — amber caution + terminal mono + radar/checklist signature.
65
+ **Validate:** Screenshot recognition without logo text in informal test.
66
+
67
+ ### F7 — Empty / error / success states underdesigned
68
+ **Severity:** Medium · **Class:** product-fix-required
69
+ **Evidence:** Empty = blank textarea; error = red line; success = scorecard only.
70
+ **Impact:** No guided first action; recovery is weak for URL/CORS failures.
71
+ **Recommendation:** Empty invites one demo; errors name cause + paste fallback; success has completion checklist.
72
+ **Validate:** Empty state has single recommended action; URL error offers paste.
73
+
74
+ ### F8 — No outcome instrumentation
75
+ **Severity:** Medium · **Class:** product-fix-required
76
+ **Evidence:** No events for score/fix/share (by design privacy-first, but no measurement plan either).
77
+ **Impact:** Cannot learn if UX works post-launch.
78
+ **Recommendation:** Privacy-safe local counters + optional anonymous funnel (no prompt text). See `06-outcome-measurement-plan.md`.
79
+
80
+ ### F9 — CLI/Action continuity weak on web
81
+ **Severity:** Low · **Class:** informational
82
+ **Evidence:** Footer codes only.
83
+ **Impact:** Web users don't graduate to CI.
84
+ **Recommendation:** After score, "Gate this in CI" copy-paste block with scan/action snippet.
85
+
86
+ ## Prioritized decision list
87
+
88
+ 1. Redesign primary workflow stages + fix plan/apply (F1, F2)
89
+ 2. Capability stamp + decision-first results (F3, F4)
90
+ 3. Demote vendor IA (F5)
91
+ 4. Distinctive preflight visual system (F6)
92
+ 5. Empty/error/success states (F7)
93
+ 6. Measurement without prompt capture (F8)
94
+ 7. CI handoff CTA (F9)
@@ -0,0 +1,60 @@
1
+ # CrewScore — Workflow & feature design
2
+
3
+ ## Outcome
4
+
5
+ Builder completes a **structural preflight** on an agent prompt: knows overall hygiene, top missing signals, optionally applies text templates with consent, and leaves with share/CI next step — without believing the product is a red-team.
6
+
7
+ ## Target primary workflow
8
+
9
+ ```
10
+ Entry (cold / return)
11
+ → Stage 1 PROMPT: paste | template | URL
12
+ → Stage 2 INSPECT: score + capability stamp + top gaps + dims
13
+ → Stage 3 ACT: plan fix → approve → re-inspect
14
+ → Stage 4 EXPORT: share image/text | CI snippet | improved prompt
15
+ ```
16
+
17
+ ### Stage details
18
+
19
+ | Stage | User intent | Happy path | Empty | Error | Recovery |
20
+ | --- | --- | --- | --- | --- | --- |
21
+ | PROMPT | Provide artifact | Template or paste | Invite "Weak demo" | — | — |
22
+ | INSPECT | Understand gaps | Score + top 3 gaps | — | Engine missing | Reload |
23
+ | ACT | Improve text | Preview sections → Apply | No weak dims → "Already covered" | — | Cancel keeps original |
24
+ | EXPORT | Take result elsewhere | Share / download / CI | — | Clipboard denied | Manual select |
25
+
26
+ ## Secondary workflow
27
+
28
+ **Vendor self-attest** — linked, not staged with primary. Self-attest stamp. No equal chrome.
29
+
30
+ ## Information architecture
31
+
32
+ - **Primary nav:** none (single job page)
33
+ - **Progress:** step indicator 1–4 (or Prompt / Inspect / Act / Export)
34
+ - **Results hierarchy:** (1) overall + soft verdict (2) capability stamp (3) top gaps (4) dimensions (5) all rules (6) export
35
+ - **Progressive disclosure:** full rules collapsed; formula collapsed
36
+
37
+ ## Agentic / AI interaction contract
38
+
39
+ CrewScore scoring is **deterministic**, not generative. The only "agent-like" action is **fix** (mutates user content).
40
+
41
+ | Action | Autonomy | Control |
42
+ | --- | --- | --- |
43
+ | Score | Auto on request | User initiates; no network model |
44
+ | Explain findings | Auto with score | Expand/collapse; open rule_ids |
45
+ | Fix plan | Preview only | List dimensions/sections that would be appended |
46
+ | Apply fix | Explicit approval | Apply / Cancel; original retained until Apply |
47
+ | Share | User-triggered | Copy text / image; disclaimer in share string |
48
+ | URL load | User-triggered | CORS failure → paste fallback |
49
+
50
+ **Prohibited claims in UI:** certification, audit complete, red-team pass, runtime safety proven.
51
+
52
+ **Uncertainty:** Always structural-only; template_boilerplate warning when fix templates detected.
53
+
54
+ ## Acceptance signals
55
+
56
+ - Time from land → first score ≤ 20s with template
57
+ - Fix cannot change text without Apply
58
+ - After score, user sees capability stamp without scrolling past fold on desktop
59
+ - Vendor not equal-weight with primary CTA
60
+ - Share text includes "not a red-team"
@@ -0,0 +1,59 @@
1
+ # CrewScore — Outcome measurement plan
2
+
3
+ ## Decision this plan supports
4
+
5
+ Did the preflight web experience improve **successful structural pre-gates** (score → understand → act → export) without increasing overclaim/trust failure?
6
+
7
+ ## Population & journey
8
+
9
+ - **Population:** Anonymous visitors to crewscore.ai + optional CLI users (separate)
10
+ - **Journey boundary:** First paint → first score → optional fix apply → optional share/CI copy
11
+ - **Baseline:** Pre-redesign qualitative (manual Playwright + audit F1–F9)
12
+ - **Comparison:** Post-deploy week-over-week funnel (if analytics enabled)
13
+
14
+ ## Metrics (balanced)
15
+
16
+ | Outcome | Metric | Guardrail |
17
+ | --- | --- | --- |
18
+ | Activation | % sessions with ≥1 score | Bounce without interaction |
19
+ | Understanding | % expand "top gaps" or dim row | Time-to-score only |
20
+ | Controlled fix | % fix_plan_shown → fix_applied | fix_applied without plan (should be ~0) |
21
+ | Export | % share_or_download_or_ci_copy | Share without score (should be 0) |
22
+ | Trust | Qualitative: can restate "not red-team" | Support/issue reports of overclaim |
23
+ | Privacy | Zero prompt text in analytics | Any payload containing prompt body = fail |
24
+
25
+ ## Event semantics (privacy-safe)
26
+
27
+ No prompt text, no URLs of private gists content, no PII.
28
+
29
+ | Event | Properties |
30
+ | --- | --- |
31
+ | `cs_score` | `source`: template\|paste\|url; `overall_bucket`: 0\|1-49\|50-69\|70-89\|90-100; `ruleset` |
32
+ | `cs_fix_plan` | `dims_to_fix_count` |
33
+ | `cs_fix_apply` | `delta_bucket` |
34
+ | `cs_fix_cancel` | — |
35
+ | `cs_export` | `kind`: share_text\|share_img\|download_prompt\|ci_copy |
36
+ | `cs_rules_expand` | — |
37
+ | `cs_vendor_open` | — |
38
+
39
+ **Implementation default for OSS launch:** localStorage counters only (`crewscore_metrics_v1`) + optional `console.debug` — no third-party by default. Document how to wire Plausible/Umami later with same events.
40
+
41
+ **Schema authority:** Python module `crewscore/metrics.py` (`bucket_score`, `append_event`, `validate_props`) is the testable source of truth for event buckets and forbidden prop keys; web localStorage must keep matching event names/buckets.
42
+
43
+ ## Decision thresholds (pre-commit)
44
+
45
+ | Signal | Green | Yellow | Red |
46
+ | --- | --- | --- | --- |
47
+ | Score rate (engaged sessions) | ≥40% | 20–40% | <20% |
48
+ | Fix apply / plan ratio | 30–80% | <30% or >90% | plan never shown |
49
+ | Export after score | ≥15% | 5–15% | <5% |
50
+ | Trust issues (manual) | 0 severe | wording nits | "certification" complaints |
51
+
52
+ ## Qualitative
53
+
54
+ - 5 builder think-alouds: "what does this score mean?"
55
+ - Show HN first-comment monitoring for black-box / overclaim language
56
+
57
+ ## Limitations
58
+
59
+ No production analytics currently; local counters only until owner enables privacy-respecting host.
@@ -27,7 +27,7 @@ inputs:
27
27
  required: false
28
28
  default: "crewscore-summary.md"
29
29
  pr-comment:
30
- description: On pull_request events, post or update a sticky PR comment with the summary (requires pull-requests: write)
30
+ description: "On pull_request events, post or update a sticky PR comment with the summary (requires pull-requests write permission)"
31
31
  required: false
32
32
  default: "true"
33
33
 
@@ -1,3 +1,3 @@
1
1
  """CrewScore — offline structural production-readiness scorecard for AI agent prompts."""
2
2
 
3
- __version__ = "0.2.4"
3
+ __version__ = "0.2.6"
@@ -429,13 +429,21 @@ def export_eval(prompt, prompt_file, output_dir):
429
429
  type=click.Path(),
430
430
  help="Write enhanced prompt to a new file",
431
431
  )
432
+ @click.option(
433
+ "--plan",
434
+ "--dry-run",
435
+ "plan",
436
+ is_flag=True,
437
+ default=False,
438
+ help="List planned fix dimensions without writing (dry-run)",
439
+ )
432
440
  @click.option(
433
441
  "--json",
434
442
  "as_json",
435
443
  is_flag=True,
436
444
  help="Emit JSON summary of applied dimensions and score delta",
437
445
  )
438
- def fix(prompt, prompt_file, apply, output, as_json):
446
+ def fix(prompt, prompt_file, apply, output, plan, as_json):
439
447
  """Append recommended guardrail patterns to a system prompt."""
440
448
  from crewscore.scorers.fix_patterns import apply_fixes, explain_fixes, generate_fixes
441
449
 
@@ -454,6 +462,13 @@ def fix(prompt, prompt_file, apply, output, as_json):
454
462
  )
455
463
  sys.exit(1)
456
464
 
465
+ if plan and (apply or output):
466
+ err_console.print(
467
+ "[red]Error: --plan/--dry-run is mutually exclusive with "
468
+ "--apply and --output[/red]"
469
+ )
470
+ sys.exit(1)
471
+
457
472
  before = structural_analysis.analyze(system_prompt)
458
473
  before_result = build_result(before)
459
474
  fixes = generate_fixes(before)
@@ -463,6 +478,51 @@ def fix(prompt, prompt_file, apply, output, as_json):
463
478
  "(tool allowlists, human approval hooks, logging, and policy enforcement)"
464
479
  )
465
480
 
481
+ if plan:
482
+ planned = list(fixes.keys())
483
+ if as_json:
484
+ click.echo(
485
+ json.dumps(
486
+ {
487
+ "fixes_planned": planned,
488
+ "before": before_result.to_dict(),
489
+ "written": False,
490
+ "note": honesty_note,
491
+ },
492
+ indent=2,
493
+ sort_keys=True,
494
+ )
495
+ )
496
+ return
497
+ console.print()
498
+ console.print(
499
+ Panel(
500
+ f"[bold]{BRAND.upper()} — Fix Plan (dry-run)[/bold]",
501
+ border_style="cyan",
502
+ expand=False,
503
+ )
504
+ )
505
+ console.print()
506
+ if not planned:
507
+ console.print(
508
+ " [green]No fixes needed — structural score is already strong.[/green]"
509
+ )
510
+ else:
511
+ console.print(
512
+ f" [cyan]Would apply[/cyan] templates for {len(planned)} dimension(s):"
513
+ )
514
+ for key in planned:
515
+ console.print(f" · {key}")
516
+ console.print()
517
+ console.print(explain_fixes(fixes))
518
+ console.print()
519
+ console.print(
520
+ "[dim]Plan only — file not modified. "
521
+ "Re-run without --plan and with --apply or --output to write.[/dim]"
522
+ )
523
+ console.print()
524
+ return
525
+
466
526
  if not fixes:
467
527
  if as_json:
468
528
  click.echo(
@@ -0,0 +1,81 @@
1
+ """Privacy-safe local metrics schema (no network, never store prompt text).
2
+
3
+ Schema authority for event names/buckets used by web localStorage
4
+ (`crewscore_metrics_v1`) and any future CLI counters.
5
+ """
6
+
7
+ from __future__ import annotations
8
+
9
+ import time
10
+ from typing import Any
11
+
12
+ # Keys that must never appear in metrics payloads (case-insensitive).
13
+ FORBIDDEN_PROP_KEYS = frozenset(
14
+ {
15
+ "prompt",
16
+ "text",
17
+ "body",
18
+ "system_prompt",
19
+ "content",
20
+ }
21
+ )
22
+
23
+
24
+ def bucket_score(n: int | float) -> str:
25
+ """Map an overall score to a privacy-safe bucket string."""
26
+ score = int(n) if n is not None else 0
27
+ if score <= 0:
28
+ return "0"
29
+ if score < 50:
30
+ return "1-49"
31
+ if score < 70:
32
+ return "50-69"
33
+ if score < 90:
34
+ return "70-89"
35
+ return "90-100"
36
+
37
+
38
+ def validate_props(props: dict[str, Any] | None) -> bool:
39
+ """Reject payloads that could carry prompt or free-text body content.
40
+
41
+ Raises ValueError when a forbidden key is present (case-insensitive).
42
+ Returns True when props are safe.
43
+ """
44
+ if not props:
45
+ return True
46
+ for key in props:
47
+ if str(key).lower() in FORBIDDEN_PROP_KEYS:
48
+ raise ValueError(
49
+ f"metrics props must not include prompt text key: {key!r}"
50
+ )
51
+ return True
52
+
53
+
54
+ def append_event(
55
+ store: dict[str, Any] | None,
56
+ event: str,
57
+ props: dict[str, Any] | None = None,
58
+ *,
59
+ max_events: int = 200,
60
+ ) -> dict[str, Any]:
61
+ """Append a privacy-checked event; cap store size to max_events (newest kept).
62
+
63
+ Store shape matches web localStorage:
64
+ {"events": [{"e": name, "t": epoch_ms, "p": props}, ...]}
65
+ """
66
+ props = dict(props or {})
67
+ validate_props(props)
68
+
69
+ out: dict[str, Any] = dict(store or {})
70
+ events = list(out.get("events") or [])
71
+ events.append(
72
+ {
73
+ "e": event,
74
+ "t": int(time.time() * 1000),
75
+ "p": props,
76
+ }
77
+ )
78
+ if max_events > 0 and len(events) > max_events:
79
+ events = events[-max_events:]
80
+ out["events"] = events
81
+ return out
@@ -0,0 +1,105 @@
1
+ # Plan: Preflight completion slice (SDD + TDD)
2
+
3
+ **Repo:** `C:\Repos\shmindmaster\crewscore`
4
+ **Branch:** `feat/preflight-completion` (off main)
5
+ **Goal:** Finish audit leftovers with TDD-backed, shippable product work — CLI fix plan parity, locked UX contracts, metrics module, export completion, 0.2.6.
6
+
7
+ ## Global constraints
8
+
9
+ - TDD: failing test first for every production behavior change.
10
+ - Honesty: structural pre-gate only; no red-team / certification claims.
11
+ - Keep existing HTML contract strings: builder-first hero, "Vendor checklist (self-attest)", not-a-red-team, templates inflate.
12
+ - Web scoring stays offline (`score-engine.js`); no LLM SDKs.
13
+ - Tests: `py -3.13 -m pytest` must stay green.
14
+ - Prefer small commits per task.
15
+ - Do not force-push main; work on feature branch; PR or merge when green.
16
+
17
+ ## Tasks
18
+
19
+ ### Task 1 — CLI fix plan / dry-run (agentic parity with web)
20
+
21
+ **Why:** Web has plan→apply; CLI `fix` still mutates or dumps with less control.
22
+
23
+ **TDD:**
24
+ 1. RED: `test_fix_plan_json_lists_dimensions_without_writing` — `crewscore fix --prompt-file X --plan --json` exits 0, lists `fixes_planned` keys, does not change file.
25
+ 2. RED: `test_fix_plan_human_lists_dimensions` — human output mentions planned dimensions / "plan" language.
26
+ 3. GREEN: implement `--plan` / `--dry-run` on `fix` command.
27
+ 4. RED/GREEN: without `--plan` or `--apply`/`--output`, behavior unchanged (print enhanced prompt).
28
+
29
+ **Files:** `crewscore/cli.py`, `tests/test_cli.py`, optionally thin helper in `scorers/fix_patterns.py`.
30
+
31
+ **Done when:** pytest for plan flags pass; applying still works.
32
+
33
+ ---
34
+
35
+ ### Task 2 — Web UX contract tests (lock preflight workflow)
36
+
37
+ **Why:** Audit F1–F5 implemented in HTML without strong regression tests.
38
+
39
+ **TDD:**
40
+ 1. RED tests in `tests/test_web_ux.py` asserting `index.html` contains:
41
+ - stage markers: Prompt / Inspect / Act / Export (or `stg-prompt` … `stg-export`)
42
+ - "Plan fix" or plan-before-mutate language
43
+ - "Apply plan" and Cancel
44
+ - capability stamp "Structural pre-gate"
45
+ - vendor secondary (not equal primary tab list as only chrome)
46
+ - `crewscore_metrics_v1` or track() privacy events
47
+ 2. GREEN: only if HTML missing pieces — fix HTML to match; do not remove honesty strings.
48
+
49
+ **Files:** `tests/test_web_ux.py`, possibly `index.html`.
50
+
51
+ **Done when:** new tests pass; existing vendor/web_engine tests pass.
52
+
53
+ ---
54
+
55
+ ### Task 3 — Privacy-safe metrics module (testable pure core)
56
+
57
+ **Why:** F8 — metrics only inlined in index.html; untestable.
58
+
59
+ **TDD:**
60
+ 1. RED: pure functions in Python or small JS-testable contract — prefer Python helper `crewscore/metrics.py`:
61
+ - `bucket_score(n) -> str` buckets
62
+ - `append_event(store, event, props, *, max_events=200) -> store` no prompt keys allowed
63
+ - `assert_privacy(props)` rejects keys like `prompt`, `text`, `body`
64
+ 2. GREEN: implement module; wire optional CLI `crewscore` does not require metrics.
65
+ 3. Document web localStorage schema matches event names in `_product-experience/06-outcome-measurement-plan.md`.
66
+
67
+ **Files:** `crewscore/metrics.py`, `tests/test_metrics.py`, doc touch if needed.
68
+
69
+ **Done when:** metrics unit tests pass; no prompt capture.
70
+
71
+ ---
72
+
73
+ ### Task 4 — Export completion checklist + reduced-motion
74
+
75
+ **Why:** F7 success state incomplete; a11y floor.
76
+
77
+ **TDD:**
78
+ 1. RED: `test_web_ux` asserts export deck has completion checklist markers (e.g. `export-checklist` or three completion items: share/CI/prompt).
79
+ 2. RED: CSS or HTML has `prefers-reduced-motion` rule.
80
+ 3. GREEN: update `index.html` export stage + CSS.
81
+
82
+ **Files:** `index.html`, `tests/test_web_ux.py`.
83
+
84
+ **Done when:** tests pass.
85
+
86
+ ---
87
+
88
+ ### Task 5 — Version 0.2.6 + README CLI docs + ship
89
+
90
+ **Why:** Surface new `--plan` flag; ship coherent release.
91
+
92
+ **TDD:**
93
+ 1. RED: version string expectations if any (prefer dynamic RULESET/version tests already).
94
+ 2. GREEN: bump `__version__` / pyproject to 0.2.6; export web engine; README documents `crewscore fix --plan`.
95
+ 3. Run full pytest; commit; open PR or merge to main; tag/publish only if existing publish pattern allowed (PyPI via 1Password).
96
+
97
+ **Files:** `crewscore/__init__.py`, `pyproject.toml`, `README.md`, `score-engine.js`, AGENTS.md if needed.
98
+
99
+ **Done when:** full suite green; package version 0.2.6; docs mention plan.
100
+
101
+ ## Parallelism note
102
+
103
+ Tasks 1 and 3 are file-independent (can parallel via worktrees).
104
+ Tasks 2 and 4 share `index.html` / `test_web_ux.py` — sequential after 1 or with 1 done.
105
+ Task 5 last.
Binary file
@@ -0,0 +1,5 @@
1
+ <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
2
+ <rect width="32" height="32" rx="3" fill="#0a0c0b"/>
3
+ <rect x="2" y="2" width="28" height="28" rx="2" fill="none" stroke="#e8a317" stroke-width="1.5" opacity="0.7"/>
4
+ <text x="16" y="21" text-anchor="middle" font-family="IBM Plex Mono, ui-monospace, monospace" font-size="11" font-weight="700" fill="#e8a317">CS</text>
5
+ </svg>