crewscore 0.2.4__tar.gz → 0.2.6__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {crewscore-0.2.4 → crewscore-0.2.6}/AGENTS.md +2 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/PKG-INFO +10 -3
- {crewscore-0.2.4 → crewscore-0.2.6}/README.md +9 -2
- crewscore-0.2.6/_product-experience/00-discovery.md +51 -0
- crewscore-0.2.6/_product-experience/01-audit.md +94 -0
- crewscore-0.2.6/_product-experience/02-workflow-and-feature-design.md +60 -0
- crewscore-0.2.6/_product-experience/06-outcome-measurement-plan.md +59 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/action.yml +1 -1
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/__init__.py +1 -1
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/cli.py +61 -1
- crewscore-0.2.6/crewscore/metrics.py +81 -0
- crewscore-0.2.6/docs/superpowers/plans/2026-07-28-preflight-completion-sdd.md +105 -0
- crewscore-0.2.6/docs/ui-after-polish.png +0 -0
- crewscore-0.2.6/docs/ui-preflight-workflow.png +0 -0
- crewscore-0.2.6/favicon.svg +5 -0
- crewscore-0.2.6/index.html +1025 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/pyproject.toml +1 -1
- {crewscore-0.2.4 → crewscore-0.2.6}/score-engine.js +1 -1
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_cli.py +83 -0
- crewscore-0.2.6/tests/test_metrics.py +56 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_vendor.py +3 -1
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_web_engine.py +6 -1
- crewscore-0.2.6/tests/test_web_ux.py +69 -0
- crewscore-0.2.4/favicon.svg +0 -5
- crewscore-0.2.4/index.html +0 -556
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/ISSUE_TEMPLATE/scoring_feedback.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/crewscore-selftest.yml +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/example-ci.yml +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.github/workflows/pytest.yml +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/.gitignore +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/CNAME +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/CONTRIBUTING.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/LICENSE +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/export_eval.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/report.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/rules_catalog.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scan.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/__init__.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/fix_patterns.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scorers/structural_analysis.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/scoring.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/summary.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/vendor_scorecard.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/crewscore/web_export.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/demo.svg +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/README.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/devto.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/linkedin.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/reddit.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/show-hn.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/launch/x-post.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/next-steps-eval.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/pmf-research-2026-07-28.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/publish-checklist.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/research/2026-07-28-useful-product-and-virality-research.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-crewscore-viral-launch.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-useful-product-slice.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-a-scoring-report.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-b-scan-report.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/superpowers/plans/2026-07-28-workstream-cd-product-action-report.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/docs/viral-product-spec-2026-07-28.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/LEADERBOARD.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/README.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/01-bare-assistant.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/02-demo-agent.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/03-agents-md-weak.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/04-partial-hygiene.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/corpus/prompts/05-hardened-ops.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/examples/sample-prompt.md +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/favicon.ico +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/scripts/export_web_engine.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/scripts/score_corpus.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_action_manifest.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_corpus.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_explain.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_export_eval.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_report.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_rules_catalog.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_scan.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_structural_analysis.py +0 -0
- {crewscore-0.2.4 → crewscore-0.2.6}/tests/test_summary.py +0 -0
|
@@ -33,6 +33,8 @@ crewscore test --prompt-file ./system-prompt.md --json --threshold 50
|
|
|
33
33
|
crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
|
|
34
34
|
|
|
35
35
|
# apply guardrail patterns
|
|
36
|
+
crewscore fix --prompt-file ./system-prompt.md --plan # dry-run: list planned dimensions, no write
|
|
37
|
+
crewscore fix --prompt-file ./system-prompt.md --dry-run # alias for --plan
|
|
36
38
|
crewscore fix --prompt-file ./system-prompt.md
|
|
37
39
|
crewscore fix --prompt-file ./system-prompt.md --apply
|
|
38
40
|
crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: crewscore
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.6
|
|
4
4
|
Summary: CrewScore — offline structural production-readiness scorecard for AI agent system prompts.
|
|
5
5
|
Project-URL: Homepage, https://crewscore.ai
|
|
6
6
|
Project-URL: Repository, https://github.com/shmindmaster/crewscore
|
|
@@ -189,7 +189,12 @@ Human mode also prints a one-line share blurb with your overall score and [crews
|
|
|
189
189
|
### Apply guardrail patterns
|
|
190
190
|
|
|
191
191
|
```bash
|
|
192
|
-
#
|
|
192
|
+
# plan only — list dimensions that would be fixed (no file write)
|
|
193
|
+
crewscore fix --prompt-file ./system-prompt.md --plan
|
|
194
|
+
# alias:
|
|
195
|
+
crewscore fix --prompt-file ./system-prompt.md --dry-run
|
|
196
|
+
|
|
197
|
+
# print enhanced prompt (stdout only)
|
|
193
198
|
crewscore fix --prompt-file ./system-prompt.md
|
|
194
199
|
|
|
195
200
|
# write in place and show score delta
|
|
@@ -200,9 +205,11 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
|
|
|
200
205
|
|
|
201
206
|
# machine-readable summary
|
|
202
207
|
crewscore fix --prompt-file ./system-prompt.md --apply --json
|
|
208
|
+
# plan as JSON (fixes_planned, written: false)
|
|
209
|
+
crewscore fix --prompt-file ./system-prompt.md --plan --json
|
|
203
210
|
```
|
|
204
211
|
|
|
205
|
-
These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
|
|
212
|
+
`--plan` / `--dry-run` is mutually exclusive with `--apply` and `--output`. These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
|
|
206
213
|
|
|
207
214
|
### Vendor checklist (self-attest, secondary)
|
|
208
215
|
|
|
@@ -162,7 +162,12 @@ Human mode also prints a one-line share blurb with your overall score and [crews
|
|
|
162
162
|
### Apply guardrail patterns
|
|
163
163
|
|
|
164
164
|
```bash
|
|
165
|
-
#
|
|
165
|
+
# plan only — list dimensions that would be fixed (no file write)
|
|
166
|
+
crewscore fix --prompt-file ./system-prompt.md --plan
|
|
167
|
+
# alias:
|
|
168
|
+
crewscore fix --prompt-file ./system-prompt.md --dry-run
|
|
169
|
+
|
|
170
|
+
# print enhanced prompt (stdout only)
|
|
166
171
|
crewscore fix --prompt-file ./system-prompt.md
|
|
167
172
|
|
|
168
173
|
# write in place and show score delta
|
|
@@ -173,9 +178,11 @@ crewscore fix --prompt-file ./system-prompt.md --output ./system-prompt-guarded.
|
|
|
173
178
|
|
|
174
179
|
# machine-readable summary
|
|
175
180
|
crewscore fix --prompt-file ./system-prompt.md --apply --json
|
|
181
|
+
# plan as JSON (fixes_planned, written: false)
|
|
182
|
+
crewscore fix --prompt-file ./system-prompt.md --plan --json
|
|
176
183
|
```
|
|
177
184
|
|
|
178
|
-
These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
|
|
185
|
+
`--plan` / `--dry-run` is mutually exclusive with `--apply` and `--output`. These are **prompt text templates**. They can raise the structural score without changing runtime behavior — wire matching controls (tool gates, logging, budgets) in your application.
|
|
179
186
|
|
|
180
187
|
### Vendor checklist (self-attest, secondary)
|
|
181
188
|
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# CrewScore — Discovery
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-07-28
|
|
4
|
+
**Repo:** `shmindmaster/crewscore`
|
|
5
|
+
**Surfaces in scope:** Public web (crewscore.ai / `index.html`), CLI, GitHub Action, docs/launch.
|
|
6
|
+
|
|
7
|
+
## Product promise
|
|
8
|
+
|
|
9
|
+
Offline **structural pre-gate** for AI agent system prompts: discover missing hygiene signals, explain with open rules, fix text templates, gate CI — never claim red-team or certification.
|
|
10
|
+
|
|
11
|
+
## Users & jobs
|
|
12
|
+
|
|
13
|
+
| Persona | Job | Success looks like |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| **Agent builder** (primary) | Before merge/demo, know if system prompt lacks production hygiene | Score + top gaps + fix or CI gate in <2 min |
|
|
16
|
+
| **OSS / HN visitor** | Understand product in 15s without install | One click demo → memorable number → trust honesty |
|
|
17
|
+
| **AppSec-curious** | Prefer open rules over black-box scores | Expand full rule IDs; match CLI |
|
|
18
|
+
| **Buyer / ops** (secondary) | Self-attest vendor diligence | Separate, demoted checklist — not the hero |
|
|
19
|
+
|
|
20
|
+
## Domain objects
|
|
21
|
+
|
|
22
|
+
- **Prompt artifact** (text / file / URL)
|
|
23
|
+
- **Ruleset** (`crewscore-hygiene@x.y.z`) + **rule_id** findings
|
|
24
|
+
- **Score result** (8 dimensions, overall, structural tier, warnings)
|
|
25
|
+
- **Fix pack** (appended templates; mutates text only)
|
|
26
|
+
- **Scan set** (repo paths; CLI/Action)
|
|
27
|
+
- **Share artifact** (image / text; must not overclaim)
|
|
28
|
+
|
|
29
|
+
## Capability map (what exists)
|
|
30
|
+
|
|
31
|
+
| Surface | Status | Notes |
|
|
32
|
+
| --- | --- | --- |
|
|
33
|
+
| Public marketing / demo site | Deep reviewed | Single-page score tool; primary viral surface |
|
|
34
|
+
| CLI `test` / `scan` / `fix` / `rules` / `export-eval` | Deep reviewed | Strong; truth source for web |
|
|
35
|
+
| GitHub Action + PR sticky summary | Shallow reviewed | Functional; not browser UX |
|
|
36
|
+
| Vendor self-attest | Shallow reviewed | Demoted; still competes for attention if tab-equal |
|
|
37
|
+
| Auth / billing / admin | Not applicable | Zero-install OSS |
|
|
38
|
+
| Live LLM / agent runtime | Not applicable | Explicitly out of scope |
|
|
39
|
+
|
|
40
|
+
## Constraints
|
|
41
|
+
|
|
42
|
+
- Deterministic scoring (no model calls on critical path)
|
|
43
|
+
- Browser must stay offline-capable (local `score-engine.js`)
|
|
44
|
+
- Honesty > virality theater
|
|
45
|
+
- Keep HTML contract tests (builder-first hero, authenticity, escapeHtml, engine parity)
|
|
46
|
+
|
|
47
|
+
## Evidence sources
|
|
48
|
+
|
|
49
|
+
- Runtime: live site + Playwright primary journey
|
|
50
|
+
- Code: `index.html`, `score-engine.js`, CLI
|
|
51
|
+
- Research: `docs/research/2026-07-28-useful-product-and-virality-research.md`
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# CrewScore — Product experience audit
|
|
2
|
+
|
|
3
|
+
**Date:** 2026-07-28
|
|
4
|
+
**Method:** Discovery + live journey (Playwright) + implementation review
|
|
5
|
+
**Scope:** Public web primary; CLI/Action as continuity targets
|
|
6
|
+
|
|
7
|
+
## Coverage ledger
|
|
8
|
+
|
|
9
|
+
| Surface | Status | Notes |
|
|
10
|
+
| --- | --- | --- |
|
|
11
|
+
| Public marketing / zero-install demo | Deep reviewed | Core product surface for cold traffic |
|
|
12
|
+
| Customer product (web score/fix) | Deep reviewed | Same page as marketing |
|
|
13
|
+
| Developer integrations (CLI/Action) | Shallow reviewed | Strong utility; linked in footer |
|
|
14
|
+
| Vendor diligence | Shallow reviewed | Secondary; over-weighted in IA if equal tab |
|
|
15
|
+
| Auth / billing / ops / support portals | Not applicable | |
|
|
16
|
+
|
|
17
|
+
## Primary journey audited
|
|
18
|
+
|
|
19
|
+
**Try template → score → understand → fix → share**
|
|
20
|
+
|
|
21
|
+
Evidence (2026-07-28): Weak demo → 0/100 → Fix → 46/100 works. Full rule dump previously overloaded; polish reduced dump but **workflow still single-panel form**, not a task lifecycle.
|
|
22
|
+
|
|
23
|
+
## Findings
|
|
24
|
+
|
|
25
|
+
### F1 — Workflow is a form, not a preflight job
|
|
26
|
+
**Severity:** High · **Class:** product-fix-required
|
|
27
|
+
**Evidence:** One panel with score, fix, URL, vendor tab; no explicit stages, plan-before-mutate, or completion.
|
|
28
|
+
**Impact:** Builders don't feel "I completed a gate"; visitors don't learn the product story.
|
|
29
|
+
**Recommendation:** Stage the job: Prompt → Inspect → Act → Export.
|
|
30
|
+
**Validate:** Task completion without reading docs; time-to-first-insight < 20s.
|
|
31
|
+
|
|
32
|
+
### F2 — "Fix" mutates without plan preview (agentic control gap)
|
|
33
|
+
**Severity:** High · **Class:** product-fix-required
|
|
34
|
+
**Evidence:** `fixAndRescore` replaces textarea immediately; no preview of sections that will be appended.
|
|
35
|
+
**Impact:** Surprising text mutation; reduces trust for production prompts.
|
|
36
|
+
**Recommendation:** Preview plan (dimensions/sections to add) → explicit Apply.
|
|
37
|
+
**Validate:** User can cancel fix; prompt unchanged until Apply.
|
|
38
|
+
|
|
39
|
+
### F3 — Capability boundary is easy to misread as "AI safety product"
|
|
40
|
+
**Severity:** High · **Class:** product-fix-required
|
|
41
|
+
**Evidence:** Hero about agent prompts; scoring is regex. Honesty present but competes with marketing.
|
|
42
|
+
**Impact:** Trust collapse on HN; wrong mental model.
|
|
43
|
+
**Recommendation:** Persistent capability chip + result stamp: "Structural pre-gate · not red-team".
|
|
44
|
+
**Validate:** After first score, user can restate what product is/isn't.
|
|
45
|
+
|
|
46
|
+
### F4 — Results still present as dashboard, not decisions
|
|
47
|
+
**Severity:** Medium · **Class:** product-fix-required
|
|
48
|
+
**Evidence:** 8 bars + optional full rules; top gaps improved but no "next action" hierarchy beyond Fix.
|
|
49
|
+
**Impact:** Cognitive load; weak link to CI/export-eval.
|
|
50
|
+
**Recommendation:** Decision stack: Top gaps (act) → dimension radar/checklist → optional rules → CI/export CTA.
|
|
51
|
+
**Validate:** First click after score is gap-relevant action ≥70% of sessions (instrument).
|
|
52
|
+
|
|
53
|
+
### F5 — Vendor path steals IA equality
|
|
54
|
+
**Severity:** Medium · **Class:** product-fix-required
|
|
55
|
+
**Evidence:** Tab strip co-primary with agent scoring. Research demoted vendor.
|
|
56
|
+
**Impact:** Dilutes builder product; resurrects questionnaire theater.
|
|
57
|
+
**Recommendation:** Vendor as secondary link / `#vendor`, not equal tab chrome.
|
|
58
|
+
**Validate:** Cold open shows only agent job.
|
|
59
|
+
|
|
60
|
+
### F6 — Visual system still "AI product default"
|
|
61
|
+
**Severity:** Medium · **Class:** product-fix-required
|
|
62
|
+
**Evidence:** Dark slate + cyan gradient + system sans; generic after 2026 AI sites.
|
|
63
|
+
**Impact:** Low memorability; fails distinctive frontend-design bar.
|
|
64
|
+
**Recommendation:** Preflight/instrument aesthetic — amber caution + terminal mono + radar/checklist signature.
|
|
65
|
+
**Validate:** Screenshot recognition without logo text in informal test.
|
|
66
|
+
|
|
67
|
+
### F7 — Empty / error / success states underdesigned
|
|
68
|
+
**Severity:** Medium · **Class:** product-fix-required
|
|
69
|
+
**Evidence:** Empty = blank textarea; error = red line; success = scorecard only.
|
|
70
|
+
**Impact:** No guided first action; recovery is weak for URL/CORS failures.
|
|
71
|
+
**Recommendation:** Empty invites one demo; errors name cause + paste fallback; success has completion checklist.
|
|
72
|
+
**Validate:** Empty state has single recommended action; URL error offers paste.
|
|
73
|
+
|
|
74
|
+
### F8 — No outcome instrumentation
|
|
75
|
+
**Severity:** Medium · **Class:** product-fix-required
|
|
76
|
+
**Evidence:** No events for score/fix/share (by design privacy-first, but no measurement plan either).
|
|
77
|
+
**Impact:** Cannot learn if UX works post-launch.
|
|
78
|
+
**Recommendation:** Privacy-safe local counters + optional anonymous funnel (no prompt text). See `06-outcome-measurement-plan.md`.
|
|
79
|
+
|
|
80
|
+
### F9 — CLI/Action continuity weak on web
|
|
81
|
+
**Severity:** Low · **Class:** informational
|
|
82
|
+
**Evidence:** Footer codes only.
|
|
83
|
+
**Impact:** Web users don't graduate to CI.
|
|
84
|
+
**Recommendation:** After score, "Gate this in CI" copy-paste block with scan/action snippet.
|
|
85
|
+
|
|
86
|
+
## Prioritized decision list
|
|
87
|
+
|
|
88
|
+
1. Redesign primary workflow stages + fix plan/apply (F1, F2)
|
|
89
|
+
2. Capability stamp + decision-first results (F3, F4)
|
|
90
|
+
3. Demote vendor IA (F5)
|
|
91
|
+
4. Distinctive preflight visual system (F6)
|
|
92
|
+
5. Empty/error/success states (F7)
|
|
93
|
+
6. Measurement without prompt capture (F8)
|
|
94
|
+
7. CI handoff CTA (F9)
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# CrewScore — Workflow & feature design
|
|
2
|
+
|
|
3
|
+
## Outcome
|
|
4
|
+
|
|
5
|
+
Builder completes a **structural preflight** on an agent prompt: knows overall hygiene, top missing signals, optionally applies text templates with consent, and leaves with share/CI next step — without believing the product is a red-team.
|
|
6
|
+
|
|
7
|
+
## Target primary workflow
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
Entry (cold / return)
|
|
11
|
+
→ Stage 1 PROMPT: paste | template | URL
|
|
12
|
+
→ Stage 2 INSPECT: score + capability stamp + top gaps + dims
|
|
13
|
+
→ Stage 3 ACT: plan fix → approve → re-inspect
|
|
14
|
+
→ Stage 4 EXPORT: share image/text | CI snippet | improved prompt
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
### Stage details
|
|
18
|
+
|
|
19
|
+
| Stage | User intent | Happy path | Empty | Error | Recovery |
|
|
20
|
+
| --- | --- | --- | --- | --- | --- |
|
|
21
|
+
| PROMPT | Provide artifact | Template or paste | Invite "Weak demo" | — | — |
|
|
22
|
+
| INSPECT | Understand gaps | Score + top 3 gaps | — | Engine missing | Reload |
|
|
23
|
+
| ACT | Improve text | Preview sections → Apply | No weak dims → "Already covered" | — | Cancel keeps original |
|
|
24
|
+
| EXPORT | Take result elsewhere | Share / download / CI | — | Clipboard denied | Manual select |
|
|
25
|
+
|
|
26
|
+
## Secondary workflow
|
|
27
|
+
|
|
28
|
+
**Vendor self-attest** — linked, not staged with primary. Self-attest stamp. No equal chrome.
|
|
29
|
+
|
|
30
|
+
## Information architecture
|
|
31
|
+
|
|
32
|
+
- **Primary nav:** none (single job page)
|
|
33
|
+
- **Progress:** step indicator 1–4 (or Prompt / Inspect / Act / Export)
|
|
34
|
+
- **Results hierarchy:** (1) overall + soft verdict (2) capability stamp (3) top gaps (4) dimensions (5) all rules (6) export
|
|
35
|
+
- **Progressive disclosure:** full rules collapsed; formula collapsed
|
|
36
|
+
|
|
37
|
+
## Agentic / AI interaction contract
|
|
38
|
+
|
|
39
|
+
CrewScore scoring is **deterministic**, not generative. The only "agent-like" action is **fix** (mutates user content).
|
|
40
|
+
|
|
41
|
+
| Action | Autonomy | Control |
|
|
42
|
+
| --- | --- | --- |
|
|
43
|
+
| Score | Auto on request | User initiates; no network model |
|
|
44
|
+
| Explain findings | Auto with score | Expand/collapse; open rule_ids |
|
|
45
|
+
| Fix plan | Preview only | List dimensions/sections that would be appended |
|
|
46
|
+
| Apply fix | Explicit approval | Apply / Cancel; original retained until Apply |
|
|
47
|
+
| Share | User-triggered | Copy text / image; disclaimer in share string |
|
|
48
|
+
| URL load | User-triggered | CORS failure → paste fallback |
|
|
49
|
+
|
|
50
|
+
**Prohibited claims in UI:** certification, audit complete, red-team pass, runtime safety proven.
|
|
51
|
+
|
|
52
|
+
**Uncertainty:** Always structural-only; template_boilerplate warning when fix templates detected.
|
|
53
|
+
|
|
54
|
+
## Acceptance signals
|
|
55
|
+
|
|
56
|
+
- Time from land → first score ≤ 20s with template
|
|
57
|
+
- Fix cannot change text without Apply
|
|
58
|
+
- After score, user sees capability stamp without scrolling past fold on desktop
|
|
59
|
+
- Vendor not equal-weight with primary CTA
|
|
60
|
+
- Share text includes "not a red-team"
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# CrewScore — Outcome measurement plan
|
|
2
|
+
|
|
3
|
+
## Decision this plan supports
|
|
4
|
+
|
|
5
|
+
Did the preflight web experience improve **successful structural pre-gates** (score → understand → act → export) without increasing overclaim/trust failure?
|
|
6
|
+
|
|
7
|
+
## Population & journey
|
|
8
|
+
|
|
9
|
+
- **Population:** Anonymous visitors to crewscore.ai + optional CLI users (separate)
|
|
10
|
+
- **Journey boundary:** First paint → first score → optional fix apply → optional share/CI copy
|
|
11
|
+
- **Baseline:** Pre-redesign qualitative (manual Playwright + audit F1–F9)
|
|
12
|
+
- **Comparison:** Post-deploy week-over-week funnel (if analytics enabled)
|
|
13
|
+
|
|
14
|
+
## Metrics (balanced)
|
|
15
|
+
|
|
16
|
+
| Outcome | Metric | Guardrail |
|
|
17
|
+
| --- | --- | --- |
|
|
18
|
+
| Activation | % sessions with ≥1 score | Bounce without interaction |
|
|
19
|
+
| Understanding | % expand "top gaps" or dim row | Time-to-score only |
|
|
20
|
+
| Controlled fix | % fix_plan_shown → fix_applied | fix_applied without plan (should be ~0) |
|
|
21
|
+
| Export | % share_or_download_or_ci_copy | Share without score (should be 0) |
|
|
22
|
+
| Trust | Qualitative: can restate "not red-team" | Support/issue reports of overclaim |
|
|
23
|
+
| Privacy | Zero prompt text in analytics | Any payload containing prompt body = fail |
|
|
24
|
+
|
|
25
|
+
## Event semantics (privacy-safe)
|
|
26
|
+
|
|
27
|
+
No prompt text, no URLs of private gists content, no PII.
|
|
28
|
+
|
|
29
|
+
| Event | Properties |
|
|
30
|
+
| --- | --- |
|
|
31
|
+
| `cs_score` | `source`: template\|paste\|url; `overall_bucket`: 0\|1-49\|50-69\|70-89\|90-100; `ruleset` |
|
|
32
|
+
| `cs_fix_plan` | `dims_to_fix_count` |
|
|
33
|
+
| `cs_fix_apply` | `delta_bucket` |
|
|
34
|
+
| `cs_fix_cancel` | — |
|
|
35
|
+
| `cs_export` | `kind`: share_text\|share_img\|download_prompt\|ci_copy |
|
|
36
|
+
| `cs_rules_expand` | — |
|
|
37
|
+
| `cs_vendor_open` | — |
|
|
38
|
+
|
|
39
|
+
**Implementation default for OSS launch:** localStorage counters only (`crewscore_metrics_v1`) + optional `console.debug` — no third-party by default. Document how to wire Plausible/Umami later with same events.
|
|
40
|
+
|
|
41
|
+
**Schema authority:** Python module `crewscore/metrics.py` (`bucket_score`, `append_event`, `validate_props`) is the testable source of truth for event buckets and forbidden prop keys; web localStorage must keep matching event names/buckets.
|
|
42
|
+
|
|
43
|
+
## Decision thresholds (pre-commit)
|
|
44
|
+
|
|
45
|
+
| Signal | Green | Yellow | Red |
|
|
46
|
+
| --- | --- | --- | --- |
|
|
47
|
+
| Score rate (engaged sessions) | ≥40% | 20–40% | <20% |
|
|
48
|
+
| Fix apply / plan ratio | 30–80% | <30% or >90% | plan never shown |
|
|
49
|
+
| Export after score | ≥15% | 5–15% | <5% |
|
|
50
|
+
| Trust issues (manual) | 0 severe | wording nits | "certification" complaints |
|
|
51
|
+
|
|
52
|
+
## Qualitative
|
|
53
|
+
|
|
54
|
+
- 5 builder think-alouds: "what does this score mean?"
|
|
55
|
+
- Show HN first-comment monitoring for black-box / overclaim language
|
|
56
|
+
|
|
57
|
+
## Limitations
|
|
58
|
+
|
|
59
|
+
No production analytics currently; local counters only until owner enables privacy-respecting host.
|
|
@@ -27,7 +27,7 @@ inputs:
|
|
|
27
27
|
required: false
|
|
28
28
|
default: "crewscore-summary.md"
|
|
29
29
|
pr-comment:
|
|
30
|
-
description: On pull_request events, post or update a sticky PR comment with the summary (requires pull-requests
|
|
30
|
+
description: "On pull_request events, post or update a sticky PR comment with the summary (requires pull-requests write permission)"
|
|
31
31
|
required: false
|
|
32
32
|
default: "true"
|
|
33
33
|
|
|
@@ -429,13 +429,21 @@ def export_eval(prompt, prompt_file, output_dir):
|
|
|
429
429
|
type=click.Path(),
|
|
430
430
|
help="Write enhanced prompt to a new file",
|
|
431
431
|
)
|
|
432
|
+
@click.option(
|
|
433
|
+
"--plan",
|
|
434
|
+
"--dry-run",
|
|
435
|
+
"plan",
|
|
436
|
+
is_flag=True,
|
|
437
|
+
default=False,
|
|
438
|
+
help="List planned fix dimensions without writing (dry-run)",
|
|
439
|
+
)
|
|
432
440
|
@click.option(
|
|
433
441
|
"--json",
|
|
434
442
|
"as_json",
|
|
435
443
|
is_flag=True,
|
|
436
444
|
help="Emit JSON summary of applied dimensions and score delta",
|
|
437
445
|
)
|
|
438
|
-
def fix(prompt, prompt_file, apply, output, as_json):
|
|
446
|
+
def fix(prompt, prompt_file, apply, output, plan, as_json):
|
|
439
447
|
"""Append recommended guardrail patterns to a system prompt."""
|
|
440
448
|
from crewscore.scorers.fix_patterns import apply_fixes, explain_fixes, generate_fixes
|
|
441
449
|
|
|
@@ -454,6 +462,13 @@ def fix(prompt, prompt_file, apply, output, as_json):
|
|
|
454
462
|
)
|
|
455
463
|
sys.exit(1)
|
|
456
464
|
|
|
465
|
+
if plan and (apply or output):
|
|
466
|
+
err_console.print(
|
|
467
|
+
"[red]Error: --plan/--dry-run is mutually exclusive with "
|
|
468
|
+
"--apply and --output[/red]"
|
|
469
|
+
)
|
|
470
|
+
sys.exit(1)
|
|
471
|
+
|
|
457
472
|
before = structural_analysis.analyze(system_prompt)
|
|
458
473
|
before_result = build_result(before)
|
|
459
474
|
fixes = generate_fixes(before)
|
|
@@ -463,6 +478,51 @@ def fix(prompt, prompt_file, apply, output, as_json):
|
|
|
463
478
|
"(tool allowlists, human approval hooks, logging, and policy enforcement)"
|
|
464
479
|
)
|
|
465
480
|
|
|
481
|
+
if plan:
|
|
482
|
+
planned = list(fixes.keys())
|
|
483
|
+
if as_json:
|
|
484
|
+
click.echo(
|
|
485
|
+
json.dumps(
|
|
486
|
+
{
|
|
487
|
+
"fixes_planned": planned,
|
|
488
|
+
"before": before_result.to_dict(),
|
|
489
|
+
"written": False,
|
|
490
|
+
"note": honesty_note,
|
|
491
|
+
},
|
|
492
|
+
indent=2,
|
|
493
|
+
sort_keys=True,
|
|
494
|
+
)
|
|
495
|
+
)
|
|
496
|
+
return
|
|
497
|
+
console.print()
|
|
498
|
+
console.print(
|
|
499
|
+
Panel(
|
|
500
|
+
f"[bold]{BRAND.upper()} — Fix Plan (dry-run)[/bold]",
|
|
501
|
+
border_style="cyan",
|
|
502
|
+
expand=False,
|
|
503
|
+
)
|
|
504
|
+
)
|
|
505
|
+
console.print()
|
|
506
|
+
if not planned:
|
|
507
|
+
console.print(
|
|
508
|
+
" [green]No fixes needed — structural score is already strong.[/green]"
|
|
509
|
+
)
|
|
510
|
+
else:
|
|
511
|
+
console.print(
|
|
512
|
+
f" [cyan]Would apply[/cyan] templates for {len(planned)} dimension(s):"
|
|
513
|
+
)
|
|
514
|
+
for key in planned:
|
|
515
|
+
console.print(f" · {key}")
|
|
516
|
+
console.print()
|
|
517
|
+
console.print(explain_fixes(fixes))
|
|
518
|
+
console.print()
|
|
519
|
+
console.print(
|
|
520
|
+
"[dim]Plan only — file not modified. "
|
|
521
|
+
"Re-run without --plan and with --apply or --output to write.[/dim]"
|
|
522
|
+
)
|
|
523
|
+
console.print()
|
|
524
|
+
return
|
|
525
|
+
|
|
466
526
|
if not fixes:
|
|
467
527
|
if as_json:
|
|
468
528
|
click.echo(
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
"""Privacy-safe local metrics schema (no network, never store prompt text).
|
|
2
|
+
|
|
3
|
+
Schema authority for event names/buckets used by web localStorage
|
|
4
|
+
(`crewscore_metrics_v1`) and any future CLI counters.
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
|
|
9
|
+
import time
|
|
10
|
+
from typing import Any
|
|
11
|
+
|
|
12
|
+
# Keys that must never appear in metrics payloads (case-insensitive).
|
|
13
|
+
FORBIDDEN_PROP_KEYS = frozenset(
|
|
14
|
+
{
|
|
15
|
+
"prompt",
|
|
16
|
+
"text",
|
|
17
|
+
"body",
|
|
18
|
+
"system_prompt",
|
|
19
|
+
"content",
|
|
20
|
+
}
|
|
21
|
+
)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
def bucket_score(n: int | float) -> str:
|
|
25
|
+
"""Map an overall score to a privacy-safe bucket string."""
|
|
26
|
+
score = int(n) if n is not None else 0
|
|
27
|
+
if score <= 0:
|
|
28
|
+
return "0"
|
|
29
|
+
if score < 50:
|
|
30
|
+
return "1-49"
|
|
31
|
+
if score < 70:
|
|
32
|
+
return "50-69"
|
|
33
|
+
if score < 90:
|
|
34
|
+
return "70-89"
|
|
35
|
+
return "90-100"
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def validate_props(props: dict[str, Any] | None) -> bool:
|
|
39
|
+
"""Reject payloads that could carry prompt or free-text body content.
|
|
40
|
+
|
|
41
|
+
Raises ValueError when a forbidden key is present (case-insensitive).
|
|
42
|
+
Returns True when props are safe.
|
|
43
|
+
"""
|
|
44
|
+
if not props:
|
|
45
|
+
return True
|
|
46
|
+
for key in props:
|
|
47
|
+
if str(key).lower() in FORBIDDEN_PROP_KEYS:
|
|
48
|
+
raise ValueError(
|
|
49
|
+
f"metrics props must not include prompt text key: {key!r}"
|
|
50
|
+
)
|
|
51
|
+
return True
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def append_event(
|
|
55
|
+
store: dict[str, Any] | None,
|
|
56
|
+
event: str,
|
|
57
|
+
props: dict[str, Any] | None = None,
|
|
58
|
+
*,
|
|
59
|
+
max_events: int = 200,
|
|
60
|
+
) -> dict[str, Any]:
|
|
61
|
+
"""Append a privacy-checked event; cap store size to max_events (newest kept).
|
|
62
|
+
|
|
63
|
+
Store shape matches web localStorage:
|
|
64
|
+
{"events": [{"e": name, "t": epoch_ms, "p": props}, ...]}
|
|
65
|
+
"""
|
|
66
|
+
props = dict(props or {})
|
|
67
|
+
validate_props(props)
|
|
68
|
+
|
|
69
|
+
out: dict[str, Any] = dict(store or {})
|
|
70
|
+
events = list(out.get("events") or [])
|
|
71
|
+
events.append(
|
|
72
|
+
{
|
|
73
|
+
"e": event,
|
|
74
|
+
"t": int(time.time() * 1000),
|
|
75
|
+
"p": props,
|
|
76
|
+
}
|
|
77
|
+
)
|
|
78
|
+
if max_events > 0 and len(events) > max_events:
|
|
79
|
+
events = events[-max_events:]
|
|
80
|
+
out["events"] = events
|
|
81
|
+
return out
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# Plan: Preflight completion slice (SDD + TDD)
|
|
2
|
+
|
|
3
|
+
**Repo:** `C:\Repos\shmindmaster\crewscore`
|
|
4
|
+
**Branch:** `feat/preflight-completion` (off main)
|
|
5
|
+
**Goal:** Finish audit leftovers with TDD-backed, shippable product work — CLI fix plan parity, locked UX contracts, metrics module, export completion, 0.2.6.
|
|
6
|
+
|
|
7
|
+
## Global constraints
|
|
8
|
+
|
|
9
|
+
- TDD: failing test first for every production behavior change.
|
|
10
|
+
- Honesty: structural pre-gate only; no red-team / certification claims.
|
|
11
|
+
- Keep existing HTML contract strings: builder-first hero, "Vendor checklist (self-attest)", not-a-red-team, templates inflate.
|
|
12
|
+
- Web scoring stays offline (`score-engine.js`); no LLM SDKs.
|
|
13
|
+
- Tests: `py -3.13 -m pytest` must stay green.
|
|
14
|
+
- Prefer small commits per task.
|
|
15
|
+
- Do not force-push main; work on feature branch; PR or merge when green.
|
|
16
|
+
|
|
17
|
+
## Tasks
|
|
18
|
+
|
|
19
|
+
### Task 1 — CLI fix plan / dry-run (agentic parity with web)
|
|
20
|
+
|
|
21
|
+
**Why:** Web has plan→apply; CLI `fix` still mutates or dumps with less control.
|
|
22
|
+
|
|
23
|
+
**TDD:**
|
|
24
|
+
1. RED: `test_fix_plan_json_lists_dimensions_without_writing` — `crewscore fix --prompt-file X --plan --json` exits 0, lists `fixes_planned` keys, does not change file.
|
|
25
|
+
2. RED: `test_fix_plan_human_lists_dimensions` — human output mentions planned dimensions / "plan" language.
|
|
26
|
+
3. GREEN: implement `--plan` / `--dry-run` on `fix` command.
|
|
27
|
+
4. RED/GREEN: without `--plan` or `--apply`/`--output`, behavior unchanged (print enhanced prompt).
|
|
28
|
+
|
|
29
|
+
**Files:** `crewscore/cli.py`, `tests/test_cli.py`, optionally thin helper in `scorers/fix_patterns.py`.
|
|
30
|
+
|
|
31
|
+
**Done when:** pytest for plan flags pass; applying still works.
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
### Task 2 — Web UX contract tests (lock preflight workflow)
|
|
36
|
+
|
|
37
|
+
**Why:** Audit F1–F5 implemented in HTML without strong regression tests.
|
|
38
|
+
|
|
39
|
+
**TDD:**
|
|
40
|
+
1. RED tests in `tests/test_web_ux.py` asserting `index.html` contains:
|
|
41
|
+
- stage markers: Prompt / Inspect / Act / Export (or `stg-prompt` … `stg-export`)
|
|
42
|
+
- "Plan fix" or plan-before-mutate language
|
|
43
|
+
- "Apply plan" and Cancel
|
|
44
|
+
- capability stamp "Structural pre-gate"
|
|
45
|
+
- vendor secondary (not equal primary tab list as only chrome)
|
|
46
|
+
- `crewscore_metrics_v1` or track() privacy events
|
|
47
|
+
2. GREEN: only if HTML missing pieces — fix HTML to match; do not remove honesty strings.
|
|
48
|
+
|
|
49
|
+
**Files:** `tests/test_web_ux.py`, possibly `index.html`.
|
|
50
|
+
|
|
51
|
+
**Done when:** new tests pass; existing vendor/web_engine tests pass.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
### Task 3 — Privacy-safe metrics module (testable pure core)
|
|
56
|
+
|
|
57
|
+
**Why:** F8 — metrics only inlined in index.html; untestable.
|
|
58
|
+
|
|
59
|
+
**TDD:**
|
|
60
|
+
1. RED: pure functions in Python or small JS-testable contract — prefer Python helper `crewscore/metrics.py`:
|
|
61
|
+
- `bucket_score(n) -> str` buckets
|
|
62
|
+
- `append_event(store, event, props, *, max_events=200) -> store` no prompt keys allowed
|
|
63
|
+
- `assert_privacy(props)` rejects keys like `prompt`, `text`, `body`
|
|
64
|
+
2. GREEN: implement module; wire optional CLI `crewscore` does not require metrics.
|
|
65
|
+
3. Document web localStorage schema matches event names in `_product-experience/06-outcome-measurement-plan.md`.
|
|
66
|
+
|
|
67
|
+
**Files:** `crewscore/metrics.py`, `tests/test_metrics.py`, doc touch if needed.
|
|
68
|
+
|
|
69
|
+
**Done when:** metrics unit tests pass; no prompt capture.
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
### Task 4 — Export completion checklist + reduced-motion
|
|
74
|
+
|
|
75
|
+
**Why:** F7 success state incomplete; a11y floor.
|
|
76
|
+
|
|
77
|
+
**TDD:**
|
|
78
|
+
1. RED: `test_web_ux` asserts export deck has completion checklist markers (e.g. `export-checklist` or three completion items: share/CI/prompt).
|
|
79
|
+
2. RED: CSS or HTML has `prefers-reduced-motion` rule.
|
|
80
|
+
3. GREEN: update `index.html` export stage + CSS.
|
|
81
|
+
|
|
82
|
+
**Files:** `index.html`, `tests/test_web_ux.py`.
|
|
83
|
+
|
|
84
|
+
**Done when:** tests pass.
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
### Task 5 — Version 0.2.6 + README CLI docs + ship
|
|
89
|
+
|
|
90
|
+
**Why:** Surface new `--plan` flag; ship coherent release.
|
|
91
|
+
|
|
92
|
+
**TDD:**
|
|
93
|
+
1. RED: version string expectations if any (prefer dynamic RULESET/version tests already).
|
|
94
|
+
2. GREEN: bump `__version__` / pyproject to 0.2.6; export web engine; README documents `crewscore fix --plan`.
|
|
95
|
+
3. Run full pytest; commit; open PR or merge to main; tag/publish only if existing publish pattern allowed (PyPI via 1Password).
|
|
96
|
+
|
|
97
|
+
**Files:** `crewscore/__init__.py`, `pyproject.toml`, `README.md`, `score-engine.js`, AGENTS.md if needed.
|
|
98
|
+
|
|
99
|
+
**Done when:** full suite green; package version 0.2.6; docs mention plan.
|
|
100
|
+
|
|
101
|
+
## Parallelism note
|
|
102
|
+
|
|
103
|
+
Tasks 1 and 3 are file-independent (can parallel via worktrees).
|
|
104
|
+
Tasks 2 and 4 share `index.html` / `test_web_ux.py` — sequential after 1 or with 1 done.
|
|
105
|
+
Task 5 last.
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
|
2
|
+
<rect width="32" height="32" rx="3" fill="#0a0c0b"/>
|
|
3
|
+
<rect x="2" y="2" width="28" height="28" rx="2" fill="none" stroke="#e8a317" stroke-width="1.5" opacity="0.7"/>
|
|
4
|
+
<text x="16" y="21" text-anchor="middle" font-family="IBM Plex Mono, ui-monospace, monospace" font-size="11" font-weight="700" fill="#e8a317">CS</text>
|
|
5
|
+
</svg>
|