crewscore 0.2.0__tar.gz → 0.2.2__tar.gz

crewscore-0.2.2/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,22 @@
+---
+name: Bug report
+about: Something broken in CLI, Action, or web scorecard
+title: "[bug] "
+labels: bug
+---
+
+## What happened
+
+## Expected
+
+## Repro
+
+```bash
+crewscore test --prompt "..."
+```
+
+## Environment
+
+- OS:
+- `crewscore --version`:
+- Install method: pip / source

crewscore-0.2.2/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: New dimension, fix template, or CI surface
+title: "[feat] "
+labels: enhancement
+---
+
+## Problem
+
+## Proposed solution
+
+## Why not a runtime / adversarial tool instead?
+
+CrewScore stays offline structural by default. Say if this must stay structural.

crewscore-0.2.2/.github/ISSUE_TEMPLATE/scoring_feedback.md

@@ -0,0 +1,24 @@
+---
+name: Scoring false positive/negative
+about: Pattern matched wrong or missed a real guardrail
+title: "[scoring] "
+labels: scoring
+---
+
+## Dimension
+
+injection / hallucination / citation / cost / human_gate / safe_stop / audit / compliance
+
+## Prompt excerpt (redact secrets)
+
+```
+```
+
+## What CrewScore did
+
+- Score:
+- Matched / missing (from `--explain` if possible):
+
+## What you expected
+
+## Suggested pattern (optional)

{crewscore-0.2.0 → crewscore-0.2.2}/.github/workflows/crewscore-selftest.yml

@@ -1,46 +1,46 @@
-# Self-test for the composite CrewScore action in this repo.
-# Installs from github.action_path so it works before PyPI publish.
-name: CrewScore Self-Test
-
-on:
-  push:
-  pull_request:
-
-jobs:
-  selftest:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Score sample prompt (smoke, threshold 0)
-        id: score
-        uses: ./
-        with:
-          prompt-file: examples/sample-prompt.md
-          threshold: "0"
-
-      - name: Assert score and tier outputs parse
-        run: |
-          echo "score=${{ steps.score.outputs.score }}"
-          echo "tier=${{ steps.score.outputs.tier }}"
-          python -c "
-          import sys
-          score = int(sys.argv[1])
-          tier = sys.argv[2]
-          assert 0 <= score <= 100, score
-          assert tier, 'tier output empty'
-          print(f'OK score={score} tier={tier!r}')
-          " "${{ steps.score.outputs.score }}" "${{ steps.score.outputs.tier }}"
-
-      - name: Threshold gate fails when score is below threshold
-        run: |
-          set +e
-          crewscore test --prompt-file examples/sample-prompt.md --json --threshold 50
-          EXIT=$?
-          set -e
-          if [ "$EXIT" -ne 2 ]; then
-            echo "Expected exit code 2 for bare sample prompt under threshold 50, got $EXIT"
-            exit 1
-          fi
-          echo "Threshold failure behaved as expected (exit 2)"
+# Self-test for the composite CrewScore action in this repo.
+# Installs from github.action_path so it works before PyPI publish.
+name: CrewScore Self-Test
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  selftest:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Score sample prompt (smoke, threshold 0)
+        id: score
+        uses: ./
+        with:
+          prompt-file: examples/sample-prompt.md
+          threshold: "0"
+
+      - name: Assert score and tier outputs parse
+        run: |
+          echo "score=${{ steps.score.outputs.score }}"
+          echo "tier=${{ steps.score.outputs.tier }}"
+          python -c "
+          import sys
+          score = int(sys.argv[1])
+          tier = sys.argv[2]
+          assert 0 <= score <= 100, score
+          assert tier, 'tier output empty'
+          print(f'OK score={score} tier={tier!r}')
+          " "${{ steps.score.outputs.score }}" "${{ steps.score.outputs.tier }}"
+
+      - name: Threshold gate fails when score is below threshold
+        run: |
+          set +e
+          crewscore test --prompt-file examples/sample-prompt.md --json --threshold 50
+          EXIT=$?
+          set -e
+          if [ "$EXIT" -ne 2 ]; then
+            echo "Expected exit code 2 for bare sample prompt under threshold 50, got $EXIT"
+            exit 1
+          fi
+          echo "Threshold failure behaved as expected (exit 2)"

{crewscore-0.2.0 → crewscore-0.2.2}/.github/workflows/example-ci.yml

@@ -1,46 +1,65 @@
-# Documented copy-paste example for consumers.
-#
-# In your own repo, prefer the published tag:
-#   uses: shmindmaster/crewscore@v1
-#
-# When developing this monorepo (or a fork), local path works:
-#   uses: ./
-#
-# Point prompt-file at your agent system prompt. The sample path below is only
-# for illustration; examples/sample-prompt.md intentionally scores low.
-#
-# workflow_dispatch only so this example does not fail default CI on the bare
-# sample prompt. Use crewscore-selftest.yml for green self-tests.
-name: CrewScore CI Example
-
-on:
-  workflow_dispatch:
-
-jobs:
-  score:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      # Published action (recommended for consumer repos):
-      # - name: CrewScore
-      #   uses: shmindmaster/crewscore@v1
-      #   with:
-      #     prompt-file: agents/system-prompt.md
-      #     threshold: "50"
-      #     # explain: "true"
-
-      # Local composite action (this repo / monorepo self-check):
-      - name: CrewScore
-        id: crewscore
-        uses: ./
-        with:
-          prompt-file: examples/sample-prompt.md
-          threshold: "50"
-
-      - name: Use score outputs
-        if: always()
-        run: |
-          echo "score=${{ steps.crewscore.outputs.score }}"
-          echo "tier=${{ steps.crewscore.outputs.tier }}"
+# Documented copy-paste example for consumers.
+#
+# In your own repo, prefer the published tag:
+#   uses: shmindmaster/crewscore@v1
+#
+# When developing this monorepo (or a fork), local path works:
+#   uses: ./
+#
+# Point prompt-file at your agent system prompt. The sample path below is only
+# for illustration; examples/sample-prompt.md intentionally scores low.
+#
+# Prefer scan-path when the repo has multiple agent artifacts (AGENTS.md,
+# system-prompt.md, prompts/*, etc.). The Action reports the worst (min) overall.
+#
+# workflow_dispatch only so this example does not fail default CI on the bare
+# sample prompt. Use crewscore-selftest.yml for green self-tests.
+name: CrewScore CI Example
+
+on:
+  workflow_dispatch:
+
+jobs:
+  score:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      # Published action (recommended for consumer repos):
+      # - name: CrewScore
+      #   uses: shmindmaster/crewscore@v1
+      #   with:
+      #     prompt-file: agents/system-prompt.md
+      #     threshold: "50"
+      #     # explain: "true"
+
+      # Repo-wide structural scan (preferred when you have multiple agent files):
+      # - name: CrewScore scan
+      #   id: crewscore
+      #   uses: shmindmaster/crewscore@v1
+      #   with:
+      #     scan-path: "."
+      #     threshold: "50"
+
+      # Local composite action (this repo / monorepo self-check) — single file:
+      - name: CrewScore
+        id: crewscore
+        uses: ./
+        with:
+          prompt-file: examples/sample-prompt.md
+          threshold: "50"
+
+      # Local scan example (uncomment when `crewscore scan` is available):
+      # - name: CrewScore scan
+      #   id: crewscore-scan
+      #   uses: ./
+      #   with:
+      #     scan-path: "."
+      #     threshold: "50"
+
+      - name: Use score outputs
+        if: always()
+        run: |
+          echo "score=${{ steps.crewscore.outputs.score }}"
+          echo "tier=${{ steps.crewscore.outputs.tier }}"

crewscore-0.2.2/.github/workflows/pytest.yml

@@ -0,0 +1,29 @@
+name: pytest
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: pip install -e ".[dev]"
+      - name: Export web engine (freshness check via tests)
+        run: python scripts/export_web_engine.py
+      - name: Test
+        run: pytest -q
+      - name: Node parity (optional extra)
+        run: |
+          if command -v node >/dev/null; then
+            node -e "console.log('node', process.version)"
+          fi

{crewscore-0.2.0 → crewscore-0.2.2}/.gitignore

@@ -1,14 +1,14 @@
-__pycache__/
-*.pyc
-*.egg-info/
-dist/
-build/
-.venv/
-.env
-*.sqlite3
-agent-guard-report-*.html
-.DS_Store
-
-.superpowers/
-.worktrees/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.venv/
+.env
+*.sqlite3
+agent-guard-report-*.html
+.DS_Store
+
+.superpowers/
+.worktrees/
 

{crewscore-0.2.0 → crewscore-0.2.2}/AGENTS.md

@@ -1,72 +1,90 @@
-# CrewScore — agent instructions
-
-## What this is
-
-Offline CLI that **structurally** scores AI agent system prompts for production-readiness signals (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance), applies fix patterns, and runs a non-technical AI vendor checklist.
-
-Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
-
-It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes.
-
-## Stack
-
-- Python 3.11+
-- click + rich
-- hatchling packaging
-- pytest
-
-## Commands
-
-```bash
-# install (editable)
-pip install -e ".[dev]"
-
-# score a prompt
-crewscore test --prompt "You are a helpful assistant..."
-crewscore test --prompt-file ./system-prompt.md
-crewscore test --prompt-file ./system-prompt.md --json
-crewscore test --prompt-file ./system-prompt.md --json --threshold 50
-
-# apply guardrail patterns
-crewscore fix --prompt-file ./system-prompt.md
-crewscore fix --prompt-file ./system-prompt.md --apply
-crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
-
-# vendor checklist
-crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
-
-# tests
-pytest
-```
-
-Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
-
-## Layout
-
-```
-crewscore/
-  cli.py                 # click entry (test, fix)
-  scoring.py             # shared result model / tiers
-  vendor_scorecard.py    # assess-vendor command
-  scorers/
-    structural_analysis.py
-    fix_patterns.py
-tests/
-index.html               # static browser demo (client-side structural scan)
-```
-
-## Product constraints
-
-- Prefer honest capability claims over roadmap theater.
-- Structural scores are pattern matches on prompt text, not proof of runtime behavior.
-- Keep the package dependency-light (no LLM SDKs required for the core path).
-- Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
-- Breaking CLI flags are acceptable if all docs and tests update in the same change.
-- Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
-
-## Do not
-
-- Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
-- Link to non-existent report hosts or wrong GitHub/PyPI names.
-- Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
-- Overclaim “production safety certification” or “7 regulated systems” beyond structural scanning.
+# CrewScore — agent instructions
+
+## What this is
+
+Offline CLI that **structurally** scores AI agent system prompts for production-readiness signals (injection defense, hallucination policy, citations, cost limits, human gates, safe-stop, audit, compliance), applies fix patterns, and optionally runs a non-technical AI vendor checklist (self-attest only).
+
+Public brand: **CrewScore** · Domain: **https://crewscore.ai** · PyPI: **`crewscore`** · Repo: **shmindmaster/crewscore**
+
+It does **not** (yet) run live adversarial LLM attacks or parse LangGraph/CrewAI runtimes. For live testing, hand off to Promptfoo / garak — see `docs/next-steps-eval.md`.
+
+## Stack
+
+- Python 3.11+
+- click + rich
+- hatchling packaging
+- pytest
+
+## Commands
+
+```bash
+# install (editable)
+pip install -e ".[dev]"
+
+# repo scan (preferred for CI / monorepos)
+crewscore scan .
+crewscore scan . --json --threshold 50
+
+# score a single prompt
+crewscore test --prompt "You are a helpful assistant..."
+crewscore test --prompt-file ./system-prompt.md --explain
+crewscore test --prompt-file ./system-prompt.md --json --explain
+crewscore test --prompt-file ./system-prompt.md --json --threshold 50
+crewscore test --prompt-file ./system-prompt.md --report out.html --badge badge.svg
+
+# apply guardrail patterns
+crewscore fix --prompt-file ./system-prompt.md
+crewscore fix --prompt-file ./system-prompt.md --apply
+crewscore fix --prompt-file ./system-prompt.md --output ./guarded.md --json
+
+# vendor checklist (secondary / self-attest)
+crewscore assess-vendor --name "Acme AI" --answers "y,y,n,dk,y,y,n,y,n,y" --json
+
+# after pattern changes: keep web in lockstep
+python scripts/export_web_engine.py
+
+# tests
+pytest
+```
+
+Legacy CLI entry point `agent-guard` still maps to the same `crewscore.cli:main` after install.
+
+## Layout
+
+```
+crewscore/
+  cli.py                 # click entry (test, fix, scan)
+  scoring.py             # shared result model / tiers
+  vendor_scorecard.py    # assess-vendor command
+  web_export.py          # builds score-engine.js payload
+  report.py              # HTML report + SVG badge
+  scorers/
+    structural_analysis.py
+    fix_patterns.py
+scripts/export_web_engine.py
+score-engine.js          # generated — commit after pattern changes
+index.html               # builder-first site (uses score-engine.js)
+action.yml               # composite GH Action (prompt-file or scan-path)
+docs/launch/             # launch copy kit
+docs/next-steps-eval.md  # Promptfoo / garak handoff
+tests/
+```
+
+## Product constraints
+
+- Prefer honest capability claims over roadmap theater.
+- Structural scores are pattern matches on prompt text, not proof of runtime behavior.
+- **Honesty scoring:** do not claim certification, audit, or red-team results; templates can inflate scores; under-score rather than over-claim.
+- Prefer `crewscore scan .` / Action `scan-path` for repo-native hygiene; demote vendor checklist in UX and docs.
+- Keep the package dependency-light (no LLM SDKs required for the core path).
+- Fame follows usefulness: explainable findings, fix, CI gate before launch theater.
+- Breaking CLI flags are acceptable if all docs and tests update in the same change.
+- Never document `pip install agent-guard` as *this* product (that PyPI name is taken by another package).
+
+## Do not
+
+- Reintroduce fake `--langgraph` / `--crewai` loaders or adversarial mode stubs without real implementations.
+- Link to non-existent report hosts or wrong GitHub/PyPI names.
+- Add empty `examples/` / `evaluator/` / `patterns/` directories without content.
+- Overclaim “production safety certification” or “7 regulated systems” beyond structural scanning.
+- Elevate the vendor self-attest checklist to equal primary product surface.

crewscore-0.2.2/CONTRIBUTING.md

@@ -0,0 +1,49 @@
+# Contributing to CrewScore
+
+Thanks for helping improve an offline structural scorecard for AI agent prompts.
+
+## Dev setup
+
+```bash
+git clone https://github.com/shmindmaster/crewscore.git
+cd crewscore
+pip install -e ".[dev]"
+pytest -q
+```
+
+## How scoring works
+
+- Patterns live in `crewscore/scorers/structural_analysis.py` (`SCORER_MAP`).
+- Scores are match counts mapped to 0–100 (see `_score_from_match_count`).
+- Explain labels: `DIMENSION_SIGNAL_LABELS` (pattern → human label pairs).
+- CLI: `crewscore test`, `fix`, `assess-vendor`.
+- **Web uses the same engine:** after changing patterns, regenerate:
+
+```bash
+python scripts/export_web_engine.py
+pytest tests/test_web_engine.py -q
+```
+
+Commit the updated `score-engine.js` with your pattern change.
+
+## Adding a pattern
+
+1. Add a regex to the right list in `structural_analysis.py`.
+2. Optionally add a `(pattern, human_label)` to `DIMENSION_SIGNAL_LABELS`.
+3. Add/adjust a unit test in `tests/test_structural_analysis.py` or `tests/test_explain.py`.
+4. Run `python scripts/export_web_engine.py` and `pytest -q`.
+
+## Adding a fix template
+
+Edit `crewscore/scorers/fix_patterns.py` `FIX_TEMPLATES` for the dimension key.
+
+## PR rules
+
+- Prefer TDD for behavior changes.
+- Keep claims honest: structural ≠ runtime red-team.
+- Never document `pip install agent-guard` as this product.
+- Do not add LLM API dependencies for the core path.
+
+## Code of conduct
+
+Be respectful. File false-positive / false-negative scoring issues with the prompt text and expected dimension.