healthcare-agents 1.2.0

package/.claude/commands/eval.md

@@ -0,0 +1,350 @@
+Evaluate and improve one healthcare agent's system prompt. Run up to 5 iterations of: prepare fixed questions -> answer -> judge -> improve -> re-score -> commit if better.
+
+**Target agent:** agents/$ARGUMENTS.md
+
+---
+
+## Support Docs
+
+Read these as operating instructions for the eval run:
+
+- `eval/rubric.md` — frozen scoring metric.
+- `eval/role-baselines/$ARGUMENTS.md` — frozen expected-capability baseline for this agent.
+- `eval/role-baselines/INDEX.md` — confirms baseline coverage for all installable agents.
+- `docs/eval/exam-architect-playbook.md` — scorer and question-writing guidance.
+- `docs/eval/model-tuning.md` — model-role and manifest guidance for current SOTA models.
+- `eval/meta/README.md` and linked meta-eval docs — optional calibration checks for release or close-call scoring.
+
+Do not use or recreate the retired Python/DSPy harness. The active system is this command, the frozen rubric, role baselines, optional local run logs, and git.
+
+---
+
+## Preferred Execution Mode
+
+When the runtime supports native subagents, model routing, or specialist workers, prefer a four-role loop:
+
+- **Parent orchestrator** — owns preflight checks, fixed-question persistence within an iteration, line-cap enforcement, local run logs, `eval/results.tsv` append, and commit/revert.
+- **Scorer/judge** — strongest available reasoning model; read-only; generates the exam, scores answers, checks calibration risk, and returns structured critique.
+- **Editor** — faster strong model; edits only `agents/$ARGUMENTS.md` using the scorer's brief.
+- **Adjudicator** — optional different model family for close deltas, high-risk roles, suspicious scoring, or release scoring.
+
+Do not recursively invoke a full agent CLI inside itself when native subagents are available. If no subagents are available, a single agent may run the same workflow end-to-end.
+
+Use `docs/eval/model-tuning.md` for model-role guidance. Pin exact model IDs in run logs when the runtime exposes them; do not rely only on UI or marketing names.
+
+---
+
+## Rules
+
+- Never modify `eval/rubric.md`.
+- Never modify files in `eval/role-baselines/` during an eval run.
+- Never modify `eval/results.tsv` except to append rows.
+- The editor may edit only `agents/$ARGUMENTS.md`.
+- The parent may write local run artifacts under `eval/run-logs/`.
+- One agent per session. After 5 iterations, print the summary and stop.
+- Each iteration is independent. Read the agent file fresh each time.
+- Scores across iterations are not comparable because questions differ. Only the same-question pre/post delta within an iteration is meaningful.
+- Any before/after or score-only baseline run must persist the full question set before answers are generated. Focus rows, weak-area labels, or paraphrases are not enough for retesting.
+- Do not broaden the role into generic healthcare administration. Preserve the agent's specialty, practitioner voice, and strongest differentiators.
+- Prefer sharpening missing mechanics, formulas, workflows, citations, and deliverables over adding broad best-practices boilerplate.
+- Do not log API keys, secrets, PHI, patient data, or private operational credentials.
+
+---
+
+## Preflight Checks
+
+Run these checks before entering the loop. If any fail, print the error message and stop immediately.
+
+1. Verify not on main:
+   Run: `git branch --show-current`
+   If the result is `main` or `master`, stop with: "Switch to a feature branch first: `git checkout -b eval/$ARGUMENTS`"
+
+2. Verify clean index:
+   Run: `git diff --cached --name-only`
+   If there is any output, stop with: "You have staged changes. Commit or unstage them first: `git reset HEAD`"
+
+3. Verify clean target file:
+   Run: `git diff --name-only -- agents/$ARGUMENTS.md`
+   If there is any output, stop with: "agents/$ARGUMENTS.md has uncommitted changes. Commit or stash first."
+
+4. Verify agent file exists:
+   Check that `agents/$ARGUMENTS.md` exists. If not, stop with: "No agent file found at agents/$ARGUMENTS.md"
+
+5. Verify required eval files exist:
+   Required:
+   - `eval/rubric.md`
+   - `eval/role-baselines/$ARGUMENTS.md`
+   - `docs/eval/exam-architect-playbook.md`
+   - `docs/eval/model-tuning.md`
+
+   If the role baseline is missing, stop and create the baseline first. Baselines are now expected for all 51 installable agents.
+
+6. Record session-start line count:
+   Run: `wc -l < agents/$ARGUMENTS.md`
+   Store as `BASELINE_LINES` for the whole session. Do not recompute it in later iterations.
+   Compute `LINE_CAP = max(BASELINE_LINES * 1.2, BASELINE_LINES + 50)`, rounded up.
+   Print: "Preflight passed. Baseline: {BASELINE_LINES} lines. Cap: {LINE_CAP} lines."
+
+7. Create a local run-log directory:
+   Use `eval/run-logs/<timestamp>-<agent-slug>/`.
+   Raw run logs are ignored by git by default. See `eval/run-logs/README.md`.
+
+---
+
+## The Loop
+
+Repeat the following up to 5 times. Number each iteration starting from 1.
+
+### Step 1: Read Inputs
+
+Read fresh:
+
+- `agents/$ARGUMENTS.md`
+- `eval/role-baselines/$ARGUMENTS.md`
+- `eval/rubric.md`
+- `docs/eval/exam-architect-playbook.md`
+- `docs/eval/model-tuning.md`
+
+For release scoring, close deltas, or high-risk agents, also read:
+
+- `eval/meta/judge-calibration-cases.md`
+- `eval/meta/scorer-consistency-check.md`
+- `eval/meta/prompt-overfitting-check.md`
+
+For usability release checks, also read `docs/eval/usability-release-check.md` and run its scenarios as smoke tests. These checks are release-only confidence checks; do not modify `eval/rubric.md`, `eval/role-baselines/`, or append `eval/results.tsv` for a usability smoke run.
+
+### Step 2: Prepare 25 Questions
+
+Choose the question source in this order:
+
+1. If a deliberate train bank exists at `eval/question-banks/train/$ARGUMENTS.md`, use it for weakness discovery.
+2. Otherwise generate 25 fresh questions from the agent prompt, role baseline, and `docs/eval/exam-architect-playbook.md`.
+3. Use validation-bank questions only after a retained edit, and only for confidence checking.
+4. Never use holdout-bank questions to guide edits.
+
+Draw questions from both:
+
+- The agent prompt: what it claims to know.
+- The role baseline: what the role should know, including omitted responsibilities.
+
+Default mix:
+
+- 5 factual mechanics
+- 8 applied reasoning
+- 5 edge cases
+- 4 cross-domain scenarios
+- 3 deliverable-production prompts
+
+Persist the exact questions for the iteration in the run log before answering them. Reuse these same questions for the post-edit re-score within the same iteration.
+
+The question artifact is mandatory. Write `questions.md` and, when practical, `questions.json` with all 25 full questions. Each question must include:
+
+- stable question ID, `Q001` through `Q025`
+- full prompt text exactly as shown to the answerer
+- question type: factual mechanics, applied reasoning, edge case, cross-domain scenario, or deliverable production
+- source basis: agent prompt, role baseline, playbook blueprint, train bank, validation bank, holdout bank, or calibration case
+- expected coverage: the capabilities, source families, or deliverable elements the answer should address
+- scoring emphasis: accuracy, completeness, specificity, or mixed
+
+Do not substitute short focus labels such as "network adequacy reporting" for the full prompt. If the full question text is not recoverable later, any retest must be labeled `fresh-comparable`, not `baseline-exact` or `same-question`.
+
+### Step 3: Answer All Questions
+
+Answer each question as if you are the target agent, with `agents/$ARGUMENTS.md` as the system prompt.
+
+Use only the agent prompt as the answerer's authority. If the prompt does not cover a topic, acknowledge the gap instead of fabricating detail.
+
+### Step 4: Judge Each Answer
+
+The scorer reads `eval/rubric.md` and scores each question-answer pair independently:
+
+- Accuracy: 0-4
+- Completeness: 0-4
+- Specificity: 0-4
+
+Apply the rubric strictly:
+
+- Accuracy 3+ requires specific codes, sections, standards, source families, or concrete authority references when the role calls for them.
+- Accuracy 4 requires the most specific authority level the prompt makes available.
+- Do not grant high accuracy for invented citations.
+- Do not reward verbosity when the answer is vague or unsafe.
+- Cite evidence from the answer and prompt; do not score from preference alone.
+
+Compute weighted score per question:
+
+```text
+(Accuracy * 0.40) + (Completeness * 0.35) + (Specificity * 0.25)
+```
+
+Average across 25 questions and multiply by 25 for a 0-100 score. This is `score_pre_edit`.
+
+The scorer output must include structured criterion detail:
+
+```json
+{
+  "question_source": "generated | train-bank | validation-bank | holdout-bank | calibration",
+  "score_pre_edit": 0,
+  "criteria_summary": {
+    "accuracy": {"score": 0, "evidence": "..."},
+    "completeness": {"score": 0, "evidence": "..."},
+    "specificity": {"score": 0, "evidence": "..."}
+  },
+  "weaknesses": [
+    {
+      "criterion": "accuracy | completeness | specificity",
+      "question_ids": ["Q003"],
+      "issue": "...",
+      "proposed_prompt_change": "..."
+    }
+  ],
+  "identity_to_preserve": [],
+  "anti_patterns_to_avoid": []
+}
+```
+
+### Step 5: Produce Improvement Brief
+
+From the scores, identify the 2-3 weakest areas. The scorer gives the editor a narrow improvement brief containing:
+
+- 2-4 representative low-scoring questions or paraphrased failure patterns.
+- 2-4 targeted prompt changes with expected gain.
+- The rubric criterion each change should improve.
+- `identity_to_preserve`: role traits, sections, and differentiators that must survive editing.
+- `anti_patterns_to_avoid`: generic broadening, duplicated boilerplate, bland executive-speak, or question-specific patching.
+
+Do not reveal hidden holdout material or answer-key wording to the editor.
+
+### Step 6: Edit The Agent
+
+Edit `agents/$ARGUMENTS.md` to strengthen the weak areas identified in Step 5.
+
+Editor constraints:
+
+- Implement the highest-leverage 1-3 changes first.
+- Prefer adding specific guidance to existing sections over rewriting or reorganizing entire sections.
+- Preserve all items under `identity_to_preserve`.
+- Avoid all listed anti-patterns.
+- Do not edit eval files, baselines, docs, or other agents.
+
+After editing, check line count:
+
+```bash
+wc -l < agents/$ARGUMENTS.md
+```
+
+If the count exceeds `LINE_CAP`:
+
+1. Immediately run `git restore agents/$ARGUMENTS.md`.
+2. Append a row to `eval/results.tsv` with status `capped`, `score_post_edit` as `N/A`, and `delta` as `N/A`.
+3. Write a capped summary in the run log.
+4. Run `git add eval/results.tsv && git commit -m "eval: $ARGUMENTS capped (exceeded line limit)"`.
+5. Skip to the next iteration.
+
+### Step 7: Re-Score
+
+Re-answer the same 25 questions from Step 2 using the edited agent prompt.
+
+Re-judge using the same rubric and scoring method. This is `score_post_edit`.
+
+If the delta is small, the role is high-risk, or the scorer behavior looks suspect, run the adjudicator or meta-eval checks described in `eval/meta/`.
+
+### Step 8: Log And Commit
+
+Before any commit/revert decision:
+
+1. Write run-log artifacts for the iteration, including:
+   - `manifest.json`
+   - `questions.md`
+   - `questions.json` when practical
+   - `scorer-output-pre.json`
+   - `editor-brief.md`
+   - `scorer-output-post.json`
+   - `summary.md`
+2. Verify `questions.md` contains all 25 complete question prompts. If not, stop and repair the artifact before editing or scoring.
+3. Ensure the manifest records exact model IDs when available, git state, file hashes, rubric hash, baseline hash, question source, question artifact paths, line cap, status, and calibration status.
+4. Append a tab-separated row to `eval/results.tsv`:
+
+```text
+{iteration}\t{$ARGUMENTS}\t{score_pre_edit}\t{score_post_edit}\t{delta}\t{status}\t{weak_areas}\t{description}
+```
+
+The existing TSV schema remains unchanged. Put the run-log directory path in `description` when one exists.
+
+Commit decision:
+
+- If `score_post_edit > score_pre_edit`:
+
+  ```bash
+  git add agents/$ARGUMENTS.md eval/results.tsv
+  git commit -m "eval: $ARGUMENTS {score_pre_edit}->{score_post_edit} (+{delta})"
+  ```
+
+- If `score_post_edit <= score_pre_edit`:
+
+  ```bash
+  git restore agents/$ARGUMENTS.md
+  git add eval/results.tsv
+  git commit -m "eval: $ARGUMENTS reverted ({delta})"
+  ```
+
+Do not commit raw `eval/run-logs/` artifacts unless a human explicitly asks to promote a run log for review.
+
+---
+
+## Score-Only And Before/After Runs
+
+For score-only baselines, multi-agent scorecards, or before/after experiments:
+
+1. Create a run directory under `eval/run-logs/<timestamp>-<experiment-name>/`.
+2. For each agent, write the full 25-question set before answering:
+   - `baseline/<batch-or-agent>/questions.md`
+   - `baseline/<batch-or-agent>/questions.json` when practical
+3. Write baseline answers and scores separately from the question artifact.
+4. Give editors only the improvement brief, not hidden holdout material or answer keys.
+5. During retest, first load the baseline `questions.md`.
+6. If all 25 full prompts are present, retest with exactly those prompts and mark `question_source=baseline-exact`.
+7. If only focus rows, weak areas, or paraphrases are present, do not reconstruct them as exact. Generate comparable questions if useful and mark `question_source=fresh-comparable`.
+8. The final scorecard must include `question_source` for every row.
+
+This rule is required because before/after deltas are only defensible when the post-edit score uses the same questions as the pre-edit score.
+
+---
+
+## Completion
+
+After 5 iterations, or if interrupted, print:
+
+```text
+=== Eval Complete: $ARGUMENTS ===
+Iterations: {N}
+Results: {improved} improved, {reverted} reverted, {capped} capped
+Starting score: {iteration 1 score_pre_edit}
+Retained score on disk: {score_post_edit from most recent improved iteration, or starting score if none improved}
+Last attempted score: {score_post_edit from the last non-capped iteration, or N/A}
+Results log: eval/results.tsv
+Run log: eval/run-logs/<timestamp>-<agent-slug>/
+Models: parent={id}, scorer={id}, editor={id}, adjudicator={id or none}
+Calibration: {not-run | passed | warning | failed}
+```
+
+Remember: retained score reflects the actual agent prompt on disk. Last attempted score may include a reverted edit.
+
+---
+
+## Scaling Pattern
+
+When improving many agents:
+
+- Run one agent per branch or worktree to avoid file and commit collisions.
+- Keep scorers and adjudicators read-only.
+- Keep editors single-file.
+- Let the parent orchestrator own git writes and `eval/results.tsv`.
+- Stop early when an agent clears the target score or recent deltas are too small to justify another pass.
+- Use role baselines for every agent; missing baselines are now a setup failure.
+
+For Codex and Claude Code under newer models:
+
+- Scorer/judge: strongest available reasoning model.
+- Editor: faster strong model.
+- Parent: reliable tool-using orchestrator.
+- Adjudicator: different strong model family when available.

package/AGENTS.md

@@ -0,0 +1,26 @@
+# Healthcare Agents Repository Instructions
+
+## Repository Map
+
+- Agent prompts live in `agents/*.md`.
+- The simple self-improvement kit lives in `.claude/commands/eval.md`, `eval/rubric.md`, `eval/results.tsv`, and `eval/role-baselines/`.
+- The old Python eval harness has been removed. The active eval path is the simple self-improvement kit above.
+
+## Git Workflow
+
+- Do not push directly to `main`.
+- For requested edits, create a short-lived branch, commit there, push the branch, open a PR, and merge the PR with `gh pr merge`.
+- Do not merge the feature branch into local `main` before opening or merging the PR. That creates duplicate local merge commits and makes `main` appear ahead/behind after GitHub merges the PR.
+- After a PR is merged, run `git fetch origin --prune` and align local `main` to `origin/main` before continuing work.
+- For docs-only or metadata-only changes, the streamlined path is: branch -> commit -> push branch -> `gh pr create` -> `gh pr merge --merge --delete-branch` -> sync local `main`.
+
+## Self-Improvement Loop
+
+- When asked to run the healthcare self-improvement loop for an agent, first read `.claude/commands/eval.md` and execute that procedure as a normal task, substituting `$ARGUMENTS` with the requested agent slug.
+- Treat `.claude/commands/eval.md` as the canonical workflow for both Claude Code and Codex.
+- If the runtime supports native subagents or model specialization, prefer a strongest scorer/judge plus a faster editor, with the parent agent owning git writes and `eval/results.tsv`.
+- Avoid recursive CLI invocation when native subagents are available.
+- Never modify `eval/rubric.md` or any file under `eval/role-baselines/`.
+- Never modify `eval/results.tsv` except to append rows.
+- Preserve the agent's distinctive role identity; do not flatten prompts into generic "best practices" boilerplate.
+- During a normal eval run, only edit the requested `agents/<slug>.md`, append `eval/results.tsv`, and write local ignored artifacts under `eval/run-logs/`.

package/CHANGELOG.md

@@ -0,0 +1,105 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## Unreleased
+
+### Added
+
+- Added `agents/registry.json` with discovery, routing, handoff, trust, and provenance metadata for all 51 agents.
+- Added CLI commands for `list`, `show`, `choose`, `prompt`, and `doctor`, plus slug-validated single-agent installs.
+- Added CI gates for installer syntax, agent linting/audit, package dry-run, CLI smoke tests, and installer dry-run.
+- Added public eval scorecard generation under `docs/eval/scorecard.md` and `docs/eval/scorecard.json`.
+- Added `docs/trust-and-safety.md` for scope, PHI, human escalation, source freshness, and eval limitations.
+
+### Changed
+
+- Improved installer dry-run output to show exact planned writes and avoid dry-run directory creation.
+- Added npm package scripts and Node engine metadata for distribution readiness.
+
+## [1.2.0] - 2026-05-05
+
+Usability release for task-based agent selection, output modes, and handoffs.
+
+### Added
+
+- Added `docs/usage/agent-selection-guide.md` with task-to-agent routing for common healthcare administration jobs.
+- Added `docs/usage/starter-prompts.md` with copy-ready prompts across all 10 domains.
+- Added `docs/usage/handoff-map.md` for cross-functional workflows and human escalation owners.
+- Added release-only usability smoke scenarios in `docs/eval/usability-release-check.md`.
+
+### Changed
+
+- Added role-tailored `Best Inputs`, `Output Modes`, and `Collaboration & Handoffs` sections to all 51 agents.
+- Updated README discovery flow, installer-managed Codex guidance, contribution template, lint checks, and audit scoring for the new usability contract.
+- Bumped package and installer metadata to v1.2.0 for GitHub-backed installs.
+
+See [docs/release-notes/2026-05-05-usability-release.md](docs/release-notes/2026-05-05-usability-release.md) for full details.
+
+## [1.1.2] - 2026-04-23
+
+Documentation correction for npm-backed install commands.
+
+### Changed
+
+- Updated README and INSTALL examples to use `npx --yes github:ajhcs/healthcare-agents` because the package name is not published to npm from this environment yet.
+- Bumped installer/package metadata to v1.1.2 so GitHub-backed npx installs report the latest compatibility release.
+
+## [1.1.1] - 2026-04-23
+
+Installer and documentation compatibility release.
+
+### Changed
+
+- Normalized all agent frontmatter `name` fields to lowercase hyphen slugs for Claude Code and OpenCode compatibility.
+- Added `display_name` frontmatter to preserve human-readable agent names.
+- Expanded the installer with aliases for Codex App, Claude Desktop, Claude Cowork, OpenCode, and portable SKILL.md targets.
+- Generated valid per-agent `SKILL.md` folders for Claude Skills, OpenCode skills, and the open `.agents/skills` convention.
+- Updated Codex install behavior to write a managed `~/.codex/AGENTS.md` discovery block.
+- Refreshed `README.md` and `INSTALL.md` for v1.1.x, current eval status, and cross-tool install paths.
+- Updated the self-improvement kit installer to copy all role baselines, not only the medical coding baseline.
+
+## [1.1.0] - 2026-04-23
+
+Agent-stack optimization release for the full 51-agent healthcare administration library.
+
+### Changed
+
+- Improved all 51 healthcare agent prompts through same-question before/after eval passes.
+- Raised the first 15 evaluated agents from an 85.0 average score to 93.9.
+- Raised the remaining 36 evaluated agents from an 85.11 average score to 95.50.
+- Added more role-specific mechanics, compliance boundaries, source hierarchies, workflow handoffs, and deliverable requirements across clinical, operations, payer, quality, health IT, population health, pharmacy, revenue, and strategy agents.
+- Reworked the eval workflow for modern SOTA model routing with parent orchestrator, scorer/judge, editor, and optional adjudicator roles.
+- Required reusable full-question artifacts for before/after evals so score deltas can be audited against exact Q001-Q025 prompts.
+
+### Added
+
+- Role baselines for all 51 installable healthcare agents.
+- Eval scorer guidance in `docs/eval/exam-architect-playbook.md`.
+- Current model-routing guidance in `docs/eval/model-tuning.md`.
+- Meta-eval checks for judge calibration, scorer consistency, and prompt-overfitting risk.
+- Local run-log documentation for retained questions, scorer outputs, editor briefs, and summaries.
+
+### Removed
+
+- Retired the unused Python/DSPy eval harness and related schema, rubric, test, and runner files.
+- Removed the standalone eval exam architect agent prompt in favor of the documented eval skill/playbook workflow.
+
+See [docs/release-notes/2026-04-23-agent-stack-optimization.md](docs/release-notes/2026-04-23-agent-stack-optimization.md) for full details.
+
+## [1.0.0] - 2026-04-09
+
+Initial release of the healthcare-agents repository.
+
+### Added
+
+- 51 healthcare administration agent system prompts across 10 categories: Revenue, Clinical, Quality, Payer, Operations, Health IT, Population Health, Pharmacy, Strategy, and Emergency Preparedness.
+- Karpathy-style automated eval loop (`/eval` command) with frozen rubric (Accuracy 0.40, Completeness 0.35, Specificity 0.25).
+- Split-role scoring architecture: strong judge model generates exams and critiques, fast editor model patches prompts, parent orchestrator owns git writes.
+- Identity-preservation constraints to prevent prompt drift during automated improvement.
+- Git-ratcheted commit strategy: improvements commit atomically, regressions revert automatically.
+- Append-only results log at `eval/results.tsv`.
+- 10 agents improved to 80+ scores through the eval loop, including Revenue Medical Coding Specialist (82.15), Revenue Finance Manager (81.55), and Revenue 340B Program Manager (81.20).
+- Cross-tool self-improvement kit and agent quality review infrastructure.
+
+See [docs/release-notes/2026-04-09-eval-loop-milestone.md](docs/release-notes/2026-04-09-eval-loop-milestone.md) for full details.