neurodock-evals 0.0.1__tar.gz

neurodock_evals-0.0.1/.gitignore

@@ -0,0 +1,189 @@
+# ─────────────────────────────────────────────────────────────
+# Node / pnpm (TypeScript packages: core, cli, extension-browser, docs)
+# ─────────────────────────────────────────────────────────────
+node_modules/
+.pnpm-store/
+.pnpm-debug.log*
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Build outputs
+dist/
+build/
+out/
+*.tsbuildinfo
+.next/
+.nuxt/
+
+# Turborepo cache
+.turbo/
+
+# ─────────────────────────────────────────────────────────────
+# Python / uv (MCP servers, clinical, evals)
+# ─────────────────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# uv / virtualenvs
+.venv/
+venv/
+env/
+.python-version
+
+# Packaging
+*.egg-info/
+*.egg
+dist/
+build/
+wheels/
+*.whl
+
+# Test / coverage
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.coverage.*
+coverage.xml
+htmlcov/
+.tox/
+.hypothesis/
+
+# Jupyter (in case any analysis notebooks land in scripts/)
+.ipynb_checkpoints/
+
+# ─────────────────────────────────────────────────────────────
+# WXT browser extension (packages/extension-browser/)
+# ─────────────────────────────────────────────────────────────
+.output/
+.wxt/
+web-ext-artifacts/
+*.zip
+*.crx
+*.xpi
+
+# Extension store submission artefacts — keep these local
+extension-submission/
+
+# ─────────────────────────────────────────────────────────────
+# Astro / Starlight docs (docs/)
+# ─────────────────────────────────────────────────────────────
+.astro/
+
+# ─────────────────────────────────────────────────────────────
+# Local NeuroDock runtime data — NEVER commit
+# ─────────────────────────────────────────────────────────────
+# Profile files contain user neurotype declarations
+profile.yaml
+profile.yml
+.neurodock/
+**/.neurodock/
+
+# SQLite stores (cognitive graph, embeddings, traces)
+*.sqlite
+*.sqlite-journal
+*.sqlite-wal
+*.sqlite-shm
+*.db
+*.db-journal
+
+# JSONL event logs
+*.jsonl
+!packages/evals/**/*.jsonl       # eval corpora are checked in
+!**/fixtures/*.jsonl              # test fixtures are checked in
+
+# Local embedding caches
+.embeddings/
+embeddings-cache/
+
+# OpenTelemetry local trace dumps
+.otel/
+traces/
+
+# ─────────────────────────────────────────────────────────────
+# Secrets / environment
+# ─────────────────────────────────────────────────────────────
+.env
+.env.*
+!.env.example
+!.env.template
+*.pem
+*.key
+*.p12
+secrets/
+
+# Anthropic / LLM provider keys (defence in depth)
+.anthropic/
+.openai/
+
+# Browser extension signing keys — local only
+*.signing-key
+extension-private-key.pem
+
+# ─────────────────────────────────────────────────────────────
+# OS / editor (you're on Windows + VS Code)
+# ─────────────────────────────────────────────────────────────
+# Windows
+Thumbs.db
+ehthumbs.db
+ehthumbs_vista.db
+Desktop.ini
+$RECYCLE.BIN/
+*.lnk
+
+# macOS (in case any contributor PRs from a Mac)
+.DS_Store
+.AppleDouble
+.LSOverride
+._*
+
+# Linux
+*~
+.directory
+.Trash-*
+
+# VS Code — share recommendations, ignore personal state
+.vscode/*
+!.vscode/extensions.json
+!.vscode/settings.shared.json
+!.vscode/launch.shared.json
+
+# JetBrains
+.idea/
+*.iml
+*.iws
+
+# Vim / Emacs
+*.swp
+*.swo
+*~
+.\#*
+\#*\#
+
+# ─────────────────────────────────────────────────────────────
+# Claude Code / agent runtime — keep agents tracked, ignore state
+# ─────────────────────────────────────────────────────────────
+.claude/cache/
+.claude/history/
+.claude/sessions/
+.claude/local/
+
+# Do keep: .claude/agents/, .claude/skills/, .claude/settings.json
+
+# ─────────────────────────────────────────────────────────────
+# Misc
+# ─────────────────────────────────────────────────────────────
+*.log
+logs/
+tmp/
+.tmp/
+.cache/
+.parcel-cache/
+
+# Lockfiles policy: KEEP these committed
+# !pnpm-lock.yaml
+# !uv.lock
+# (intentionally not listed above so they're tracked)

neurodock_evals-0.0.1/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to `neurodock-evals`. Format loosely follows
+[Keep a Changelog](https://keepachangelog.com).
+
+## [0.0.1] — 2026-05-17
+
+### Added
+
+- Initial harness scaffold: `harness.py`, `corpus.py`, `runner.py`, `scoring.py`,
+  `anonymise.py`, `dedupe.py`, `types.py`.
+- JSON Schemas: `schemas/example.schema.json`, `schemas/annotation.schema.json`.
+- Ten hand-authored, **synthesised** seed examples across four translation
+  slices: `translation/incoming/` (3), `translation/tone/` (2),
+  `translation/outgoing/` (2), `translation/meetings/` (3, including a
+  multi-rater agreement metric demonstration).
+- `corpora/guardrail/` placeholder for the  .
+- `CONTRIBUTING.md` describing the consent + anonymisation pipeline.
+- Pytest suite covering corpus load, anonymisation, dedupe, runner, scoring,
+  and harness CLI.
+
+### Not yet
+
+- Real contributed corpora — gated on the  pipeline.
+- HuggingFace publication — deferred to v0.1.0.
+- Tuned CI thresholds — the harness ships with permissive defaults; tuning
+  follows the first real contributions.
+- rater agreement metric adjudication workflow beyond a single demo example.

neurodock_evals-0.0.1/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: neurodock-evals
+Version: 0.0.1
+Summary: NeuroDock eval corpora and harness — versioned datasets for translation, skills, and guardrails.
+Author: NeuroDock contributors
+License: AGPL-3.0-or-later
+Requires-Python: >=3.11
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: neurodock-mcp-translation
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: test
+Requires-Dist: pytest-asyncio>=1.3.0; extra == 'test'
+Requires-Dist: pytest>=8.3.0; extra == 'test'
+Description-Content-Type: text/markdown
+
+# neurodock-evals
+
+The versioned eval corpora and the air-gapped harness that runs ND prompts
+against them.
+
+The corpus is the strategic asset that makes the translation layer honest. We
+prove that ND-aware prompts help neurodivergent users in real situations, and
+we catch regressions when prompts change. The harness gates prompt PRs in CI.
+
+This package is **v0.0.1** — the scaffold, the harness, and 6-10 hand-authored
+seed examples. The seeds are **synthesised by  to
+demonstrate the format** — they are NOT real corporate messages. Real
+contributed corpora arrive over Phase 2 (target ~300 examples by month 6, per
+).
+
+## What's here
+
+```
+packages/evals/
+├── src/neurodock_evals/        # Harness, anonymiser, deduper, scorer
+├── corpora/                    # Versioned YAML eval examples by slice
+├── schemas/                    # JSON Schemas for examples + annotations
+└── tests/                      # Tests for the harness itself
+```
+
+## Quick start
+
+Run the harness against the seed corpora:
+
+```bash
+uv run python -m neurodock_evals.harness --corpus translation/incoming \
+    --tool translate_incoming
+```
+
+Run all four translation slices:
+
+```bash
+uv run python -m neurodock_evals.harness --ci
+```
+
+Anonymise a contribution before opening a PR:
+
+```bash
+uv run python -m neurodock_evals.anonymise path/to/example.yaml
+```
+
+## Air-gapped by design
+
+The harness never calls an LLM. It exercises each tool's **deterministic
+baseline** (the heuristic layer the translation server returns even before any
+LLM refinement) and scores the baseline against the human-rated `expected`
+block. Any LLM-side eval is a separate concern that the maintainer reviews
+under a different policy.
+
+## Privacy
+
+- The harness never logs example contents to stdout or to anywhere outside
+  `.eval-reports/`.
+- Reports contain example IDs and scores only — never verbatim text.
+- The contribution pipeline (`anonymise.py`) is a safety net, NOT a substitute
+  for contributor judgement. See `CONTRIBUTING.md`.
+- All corpora are licensed **AGPL-3.0-or-later**.
+
+## Glossary
+
+| Term | Meaning |
+| ---- | ------- |
+| corpus slice | a directory under `corpora/<server>/<slice>/`; the unit of versioning |
+| example | one YAML file under a slice — one input, one `expected` block, multiple ratings |
+| rating | one ND-rater's judgement of how close the `expected` block matches their read |
+| deterministic baseline | the heuristic output a translation tool returns without invoking an LLM |
+| eval-corpus binding | every `mcp-translation` tool cites the slice that validates it (ADR 0005 §4) |
+
+## Status
+
+- v0.0.1 (current): scaffold + harness + 10 synthesised seed examples
+- v0.0.2 (planned): first contributed corpus (after )
+- v0.1.0 (planned): HuggingFace publication pipeline under the `neurodock` org
+
+See `CHANGELOG.md` for detail.

neurodock_evals-0.0.1/README.md

@@ -0,0 +1,80 @@
+# neurodock-evals
+
+The versioned eval corpora and the air-gapped harness that runs ND prompts
+against them.
+
+The corpus is the strategic asset that makes the translation layer honest. We
+prove that ND-aware prompts help neurodivergent users in real situations, and
+we catch regressions when prompts change. The harness gates prompt PRs in CI.
+
+This package is **v0.0.1** — the scaffold, the harness, and 6-10 hand-authored
+seed examples. The seeds are **synthesised by  to
+demonstrate the format** — they are NOT real corporate messages. Real
+contributed corpora arrive over Phase 2 (target ~300 examples by month 6, per
+).
+
+## What's here
+
+```
+packages/evals/
+├── src/neurodock_evals/        # Harness, anonymiser, deduper, scorer
+├── corpora/                    # Versioned YAML eval examples by slice
+├── schemas/                    # JSON Schemas for examples + annotations
+└── tests/                      # Tests for the harness itself
+```
+
+## Quick start
+
+Run the harness against the seed corpora:
+
+```bash
+uv run python -m neurodock_evals.harness --corpus translation/incoming \
+    --tool translate_incoming
+```
+
+Run all four translation slices:
+
+```bash
+uv run python -m neurodock_evals.harness --ci
+```
+
+Anonymise a contribution before opening a PR:
+
+```bash
+uv run python -m neurodock_evals.anonymise path/to/example.yaml
+```
+
+## Air-gapped by design
+
+The harness never calls an LLM. It exercises each tool's **deterministic
+baseline** (the heuristic layer the translation server returns even before any
+LLM refinement) and scores the baseline against the human-rated `expected`
+block. Any LLM-side eval is a separate concern that the maintainer reviews
+under a different policy.
+
+## Privacy
+
+- The harness never logs example contents to stdout or to anywhere outside
+  `.eval-reports/`.
+- Reports contain example IDs and scores only — never verbatim text.
+- The contribution pipeline (`anonymise.py`) is a safety net, NOT a substitute
+  for contributor judgement. See `CONTRIBUTING.md`.
+- All corpora are licensed **AGPL-3.0-or-later**.
+
+## Glossary
+
+| Term | Meaning |
+| ---- | ------- |
+| corpus slice | a directory under `corpora/<server>/<slice>/`; the unit of versioning |
+| example | one YAML file under a slice — one input, one `expected` block, multiple ratings |
+| rating | one ND-rater's judgement of how close the `expected` block matches their read |
+| deterministic baseline | the heuristic output a translation tool returns without invoking an LLM |
+| eval-corpus binding | every `mcp-translation` tool cites the slice that validates it (ADR 0005 §4) |
+
+## Status
+
+- v0.0.1 (current): scaffold + harness + 10 synthesised seed examples
+- v0.0.2 (planned): first contributed corpus (after )
+- v0.1.0 (planned): HuggingFace publication pipeline under the `neurodock` org
+
+See `CHANGELOG.md` for detail.

neurodock_evals-0.0.1/corpora/README.md

@@ -0,0 +1,30 @@
+# Eval corpora
+
+Versioned datasets that anchor the prompt regression suite. Every directory
+under here is **one slice**: one server, one tool-family, one set of
+hand-rated examples.
+
+## Layout
+
+```
+corpora/
+├── translation/        # Slices for the mcp-translation server
+│   ├── incoming/       # translate_incoming evaluator
+│   ├── tone/           # check_tone evaluator
+│   ├── outgoing/       # rewrite_outgoing evaluator
+│   └── meetings/       # brief_meeting evaluator
+└── guardrail/          # Reserved for the  
+```
+
+## File format
+
+One YAML file per example, named `NNN-slug.example.yaml`. Validated against
+`packages/evals/schemas/example.schema.json`. See `CONTRIBUTING.md` for the
+contribution flow.
+
+## Provenance
+
+Every example in v0.0.1 is **synthesised** by  to
+demonstrate the format and exercise the harness end-to-end. These are NOT
+real corporate messages. Phase 2 brings the first contributed examples
+through the consent pipeline at `evals.neurodock.org/contribute`.

neurodock_evals-0.0.1/corpora/translation/README.md

@@ -0,0 +1,14 @@
+# translation/ slices
+
+Eval corpora for the four `mcp-translation` tools, per ADR 0005 §4
+(eval-corpus binding).
+
+| Slice | Tool | What it tests |
+| ----- | ---- | ------------- |
+| `translation/incoming` | `translate_incoming` | Subtext, ambiguity, recommended next action on incoming messages |
+| `translation/tone` | `check_tone` | Directness/warmth/urgency scoring and flagged phrases |
+| `translation/outgoing` | `rewrite_outgoing` | Register-targeted rewrites that preserve specified terms |
+| `translation/meetings` | `brief_meeting` | Verbatim-anchored partition of a transcript into asks/decisions/ambiguities |
+
+Each slice is loaded by `neurodock_evals.corpus.load_slice("translation/<name>")`
+and run via `neurodock_evals.runner.run_example`.

neurodock_evals-0.0.1/corpora/translation/incoming/001-circle-back.example.yaml

@@ -0,0 +1,27 @@
+id: "translation.incoming.001"
+slice: "translation/incoming"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  text: "I'll circle back on the migration plan next week — quick one, just need a sanity check."
+  channel: "slack"
+expected:
+  explicit_ask: null
+  ambiguity:
+    detected: true
+  recommended_next_action:
+    action: "set_reminder"
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["adhd"]
+    agreement_with_expected: 0.9
+    notes: "Classic soft-deferral. 'I'll circle back' + 'next week' both signal the response will not self-fulfil."
+notes: |
+  Demonstrates the hedged-commitment / vague-timeline combination. The baseline
+  should flag both phrases and recommend `set_reminder` because there is no
+  explicit ask to act on.

neurodock_evals-0.0.1/corpora/translation/incoming/002-loop-you-in.example.yaml

@@ -0,0 +1,32 @@
+id: "translation.incoming.002"
+slice: "translation/incoming"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  text: "Have you had a chance to look at the rollout doc? No rush but it's been a while."
+  channel: "email"
+  thread_context:
+    - "Sharing the rollout doc — would love your thoughts before Friday."
+    - "(no reply)"
+expected:
+  ambiguity:
+    detected: true
+  recommended_next_action:
+    action: "reply"
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["audhd"]
+    agreement_with_expected: 0.85
+    notes: |
+      Polite-overdue framing. "Have you had a chance" + "no rush" reads as
+      implied urgency once you account for the thread context. The baseline
+      should flag this and recommend acknowledging receipt with a concrete
+      next step.
+notes: |
+  Implied-urgency pattern: polite words masking a follow-up. The baseline
+  flags both `have you had a chance` and `no rush` as implied_urgency.

neurodock_evals-0.0.1/corpora/translation/incoming/003-revisit-next-week.example.yaml

@@ -0,0 +1,37 @@
+id: "translation.incoming.003"
+slice: "translation/incoming"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  text: "Hey — can we revisit the rollout timeline? I'm not sure everyone is aligned."
+  channel: "slack"
+  thread_context:
+    - "We agreed last sprint to ship the rollout by end of May."
+    - "Two engineers flagged risk on the migration script."
+expected:
+  ambiguity:
+    detected: true
+  recommended_next_action:
+    action: "clarify"
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["asd"]
+    agreement_with_expected: 0.95
+    notes: |
+      Soft-request + implied-blame combo. The 'can we revisit' phrase reads as
+      'I want to change' rather than a neutral re-open; 'not sure everyone is
+      aligned' softly attributes misalignment without naming who.
+  - rater_id: "rater-curator-B"
+    rater_neurotypes: ["adhd", "asd"]
+    agreement_with_expected: 0.8
+    notes: |
+      Agree on subtext, but I'd want the recommendation to push harder on
+      naming the specific stakeholders before agreeing to reopen scope.
+notes: |
+  The canonical 'can we revisit' example from the v0.1.0 schema. Carries two
+  raters so the harness can demonstrate inter-rater agreement reporting.

neurodock_evals-0.0.1/corpora/translation/incoming/README.md

@@ -0,0 +1,8 @@
+# translation/incoming/
+
+Examples for the `translate_incoming` tool. Each example carries an `input`
+matching `TranslateIncomingInput` and an `expected` block naming the fields
+the rater considers diagnostic.
+
+The runner scores against the deterministic baseline (no LLM). v0.0.1 ships
+three synthesised seed examples; real contributed examples arrive in Phase 2.

neurodock_evals-0.0.1/corpora/translation/meetings/001-short-product-meeting.example.yaml

@@ -0,0 +1,38 @@
+id: "translation.meetings.001"
+slice: "translation/meetings"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  transcript: |
+    Alex: Sam, can you own the migration script for the rollout?
+    Sam: Yes — I'll have it by Wednesday.
+    Alex: Great. We agreed to ship by end of May.
+  me: "Sam"
+  speakers: ["Alex", "Sam"]
+  project: "rollout"
+expected:
+  my_asks:
+    __len__: 1
+  others_asks:
+    __len__: 0
+  decisions:
+    __len__: 2
+  ambiguous_items:
+    __len__: 0
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["asd"]
+    agreement_with_expected: 0.95
+    notes: |
+      Clean partition: Alex's 'Can you own ...' becomes Sam's my_ask; Sam's
+      'Yes — I'll have it' is a decision; Alex's 'We agreed' is also a
+      decision. No ambiguity in this transcript.
+notes: |
+  Demonstrates the four-section partition with verbatim-anchored spans. Both
+  the 'me' ask and the two decision lines are line-anchored in the transcript;
+  the runner enforces the verbatim invariant on every load.

neurodock_evals-0.0.1/corpora/translation/meetings/002-hedged-decision.example.yaml

@@ -0,0 +1,37 @@
+id: "translation.meetings.002"
+slice: "translation/meetings"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  transcript: |
+    Alex: Sam, can you draft the migration plan?
+    Sam: I'll circle back next week with thoughts.
+    Alex: Someone should also think about the rollback path.
+  me: "Sam"
+  speakers: ["Alex", "Sam"]
+  project: "rollout"
+expected:
+  my_asks:
+    __len__: 1
+  decisions:
+    __len__: 0
+  ambiguous_items:
+    __len__: 2
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["audhd"]
+    agreement_with_expected: 0.85
+    notes: |
+      Sam's 'I'll circle back next week' is both a (weak) decision and a
+      hedged commitment — the baseline surfaces it under both categories.
+      Alex's 'Someone should also think' has unassigned owner + hedged commitment.
+notes: |
+  Tests the verbatim-anchor invariant on ambiguous_items: every quoted_span
+  must slice cleanly from the transcript. This example also documents the
+  case where one line becomes both a decision and an ambiguous_item, which
+  the LLM-refinement layer can later collapse.

neurodock_evals-0.0.1/corpora/translation/meetings/README.md

@@ -0,0 +1,9 @@
+# translation/meetings/
+
+Examples for the `brief_meeting` tool. Each example carries an `input`
+matching `BriefMeetingInput` (a transcript, the rater's `me` handle, and
+optional speakers/project) and an `expected` block over my_asks /
+others_asks / decisions / ambiguous_items counts and salient fields.
+
+Per ADR 0005 §5, every `ambiguous_items[*].quoted_span` is verbatim-anchored.
+The runner exercises that invariant on every meeting example.

neurodock_evals-0.0.1/corpora/translation/outgoing/001-rewrite-for-warmth.example.yaml

@@ -0,0 +1,29 @@
+id: "translation.outgoing.001"
+slice: "translation/outgoing"
+created_at: "2026-05-17"
+consent:
+  contributor: "synth-curator-001"
+  consent_token: "sha256:synthesised-seed-2026-05-17"
+  anonymisation_pass: 1
+status: "synthesised"
+license: "AGPL-3.0-or-later"
+input:
+  text: "Strong nack. This will block the release."
+  target_register: "warm"
+  channel: "github"
+expected:
+  preserved_terms: []
+  unpreserved_terms: []
+  diff_summary:
+    tone_shift: "Lifted warmth by opening with a relational acknowledgement; preserved the underlying ask."
+ratings:
+  - rater_id: "rater-curator-A"
+    rater_neurotypes: ["asd"]
+    agreement_with_expected: 0.85
+    notes: |
+      The baseline drops 'strong nack' and prepends a 'Hey — ' opener. The
+      underlying concern survives; the relational opener softens the blow.
+notes: |
+  Warmth-rewrite baseline test. The structural_changes array should contain
+  "Removed opening rejection token" and "Added relational opener". The
+  `tone_shift` summary is deterministic and matched verbatim.