confusion-mapper 1.0.0__tar.gz

confusion_mapper-1.0.0/CHANGELOG.md

@@ -0,0 +1,39 @@
+# Changelog
+
+All notable changes to ConfusionMapper are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2026-06-23
+
+First public release.
+
+### Added
+
+- `compute_cohens_kappa(human, ai)` for nominal Cohen's kappa with Landis-Koch interpretation.
+- `compute_cohens_kappa(..., weights="linear")` and `weights="quadratic"` for weighted kappa over ordinal taxonomies (Cicchetti-Allison and Fleiss-Cohen schemes).
+- `build_confusion_matrix(human, ai)` returning the full N x N category cross-tabulation as a nested dict.
+- `get_per_type_stats(human, ai)` returning per-category agreement counts and percentages.
+- `bootstrap_kappa_ci(human, ai, method="bca", seed=...)` for percentile and bias-corrected and accelerated (BCa) bootstrap confidence intervals on kappa, with a seed parameter for bit-identical reproducibility.
+- `load_taxonomy_from_json(path)` for swapping the default Confusion Fingerprint Index categories for any two-or-more category nominal scheme.
+- Built-in NCERT-aligned MCQ bank covering five Class 7 science questions with fifteen pre-labelled distractors.
+- OpenAI-backed AI rater (`classify_with_ai`) with `model="gpt-4o"`, `temperature=0.0`, and a deterministic structured prompt.
+- Optional AI question generator (`generate_ai_questions`).
+- Tkinter results dashboard (`show_results_window`) with a kappa gauge, 4 x 4 confusion matrix, and per-type bar chart. Falls back to console output when no display is available.
+- JSON session export for full reproducibility records.
+- Headless tutorial at `examples/demo.py` reproducing a complete kappa analysis on the bundled 30-item paired-label set.
+- Case study at `case_study/` with a fully reproducible IRR analysis (kappa, weighted kappa, bootstrap CI, confusion matrix, per-type stats) on the NCERT-aligned distractor set.
+- 55-test pytest suite covering the kappa formula, weighted kappa, bootstrap CI determinism, custom taxonomy loading, and confusion matrix invariants.
+- GitHub Actions CI building the pytest matrix on Python 3.9 through 3.12 and producing a JOSS-formatted paper PDF as a downloadable artifact on every push.
+
+### Documentation
+
+- README with install, usage, advanced features, and citation sections.
+- JOSS-ready `paper.md` and `paper.bib` (994 body words, ten resolved citations).
+- `CITATION.cff` validated and including the Zenodo software DOI.
+- `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), bug-report and feature-request issue templates, pull-request template.
+- `docs/` containing taxonomy, methodology, and research-context notes.
+
+### Archive
+
+- Version 1.0.0 archived on Zenodo: <https://doi.org/10.5281/zenodo.20807432>.

confusion_mapper-1.0.0/CITATION.cff

@@ -0,0 +1,87 @@
+cff-version: 1.2.0
+message: >-
+  If you use ConfusionMapper in your research, please cite it using
+  the metadata below.
+type: software
+title: "ConfusionMapper"
+abstract: >-
+  ConfusionMapper is a Python tool for human-AI hybrid inter-rater
+  reliability (IRR) assessment of cognitive error taxonomy classification
+  in multiple-choice question distractors. It implements the Confusion
+  Fingerprint Index (CFI) taxonomy, computes Cohen's kappa, builds a
+  full 4x4 confusion matrix across all category pairs, and enforces a
+  pre-registration-compliant reliability gate (kappa >= 0.70) before
+  downstream data collection may proceed.
+authors:
+  - family-names: "Maurya"
+    given-names: "Manik"
+    orcid: "https://orcid.org/0009-0005-3554-693X"
+    email: "manikmaurya.in@gmail.com"
+    affiliation: >-
+      Department of Cognitive Science, Indian Institute of Technology
+      Kanpur, India
+version: "1.0.0"
+date-released: "2026-06-22"
+license: MIT
+repository-code: "https://github.com/Manik-Maurya/Confusion-Mapper"
+url: "https://github.com/Manik-Maurya/Confusion-Mapper"
+doi: "10.5281/zenodo.20807432"
+keywords:
+  - inter-rater reliability
+  - Cohen's kappa
+  - cognitive science
+  - educational assessment
+  - multiple-choice questions
+  - distractor classification
+  - Confusion Fingerprint Index
+  - human-AI collaboration
+  - Python
+  - pre-registration
+preferred-citation:
+  type: article
+  title: >-
+    ConfusionMapper: A Human-AI Hybrid Inter-Rater Reliability Tool for
+    Cognitive Error Taxonomy Classification
+  authors:
+    - family-names: "Maurya"
+      given-names: "Manik"
+      orcid: "https://orcid.org/0009-0005-3554-693X"
+  year: 2026
+  journal: "Journal of Open Source Software"
+  # volume and pages to be updated post-publication
+references:
+  - type: article
+    title: "A coefficient of agreement for nominal scales"
+    authors:
+      - family-names: "Cohen"
+        given-names: "Jacob"
+    year: 1960
+    journal: "Educational and Psychological Measurement"
+    volume: 20
+    issue: 1
+    start: 37
+    end: 46
+    doi: "10.1177/001316446002000104"
+  - type: article
+    title: "The measurement of observer agreement for categorical data"
+    authors:
+      - family-names: "Landis"
+        given-names: "J. Richard"
+      - family-names: "Koch"
+        given-names: "Gary G."
+    year: 1977
+    journal: "Biometrics"
+    volume: 33
+    issue: 1
+    start: 159
+    end: 174
+    doi: "10.2307/2529310"
+  - type: misc
+    title: >-
+      Confusion Fingerprint Index: A Taxonomy of Cognitive Error Types
+      in Multiple-Choice Assessment
+    authors:
+      - family-names: "Maurya"
+        given-names: "Manik"
+    year: 2026
+    doi: "10.35542/osf.io/cr53e_v1"

confusion_mapper-1.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,52 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our community include:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+- The use of sexualized language or imagery, and sexual attention or advances of any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or email address, without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement Responsibilities
+
+The project maintainer is responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, issues, pull requests, discussions, and any other project-related communication, and also applies when an individual is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the project maintainer at **manikmaurya.in@gmail.com**. All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+1. **Correction**, A private, written warning, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate.
+2. **Warning**, A warning with consequences for continued behavior. No interaction with the people involved for a specified period of time.
+3. **Temporary Ban**, A temporary ban from any sort of interaction or public communication with the community for a specified period of time.
+4. **Permanent Ban**, A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.

confusion_mapper-1.0.0/CONTRIBUTING.md

@@ -0,0 +1,51 @@
+# Contributing to ConfusionMapper
+
+Thank you for your interest in contributing. ConfusionMapper is a research-grade tool used as a methodological reliability gate for pre-registered educational RCTs, so changes must preserve numerical correctness and reproducibility.
+
+## Ways to Contribute
+
+- **Report a bug**, open an issue using the Bug Report template. Include a minimal reproducible example, the expected output, and the actual output.
+- **Request a feature**, open an issue using the Feature Request template. Explain the research use case the feature would unlock.
+- **Submit a pull request**, see the workflow below.
+- **Improve documentation**, typo fixes, clearer examples, and translation of the docstrings are all welcome.
+
+## Development Setup
+
+```bash
+git clone https://github.com/Manik-Maurya/Confusion-Mapper.git
+cd Confusion-Mapper
+pip install -e ".[dev]"
+pytest tests/ -v
+python examples/demo.py
+```
+
+The full test suite (33 tests) runs in under a second and requires no API key or display.
+
+## Pull Request Workflow
+
+1. Fork the repository and create a feature branch from `main` (e.g. `fix/kappa-edge-case` or `feat/weighted-kappa`).
+2. Make your changes in small, focused commits with clear messages.
+3. Add or update tests covering the change. New numerical code requires at least one hand-verified worked example in the test docstring.
+4. Run the full suite locally: `pytest tests/ -v`, every test must pass.
+5. Run the headless demo: `python examples/demo.py`, must exit 0.
+6. Update `README.md`, `paper.md`, or `CITATION.cff` if the change affects user-facing behaviour or scholarly attribution.
+7. Open a pull request describing what changed and why. Link the issue it resolves.
+
+## Coding Standards
+
+- **Python 3.9+**, no third-party dependencies in the core module beyond `openai` and the standard library.
+- **No silent behaviour changes** in `compute_cohens_kappa`, `build_confusion_matrix`, or `get_per_type_stats` without an explicit version bump and migration note.
+- **Docstrings** on every public function: one-line summary, parameters, returns, and at least one worked example for numerical functions.
+- **Deterministic output**, no randomness in user-facing functions; tests use seeded RNG.
+
+## Reporting Security Issues
+
+Email security concerns privately to manikmaurya.in@gmail.com rather than opening a public issue.
+
+## Code of Conduct
+
+Participation in this project is governed by the [Code of Conduct](CODE_OF_CONDUCT.md). By contributing, you agree to abide by its terms.
+
+## Attribution and Licensing
+
+All contributions are licensed under the project's MIT License. Significant code contributions will be acknowledged in release notes; contributions that meet the [JOSS authorship criteria](https://joss.readthedocs.io/en/latest/submitting.html#authorship) (substantive design, code, or scholarly contribution) may be added to the author list of future paper versions.

confusion_mapper-1.0.0/LICENSE

@@ -0,0 +1,6 @@
+MIT License
+
+Copyright (c) 2026 Manik Maurya
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files...

confusion_mapper-1.0.0/MANIFEST.in

@@ -0,0 +1,19 @@
+include LICENSE
+include README.md
+include CITATION.cff
+include CONTRIBUTING.md
+include CODE_OF_CONDUCT.md
+include CHANGELOG.md
+include paper.md
+include paper.bib
+include pyproject.toml
+
+recursive-include sample_data *.csv *.json
+recursive-include examples *.py
+recursive-include tests *.py
+recursive-include docs *.md
+
+global-exclude __pycache__
+global-exclude *.py[cod]
+global-exclude .pytest_cache
+global-exclude *.egg-info

confusion_mapper-1.0.0/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: confusion-mapper
+Version: 1.0.0
+Summary: Human-AI hybrid inter-rater reliability tool for cognitive error taxonomy classification in MCQ distractors.
+Author-email: Manik Maurya <manikmaurya.in@gmail.com>
+Maintainer-email: Manik Maurya <manikmaurya.in@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Manik Maurya
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files...
+Project-URL: Homepage, https://github.com/Manik-Maurya/Confusion-Mapper
+Project-URL: Repository, https://github.com/Manik-Maurya/Confusion-Mapper
+Project-URL: Source, https://github.com/Manik-Maurya/Confusion-Mapper
+Project-URL: Documentation, https://github.com/Manik-Maurya/Confusion-Mapper#readme
+Project-URL: Issues, https://github.com/Manik-Maurya/Confusion-Mapper/issues
+Project-URL: Changelog, https://github.com/Manik-Maurya/Confusion-Mapper/blob/main/CHANGELOG.md
+Project-URL: Archive, https://doi.org/10.5281/zenodo.20807432
+Keywords: inter-rater-reliability,cohens-kappa,weighted-kappa,bootstrap-confidence-interval,cognitive-science,educational-assessment,multiple-choice-questions,distractor-classification,human-ai-collaboration,preregistration
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Education
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Education :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=4.0; extra == "dev"
+Dynamic: license-file
+
+# ConfusionMapper
+
+A Python tool for classifying wrong answer choices in multiple-choice questions by the type of cognitive error they represent, and for computing how well a human researcher and an AI rater agree on those labels.
+
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.20807432-1682D4?logo=zenodo&logoColor=white)](https://doi.org/10.5281/zenodo.20807432)
+[![Tests](https://github.com/Manik-Maurya/Confusion-Mapper/actions/workflows/tests.yml/badge.svg)](https://github.com/Manik-Maurya/Confusion-Mapper/actions/workflows/tests.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](https://www.python.org/)
+
+## What it does
+
+In educational research, you cannot use qualitative labels in a statistical analysis unless two independent raters can produce roughly the same labels on the same items. The usual way to check this is Cohen's kappa.
+
+ConfusionMapper handles the full workflow. You go through a set of distractors one by one and pick a label for each. If you have an OpenAI API key, an AI rater labels the same distractors in parallel. At the end, the program computes Cohen's kappa, draws a 4x4 confusion matrix of where the two raters agreed and disagreed, and saves everything to JSON.
+
+It was built as the reliability check for a preregistered RCT on cognitive error feedback in government junior high schools (OSF: https://doi.org/10.17605/OSF.IO/YU6P5). Data collection in that study could only start once kappa cleared 0.70.
+
+## The four error types
+
+The taxonomy is called the Confusion Fingerprint Index (CFI). It splits wrong answers into four categories:
+
+| Code | Name | What it looks like |
+|------|------|---------------------|
+| RF | Recall Failure | No memory trace at all; the answer is essentially random |
+| PK | Partial Knowledge | Almost right; the direction is correct but the model is incomplete |
+| CF | Confabulation | A coherent misconception held with confidence |
+| INT | Interference | A correct answer pulled from the wrong topic |
+
+## Install
+
+Once a versioned release is on PyPI:
+
+```bash
+pip install confusion-mapper
+```
+
+From source (until the first PyPI release, this is the recommended path):
+
+```bash
+git clone https://github.com/Manik-Maurya/Confusion-Mapper.git
+cd Confusion-Mapper
+pip install -e .
+```
+
+The only runtime dependency is `openai>=1.0.0`. `tkinter` ships with Python on most systems; on a minimal Debian or Ubuntu image install it with `sudo apt-get install python3-tk`. If `tkinter` is missing, ConfusionMapper drops to console output instead of the GUI.
+
+Python 3.9 or higher.
+
+For the test extras:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Run
+
+Without an API key (you label everything by hand):
+
+```bash
+python confusion_mapper.py
+```
+
+With the AI rater on:
+
+```bash
+# macOS / Linux
+export OPENAI_API_KEY="your-key-here"
+python confusion_mapper.py
+
+# Windows
+set OPENAI_API_KEY=your-key-here
+python confusion_mapper.py
+```
+
+For each distractor, type `1` for RF, `2` for PK, `3` for CF, or `4` for INT. When you finish, the dashboard opens and the session writes itself to JSON.
+
+## Case study
+
+A fully reproducible worked example lives in [`case_study/`](case_study/). It runs the entire
+pipeline (nominal kappa, weighted kappa under linear and quadratic schemes, BCa
+bootstrap 95% CI, confusion matrix, per-category stats) on the bundled 30-item
+paired-label set with a fixed seed, then writes a publication-ready bundle to
+`case_study/results/` (JSON summary, CSV matrix, CSV per-type stats, full bootstrap
+distribution, and a Markdown report). Regenerate with one command:
+
+```bash
+python case_study/run_case_study.py
+```
+
+Headline result on the bundled data: nominal kappa = 0.8653, BCa 95% CI = (0.6856, 1.0000),
+pre-registration gate PASSES at the 0.70 threshold.
+
+## Headless example
+
+A non-interactive demo runs the three core functions on 30 pre-labelled distractors in under a second. No API key, no display:
+
+```bash
+python examples/demo.py
+```
+
+It prints kappa, the full confusion matrix, per-category agreement, and a PASS / HOLD verdict against the 0.70 gate. Swap in your own paired labels by replacing `sample_data/example_labels.csv`.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+55 tests, runs in under a second. Covers the kappa formula (with hand-verified worked examples), weighted kappa (nominal / linear / quadratic), bootstrap confidence intervals (percentile + BCa, seedable), custom-taxonomy loading, the confusion matrix, and the mathematical invariants that tie everything together. Doesn't need an API key or a display.
+
+## Advanced features
+
+All three are stdlib-only and tested.
+
+**Weighted kappa for ordinal taxonomies.** Pass `weights="linear"` or
+`weights="quadratic"` to penalise far-apart disagreements more than adjacent ones:
+
+```python
+from confusion_mapper import compute_cohens_kappa
+r = compute_cohens_kappa(human, ai, weights="linear")
+print(r["kappa"], r["weights"])
+```
+
+**Bootstrap 95% confidence interval (BCa or percentile).** Cohen's kappa is a point
+estimate; this gives you uncertainty around it. The `seed` makes the interval
+bit-identically reproducible:
+
+```python
+from confusion_mapper import bootstrap_kappa_ci
+ci = bootstrap_kappa_ci(human, ai, n_resamples=10000, method="bca", seed=42)
+print(f"kappa = {ci['point_estimate']} (95% CI {ci['ci_lower']} to {ci['ci_upper']})")
+```
+
+**Custom taxonomy via JSON.** Swap the default CFI categories for any 2-or-more
+category nominal scheme. A working example sits at
+`sample_data/custom_taxonomy.json`:
+
+```python
+from confusion_mapper import load_taxonomy_from_json, compute_cohens_kappa
+codes, tax = load_taxonomy_from_json("sample_data/custom_taxonomy.json")
+r = compute_cohens_kappa(human, ai, categories=codes)
+```
+
+## Why a 4x4 confusion matrix instead of just kappa
+
+The single kappa number tells you how much you and the AI agree overall. It does not tell you which category boundary is causing the disagreement. CF vs INT is the hardest distinction in the CFI taxonomy (a confident wrong belief looks a lot like a correct answer applied to the wrong topic), and the [CF, INT] cell of the confusion matrix is where most disagreement tends to land. Looking at the full 4x4 matrix lets you see exactly that, and rewrite the AI prompt or your own rubric until the cell stops glowing.
+
+## A note on ethics
+
+The AI rater is a starting point, not a substitute for a second human. For published research you should still run kappa between two trained human raters on the same calibration set. Treat the AI labels as a way to surface boundaries that need rubric work, not as a replacement for human judgment.
+
+## How to cite
+
+Archived release on Zenodo: https://doi.org/10.5281/zenodo.20807432
+
+APA:
+
+> Maurya, M. (2026). *ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research* (Version 1.0.0) [Software]. Zenodo. https://doi.org/10.5281/zenodo.20807432
+
+BibTeX:
+
+```bibtex
+@software{maurya2026confusionmapper,
+  author    = {Maurya, Manik},
+  title     = {ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification
+               and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research},
+  year      = {2026},
+  version   = {1.0.0},
+  doi       = {10.5281/zenodo.20807432},
+  url       = {https://doi.org/10.5281/zenodo.20807432}
+}
+```
+
+## Contributing
+
+Bug reports, feature requests, and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
+
+## Acknowledgements
+
+Initial development was completed as part of Stanford Code in Place 2026. The tool is used in the Confusion Fingerprint Index research programme at the Department of Cognitive Science, IIT Kanpur.
+
+## License
+
+MIT License © 2026 Manik Maurya. See [LICENSE](LICENSE) for details.

confusion_mapper-1.0.0/README.md

@@ -0,0 +1,185 @@
+# ConfusionMapper
+
+A Python tool for classifying wrong answer choices in multiple-choice questions by the type of cognitive error they represent, and for computing how well a human researcher and an AI rater agree on those labels.
+
+[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.20807432-1682D4?logo=zenodo&logoColor=white)](https://doi.org/10.5281/zenodo.20807432)
+[![Tests](https://github.com/Manik-Maurya/Confusion-Mapper/actions/workflows/tests.yml/badge.svg)](https://github.com/Manik-Maurya/Confusion-Mapper/actions/workflows/tests.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](https://www.python.org/)
+
+## What it does
+
+In educational research, you cannot use qualitative labels in a statistical analysis unless two independent raters can produce roughly the same labels on the same items. The usual way to check this is Cohen's kappa.
+
+ConfusionMapper handles the full workflow. You go through a set of distractors one by one and pick a label for each. If you have an OpenAI API key, an AI rater labels the same distractors in parallel. At the end, the program computes Cohen's kappa, draws a 4x4 confusion matrix of where the two raters agreed and disagreed, and saves everything to JSON.
+
+It was built as the reliability check for a preregistered RCT on cognitive error feedback in government junior high schools (OSF: https://doi.org/10.17605/OSF.IO/YU6P5). Data collection in that study could only start once kappa cleared 0.70.
+
+## The four error types
+
+The taxonomy is called the Confusion Fingerprint Index (CFI). It splits wrong answers into four categories:
+
+| Code | Name | What it looks like |
+|------|------|---------------------|
+| RF | Recall Failure | No memory trace at all; the answer is essentially random |
+| PK | Partial Knowledge | Almost right; the direction is correct but the model is incomplete |
+| CF | Confabulation | A coherent misconception held with confidence |
+| INT | Interference | A correct answer pulled from the wrong topic |
+
+## Install
+
+Once a versioned release is on PyPI:
+
+```bash
+pip install confusion-mapper
+```
+
+From source (until the first PyPI release, this is the recommended path):
+
+```bash
+git clone https://github.com/Manik-Maurya/Confusion-Mapper.git
+cd Confusion-Mapper
+pip install -e .
+```
+
+The only runtime dependency is `openai>=1.0.0`. `tkinter` ships with Python on most systems; on a minimal Debian or Ubuntu image install it with `sudo apt-get install python3-tk`. If `tkinter` is missing, ConfusionMapper drops to console output instead of the GUI.
+
+Python 3.9 or higher.
+
+For the test extras:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Run
+
+Without an API key (you label everything by hand):
+
+```bash
+python confusion_mapper.py
+```
+
+With the AI rater on:
+
+```bash
+# macOS / Linux
+export OPENAI_API_KEY="your-key-here"
+python confusion_mapper.py
+
+# Windows
+set OPENAI_API_KEY=your-key-here
+python confusion_mapper.py
+```
+
+For each distractor, type `1` for RF, `2` for PK, `3` for CF, or `4` for INT. When you finish, the dashboard opens and the session writes itself to JSON.
+
+## Case study
+
+A fully reproducible worked example lives in [`case_study/`](case_study/). It runs the entire
+pipeline (nominal kappa, weighted kappa under linear and quadratic schemes, BCa
+bootstrap 95% CI, confusion matrix, per-category stats) on the bundled 30-item
+paired-label set with a fixed seed, then writes a publication-ready bundle to
+`case_study/results/` (JSON summary, CSV matrix, CSV per-type stats, full bootstrap
+distribution, and a Markdown report). Regenerate with one command:
+
+```bash
+python case_study/run_case_study.py
+```
+
+Headline result on the bundled data: nominal kappa = 0.8653, BCa 95% CI = (0.6856, 1.0000),
+pre-registration gate PASSES at the 0.70 threshold.
+
+## Headless example
+
+A non-interactive demo runs the three core functions on 30 pre-labelled distractors in under a second. No API key, no display:
+
+```bash
+python examples/demo.py
+```
+
+It prints kappa, the full confusion matrix, per-category agreement, and a PASS / HOLD verdict against the 0.70 gate. Swap in your own paired labels by replacing `sample_data/example_labels.csv`.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+```
+
+55 tests, runs in under a second. Covers the kappa formula (with hand-verified worked examples), weighted kappa (nominal / linear / quadratic), bootstrap confidence intervals (percentile + BCa, seedable), custom-taxonomy loading, the confusion matrix, and the mathematical invariants that tie everything together. Doesn't need an API key or a display.
+
+## Advanced features
+
+All three are stdlib-only and tested.
+
+**Weighted kappa for ordinal taxonomies.** Pass `weights="linear"` or
+`weights="quadratic"` to penalise far-apart disagreements more than adjacent ones:
+
+```python
+from confusion_mapper import compute_cohens_kappa
+r = compute_cohens_kappa(human, ai, weights="linear")
+print(r["kappa"], r["weights"])
+```
+
+**Bootstrap 95% confidence interval (BCa or percentile).** Cohen's kappa is a point
+estimate; this gives you uncertainty around it. The `seed` makes the interval
+bit-identically reproducible:
+
+```python
+from confusion_mapper import bootstrap_kappa_ci
+ci = bootstrap_kappa_ci(human, ai, n_resamples=10000, method="bca", seed=42)
+print(f"kappa = {ci['point_estimate']} (95% CI {ci['ci_lower']} to {ci['ci_upper']})")
+```
+
+**Custom taxonomy via JSON.** Swap the default CFI categories for any 2-or-more
+category nominal scheme. A working example sits at
+`sample_data/custom_taxonomy.json`:
+
+```python
+from confusion_mapper import load_taxonomy_from_json, compute_cohens_kappa
+codes, tax = load_taxonomy_from_json("sample_data/custom_taxonomy.json")
+r = compute_cohens_kappa(human, ai, categories=codes)
+```
+
+## Why a 4x4 confusion matrix instead of just kappa
+
+The single kappa number tells you how much you and the AI agree overall. It does not tell you which category boundary is causing the disagreement. CF vs INT is the hardest distinction in the CFI taxonomy (a confident wrong belief looks a lot like a correct answer applied to the wrong topic), and the [CF, INT] cell of the confusion matrix is where most disagreement tends to land. Looking at the full 4x4 matrix lets you see exactly that, and rewrite the AI prompt or your own rubric until the cell stops glowing.
+
+## A note on ethics
+
+The AI rater is a starting point, not a substitute for a second human. For published research you should still run kappa between two trained human raters on the same calibration set. Treat the AI labels as a way to surface boundaries that need rubric work, not as a replacement for human judgment.
+
+## How to cite
+
+Archived release on Zenodo: https://doi.org/10.5281/zenodo.20807432
+
+APA:
+
+> Maurya, M. (2026). *ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research* (Version 1.0.0) [Software]. Zenodo. https://doi.org/10.5281/zenodo.20807432
+
+BibTeX:
+
+```bibtex
+@software{maurya2026confusionmapper,
+  author    = {Maurya, Manik},
+  title     = {ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification
+               and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research},
+  year      = {2026},
+  version   = {1.0.0},
+  doi       = {10.5281/zenodo.20807432},
+  url       = {https://doi.org/10.5281/zenodo.20807432}
+}
+```
+
+## Contributing
+
+Bug reports, feature requests, and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) and [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
+
+## Acknowledgements
+
+Initial development was completed as part of Stanford Code in Place 2026. The tool is used in the Confusion Fingerprint Index research programme at the Department of Cognitive Science, IIT Kanpur.
+
+## License
+
+MIT License © 2026 Manik Maurya. See [LICENSE](LICENSE) for details.