a11y-fixer 0.1.0__tar.gz

a11y_fixer-0.1.0/CHANGELOG.md

@@ -0,0 +1,85 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project aims to
+adhere to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+- `--fail-on STATUSES` — narrow the exit-1 gate to specific outcomes (e.g.
+  `invalid`, or `would-fix,invalid,residual-violations`). Overrides the `--check`
+  gate; a file that errored still exits 1.
+- Eval harness `--axe` — renders each opted-in fixture's fixed component to
+  static markup and runs axe-core against it in jsdom (`lint/axe.mjs`), so a
+  fixture can assert the fix is actually clean to assistive tech, not just to
+  the linter. Fixtures opt in with `expect_axe_clean` / `axe_props`; adds an
+  `axe_clean_rate` metric.
+- Eval corpus grown from 7 to 15 fixtures: `invalid-aria-prop`, `iframe-no-title`,
+  `anchor-no-content`, `redundant-role`, `mouse-only-handlers`, a second
+  false-positive guard `decorative-img` (a correct empty `alt=""` the model must
+  not "helpfully" fill in), and two `prompt-injection` fixtures that verify
+  override instructions embedded in a JS comment or in rendered JSX text don't
+  steer the model off its task (SYSTEM_PROMPT constraint #7).
+
+### Changed
+
+- `--workers > 1` now shares a single rate-limit cooldown across all worker
+  threads: a provider 429 pauses the whole pool (with jitter) instead of each
+  thread running its own blind backoff into the same quota. Single-worker runs
+  are unchanged.
+
+### Fixed
+
+- The sdist no longer vendors `lint/node_modules/` (a stray `*.json` glob in
+  `MANIFEST.in` was pulling in every nested `package.json`). It ships only the
+  runner scripts, `package.json`, and `package-lock.json`; run `npm install` in
+  `lint/` after install as before.
+
+## [0.1.0] - 2026-09-08
+
+First tagged release. The agent went from a "trust the model and write to disk"
+script to a checked-remediation tool with CI, caching, and an eval harness.
+
+### Added
+
+- **Structural safety gate** — every fix is parsed as JSX/TSX (tree-sitter) and
+  compared against a fingerprint of the original (event handlers, hook calls,
+  identifiers). A fix that fails to parse or drops behavior is rejected as
+  `invalid`, never written.
+- **Repair pass** — a rejected fix is re-prompted once with the specific reason
+  (`--repair-attempts`, default 1) before being reported.
+- **`--lint` feedback loop** — bundled `eslint-plugin-jsx-a11y` runs in-process,
+  its violations are fed into the prompt, and the model's output is re-linted
+  (`--lint-rounds`, default 2). Lint-clean files skip the model entirely.
+- **CI modes** — `--check` (exit 1 on any residual issue, never writes),
+  `--patch FILE` (unified diff), `--json-summary FILE` (machine-readable run
+  report).
+- **`--since REF`** — restrict the run to files changed vs a git ref (changed
+  tracked files plus new untracked ones).
+- **Content-hash cache** (`.a11y-cache.json`) keyed by a prompt fingerprint
+  (model, provider, base URL, system prompt, lint ruleset); concurrency-safe
+  merge-on-save behind a lock file.
+- **Provider abstraction** — Groq (default), OpenAI, or any OpenAI-compatible
+  server via `--provider` / `--base-url`. Deterministic decoding, truncation
+  detection, retry with `Retry-After` awareness.
+- **`a11y.toml`** project config (auto-discovered) — design-system primitives to
+  leave alone, extra prompt rules, ignore globs.
+- **Observability** — `--log FILE` writes a line-flushed JSONL event stream;
+  `--verbose` / `--quiet`; `--timeout`; graceful SIGINT (drains in-flight files,
+  saves, exits 130).
+- **PR-bot** (`a11y-fixer-pr`) — runs the fixer, opens or updates a pull request
+  with a per-file summary via `gh`.
+- **Eval harness** (`evals/run.py`) — golden fixtures scored on status,
+  lint-cleanliness, and identifier preservation; aggregate `fix_rate`,
+  `false_positive_rate`, `regression_rate`.
+- Faithful write-back of original encoding and line endings (BOM / CRLF);
+  machine-readable outputs are always LF.
+- `--version`.
+- Packaging (`pyproject.toml`, console scripts `a11y-fixer` and
+  `a11y-fixer-pr`), 138-test pytest suite, and GitHub Actions workflows for the
+  PR check, the PR-bot, and the eval.
+
+[Unreleased]: https://github.com/arysarin/a11y-fixer/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/arysarin/a11y-fixer/releases/tag/v0.1.0

a11y_fixer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 a11y-fixer authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

a11y_fixer-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE README.md CHANGELOG.md requirements.txt a11y.toml.example MANIFEST.in
+recursive-include lint *.mjs
+include lint/package.json lint/package-lock.json lint/eslint.config.mjs lint/.gitignore
+prune lint/node_modules
+recursive-include tests *.py
+recursive-include evals *.py *.json *.jsx *.tsx *.toml *.md

a11y_fixer-0.1.0/PKG-INFO

@@ -0,0 +1,364 @@
+Metadata-Version: 2.4
+Name: a11y-fixer
+Version: 0.1.0
+Summary: LLM accessibility remediation agent for JSX/TSX with structural safety checks
+Author: a11y-fixer authors
+License: MIT
+Project-URL: Homepage, https://github.com/arysarin/a11y-fixer
+Project-URL: Repository, https://github.com/arysarin/a11y-fixer
+Project-URL: Changelog, https://github.com/arysarin/a11y-fixer/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/arysarin/a11y-fixer/issues
+Keywords: accessibility,a11y,wcag,aria,jsx,react,llm,codemod
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Build Tools
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: groq>=0.11.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: tree-sitter>=0.21
+Requires-Dist: tree-sitter-typescript>=0.23
+Requires-Dist: tomli>=2.0; python_version < "3.11"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Dynamic: license-file
+
+# a11y-fixer
+
+An accessibility remediation agent for React JSX/TSX. It scans a file or
+directory, sends each component through an LLM (Groq, OpenAI, or any
+OpenAI-compatible endpoint) with a WCAG 2.0/2.2 + ARIA APG system prompt,
+**validates the result structurally**, and then either rewrites the source,
+emits a patch, or reports for CI.
+
+The validation step is what makes the output safe to apply unattended: model
+output is parsed as JSX/TSX and compared against the original, and a fix is
+rejected if it fails to parse or drops an event handler or hook call.
+
+## Install
+
+```bash
+pip install a11y-fixer
+```
+
+That gives you the core agent — structural gate, `--write` / `--patch` /
+`--check` / `--fail-on`, all providers, the cache, `--since`. The optional
+eslint-jsx-a11y feedback loop (`--lint`) and the eval harness's axe pass
+(`--axe`) run a bundled Node package that isn't in the wheel; for those, work
+from a clone:
+
+```bash
+git clone https://github.com/arysarin/a11y-fixer && cd a11y-fixer
+pip install -e ".[dev]"        # editable install + pytest
+npm --prefix lint install      # only needed for --lint / --axe
+```
+
+Set an API key via environment or a local `.env` (the variable depends on the
+provider — `GROQ_API_KEY` by default, `OPENAI_API_KEY` for `--provider openai`):
+
+```bash
+echo "GROQ_API_KEY=gsk-..." > .env
+```
+
+## Usage
+
+```bash
+# dry run: print diffs, change nothing
+a11y-fixer src/
+
+# apply validated fixes in place (writes .bak alongside each changed file)
+a11y-fixer src/ --write
+
+# CI: check only what this PR changed
+a11y-fixer src/ --since origin/main --check
+
+# write validated fixes to a patch instead of touching the tree
+a11y-fixer src/ --patch a11y-fixes.patch
+
+# CI gate: exit non-zero if any file still has an accessibility issue
+a11y-fixer src/ --check --json-summary a11y-summary.json
+```
+
+### Output modes
+
+| Mode | Flag | Writes source? | Exit on findings |
+| --- | --- | --- | --- |
+| Dry run | *(default)* | no | 0 |
+| Write | `--write` | yes (validated only) | 0 |
+| Patch | `--patch FILE` | no (diff to `FILE`) | 0 |
+| Check | `--check` | no | 1 |
+
+`--write` is ignored when combined with `--check` or `--patch`.
+
+### Exit codes
+
+- `0` — clean, or fixes applied/emitted successfully
+- `1` — `--check` (or `--fail-on`) found a gated outcome, **or** a file errored
+  (API failure, truncated response, unreadable file)
+- `2` — bad invocation (missing API key, path not found, `--workers < 1`)
+- `130` — interrupted with Ctrl-C (partial results saved)
+
+### Options
+
+| Flag | Default | Purpose |
+| --- | --- | --- |
+| `--ext` | `jsx,tsx` | comma-separated extensions to scan |
+| `--exclude` | `node_modules,.git,dist,build,.next` | directory names to skip |
+| `--since REF` | — | only process files changed vs a git ref (e.g. `origin/main`) |
+| `--model` | `openai/gpt-oss-120b` | Groq model id |
+| `--write` | off | apply fixes in place |
+| `--no-backup` | off | skip `.bak` files when writing |
+| `--strict` | off | also block a fix that drops *any* identifier, not just handlers/hooks |
+| `--repair-attempts N` | `1` | re-prompt a fix that fails the structural gate, up to N times (`0` disables) |
+| `--lint` | off | run eslint-jsx-a11y, target its violations, verify the fix (needs Node) |
+| `--lint-rounds N` | `2` | max model passes to clear lint violations |
+| `--lint-config FILE` | bundled a11y config | ESLint config to lint with |
+| `--config FILE` | auto-discover `a11y.toml` | project config (see above) |
+| `--no-config` | off | skip config auto-discovery |
+| `--provider` | `groq` | `groq`, `openai`, or `openai-compatible` |
+| `--model` | provider default | model id (`openai/gpt-oss-120b` for groq, `gpt-4o-mini` for openai) |
+| `--base-url` | — | API base URL; required for `openai-compatible` |
+| `--check` | off | CI mode: never write, exit 1 on findings |
+| `--fail-on STATUSES` | — | narrow the exit-1 gate to specific outcomes, e.g. `invalid` or `would-fix,invalid,residual-violations` (overrides `--check`'s gate; a file that errored still exits 1) |
+| `--patch FILE` | — | write validated fixes to `FILE` as a unified diff |
+| `--json-summary FILE` | — | write a machine-readable run summary |
+| `--cache FILE` | `.a11y-cache.json` | cache file for skipping unchanged files |
+| `--no-cache` | off | ignore and don't update the cache |
+| `--api-key` | `$GROQ_API_KEY` | Groq key override |
+| `--workers` | `1` | files processed concurrently (with `> 1`, all workers share one rate-limit cooldown, so a provider 429 pauses the pool instead of each thread backing off blind) |
+| `--timeout SECONDS` | `120` | per model request / lint subprocess timeout (`0` = none) |
+| `--log FILE` | — | append a JSONL event log |
+| `--quiet` / `--verbose` | — | suppress per-file output / show lint+repair progress |
+
+## Observability
+
+`--log FILE` appends one JSON object per line as the run proceeds — it survives
+Ctrl-C and crashes, since each line is flushed immediately:
+
+```
+{"event":"run_start","mode":"CHECK","model":"...","files":42, ...}
+{"event":"file_start","path":"src/NavBar.jsx"}
+{"event":"lint","path":"src/NavBar.jsx","phase":"initial","violations":2}
+{"event":"model_call","path":"src/NavBar.jsx","kind":"lint_round","round":1,"prompt_tokens":970,"completion_tokens":734}
+{"event":"file_done","path":"src/NavBar.jsx","status":"would-fix","seconds":7.6,"violations_before":2,"violations_after":0,"repair_attempts":1, ...}
+{"event":"run_end","duration_seconds":8.6,"interrupted":false,"scanned":42, ...}
+```
+
+Ctrl-C lets in-flight files finish, saves the cache and prints the summary, then
+exits `130`. A second Ctrl-C bails immediately. Parallel runs sharing a
+`.a11y-cache.json` merge on save (each writes only the entries it touched) behind
+a best-effort lock file, so they don't clobber each other.
+
+## Linter feedback loop (`--lint`)
+
+By default the agent trusts the model to find and fix accessibility issues.
+`--lint` closes that loop with a real linter:
+
+1. Run `eslint-plugin-jsx-a11y` over the file.
+2. If it's clean, **skip the model entirely** (no tokens spent).
+3. Otherwise pass the exact violations into the prompt, get a fix, and re-lint it.
+4. Repeat up to `--lint-rounds` (default 2) until clean or give up.
+5. The structural gate (parse, handlers/hooks) still runs on the final result.
+
+The run summary reports `a11y lint  N -> M violations`, the JSON summary carries
+per-file `violations_before` / `violations_after` / `lint_rounds`, and under
+`--check` any file left with violations fails CI.
+
+**Setup** — needs Node.js and a one-time install of the bundled ESLint:
+
+```bash
+cd lint && npm install        # installs eslint + jsx-a11y + the TS parser
+a11y-fixer src/ --lint --check
+```
+
+`--lint-config FILE` points at your project's own ESLint config instead of the
+bundled a11y-only one. The bundled rule set is folded into the cache key, so
+changing it re-checks every file.
+
+## Project config
+
+Drop an `a11y.toml` (or `.a11y.toml`) in the directory you run from — it's
+auto-discovered (`--config FILE` to point elsewhere, `--no-config` to skip). See
+[a11y.toml.example](a11y.toml.example).
+
+```toml
+primitives = ["Button", "Link", "Modal"]   # already-accessible components; leave them alone
+rules = ["Our <Icon> is decorative unless a title prop is set."]
+ignore = ["**/*.stories.tsx", "src/legacy/**"]   # skip these files
+ext = "jsx,tsx"                              # optional; CLI --ext still wins
+exclude = "node_modules,dist,coverage"      # optional; CLI --exclude still wins
+```
+
+`primitives` and `rules` are injected into the system prompt so the model stops
+adding redundant ARIA to your wrapped components. `ignore` patterns are matched
+(with `fnmatch`) against both the working-dir-relative path and the basename.
+Changing the config changes the prompt, which invalidates the cache.
+
+## Providers
+
+The model call goes through a small `LLMClient` adapter, so any OpenAI-shaped
+API works:
+
+```bash
+# Groq (default)
+a11y-fixer src/ --check
+
+# OpenAI
+a11y-fixer src/ --provider openai --model gpt-4o-mini --check      # needs OPENAI_API_KEY
+pip install "a11y-fixer[openai]"                                    # installs the openai SDK
+
+# Any OpenAI-compatible server (Ollama, vLLM, LiteLLM, LM Studio, ...)
+a11y-fixer src/ --provider openai-compatible \
+  --base-url http://localhost:11434/v1 --model qwen2.5-coder:7b --check
+```
+
+`--base-url` also lets the `openai-compatible` provider run without an API key
+(for local servers that don't need one). Changing provider, model, or base URL
+invalidates the cache.
+
+## Caching
+
+Every run records the content hash of each file it verified as **clean**
+(`unchanged`) or **already fixed** (`fixed`) into `--cache` (default
+`.a11y-cache.json`, keyed relative to the working directory). A later run whose
+file content still matches skips the model call entirely and reports it as
+`unchanged (from cache)` with zero tokens.
+
+Only known-good outcomes are cached, so the cache can never suppress a pending
+fix, a validation failure, or a diff you need to see — `would-fix`, `invalid`,
+and `error` files are re-processed every run. The whole cache is discarded when
+the model (`--model`) or the system prompt changes, since a stale "clean"
+verdict would no longer be trustworthy.
+
+Add `.a11y-cache.json` to `.gitignore` (already done here). Use `--no-cache` in
+environments where you want every file re-checked unconditionally.
+
+## How validation works
+
+For each file the agent builds a structural fingerprint (via `tree-sitter`) of
+the original and the model output:
+
+1. **Parse gate** — if the original parsed cleanly and the fix does not, the fix
+   is rejected (`invalid`) and never written.
+2. **Handler / hook preservation** — any `on*` JSX attribute or `use*` hook call
+   present in the original but gone from the fix rejects it.
+3. **`--strict`** — additionally rejects if any other identifier drops to zero
+   references (noisier: legitimate refactors like `window.location.href = …` →
+   `<a href>` trip this).
+
+If the original file does not parse, all checks are skipped (no trustworthy
+baseline) and the fix is written as-is under `--write`.
+
+**Repair pass** — when the structural gate rejects a fix, it is sent back once
+(configurable with `--repair-attempts N`, default 1, `0` disables) with the exact
+reason (`event handlers removed: onClick`, `does not parse`, …) and re-validated.
+A recovered fix proceeds normally; the summary reports `repair  K/N recovered`.
+This costs one extra model call per rejected file.
+
+A truncated or empty model response is always an error, never a partial write.
+
+File content is treated strictly as code to remediate; the system prompt tells
+the model to ignore any instructions embedded in it (prompt-injection guard),
+and the structural gate above is the real backstop regardless. The eval corpus
+carries `prompt-injection` fixtures (override text in a JS comment and in
+rendered JSX) that check this holds against the live model.
+
+Files are read as UTF-8; a byte-order mark and CRLF line endings are detected
+and preserved on write-back. A non-UTF-8 file is reported as an `error` rather
+than guessed at.
+
+`--since REF` narrows the run to files that differ from a git ref — tracked
+modifications plus new untracked files, deletions excluded. A run with nothing
+changed exits `0` ("No changed files to process"); an unknown ref or a
+non-git directory exits `2`.
+
+## CI
+
+See [.github/workflows/a11y.yml](.github/workflows/a11y.yml): runs `--check` on
+pull requests touching `.jsx`/`.tsx`, and a manual `workflow_dispatch` produces
+a fix patch as a build artifact. Requires a `GROQ_API_KEY` repository secret
+(or `OPENAI_API_KEY` if you switch `--provider`).
+
+To stop the build only on a fix that failed the safety gate — while letting
+merely-unremediated files through — use `--fail-on`:
+
+```bash
+a11y-fixer src/ --since origin/main --lint --fail-on invalid
+```
+
+## Opening a PR with the fixes
+
+`a11y_pr.py` (console script `a11y-fixer-pr`) runs the agent and turns the result
+into a pull request. Run it from the repo root with a clean tracked tree:
+
+```bash
+a11y-fixer-pr src/ --lint --label accessibility
+```
+
+It runs `a11y-fixer src/ --lint --patch … --json-summary …` (any extra args pass
+through), and if there are validated fixes: branches from HEAD (`--branch`,
+default `a11y-fixes`), applies the patch, commits (message carries the lint
+delta), force-pushes, and creates or refreshes a PR through the `gh` CLI. The PR
+body is a per-file table (status, `violations_before → after`, repair passes)
+with token/cost totals and a note that rejected fixes were left out. `--dry-run`
+stops after the local commit; without `gh` it pushes and prints the compare
+instructions. [.github/workflows/a11y-pr.yml](.github/workflows/a11y-pr.yml) wires
+it to `workflow_dispatch`.
+
+## Limitations
+
+- Without `--lint`, semantic correctness of a fix is not verified — only that it
+  parses and keeps handlers/hooks. `--lint` verifies against
+  `eslint-plugin-jsx-a11y`; `axe-core` on rendered output would be stronger still.
+- The bundled `lint/` directory ships in source checkouts and editable installs
+  (`pip install -e .`); a plain wheel install does not carry it, so run `--lint`
+  from a checkout.
+- Each file is processed in isolation; the agent has no knowledge of shared
+  design-system components (an `a11y.toml` mitigates this).
+- Cost figures in the summary are estimates from `MODEL_PRICING` in
+  `a11y_fixer.py`; update the rates for your model.
+
+## Tests
+
+```bash
+pytest
+```
+
+`.github/workflows/ci.yml` runs the suite on Python 3.10 / 3.11 / 3.12 on every
+push and PR, plus a smoke check that the bundled `eslint-plugin-jsx-a11y` config
+still flags a known violation and that the package builds with its runtime data
+files (`lint/`, `a11y.toml.example`).
+
+## Releasing
+
+`a11y_fixer.__version__` is the single source of truth; `pyproject.toml` reads it
+dynamically, so a release is: bump that string, move the `CHANGELOG.md`
+`[Unreleased]` items under a dated version heading, tag `vX.Y.Z`. `a11y-fixer
+--version` reports the installed version.
+
+## Evals
+
+[evals/](evals/) holds golden fixtures and a scorer for measuring a prompt /
+model / validation change rather than eyeballing it:
+
+```bash
+GROQ_API_KEY=... python evals/run.py --lint --json after.json
+GROQ_API_KEY=... python evals/run.py --lint --axe   # also render + run axe-core
+```
+
+Reports `fix_rate`, `false_positive_rate`, `regression_rate`, and (with `--axe`)
+`axe_clean_rate`; exits non-zero below `--threshold` (default 1.0). `--axe`
+renders each opted-in fixture's fixed component and runs axe-core against it in
+jsdom, checking the fix actually helps assistive tech rather than just passing
+the linter. See [evals/README.md](evals/README.md).