jittest 0.2.1__tar.gz

jittest-0.2.1/.gitignore

@@ -0,0 +1,20 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+coverage.xml
+htmlcov/
+.env
+.jittest/
+jittest-results.json
+eval-results.json
+false-positives.json
+.jittest-comment.md
+.DS_Store

jittest-0.2.1/CHANGELOG.md

@@ -0,0 +1,78 @@
+# Changelog
+
+## 0.2.1 — 2026-07-26
+
+### Fixed
+- **pytest parity**: Added `pythonpath = ["src"]` to `[tool.pytest.ini_options]` so `python -m pytest` can import jittest from src-layout without `PYTHONPATH`. The unittest runner worked because CI set `PYTHONPATH=src` explicitly; pytest had no such configuration.
+- **Outcome enum restored to StrEnum**: Ruff rule UP042 was applied as `Outcome(str, Enum)` → `Outcome(Enum)`, which removed string behaviour the codebase relies on (`Outcome.PASS == "pass"`, JSON serialisation). Corrected to `Outcome(StrEnum)`. Added 4 regression tests that fail on plain Enum and pass on StrEnum.
+- **Canonical Apache-2.0 licence**: Replaced placeholder LICENSE with verbatim text from apache.org. GitHub now detects `spdx_id: Apache-2.0` instead of `NOASSERTION`.
+- **Ruff compliance**: Fixed all 7 ruff errors (UP042, UP037 ×2, F401, UP012, SIM102, SIM117). No per-file-ignores, no noqa comments, no config relaxation.
+
+### Changed
+- **ARCHITECTURE.md and STACK.md rewritten**: Both documents described a v0.1 design that depended on unidiff, litellm, pydantic, typer, rich, PyYAML and coverage. The shipped v0.2 code depends on nothing. Documents now describe what each module in `src/jittest/` actually does, module by module.
+- **install-smoke workflow**: Added `.github/workflows/install-smoke.yml` that verifies `pip install .` produces a working `jittest` command on Python 3.11, 3.12 and 3.13. All three pass.
+- **CI umbrella job**: Added a `ci` job to `.github/workflows/ci.yml` that aggregates lint, test and build results into a single check name matching the branch protection required context.
+- **Actions version bumps**: actions/checkout v4→v7, actions/setup-python v5→v7, actions/upload-artifact v4→v7, softprops/action-gh-release v2→v3.
+
+All notable changes to this project are documented here.
+Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2026-07-25
+
+### Changed - the zero-dependency rewrite
+
+- **`dependencies = []`.** jittest now installs with no third-party packages at
+  all. `unidiff`, `litellm`, `pydantic`, `typer`, `rich` and `PyYAML` are gone.
+  A tool that runs inside other people's CI should not bring a dependency tree
+  into their locked project, and every package we drop is one fewer resolver
+  conflict, one fewer supply-chain surface and one fewer reason to say no.
+  - unified diff parsing: own parser in `diff.py`
+  - model access: `urllib` in `llm.py` (Anthropic Messages + OpenAI-compatible)
+  - config: stdlib `tomllib`
+  - CLI: `argparse`
+  - litellm is still supported, opt-in, via `pip install jittest[litellm]` and
+    `JITTEST_USE_LITELLM=1`
+
+### Added
+
+- **`_minirunner`**: a stdlib test runner used when pytest is not importable,
+  with pytest-compatible exit codes (0 pass / 1 fail / 2 collection error /
+  5 no tests). jittest can now answer "does this test pass here?" in hardened
+  or air-gapped runners.
+- **Worktree reuse.** Base and head are checked out once per run instead of once
+  per candidate. This is what makes three executions per candidate affordable,
+  and three executions is what buys the flakiness rerun.
+- **Flakiness reruns** (`--reruns`, default 2). A failure on head must reproduce
+  before we spend a base checkout, let alone a reviewer's attention.
+- **`jittest doctor`**: reports python version, git, runner, key presence and
+  effective config, so a failed install is diagnosable in one command.
+- **Human outcome labels** in the ledger (`fixed_code`, `kept_test`, `intended`,
+  `false_positive`, `ignored`) plus `jittest outcome` and anonymised
+  `jittest export`. This is the corpus, and it is the only asset here that
+  cannot be reimplemented in a fortnight.
+- **Static safety gate** (`safety.py`) applied to every candidate before it is
+  executed: no sockets, no subprocesses, no `eval`, no `assert True`, no sleeps.
+- **Response cache** keyed on model, prompts and temperature, so re-running a PR
+  costs nothing.
+- **Budget cap** enforced before a request is sent rather than after.
+- **`--dry-run`** runs the entire pipeline with a stub model: no API key, no
+  network, no cost. It is how the test suite runs, and how a new user can watch
+  jittest work on their own repo before handing over a key.
+- 36 offline tests covering the diff parser, risk ranking, config precedence,
+  the safety gate, assessment parsing, the ledger, the oracle against a real
+  seeded regression in a real git repo, and the pipeline end to end.
+
+### Fixed
+
+- Candidate test files are always deleted from the checkout, including after a
+  timeout.
+- `PYTHONHASHSEED=0` and `PYTHONDONTWRITEBYTECODE=1` in the child environment,
+  removing one avoidable source of non-determinism.
+- Assessor replies with a confidence of `85` are read as `0.85`; unparseable
+  replies degrade to `unclear`, which is not reported. The layer fails closed.
+
+## [0.1.0] - 2026-07-25
+
+- First scaffold: diff parsing, risk targeting, generator prompts, differential
+  oracle, assessor, SQLite ledger, GitHub Action, docs and evaluation harness.

jittest-0.2.1/LICENSE

@@ -0,0 +1,201 @@
+Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1\. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2\. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3\. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4\. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5\. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6\. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7\. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8\. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9\. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2026 the jittest authors
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.

jittest-0.2.1/PKG-INFO

@@ -0,0 +1,248 @@
+Metadata-Version: 2.4
+Name: jittest
+Version: 0.2.1
+Summary: Generate tests that FAIL on your pull request and PASS on main. Open reference implementation of just-in-time catching test generation.
+Project-URL: Homepage, https://github.com/Kartik24Hulmukh/jittest
+Project-URL: Documentation, https://github.com/Kartik24Hulmukh/jittest/tree/main/docs
+Project-URL: Source, https://github.com/Kartik24Hulmukh/jittest
+Project-URL: Issues, https://github.com/Kartik24Hulmukh/jittest/issues
+Project-URL: Changelog, https://github.com/Kartik24Hulmukh/jittest/blob/main/CHANGELOG.md
+Author: jittest contributors
+License: Apache-2.0
+License-File: LICENSE
+Keywords: catching-tests,ci,code-review,jittest,llm,pull-request,pytest,regression,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Quality Assurance
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-timeout>=2.3; extra == 'dev'
+Requires-Dist: pytest>=8.2; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.55.0; extra == 'litellm'
+Description-Content-Type: text/markdown
+
+<h1 align="center">jittest</h1>
+
+<p align="center">
+  <b>Generate tests that FAIL on your pull request and PASS on main.</b><br>
+  The open reference implementation of just-in-time <i>catching</i> test generation.
+</p>
+
+<p align="center">
+  <img alt="license" src="https://img.shields.io/badge/license-Apache--2.0-blue">
+  <img alt="python" src="https://img.shields.io/badge/python-3.11%2B-blue">
+  <img alt="dependencies" src="https://img.shields.io/badge/dependencies-0-brightgreen">
+</p>
+
+---
+
+## The distinction the whole tool rests on
+
+Almost every AI testing tool generates tests that **pass** on your new code.
+That is a *hardening* test. It documents what the code does today, including any
+bug you just introduced. Generate tests until they go green and you can
+[reinforce the bug you were trying to find](https://arxiv.org/pdf/2412.14137).
+
+jittest generates **catching** tests:
+
+> A catching test **passes on the base commit** and **fails on the head commit**,
+> proving the change broke something.
+
+That is not a prompt instruction we hope the model follows. It is a mechanical
+gate. Every candidate is executed on both commits, and anything that does not
+satisfy the rule is thrown away before you ever see it.
+
+Meta published this method in 2026 and reported catching tests being produced at
+roughly **4x the rate of hardening tests**, with assessor agents cutting reviewer
+load by about **70%** ([arXiv 2601.22832](https://arxiv.org/abs/2601.22832)).
+There was no open implementation. This is one.
+
+## Install
+
+```bash
+pip install jittest
+```
+
+**Zero dependencies.** Not "few" — zero. jittest runs inside your CI, and a CI
+tool has no business injecting a dependency tree into your locked project. Diff
+parsing, HTTP, config and the test-runner fallback are all standard library.
+
+## Try it in 60 seconds, with no API key
+
+```bash
+cd your-repo
+jittest doctor                      # can this environment run jittest?
+jittest run --base main --head HEAD --dry-run
+```
+
+`--dry-run` runs the real diff parser, the real risk ranker, the real git
+worktrees and the real oracle, with a stub model. No key, no network, no cost.
+Watch what it targets before you decide whether to hand it a key.
+
+Then, for real:
+
+```bash
+export JITTEST_API_KEY=sk-...
+jittest run --base main --head HEAD --budget 1.00
+```
+
+## In GitHub Actions
+
+```yaml
+name: jittest
+on: pull_request
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  catching-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0            # the oracle needs both commits
+      - uses: Kartik24Hulmukh/jittest@v0
+        with:
+          base: ${{ github.event.pull_request.base.sha }}
+          head: ${{ github.event.pull_request.head.sha }}
+          budget: "1.00"
+          comment: "true"
+        env:
+          JITTEST_API_KEY: ${{ secrets.JITTEST_API_KEY }}
+```
+
+**If jittest proves nothing, it posts nothing.** No "looks good to me", no
+summary of your diff, no comment at all. Silence is the default and it is a
+feature: 2026 surveys put 15–30% of AI review comments in the low-value bucket,
+and the cost of that is not tokens, it is the maintainer who turns the bot off.
+
+## What a finding looks like
+
+> #### 1. `billing/calc.py::apply_discount`
+> **Removing the clamp lets a discount above 100% return a negative price.**
+> > Should a discount over 100% still floor the price at zero?
+>
+> <sub>assessor: likely regression (confidence 0.88, severity high) — risk score
+> 0.71 [consequential_domain, branch_density, modifies_existing]</sub>
+>
+> ```python
+> from billing.calc import apply_discount
+>
+> def test_discount_never_goes_below_zero():
+>     assert apply_discount(100.0, 150.0) == 0.0
+> ```
+>
+> <details><summary>Reproduce locally</summary>
+>
+> ```bash
+> git checkout a1b2c3d4e5f6 && pytest billing/calc.py -q   # expect FAIL
+> git checkout 0f9e8d7c6b5a && pytest billing/calc.py -q   # expect PASS
+> ```
+> </details>
+
+Every claim ships with the command that reproduces it. You should never have to
+take our word for anything.
+
+## How it works
+
+```
+git diff
+  → change targets          functions, not files
+  → risk ranking            the cost gate: only the top N symbols are worth tokens
+  → N candidate tests       the only step a model decides anything
+  → static safety gate      no sockets, no subprocesses, no eval, no assert True
+  → DIFFERENTIAL ORACLE     fails on head ∧ reproduces ∧ passes on base
+  → assessor                proven true — but would a human care?
+  → ledger + report
+```
+
+The oracle is the product. Everything else is replaceable.
+
+| Oracle result | What it means | What jittest does |
+| --- | --- | --- |
+| fails on head, passes on base | the change broke behaviour | **report it** |
+| passes on head | hardening test | discard |
+| fails on both | pre-existing fault | `--latent` only |
+| does not reproduce | flaky | discard |
+| cannot be collected | broken test | discard, retry once |
+
+## Commands
+
+```bash
+jittest run       --base main --head HEAD [--dry-run] [--comment] [--latent]
+jittest doctor    # environment check
+jittest stats     # what the local ledger has learned
+jittest outcome <test-hash> fixed_code|kept_test|intended|false_positive|ignored
+jittest export    corpus.jsonl        # anonymised by default
+```
+
+`jittest outcome` is the important one. It records what a human actually did
+after seeing a finding, which is the only honest measure of precision — and,
+accumulated across repositories, the training signal for a risk model that no
+amount of prompt engineering can substitute for.
+
+## Configuration
+
+Precedence: defaults → `.jittest.toml` or `[tool.jittest]` in `pyproject.toml`
+→ `JITTEST_*` environment variables → CLI flags.
+
+```toml
+[tool.jittest]
+model = "anthropic/claude-sonnet-4-5"
+budget_usd = 1.00          # hard cap, enforced before each request
+max_targets = 5
+candidates_per_target = 4
+risk_threshold = 0.35
+reruns = 2                 # flakiness reruns on head
+ignore = ["legacy/*"]      # added to the built-in defaults
+```
+
+Any OpenAI-compatible endpoint works via `JITTEST_API_BASE`, including local
+Ollama. For the long tail of providers, `pip install jittest[litellm]` and set
+`JITTEST_USE_LITELLM=1`.
+
+## Honest status
+
+This is **v0.2.1, alpha**. What is verified and what is not:
+
+| | Status |
+| --- | --- |
+| Oracle, pipeline, safety gate, ledger, config, CLI | 36 offline tests, green, no network required |
+| Oracle behaviour on a real seeded regression in a real git repo | tested |
+| Catch rate on a public benchmark (BugsInPy) | **not yet measured** — harness in `eval/` |
+| False-positive rate on real PRs | **not yet measured** |
+| Cost per PR in practice | target under $1.00; **not yet measured at scale** |
+
+We will publish those numbers when we have run them, with the methodology and
+the failures included. Until then they are blank rather than optimistic. If you
+see a benchmark claim in this README that is not backed by a script in `eval/`,
+it is a bug — please file it.
+
+## Docs
+
+- [`docs/QUICKSTART.md`](docs/QUICKSTART.md) — five minutes, no key
+- [`docs/DIFFERENTIATION.md`](docs/DIFFERENTIATION.md) — why another one of these
+- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — the pipeline in detail
+- [`docs/PAPERS.md`](docs/PAPERS.md) — the research this implements
+- [`docs/PRIOR-ART.md`](docs/PRIOR-ART.md) — what exists and how this differs
+- [`SECURITY.md`](SECURITY.md) — we execute model-written code; read this
+
+## Citing
+
+If jittest helps your research, cite the method
+([arXiv 2601.22832](https://arxiv.org/abs/2601.22832)) and this implementation
+(see `CITATION.cff`).
+
+## Licence
+
+Apache-2.0.