flakeval 0.1.0__tar.gz → 0.1.2__tar.gz

flakeval-0.1.2/PKG-INFO

@@ -0,0 +1,249 @@
+Metadata-Version: 2.4
+Name: flakeval
+Version: 0.1.2
+Summary: Is your LLM eval improvement real, or just noise? Repeat evals, measure variance, get a verdict.
+Author: Anchit Gupta
+License-Expression: MIT
+Project-URL: Homepage, https://anchitgupta22.github.io/flakeval/
+Project-URL: Repository, https://github.com/AnchitGupta22/flakeval
+Project-URL: Issues, https://github.com/AnchitGupta22/flakeval/issues
+Keywords: llm,evaluation,statistics,machine-learning
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Provides-Extra: examples
+Requires-Dist: datasets>=2.19; extra == "examples"
+Dynamic: license-file
+
+<p align="center">
+    <img src="assets/flakeval_logo.png" alt="Logo" width="240" />
+    <br/>
+    <br/>
+    <a href="LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/License-MIT-yellow.svg" /></a>
+    <a href="https://pypi.org/project/flakeval/"><img alt="PyPI Version" src="https://img.shields.io/pypi/v/flakeval" /></a>
+    <a href="https://pypi.org/project/flakeval/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/flakeval" /></a>
+    <a href="https://github.com/AnchitGupta22/flakeval/actions"><img alt="Tests"
+    src="https://github.com/AnchitGupta22/flakeval/actions/workflows/tests.yml/badge.svg"/></a>
+    <a href="https://anchitgupta22.github.io/flakeval/"><img alt="Documentation" src="https://img.shields.io/badge/Documentation-Read%20the%20docs-red" /></a>
+</p>
+
+---
+
+**Is your AI improvement real, or just luck?**
+
+*(Say it "flake-eval": flaky + evaluation.)*
+
+You change a prompt, a model or a setting, and your test score goes from 72% to 74%. Did it really get better, or would you get 74% just by running the same test again? `flakeval` answers that with a plain verdict.
+
+- Works on results you already have (a simple JSONL file), or runs your evaluation for you.
+- Gives a verdict, a range for the true difference, and the **smallest difference your test can reliably detect**.
+- Tells you whether you need **more runs** or **more test questions**.
+- Exits with an error on a real regression, so it works as a CI check.
+- One dependency (numpy). Runs on a laptop.
+
+**Full documentation, case studies and FAQ: https://anchitgupta22.github.io/flakeval/**
+
+---
+
+## Contents
+
+1. [The idea in plain English](#the-idea-in-plain-english)
+2. [Quick start](#quick-start)
+3. [Case studies (summary)](#case-studies-summary)
+4. [FAQ](#faq)
+5. [How it works](#how-it-works)
+6. [Limitations](#limitations)
+7. [Repository layout](#repository-layout)
+8. [Roadmap](#roadmap)
+
+---
+
+## The idea in plain English
+
+Language models answer with some randomness, and your test only has a limited number of questions. Together this wobble is called **noise**. A 2-point change in score could be a real improvement or just noise. `flakeval` measures the noise so you know which.
+
+| Verdict | Meaning | What to do |
+|---|---|---|
+| **REAL IMPROVEMENT** | B scored higher; chance is an unlikely explanation. | Trust it. |
+| **REAL REGRESSION** | B scored lower; chance is an unlikely explanation. | Investigate before shipping B. |
+| **WITHIN NOISE** | The test cannot tell (this does **not** mean "equal"). | Add more **runs** or more **questions**, as the advice line says. |
+| **IDENTICAL RESULTS** | A and B scored the same on every item. | Check your change actually reached the model. |
+
+Example (simulated: B is truly ~2 points better, 200 items, 5 runs each):
+
+```
+$ flakeval compare a.jsonl b.jsonl
+A: 0.6800 (5 runs)   B: 0.6860 (5 runs)   items: 200
+delta (B - A): +0.0060   95% CI: [-0.0310, +0.0430]   p = 0.7955
+smallest reliably detectable difference: ~0.0535
+VERDICT: WITHIN NOISE
+```
+
+The true difference could be anywhere from 3.1 points worse to 4.3 points better, and this test can only reliably spot gaps of about 5 points, so a 2-point improvement is invisible to it.
+
+---
+
+## Quick start
+
+Requires Python 3.10+.
+
+```bash
+pip install flakeval
+```
+
+### Option A: you already have scores
+
+Write one JSON line per item per run (use at least 3 runs per item, and the same ids in both files):
+
+```json
+{"item_id": "q17", "run": 2, "score": 1.0}
+```
+
+```bash
+flakeval report a.jsonl              # how noisy is one system?
+flakeval compare a.jsonl b.jsonl     # is B really different from A?
+flakeval compare a.jsonl b.jsonl --json   # machine-readable
+```
+
+`flakeval compare` exits with code 1 only on a statistically real regression, so it works as a **CI gate**.
+
+From Python:
+
+```python
+from flakeval import repeat, compare
+
+a = repeat(eval_model_a, items, runs=5, workers=8, save="a.jsonl")
+b = repeat(eval_model_b, items, runs=5, workers=8, save="b.jsonl")
+print(compare(a, b))
+```
+
+If your function uses a sampling seed, make it **different for every run** (`repeat(..., with_run=True)` passes the run number). A fixed seed hides the noise this tool measures.
+
+### Option B: let flakeval run your models
+
+```bash
+flakeval init                  # creates my_eval.py (settings) and sample_data.csv
+python my_eval.py --dry-run    # 2 questions per system, prints answers and scores; saves nothing
+python my_eval.py              # the real run (shows the number of calls and asks first)
+```
+
+Put your questions in a CSV or JSONL (`id, input text, expected answer`), then edit the settings at the top of `my_eval.py`. Example: does a new prompt beat the current one on the same model? The first system is the baseline.
+
+```python
+DATA_PATH = "tickets.csv"
+INPUT_COLUMN = "ticket"
+EXPECTED_COLUMN = "label"
+ID_COLUMN = "id"
+SCORER = "exact"
+
+SYSTEMS = [
+    {"name": "current_prompt", "model": "qwen2.5:3b", "base_url": "http://localhost:11434/v1",
+     "api_key_env": None, "temperature": 0.7, "system_prompt": "",
+     "prompt_template": "Classify this ticket as billing, bug or question.\n\nTicket: {input}"},
+    {"name": "new_prompt", "model": "qwen2.5:3b", "base_url": "http://localhost:11434/v1",
+     "api_key_env": None, "temperature": 0.7, "system_prompt": "",
+     "prompt_template": "Reply with exactly one label: billing, bug or question.\n\nTicket: {input}"},
+]
+
+RUNS = 5          # 3 to 5 is a good range
+MAX_TOKENS = 20
+```
+
+- **Models:** anything with an OpenAI-compatible endpoint (Ollama, LM Studio, vLLM, hosted APIs). Put the *name* of the environment variable holding your key in `api_key_env`, never the key itself. For anything else, pass your own function as `"call": my_function`.
+- **Built-in scorers:** `exact`, `contains`, `number`, `choice_letter`, `answer_line`, `final_number`, or `"custom"` (your own `custom_score(output, expected, row)` returning 0 to 1).
+- **Overrides:** `--max-items 10`, `--runs 2`, `--workers 4`, `--out-dir DIR`, `--scorer NAME`. Try `--max-items 10 --runs 2 --out-dir trial` for a cheap rehearsal.
+- **Output:** a printed report plus `my_results/` with scores, raw answers (`<system>.outputs.jsonl`) and `report.txt`. The baseline is run twice (A/A check) by default, showing how much two identical runs differ by chance.
+- **Safe to rerun:** finished systems are skipped, and a server failure stops the run instead of recording fake zeros.
+- **Fixed your scorer?** `python my_eval.py --rescore` re-marks saved answers without calling any model.
+- **Pure Python, no settings file:** use `default_config()` and `run_study(cfg)` from `flakeval.study`.
+
+**Read some raw answers that scored 0 before trusting the numbers.** A scorer that wrongly marks good answers down looks exactly like model noise.
+
+**Privacy:** nothing is uploaded. Rows go only to the endpoints you list. If your data is confidential, add `my_results/` to `.gitignore`.
+
+### Option C: clone the repo (tests, demo, case studies)
+
+```bash
+git clone https://github.com/AnchitGupta22/flakeval && cd flakeval
+pip install -e ".[dev,examples]"
+python -m pytest
+cd examples && python make_demo.py && flakeval compare a.jsonl b.jsonl
+```
+
+If `flakeval` is not found, use `python -m flakeval.cli compare a.jsonl b.jsonl`.
+
+---
+
+## Case studies (summary)
+
+Two worked examples live in `examples/template_case_studies/` (`arc_eval.py`, `gsm8k_eval.py`), run on `qwen2.5:3b` and `llama3.2:3b` with Ollama. Full tables, methodology and diagnostics are in the [documentation](https://anchitgupta22.github.io/flakeval/).
+
+- **ARC (science multiple choice, 200 items x 5 runs):** the same model twice scored 82.6% and 82.9% (within noise). `llama3.2:3b` scored 11.7 points lower than `qwen2.5:3b` (p < 0.001), flagged as real.
+- **GSM8K (math reasoning, 200 items x 3 runs):** under strict scoring, the same system twice differed by 4.8 points (p = 0.024). Five extra repeats ranged from 69.7% to 74.5%, so this was chance. A "double-check" prompt result of -4.7 points fell inside that natural spread, so no conclusion could be drawn.
+
+Takeaways:
+
+1. Always include the "same system twice" check; it shows how big a gap chance alone can produce.
+2. Reasoning tasks are far noisier than single-letter answers (32.5% of GSM8K items changed score across 3 runs vs 0.5% on ARC).
+3. Check the "smallest detectable difference" **before** you start. If it is bigger than the improvement you hope for, your test is too small.
+4. Treat borderline p-values (about 0.01 to 0.05) with caution, and always report how you scored the answers.
+
+---
+
+## FAQ
+
+**Isn't it all just chance?** Yes, partly, and `flakeval` cannot remove that. It **measures how much chance is in your result**, like a margin of error in a poll, and tells you whether more runs or more questions would shrink it. Halving the smallest detectable difference takes about four times the data.
+
+**Why not just set temperature to 0?** That removes most model randomness but not the sampling of your questions, and real users do not run at temperature 0.
+
+**How many questions and runs do I need?** Look at the "smallest detectable difference" in the report. A reasonable start is 100 or more questions and 3 to 5 runs.
+
+---
+
+## How it works
+
+Scores form an items x runs matrix, and comparisons are paired by item id. The 95% interval comes from a bootstrap that resamples **items** (run noise is already inside each item's mean). The p-value comes from a paired sign-flip permutation test. A verdict of "real" requires the interval to exclude zero **and** p < 0.05. The smallest detectable difference is about 2.8 x the standard error (80% power, alpha 0.05).
+
+In simulation (60 items, 5 runs), two identical systems were called "different" 5.3% of the time (target 5%), and a true 10-point gap was detected 71% of the time.
+
+**Pitfall guarded against:** many servers use a fixed default seed (Ollama's is 0), so "repeated" runs are identical and show zero noise. The template sends a different seed per question and run, and `flakeval` warns when no item changes score across runs.
+
+## Limitations
+
+- Assumes your items are a random sample of the questions you care about.
+- Compares two systems at a time; no multiple-comparison correction yet.
+- Scores are treated as plain numbers; an LLM judge's own randomness is not modeled separately.
+- p-values assume runs of one configuration are interchangeable.
+- The case studies use one benchmark each, two small models and one machine. They illustrate the method; they are not a model ranking.
+
+## Repository layout
+
+```
+src/flakeval/          the installable package (stats, io, cli, study, scoring, models, progress, templates)
+examples/              demo data and the two case studies (not part of the installed package)
+tests/                 pytest suite
+```
+
+## Roadmap
+
+- [ ] Importers for lm-evaluation-harness, Inspect, and promptfoo outputs
+- [ ] HTML report
+- [ ] Adaptive mode: keep sampling until the verdict is decided
+- [ ] LLM-judge variance and multi-system comparisons with correction
+
+## License
+
+MIT. See [LICENSE](LICENSE).
+
+## Citation
+
+If `flakeval` is useful in your work, please cite it (see [`CITATION.cff`](CITATION.cff)). **Author:** Anchit Gupta

flakeval-0.1.2/README.md

@@ -0,0 +1,223 @@
+<p align="center">
+    <img src="assets/flakeval_logo.png" alt="Logo" width="240" />
+    <br/>
+    <br/>
+    <a href="LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/License-MIT-yellow.svg" /></a>
+    <a href="https://pypi.org/project/flakeval/"><img alt="PyPI Version" src="https://img.shields.io/pypi/v/flakeval" /></a>
+    <a href="https://pypi.org/project/flakeval/"><img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/flakeval" /></a>
+    <a href="https://github.com/AnchitGupta22/flakeval/actions"><img alt="Tests"
+    src="https://github.com/AnchitGupta22/flakeval/actions/workflows/tests.yml/badge.svg"/></a>
+    <a href="https://anchitgupta22.github.io/flakeval/"><img alt="Documentation" src="https://img.shields.io/badge/Documentation-Read%20the%20docs-red" /></a>
+</p>
+
+---
+
+**Is your AI improvement real, or just luck?**
+
+*(Say it "flake-eval": flaky + evaluation.)*
+
+You change a prompt, a model or a setting, and your test score goes from 72% to 74%. Did it really get better, or would you get 74% just by running the same test again? `flakeval` answers that with a plain verdict.
+
+- Works on results you already have (a simple JSONL file), or runs your evaluation for you.
+- Gives a verdict, a range for the true difference, and the **smallest difference your test can reliably detect**.
+- Tells you whether you need **more runs** or **more test questions**.
+- Exits with an error on a real regression, so it works as a CI check.
+- One dependency (numpy). Runs on a laptop.
+
+**Full documentation, case studies and FAQ: https://anchitgupta22.github.io/flakeval/**
+
+---
+
+## Contents
+
+1. [The idea in plain English](#the-idea-in-plain-english)
+2. [Quick start](#quick-start)
+3. [Case studies (summary)](#case-studies-summary)
+4. [FAQ](#faq)
+5. [How it works](#how-it-works)
+6. [Limitations](#limitations)
+7. [Repository layout](#repository-layout)
+8. [Roadmap](#roadmap)
+
+---
+
+## The idea in plain English
+
+Language models answer with some randomness, and your test only has a limited number of questions. Together this wobble is called **noise**. A 2-point change in score could be a real improvement or just noise. `flakeval` measures the noise so you know which.
+
+| Verdict | Meaning | What to do |
+|---|---|---|
+| **REAL IMPROVEMENT** | B scored higher; chance is an unlikely explanation. | Trust it. |
+| **REAL REGRESSION** | B scored lower; chance is an unlikely explanation. | Investigate before shipping B. |
+| **WITHIN NOISE** | The test cannot tell (this does **not** mean "equal"). | Add more **runs** or more **questions**, as the advice line says. |
+| **IDENTICAL RESULTS** | A and B scored the same on every item. | Check your change actually reached the model. |
+
+Example (simulated: B is truly ~2 points better, 200 items, 5 runs each):
+
+```
+$ flakeval compare a.jsonl b.jsonl
+A: 0.6800 (5 runs)   B: 0.6860 (5 runs)   items: 200
+delta (B - A): +0.0060   95% CI: [-0.0310, +0.0430]   p = 0.7955
+smallest reliably detectable difference: ~0.0535
+VERDICT: WITHIN NOISE
+```
+
+The true difference could be anywhere from 3.1 points worse to 4.3 points better, and this test can only reliably spot gaps of about 5 points, so a 2-point improvement is invisible to it.
+
+---
+
+## Quick start
+
+Requires Python 3.10+.
+
+```bash
+pip install flakeval
+```
+
+### Option A: you already have scores
+
+Write one JSON line per item per run (use at least 3 runs per item, and the same ids in both files):
+
+```json
+{"item_id": "q17", "run": 2, "score": 1.0}
+```
+
+```bash
+flakeval report a.jsonl              # how noisy is one system?
+flakeval compare a.jsonl b.jsonl     # is B really different from A?
+flakeval compare a.jsonl b.jsonl --json   # machine-readable
+```
+
+`flakeval compare` exits with code 1 only on a statistically real regression, so it works as a **CI gate**.
+
+From Python:
+
+```python
+from flakeval import repeat, compare
+
+a = repeat(eval_model_a, items, runs=5, workers=8, save="a.jsonl")
+b = repeat(eval_model_b, items, runs=5, workers=8, save="b.jsonl")
+print(compare(a, b))
+```
+
+If your function uses a sampling seed, make it **different for every run** (`repeat(..., with_run=True)` passes the run number). A fixed seed hides the noise this tool measures.
+
+### Option B: let flakeval run your models
+
+```bash
+flakeval init                  # creates my_eval.py (settings) and sample_data.csv
+python my_eval.py --dry-run    # 2 questions per system, prints answers and scores; saves nothing
+python my_eval.py              # the real run (shows the number of calls and asks first)
+```
+
+Put your questions in a CSV or JSONL (`id, input text, expected answer`), then edit the settings at the top of `my_eval.py`. Example: does a new prompt beat the current one on the same model? The first system is the baseline.
+
+```python
+DATA_PATH = "tickets.csv"
+INPUT_COLUMN = "ticket"
+EXPECTED_COLUMN = "label"
+ID_COLUMN = "id"
+SCORER = "exact"
+
+SYSTEMS = [
+    {"name": "current_prompt", "model": "qwen2.5:3b", "base_url": "http://localhost:11434/v1",
+     "api_key_env": None, "temperature": 0.7, "system_prompt": "",
+     "prompt_template": "Classify this ticket as billing, bug or question.\n\nTicket: {input}"},
+    {"name": "new_prompt", "model": "qwen2.5:3b", "base_url": "http://localhost:11434/v1",
+     "api_key_env": None, "temperature": 0.7, "system_prompt": "",
+     "prompt_template": "Reply with exactly one label: billing, bug or question.\n\nTicket: {input}"},
+]
+
+RUNS = 5          # 3 to 5 is a good range
+MAX_TOKENS = 20
+```
+
+- **Models:** anything with an OpenAI-compatible endpoint (Ollama, LM Studio, vLLM, hosted APIs). Put the *name* of the environment variable holding your key in `api_key_env`, never the key itself. For anything else, pass your own function as `"call": my_function`.
+- **Built-in scorers:** `exact`, `contains`, `number`, `choice_letter`, `answer_line`, `final_number`, or `"custom"` (your own `custom_score(output, expected, row)` returning 0 to 1).
+- **Overrides:** `--max-items 10`, `--runs 2`, `--workers 4`, `--out-dir DIR`, `--scorer NAME`. Try `--max-items 10 --runs 2 --out-dir trial` for a cheap rehearsal.
+- **Output:** a printed report plus `my_results/` with scores, raw answers (`<system>.outputs.jsonl`) and `report.txt`. The baseline is run twice (A/A check) by default, showing how much two identical runs differ by chance.
+- **Safe to rerun:** finished systems are skipped, and a server failure stops the run instead of recording fake zeros.
+- **Fixed your scorer?** `python my_eval.py --rescore` re-marks saved answers without calling any model.
+- **Pure Python, no settings file:** use `default_config()` and `run_study(cfg)` from `flakeval.study`.
+
+**Read some raw answers that scored 0 before trusting the numbers.** A scorer that wrongly marks good answers down looks exactly like model noise.
+
+**Privacy:** nothing is uploaded. Rows go only to the endpoints you list. If your data is confidential, add `my_results/` to `.gitignore`.
+
+### Option C: clone the repo (tests, demo, case studies)
+
+```bash
+git clone https://github.com/AnchitGupta22/flakeval && cd flakeval
+pip install -e ".[dev,examples]"
+python -m pytest
+cd examples && python make_demo.py && flakeval compare a.jsonl b.jsonl
+```
+
+If `flakeval` is not found, use `python -m flakeval.cli compare a.jsonl b.jsonl`.
+
+---
+
+## Case studies (summary)
+
+Two worked examples live in `examples/template_case_studies/` (`arc_eval.py`, `gsm8k_eval.py`), run on `qwen2.5:3b` and `llama3.2:3b` with Ollama. Full tables, methodology and diagnostics are in the [documentation](https://anchitgupta22.github.io/flakeval/).
+
+- **ARC (science multiple choice, 200 items x 5 runs):** the same model twice scored 82.6% and 82.9% (within noise). `llama3.2:3b` scored 11.7 points lower than `qwen2.5:3b` (p < 0.001), flagged as real.
+- **GSM8K (math reasoning, 200 items x 3 runs):** under strict scoring, the same system twice differed by 4.8 points (p = 0.024). Five extra repeats ranged from 69.7% to 74.5%, so this was chance. A "double-check" prompt result of -4.7 points fell inside that natural spread, so no conclusion could be drawn.
+
+Takeaways:
+
+1. Always include the "same system twice" check; it shows how big a gap chance alone can produce.
+2. Reasoning tasks are far noisier than single-letter answers (32.5% of GSM8K items changed score across 3 runs vs 0.5% on ARC).
+3. Check the "smallest detectable difference" **before** you start. If it is bigger than the improvement you hope for, your test is too small.
+4. Treat borderline p-values (about 0.01 to 0.05) with caution, and always report how you scored the answers.
+
+---
+
+## FAQ
+
+**Isn't it all just chance?** Yes, partly, and `flakeval` cannot remove that. It **measures how much chance is in your result**, like a margin of error in a poll, and tells you whether more runs or more questions would shrink it. Halving the smallest detectable difference takes about four times the data.
+
+**Why not just set temperature to 0?** That removes most model randomness but not the sampling of your questions, and real users do not run at temperature 0.
+
+**How many questions and runs do I need?** Look at the "smallest detectable difference" in the report. A reasonable start is 100 or more questions and 3 to 5 runs.
+
+---
+
+## How it works
+
+Scores form an items x runs matrix, and comparisons are paired by item id. The 95% interval comes from a bootstrap that resamples **items** (run noise is already inside each item's mean). The p-value comes from a paired sign-flip permutation test. A verdict of "real" requires the interval to exclude zero **and** p < 0.05. The smallest detectable difference is about 2.8 x the standard error (80% power, alpha 0.05).
+
+In simulation (60 items, 5 runs), two identical systems were called "different" 5.3% of the time (target 5%), and a true 10-point gap was detected 71% of the time.
+
+**Pitfall guarded against:** many servers use a fixed default seed (Ollama's is 0), so "repeated" runs are identical and show zero noise. The template sends a different seed per question and run, and `flakeval` warns when no item changes score across runs.
+
+## Limitations
+
+- Assumes your items are a random sample of the questions you care about.
+- Compares two systems at a time; no multiple-comparison correction yet.
+- Scores are treated as plain numbers; an LLM judge's own randomness is not modeled separately.
+- p-values assume runs of one configuration are interchangeable.
+- The case studies use one benchmark each, two small models and one machine. They illustrate the method; they are not a model ranking.
+
+## Repository layout
+
+```
+src/flakeval/          the installable package (stats, io, cli, study, scoring, models, progress, templates)
+examples/              demo data and the two case studies (not part of the installed package)
+tests/                 pytest suite
+```
+
+## Roadmap
+
+- [ ] Importers for lm-evaluation-harness, Inspect, and promptfoo outputs
+- [ ] HTML report
+- [ ] Adaptive mode: keep sampling until the verdict is decided
+- [ ] LLM-judge variance and multi-system comparisons with correction
+
+## License
+
+MIT. See [LICENSE](LICENSE).
+
+## Citation
+
+If `flakeval` is useful in your work, please cite it (see [`CITATION.cff`](CITATION.cff)). **Author:** Anchit Gupta

{flakeval-0.1.0 → flakeval-0.1.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "flakeval"
-version = "0.1.0"
+version = "0.1.2"
 description = "Is your LLM eval improvement real, or just noise? Repeat evals, measure variance, get a verdict."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -20,6 +20,19 @@ keywords = [
     "statistics",
     "machine-learning"
 ]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Operating System :: OS Independent"
+]
+
+[project.urls]
+Homepage = "https://anchitgupta22.github.io/flakeval/"
+Repository = "https://github.com/AnchitGupta22/flakeval"
+Issues = "https://github.com/AnchitGupta22/flakeval/issues"
 
 [project.optional-dependencies]
 dev = ["pytest"]
@@ -32,4 +45,4 @@ flakeval = "flakeval.cli:main"
 where = ["src"]
 
 [tool.setuptools.package-data]
-flakeval = ["templates/*"]
+flakeval = ["templates/*"]

flakeval-0.1.2/src/flakeval/__init__.py

@@ -0,0 +1,20 @@
+from importlib.metadata import version, PackageNotFoundError
+
+from .io import load_jsonl, repeat, save_jsonl
+from .stats import Comparison, Scores, Summary, compare, summarize
+
+__all__ = [
+    "Scores",
+    "Summary",
+    "Comparison",
+    "summarize",
+    "compare",
+    "repeat",
+    "load_jsonl",
+    "save_jsonl",
+]
+
+try:
+    __version__ = version("flakeval")
+except PackageNotFoundError:
+    __version__ = "0.0.0"