slm-eval 0.0.1__tar.gz

slm_eval-0.0.1/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 EleutherAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

slm_eval-0.0.1/LICENSE_NOTES.md

@@ -0,0 +1,118 @@
+# Licence notes for the supported benchmarks
+
+`slm_eval` ships **task definitions and scoring code only**. It does not vendor
+any benchmark dataset. Each task downloads its dataset at run time from the
+publisher's own distribution point, so the publisher's licence applies to the
+data and this project's MIT licence applies to the code.
+
+Read this file before you use `slm_eval` for anything beyond internal research.
+
+Metadata below was read from `https://huggingface.co/api/datasets/<id>` on
+**2026-09-20**.
+
+## Rule for this project
+
+A benchmark is only supported if it is 100% usable commercially. That means: no
+NonCommercial clause, no NoDerivatives clause. Attribution requirements and
+ShareAlike requirements are accepted, and this repository meets them. If a
+benchmark's licence cannot be verified, the benchmark is not shipped.
+
+## Summary
+
+| Benchmark | Task name | Dataset licence | Safe for commercial use? |
+|---|---|---|---|
+| HellaSwag | `hellaswag` | MIT (see note 1) | Yes |
+| ARC Easy / ARC Challenge | `arc_easy`, `arc_challenge` | CC-BY-SA-4.0 | Yes, with attribution (note 2) |
+| ArithMark 3.0 | `arithmark_3_0` | Apache-2.0 | Yes |
+| ArithMark 2.0 | `arithmark_2` | Apache-2.0 | Yes |
+| MATH-500 | `math_500` | MIT (see note 3) | Yes |
+| GPQA Diamond | `gpqa_diamond` | CC-BY-4.0 | Yes, with attribution (note 4) |
+| BFCL v4 | `bfcl_v4` | Apache-2.0 | Yes (note 6) |
+| Int Index | `int_index` | derived from the above | Yes |
+
+## Notes
+
+**1. HellaSwag.** The dataset metadata field is empty, which is why it usually
+shows up as "unknown". The dataset card body states "Licensing Information: MIT",
+and the accompanying GitHub repository (`rowanz/hellaswag`) carries an MIT
+licence file. Treated as MIT. No attribution is required, but the source is
+credited in `NOTICE` anyway.
+
+**2. ARC (AI2 Reasoning Challenge).** The card metadata is
+`['cc-by-sa-4.0']`. CC-BY-SA-4.0 permits commercial use. Two obligations apply
+to the *data*: attribution, and ShareAlike on redistribution of a modified copy.
+This project redistributes nothing: the task downloads `allenai/ai2_arc` at run
+time and stores it in the HuggingFace cache. If you republish the ARC rows
+yourself, you must carry the attribution and the same licence forward. The
+attribution line is in `NOTICE`.
+
+**3. MATH-500.** The dataset card metadata field is empty. The dataset is
+published by HuggingFace under an MIT licence, and it is a 500-problem subset of
+the MATH dataset (`hendrycks/competition_math`, MIT). Treated as MIT.
+
+**4. GPQA Diamond.** Card metadata is `cc-by-4.0`, and the repository is gated
+(`gated: auto`). CC-BY-4.0 permits commercial use and requires attribution. The
+gating is about access, not about commercial terms: you must accept the terms on
+Hugging Face and authenticate to load it. This task therefore tries the gated
+source first and falls back to the open mirror `fingertap/GPQA-Diamond`, logging
+which source it used. Attribution is in `NOTICE`.
+
+**5. BFCL v4.** The task reads the data that the `bfcl-eval` Python package
+ships, which comes from the `ShishirPatil/gorilla` repository under
+Apache-2.0. The Apache-2.0 licence covers the Berkeley Function Calling
+Leaderboard data and code.
+
+## Benchmarks that were removed
+
+Each of these was checked and then removed, either because the licence fails the
+rule above or because the benchmark is impractical for a CPU-only SLM suite.
+
+**GSM-Symbolic (`apple/ml-gsm-symbolic`) — CC-BY-NC-ND-4.0.** NonCommercial and
+NoDerivatives. Disqualified under every licence bar. Removed.
+
+**AgentIF (`THU-KEG/AgentIF`) — CC-BY-NC-4.0.** NonCommercial. Removed.
+
+**WildBench (`allenai/WildBench`).** Two problems. The benchmark is scored by an
+LLM judge, so results are subjective, judge-biased, and billed per sample. The
+underlying WildChat conversations are also released under AI2's ImpACT License
+(Low Risk Artifacts), which is not an OSI licence and adds use restrictions.
+Removed as too subjective and licence-encumbered.
+
+**BrowseComp (`openai/BrowseCompLongContext`).** MIT licensed, but the benchmark
+needs a live browsing agent and multi-step web search, which this suite cannot
+provide. Removed as too complex.
+
+**SWE-bench Lite (`princeton-nlp/SWE-bench_Lite`).** MIT licensed, but every
+sample needs a full repository checkout and a Docker sandbox. Removed as too
+heavy for a CPU-only suite.
+
+**PIQA (`ybisk/piqa`) — licence unverifiable.** The card metadata is
+`['unknown']`, and the card body states no licence. Commercial use therefore
+cannot be verified, so PIQA is not shipped. This has one visible consequence:
+PIQA is a published component of the Int Index. The published method drops an
+unavailable component from both the numerator and the denominator, so the Int
+Index here is computed as `(HellaSwag + ARC + 0.65 x ArithMark-3) / 2.65`.
+
+**LiveCodeBench (`livecodebench/code_generation_lite`) — licence
+unverifiable.** The dataset card metadata is the incomplete SPDX value `cc`,
+which names no licence variant, so it covers CC-BY, CC-BY-NC and CC-BY-ND
+alike. The repository that publishes the data
+(`LiveCodeBench/LiveCodeBench`) is MIT, but that licence covers the software, and
+the problems themselves are collected from LeetCode, AtCoder and Codeforces. No
+grant that names a commercial-use-safe licence therefore covers the problem
+data. Removed under the suite rule. This is the one removal that costs a whole
+capability: the suite no longer ships a coding benchmark. To restore it, ask the
+LiveCodeBench maintainers to state the dataset licence explicitly.
+
+**BananaMind Base Bench 1.1
+(`BananaMind/BananaMind-Base-Bench-1.1`) — licence unverifiable.** The card
+metadata declares no licence and the repository is gated. Excluded until the
+authors declare a licence. (The author's Hugging Face account is `Monster-Code`,
+the same person who maintains this suite, so the exclusion can be lifted from
+this side at any time by declaring a licence on the dataset card.)
+
+## Engine licence
+
+The evaluation engine is the MIT-licensed LM Evaluation Harness by EleutherAI.
+`LICENSE.md` retains that copyright notice, as the MIT licence requires.
+`NOTICE` records the attribution.

slm_eval-0.0.1/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include tests

slm_eval-0.0.1/NOTICE

@@ -0,0 +1,56 @@
+slm_eval
+========
+
+An evaluation suite for small language models, maintained by TheLimeDev
+(GitHub: https://github.com/TheLimeDev, Hugging Face: https://huggingface.co/Monster-Code)
+
+This product includes software developed by EleutherAI
+(https://www.eleuther.ai/): the evaluation engine is derived from the
+MIT-licensed LM Evaluation Harness. The MIT copyright notice is retained in
+LICENSE.md. The task definitions for that engine were removed and replaced by
+the benchmark tasks shipped here.
+
+
+Benchmark data attribution
+--------------------------
+
+slm_eval ships task definitions and scoring code only. No benchmark dataset is
+redistributed with this product: each task downloads its dataset at run time
+from the publisher's own distribution point.
+
+The following benchmark datasets require attribution, and are used under the
+licence stated:
+
+* ARC (AI2 Reasoning Challenge) -- tasks `arc_easy`, `arc_challenge`
+  Dataset: allenai/ai2_arc
+  Copyright Allen Institute for AI. Licensed under CC-BY-SA-4.0.
+  License text: https://creativecommons.org/licenses/by-sa/4.0/
+  Citation: Clark et al., "Think you have Solved Question Answering? Try ARC,
+  the AI2 Reasoning Challenge", arXiv:1803.05457.
+
+* GPQA Diamond -- task `gpqa_diamond`
+  Dataset: Idavidrein/gpqa, with open mirror fingertap/GPQA-Diamond
+  Copyright the GPQA authors. Licensed under CC-BY-4.0.
+  License text: https://creativecommons.org/licenses/by/4.0/
+  Citation: Rein et al., "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
+  arXiv:2311.12022.
+
+The remaining datasets are used under permissive licences and are credited here
+for completeness:
+
+* HellaSwag -- task `hellaswag` (MIT). Dataset: Rowan/hellaswag.
+  Citation: Zellers et al., "HellaSwag: Can a Machine Really Finish Your
+  Sentence?", arXiv:1905.07830.
+* ArithMark 3.0 / ArithMark 2.0 -- tasks `arithmark_3_0`, `arithmark_2`
+  (Apache-2.0). Datasets: AxiomicLabs/Arithmark-3.0,
+  AxiomicLabs/ArithMark-2.0. Copyright Axiomic Labs.
+* MATH-500 -- task `math_500` (MIT). Dataset: HuggingFaceH4/MATH-500.
+  A 500-problem subset of the MATH dataset by Hendrycks et al.,
+  "Measuring Mathematical Problem Solving With the MATH Dataset",
+  arXiv:2103.03874.
+* BFCL v4 -- task `bfcl_v4` (Apache-2.0). Data shipped by the `bfcl-eval`
+  package, from the ShishirPatil/gorilla repository. Copyright the Gorilla
+  project authors.
+* Int Index -- task group `int_index`. Not a dataset. The composite formula is
+  published by Axiomic Labs on the Open SLM Leaderboard:
+  https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard

slm_eval-0.0.1/PKG-INFO

@@ -0,0 +1,328 @@
+Metadata-Version: 2.4
+Name: slm_eval
+Version: 0.0.1
+Summary: An evaluation suite for small language models (SLMs) with objective, reproducible benchmarks
+Author: TheLimeDev (@Monster-Code)
+Maintainer: TheLimeDev (@Monster-Code)
+License: MIT
+Project-URL: Homepage, https://github.com/TheLimeDev/SLM-Eval
+Project-URL: Repository, https://github.com/TheLimeDev/SLM-Eval
+Project-URL: Author, https://github.com/TheLimeDev
+Project-URL: Hugging-Face, https://huggingface.co/Monster-Code
+Project-URL: Documentation, https://github.com/TheLimeDev/SLM-Eval/blob/main/docs/README.md
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+License-File: LICENSE_NOTES.md
+License-File: NOTICE
+Requires-Dist: datasets>=2.16.0
+Requires-Dist: numpy
+Requires-Dist: evaluate>=0.4.0
+Requires-Dist: jinja2
+Requires-Dist: pytablewriter
+Requires-Dist: rouge-score>=0.0.4
+Requires-Dist: sacrebleu>=1.5.0
+Requires-Dist: scikit-learn>=0.24.1
+Requires-Dist: sqlitedict
+Requires-Dist: dill
+Requires-Dist: word2number
+Requires-Dist: more_itertools
+Requires-Dist: typing_extensions
+Requires-Dist: tqdm>=4.59.0
+Provides-Extra: api
+Requires-Dist: requests; extra == "api"
+Requires-Dist: aiohttp; extra == "api"
+Requires-Dist: tenacity; extra == "api"
+Requires-Dist: tqdm; extra == "api"
+Requires-Dist: tiktoken; extra == "api"
+Provides-Extra: archiver
+Requires-Dist: jsonlines; extra == "archiver"
+Requires-Dist: zstandard; extra == "archiver"
+Provides-Extra: hf
+Requires-Dist: transformers>=4.1; extra == "hf"
+Requires-Dist: torch>=1.8; extra == "hf"
+Requires-Dist: accelerate>=0.26.0; extra == "hf"
+Requires-Dist: peft>=0.2.0; extra == "hf"
+Provides-Extra: vllm
+Requires-Dist: vllm>=0.18; extra == "vllm"
+Provides-Extra: gptq
+Requires-Dist: auto-gptq[triton]>=0.6.0; extra == "gptq"
+Provides-Extra: gptqmodel
+Requires-Dist: gptqmodel>=1.0.9; extra == "gptqmodel"
+Provides-Extra: ipex
+Requires-Dist: optimum-intel; extra == "ipex"
+Provides-Extra: ibm-watsonx-ai
+Requires-Dist: ibm_watsonx_ai>=1.1.22; extra == "ibm-watsonx-ai"
+Requires-Dist: python-dotenv; extra == "ibm-watsonx-ai"
+Provides-Extra: onnxruntime-genai
+Requires-Dist: onnxruntime-genai; extra == "onnxruntime-genai"
+Requires-Dist: transformers; extra == "onnxruntime-genai"
+Requires-Dist: numpy; extra == "onnxruntime-genai"
+Provides-Extra: onnxruntime
+Requires-Dist: onnxruntime>=1.23; extra == "onnxruntime"
+Requires-Dist: transformers; extra == "onnxruntime"
+Requires-Dist: numpy; extra == "onnxruntime"
+Provides-Extra: litellm
+Requires-Dist: litellm<1.85,>=1.60; extra == "litellm"
+Requires-Dist: aiohttp; extra == "litellm"
+Requires-Dist: requests; extra == "litellm"
+Requires-Dist: tenacity; extra == "litellm"
+Requires-Dist: tqdm; extra == "litellm"
+Provides-Extra: optimum
+Requires-Dist: optimum[openvino]; extra == "optimum"
+Provides-Extra: habana
+Requires-Dist: optimum-habana; extra == "habana"
+Provides-Extra: sparsify
+Requires-Dist: sparsify; extra == "sparsify"
+Provides-Extra: dev
+Requires-Dist: pytest>=9.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-xdist; extra == "dev"
+Requires-Dist: requests; extra == "dev"
+Requires-Dist: aiohttp; extra == "dev"
+Requires-Dist: tenacity; extra == "dev"
+Requires-Dist: tqdm; extra == "dev"
+Requires-Dist: tiktoken; extra == "dev"
+Requires-Dist: sentencepiece; extra == "dev"
+Requires-Dist: pillow; extra == "dev"
+Provides-Extra: ifeval
+Requires-Dist: langdetect; extra == "ifeval"
+Requires-Dist: immutabledict; extra == "ifeval"
+Requires-Dist: nltk>=3.9.1; extra == "ifeval"
+Provides-Extra: math
+Requires-Dist: sympy>=1.12; extra == "math"
+Requires-Dist: antlr4-python3-runtime==4.11; extra == "math"
+Requires-Dist: math_verify[antlr4_11_0]; extra == "math"
+Provides-Extra: benchmarks
+Requires-Dist: slm_eval[math]; extra == "benchmarks"
+Requires-Dist: slm_eval[ifeval]; extra == "benchmarks"
+Requires-Dist: slm_eval[api]; extra == "benchmarks"
+Provides-Extra: sentencepiece
+Requires-Dist: sentencepiece>=0.1.98; extra == "sentencepiece"
+Provides-Extra: trackio
+Requires-Dist: trackio>=0.23.0; extra == "trackio"
+Provides-Extra: wandb
+Requires-Dist: wandb>=0.16.3; extra == "wandb"
+Requires-Dist: pandas; extra == "wandb"
+Requires-Dist: numpy; extra == "wandb"
+Provides-Extra: zeno
+Requires-Dist: pandas; extra == "zeno"
+Requires-Dist: zeno-client; extra == "zeno"
+Dynamic: license-file
+
+# slm_eval
+
+An evaluation suite for **small language models (SLMs)**. It ships a small,
+hand-picked set of benchmarks, each chosen because it can be scored objectively,
+reproducibly, and on a single CPU machine.
+
+Maintained by [TheLimeDev](https://github.com/TheLimeDev)
+(Hugging Face: [@Monster-Code](https://huggingface.co/Monster-Code)).
+
+The evaluation engine, model backends, filters and metrics come from the
+MIT-licensed LM Evaluation Harness by EleutherAI. All ~14,000 of its task
+definitions were removed. Only the benchmarks below remain, so anything you
+already know about running the engine still applies — the task registry is
+different.
+
+## The supported benchmarks
+
+| Task name | Benchmark | What it measures | Data source | Licence |
+|---|---|---|---|---|
+| `hellaswag` | HellaSwag | Commonsense continuation of a described activity | `Rowan/hellaswag` | MIT |
+| `arc_easy` | ARC Easy | Grade-school science questions | `allenai/ai2_arc` (`ARC-Easy`) | CC-BY-SA-4.0 |
+| `arc_challenge` | ARC Challenge | Harder science questions that retrieval alone does not solve | `allenai/ai2_arc` (`ARC-Challenge`) | CC-BY-SA-4.0 |
+| `arithmark_3_0` | ArithMark 3.0 | Arithmetic continuation, resistant to memorisation | `AxiomicLabs/Arithmark-3.0` | Apache-2.0 |
+| `arithmark_2` | ArithMark 2.0 | The predecessor of ArithMark 3.0 | `AxiomicLabs/ArithMark-2.0` | Apache-2.0 |
+| `math_500` | MATH-500 | Competition mathematics needing multi-step reasoning | `HuggingFaceH4/MATH-500` | MIT |
+| `gpqa_diamond` | GPQA Diamond | Graduate-level "Google-proof" science questions | `Idavidrein/gpqa`, mirror `fingertap/GPQA-Diamond` | CC-BY-4.0 |
+| `bfcl_v4` | BFCL v4 | Function calling and JSON-schema adherence | `bfcl-eval` package data | Apache-2.0 |
+| `int_index` | Int Index | A composite score over HellaSwag, ARC and ArithMark-3 | derived, not a dataset | n/a |
+
+`int_index` is a group, not a dataset: it evaluates its component benchmarks and
+then combines them with the published formula. See
+`slm_eval/tasks/slm/int_index/README.md`.
+
+### Every licence here is usable commercially
+
+No benchmark in this suite is NonCommercial (`NC`) or NoDerivatives (`ND`), and
+every benchmark rests on a licence value that names a specific
+commercial-use-safe licence. Two benchmarks carry obligations, and both are met
+by this repository:
+
+- **ARC** is CC-BY-SA-4.0. Attribution is required, and redistributing a
+  modified copy of the ARC data must stay under CC-BY-SA-4.0. This repository
+  never redistributes the data: the task downloads it from Hugging Face at run
+  time. The attribution is in `NOTICE` and in the task README.
+- **GPQA** is CC-BY-4.0. Attribution is required. The public GPQA repository is
+  gated, so a Hugging Face token helps; without one the task falls back to an
+  open mirror and says so in the log. Attribution is in `NOTICE`.
+
+Full research, with sources checked against the live dataset metadata, is in
+`LICENSE_NOTES.md`.
+
+### Benchmarks that were removed, and why
+
+| Benchmark | Reason |
+|---|---|
+| GSM-Symbolic | CC-BY-NC-ND-4.0: NonCommercial and NoDerivatives |
+| AgentIF | CC-BY-NC-4.0: NonCommercial |
+| WildBench | Subjective: needs an LLM judge, so results are biased and costly |
+| BrowseComp | Too complex: needs a live browsing agent |
+| SWE-bench Lite | Too heavy: needs a full Docker sandbox per sample |
+| PIQA | The dataset card declares no licence, so commercial use cannot be verified |
+| LiveCodeBench | The card declares only the unversioned `cc` tag, and the problems come from third-party contest sites |
+| BananaMind Base Bench 1.1 | The dataset card declares no licence, and the repository is gated |
+
+PIQA is a published component of the Int Index. Because it is not shipped here,
+the Int Index is computed from HellaSwag, ARC and ArithMark-3 with a denominator
+of 2.65, which is what the published method prescribes for a missing component.
+
+LiveCodeBench is the only removal that costs a whole capability: the suite no
+longer ships a coding benchmark. `LICENSE_NOTES.md` records the full reasoning.
+
+## Install
+
+```bash
+pip install -e ".[hf,api,benchmarks]"
+```
+
+`hf` provides the HuggingFace `transformers` backend. `benchmarks` adds the
+scoring dependencies the tasks need (`math_verify`, `langdetect`, and friends).
+
+## Run
+
+```bash
+# Everything at once. `slm` is a tag group: every task tagged `slm` runs.
+python3 -m slm_eval \
+  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
+  --tasks slm --device cpu --batch_size 8
+
+# One benchmark.
+python3 -m slm_eval \
+  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
+  --tasks arithmark_3_0 --device cpu
+
+# A quick wiring check on a few examples.
+python3 -m slm_eval \
+  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
+  --tasks hellaswag --limit 16 --device cpu
+```
+
+List the available tasks:
+
+```bash
+python3 -m slm_eval ls tasks
+```
+
+## The Int Index
+
+The Int Index is the composite score published on the Open SLM Leaderboard:
+
+```
+N(score, chance) = 100 x (score - chance) / (100 - chance)
+
+Int Index = (HellaSwag + ARC + PIQA + 0.65 x ArithMark-3) / 3.65
+```
+
+Run the components in one pass, then compute the index from the saved results:
+
+```bash
+python3 -m slm_eval --model hf \
+  --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
+  --tasks int_index --device cpu --output_path logs/tiny-run
+
+# The tracker writes logs/tiny-run/<model name>/results_<timestamp>.json,
+# so let the shell expand the path.
+python3 -m slm_eval.int_index --results logs/tiny-run/*/results_*.json
+```
+
+The scale is small: chance level scores about 0, and the strongest reported
+sub-10B models reach roughly 5. It is not a percentage. Each component is
+rescaled so that guessing maps to 0 and a perfect score maps to 100.
+
+## Average percentage (`eval_avg`)
+
+`eval_avg` is the plain number: the average of the per-benchmark percentages in
+a result file, with no weighting and no chance correction.
+
+```bash
+python3 -m slm_eval.eval_avg --results logs/tiny-run/*/results_*.json
+```
+
+It picks exactly one metric per benchmark, in this order of preference:
+`acc_norm`, `acc`, `exact_match`, `function_name_match`, `ast_match`,
+`valid_json`. If a benchmark reports none of those, it uses its alphabetically
+first scored metric. `acc_norm` is preferred because raw `acc` carries a
+token-length bias.
+
+Diagnostics such as `unparseable` and `output_emitted` measure the harness, not
+the model, so they are never chosen. A benchmark with no usable metric is
+reported as skipped and leaves the average, instead of counting as zero. The
+printed table names the metric used for every benchmark, so the number stays
+auditable.
+
+`eval_avg` is a plain mean, so it is comparable only between runs over the same
+benchmark set. It is not the Int Index, which is chance-corrected, weighted and
+published.
+
+## Scoring tiers
+
+`bfcl_v4` is built in two tiers:
+
+- **Tier 1** always works. It builds the real prompt, generates a real response,
+  and computes a deterministic format or rule-based metric.
+- **Tier 2** is the official metric. It is implemented, but gated behind an
+  explicit prerequisite. If you request it without the prerequisite, the task
+  raises a `RuntimeError` that names exactly what is missing. It never silently
+  returns zero.
+
+| Task | Tier 1 | Tier 2 |
+|---|---|---|
+| `bfcl_v4` | `valid_json`, `function_name_match` | `ast_match`, needs `--metadata enable_tier2_metric=True` |
+
+No shipped benchmark executes generated code: BFCL Tier 2 runs the official AST
+checker over the predicted call. The engine still enforces
+`--confirm_run_unsafe_code` for any task that marks itself unsafe, so a task added
+later cannot execute code without that confirmation. Per-task limits and
+prerequisites are documented in each task's `README.md`.
+
+## A note on the inherited documentation
+
+`docs/`, `scripts/` and `examples/` document the engine, the CLI and the model
+backends, which this suite does not change. Some examples inside them name
+benchmarks that were removed here. For the tasks that do exist,
+`slm_eval/tasks/slm/CONVENTIONS.md` and the per-task `README.md` files are
+authoritative.
+
+## Layout
+
+```
+slm_eval/                     evaluation engine
+  int_index.py                the Int Index composite calculator
+  eval_avg.py                 the plain average-percentage calculator
+  tasks/
+    CONVENTIONS.md            the task authoring contract
+    slm/                      the supported benchmarks live here
+      hellaswag/ arc_easy/ arc_challenge/
+      arithmark_2/ arithmark_3_0/
+      math_500/ gpqa_diamond/ bfcl_v4/
+      int_index/              composite group (not a dataset)
+tests/                        tests for this suite
+LICENSE_NOTES.md              licence research for every benchmark
+NOTICE                        attribution for the CCPL benchmarks and the engine
+```
+
+## Add or change a task
+
+Read `slm_eval/tasks/slm/CONVENTIONS.md`. Task discovery globs
+`slm_eval/tasks/**/*.yaml`, so a new directory with a YAML file is enough.
+
+## Licence
+
+The engine is MIT, from EleutherAI. `LICENSE.md` keeps that notice. See `NOTICE`
+for the attribution required by the CC-BY-4.0 and CC-BY-SA-4.0 datasets, and
+`LICENSE_NOTES.md` for the per-benchmark research.