tinydistill 0.3.0__tar.gz → 0.3.1__tar.gz

{tinydistill-0.3.0 → tinydistill-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: tinydistill
-Version: 0.3.0
+Version: 0.3.1
 Summary: Distil an LLM classification prompt into a small, fast model that runs on a CPU.
 Project-URL: Homepage, https://github.com/Andreas-Hum/tinydistill
 Author: Andreas-Hum
@@ -70,7 +70,8 @@ results files in [`examples/`](https://github.com/Andreas-Hum/tinydistill/tree/m
 quotes them.
 
 **Try it:** the [live demo](https://andreas-hum.github.io/tinydistill/) runs the Banking77 student (7B 8-shot
-teacher, int8, 23 MB) in your browser.
+teacher, int8, 23 MB) in your browser and shows what happens inside it: the word pieces, what each layer would
+answer, where the classifier looks, which words matter, and the teacher's soft labels next to the student's.
 
 ### Banking77: the hard case
 
@@ -124,15 +125,18 @@ The distilled student is far better than the same student trained on the few gol
 CPU in about a millisecond. Sending its least confident texts to the teacher closes most of the remaining gap
 to the teacher.
 
-Other students, the same teacher and soft labels. `static` is the static-embedding student
-(`--student static`: token vectors from MiniLM-L6, averaged, no attention), trained for 20 epochs at learning
-rate 1e-3 instead of 5 epochs at 1e-4:
+Other students, the same teacher and soft labels. Students train for 5 epochs unless the name says otherwise.
+`static` is the static-embedding student (`--student static`: token vectors from MiniLM-L6, averaged, no
+attention), trained for 20 epochs at learning rate 1e-3:
 
 <!-- results: banking77/students.md -->
 | Run | Teacher | Teacher accuracy | Student | Parameters (M) | Accuracy | Macro-F1 | Agreement with teacher | Deployed as | Latency, batch 1 (ms) | Latency, batch 32 (ms) | Size (MB) |
 |---|---|---|---|---|---|---|---|---|---|---|---|
 | banking77-7b-8shot | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.729 ± 0.011 | 0.697 ± 0.015 | 0.800 ± 0.003 | ONNX int8 | 1.4 | 40.2 | 23 |
 | all-MiniLM-L12-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.743 ± 0.011 | 0.710 ± 0.016 | 0.815 ± 0.004 | ONNX int8 | 3.3 | 79.2 | 34 |
+| all-MiniLM-L12-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.817 ± 0.001 | 0.814 ± 0.001 | 0.875 ± 0.002 | ONNX int8 | 3.2 | 82.0 | 34 |
+| all-MiniLM-L6-v2-10-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.808 ± 0.003 | 0.801 ± 0.002 | 0.860 ± 0.004 | ONNX int8 | 1.1 | 38.1 | 23 |
+| all-MiniLM-L6-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.813 ± 0.002 | 0.811 ± 0.003 | 0.868 ± 0.003 | ONNX int8 | 1.2 | 36.3 | 23 |
 | all-mpnet-base-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-mpnet-base-v2` | 110 | 0.814 ± 0.004 | 0.805 ± 0.007 | 0.863 ± 0.001 | ONNX int8 | 7.3 | 220.4 | 110 |
 | bge-small-en-v1.5 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `BAAI/bge-small-en-v1.5` | 33 | 0.739 ± 0.008 | 0.706 ± 0.011 | 0.815 ± 0.004 | ONNX int8 | 2.7 | 75.2 | 34 |
 | static | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `static` | 8 | 0.803 ± 0.001 | 0.801 ± 0.001 | 0.824 ± 0.001 | ONNX int8 | 0.1 | 5.1 | 8 |
@@ -140,10 +144,14 @@ rate 1e-3 instead of 5 epochs at 1e-4:
 All latencies on the CPU with 2 threads.
 <!-- /results -->
 
-Two students are more accurate than their own teacher: the 110M-parameter mpnet and the static student.
-Learning from the teacher's soft labels on 10,000 texts, plus the 616 gold labels, smooths out some of its
-individual mistakes. The static student does this at about a tenth of a millisecond per text and 8 MB. It
-also trained four times longer, so the gap to the small transformers is not only about architecture.
+Training longer matters more than a bigger student: with 20 epochs instead of 5, the 23M-parameter MiniLM-L6
+student is as accurate as the 110M-parameter mpnet, at a sixth of its latency, and more accurate than its own
+7B teacher. Learning from the teacher's soft labels on 10,000 texts, plus the 616 gold labels, smooths out
+some of the teacher's individual mistakes. The static student is about a point behind the MiniLM students
+trained as long, at a tenth of a millisecond per text and 8 MB. Ten epochs get most of the way (MiniLM-L6:
+0.808), so 10 is the default from 0.3.1. On the easier financial-tweets task, 20 epochs give the same accuracy
+as 5 ([`financial-tweets-7b-20-epochs.md`](https://github.com/Andreas-Hum/tinydistill/blob/main/examples/financial-tweets/financial-tweets-7b-20-epochs.md)).
+The other tables on this page were trained for 5 epochs, the default until 0.3.1.
 
 ### Financial tweets: an easier task
 
@@ -253,7 +261,8 @@ Banking77 splits are sorted by intent, so a limited run only sees a few intents.
 Data can be `.csv`, `.jsonl` or `hf:<dataset>[:<config>]:<split>` (needs the `hf` extra). Useful options:
 `--teacher` (any Hugging Face causal LM), `--load-4bit` (bitsandbytes, CUDA only: `pip install "tinydistill[4bit]"`),
 `--student` (any Hugging Face encoder), `--epochs`, `--lr`, `--device`, `--threads`, `--seed`, and `--cache-dir`
-(runs that share it label each text only once). See `tinydistill <command> --help`.
+(runs that share it label each text only once, and a run whose texts are all cached does not load the
+teacher at all). See `tinydistill <command> --help`.
 
 Long steps print their progress, speed and time left every few seconds and write it to `RUN/status.json`;
 `tinydistill status RUN` reads it from another terminal and tells you whether the process is still alive.
@@ -261,8 +270,8 @@ Long steps print their progress, speed and time left every few seconds and write
 **Static student.** `--student static` (or `static:<encoder>`) trains a student with no attention at all: one
 vector per token, averaged, then a linear layer, started from the encoder's own token representations (as in
 Model2Vec). It is much smaller and faster than a transformer student. It ignores word order, so it can miss
-meaning that depends on it (negation, "A rather than B"); on Banking77 it beat the small transformer
-students and came close to mpnet (see the results). It exports to ONNX and runs with `predict` and `serve` like any other student, but
+meaning that depends on it (negation, "A rather than B"); on Banking77 it came within about a point of
+MiniLM students trained as long (see the results). It exports to ONNX and runs with `predict` and `serve` like any other student, but
 not in the browser page. Train it longer and at a higher learning rate, such as `--epochs 20 --lr 1e-3`.
 
 ### Choosing the teacher
@@ -307,7 +316,9 @@ using only the standard library. It listens on 127.0.0.1 unless you pass `--host
 
 **Browser.** `tinydistill web RUN/onnx --out site` writes a static page plus the model in the layout
 [transformers.js](https://github.com/huggingface/transformers.js) expects; the int8 model then runs in the
-visitor's browser with no server at all. Serve the folder with any static host (GitHub Pages works).
+visitor's browser with no server at all. Below the answer, an "Inside the model" view shows each step of the inference
+(tokens, the answer after each layer, attention, probabilities, which words matter, the teacher's soft labels
+for the examples and the cascade), so the page also explains distillation. Serve the folder with any static host (GitHub Pages works).
 
 ### A few gold labels
 

{tinydistill-0.3.0 → tinydistill-0.3.1}/README.md

@@ -40,7 +40,8 @@ results files in [`examples/`](https://github.com/Andreas-Hum/tinydistill/tree/m
 quotes them.
 
 **Try it:** the [live demo](https://andreas-hum.github.io/tinydistill/) runs the Banking77 student (7B 8-shot
-teacher, int8, 23 MB) in your browser.
+teacher, int8, 23 MB) in your browser and shows what happens inside it: the word pieces, what each layer would
+answer, where the classifier looks, which words matter, and the teacher's soft labels next to the student's.
 
 ### Banking77: the hard case
 
@@ -94,15 +95,18 @@ The distilled student is far better than the same student trained on the few gol
 CPU in about a millisecond. Sending its least confident texts to the teacher closes most of the remaining gap
 to the teacher.
 
-Other students, the same teacher and soft labels. `static` is the static-embedding student
-(`--student static`: token vectors from MiniLM-L6, averaged, no attention), trained for 20 epochs at learning
-rate 1e-3 instead of 5 epochs at 1e-4:
+Other students, the same teacher and soft labels. Students train for 5 epochs unless the name says otherwise.
+`static` is the static-embedding student (`--student static`: token vectors from MiniLM-L6, averaged, no
+attention), trained for 20 epochs at learning rate 1e-3:
 
 <!-- results: banking77/students.md -->
 | Run | Teacher | Teacher accuracy | Student | Parameters (M) | Accuracy | Macro-F1 | Agreement with teacher | Deployed as | Latency, batch 1 (ms) | Latency, batch 32 (ms) | Size (MB) |
 |---|---|---|---|---|---|---|---|---|---|---|---|
 | banking77-7b-8shot | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.729 ± 0.011 | 0.697 ± 0.015 | 0.800 ± 0.003 | ONNX int8 | 1.4 | 40.2 | 23 |
 | all-MiniLM-L12-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.743 ± 0.011 | 0.710 ± 0.016 | 0.815 ± 0.004 | ONNX int8 | 3.3 | 79.2 | 34 |
+| all-MiniLM-L12-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.817 ± 0.001 | 0.814 ± 0.001 | 0.875 ± 0.002 | ONNX int8 | 3.2 | 82.0 | 34 |
+| all-MiniLM-L6-v2-10-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.808 ± 0.003 | 0.801 ± 0.002 | 0.860 ± 0.004 | ONNX int8 | 1.1 | 38.1 | 23 |
+| all-MiniLM-L6-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.813 ± 0.002 | 0.811 ± 0.003 | 0.868 ± 0.003 | ONNX int8 | 1.2 | 36.3 | 23 |
 | all-mpnet-base-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-mpnet-base-v2` | 110 | 0.814 ± 0.004 | 0.805 ± 0.007 | 0.863 ± 0.001 | ONNX int8 | 7.3 | 220.4 | 110 |
 | bge-small-en-v1.5 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `BAAI/bge-small-en-v1.5` | 33 | 0.739 ± 0.008 | 0.706 ± 0.011 | 0.815 ± 0.004 | ONNX int8 | 2.7 | 75.2 | 34 |
 | static | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `static` | 8 | 0.803 ± 0.001 | 0.801 ± 0.001 | 0.824 ± 0.001 | ONNX int8 | 0.1 | 5.1 | 8 |
@@ -110,10 +114,14 @@ rate 1e-3 instead of 5 epochs at 1e-4:
 All latencies on the CPU with 2 threads.
 <!-- /results -->
 
-Two students are more accurate than their own teacher: the 110M-parameter mpnet and the static student.
-Learning from the teacher's soft labels on 10,000 texts, plus the 616 gold labels, smooths out some of its
-individual mistakes. The static student does this at about a tenth of a millisecond per text and 8 MB. It
-also trained four times longer, so the gap to the small transformers is not only about architecture.
+Training longer matters more than a bigger student: with 20 epochs instead of 5, the 23M-parameter MiniLM-L6
+student is as accurate as the 110M-parameter mpnet, at a sixth of its latency, and more accurate than its own
+7B teacher. Learning from the teacher's soft labels on 10,000 texts, plus the 616 gold labels, smooths out
+some of the teacher's individual mistakes. The static student is about a point behind the MiniLM students
+trained as long, at a tenth of a millisecond per text and 8 MB. Ten epochs get most of the way (MiniLM-L6:
+0.808), so 10 is the default from 0.3.1. On the easier financial-tweets task, 20 epochs give the same accuracy
+as 5 ([`financial-tweets-7b-20-epochs.md`](https://github.com/Andreas-Hum/tinydistill/blob/main/examples/financial-tweets/financial-tweets-7b-20-epochs.md)).
+The other tables on this page were trained for 5 epochs, the default until 0.3.1.
 
 ### Financial tweets: an easier task
 
@@ -223,7 +231,8 @@ Banking77 splits are sorted by intent, so a limited run only sees a few intents.
 Data can be `.csv`, `.jsonl` or `hf:<dataset>[:<config>]:<split>` (needs the `hf` extra). Useful options:
 `--teacher` (any Hugging Face causal LM), `--load-4bit` (bitsandbytes, CUDA only: `pip install "tinydistill[4bit]"`),
 `--student` (any Hugging Face encoder), `--epochs`, `--lr`, `--device`, `--threads`, `--seed`, and `--cache-dir`
-(runs that share it label each text only once). See `tinydistill <command> --help`.
+(runs that share it label each text only once, and a run whose texts are all cached does not load the
+teacher at all). See `tinydistill <command> --help`.
 
 Long steps print their progress, speed and time left every few seconds and write it to `RUN/status.json`;
 `tinydistill status RUN` reads it from another terminal and tells you whether the process is still alive.
@@ -231,8 +240,8 @@ Long steps print their progress, speed and time left every few seconds and write
 **Static student.** `--student static` (or `static:<encoder>`) trains a student with no attention at all: one
 vector per token, averaged, then a linear layer, started from the encoder's own token representations (as in
 Model2Vec). It is much smaller and faster than a transformer student. It ignores word order, so it can miss
-meaning that depends on it (negation, "A rather than B"); on Banking77 it beat the small transformer
-students and came close to mpnet (see the results). It exports to ONNX and runs with `predict` and `serve` like any other student, but
+meaning that depends on it (negation, "A rather than B"); on Banking77 it came within about a point of
+MiniLM students trained as long (see the results). It exports to ONNX and runs with `predict` and `serve` like any other student, but
 not in the browser page. Train it longer and at a higher learning rate, such as `--epochs 20 --lr 1e-3`.
 
 ### Choosing the teacher
@@ -277,7 +286,9 @@ using only the standard library. It listens on 127.0.0.1 unless you pass `--host
 
 **Browser.** `tinydistill web RUN/onnx --out site` writes a static page plus the model in the layout
 [transformers.js](https://github.com/huggingface/transformers.js) expects; the int8 model then runs in the
-visitor's browser with no server at all. Serve the folder with any static host (GitHub Pages works).
+visitor's browser with no server at all. Below the answer, an "Inside the model" view shows each step of the inference
+(tokens, the answer after each layer, attention, probabilities, which words matter, the teacher's soft labels
+for the examples and the cascade), so the page also explains distillation. Serve the folder with any static host (GitHub Pages works).
 
 ### A few gold labels
 

{tinydistill-0.3.0 → tinydistill-0.3.1}/examples/banking77/run.sh

@@ -11,7 +11,8 @@ set -euo pipefail
 cd "$(dirname "$0")/../.."
 export OMP_NUM_THREADS=${OMP_NUM_THREADS:-2} MKL_NUM_THREADS=${MKL_NUM_THREADS:-2}
 base=(examples/banking77/task.yaml --data hf:PolyAI/banking77:train --test hf:PolyAI/banking77:test
-      --cache-dir runs/cache --device cuda --threads "$OMP_NUM_THREADS" --int8 --seeds 3)
+      --cache-dir runs/cache --device cuda --threads "$OMP_NUM_THREADS" --int8 --seeds 3
+      --epochs 5)  # the default before 0.3.1; the lines that pass --epochs 20 override it
 few=(--few-shot 8 --gold-per-class 8 --gold-only)
 big=(--teacher Qwen/Qwen2.5-7B-Instruct --load-4bit)
 
@@ -28,6 +29,14 @@ done
 # a static-embedding student (no attention), which wants a higher learning rate and more epochs
 tinydistill run "${base[@]}" "${big[@]}" --few-shot 8 --gold-per-class 8 --student static --lr 1e-3 --epochs 20 \
   --out runs/banking77-students/static --skip-teacher-bench
+# MiniLM-L6 at 10 epochs (the default from 0.3.1), and the two smallest transformer students trained as long
+# as the static one, for a like-for-like comparison
+tinydistill run "${base[@]}" "${big[@]}" --few-shot 8 --gold-per-class 8 --epochs 10 \
+  --out runs/banking77-students/all-MiniLM-L6-v2-10-epochs --skip-teacher-bench
+for student in sentence-transformers/all-MiniLM-L6-v2 sentence-transformers/all-MiniLM-L12-v2; do
+  tinydistill run "${base[@]}" "${big[@]}" --few-shot 8 --gold-per-class 8 --student "$student" --epochs 20 \
+    --out "runs/banking77-students/${student##*/}-20-epochs" --skip-teacher-bench
+done
 
 for run in banking77-1.5b banking77-1.5b-8shot banking77-7b banking77-7b-8shot; do
   cp "runs/$run/results.json" "examples/banking77/$run.json"

{tinydistill-0.3.0 → tinydistill-0.3.1}/examples/banking77/students.md

@@ -2,6 +2,9 @@
 |---|---|---|---|---|---|---|---|---|---|---|---|
 | banking77-7b-8shot | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.729 ± 0.011 | 0.697 ± 0.015 | 0.800 ± 0.003 | ONNX int8 | 1.4 | 40.2 | 23 |
 | all-MiniLM-L12-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.743 ± 0.011 | 0.710 ± 0.016 | 0.815 ± 0.004 | ONNX int8 | 3.3 | 79.2 | 34 |
+| all-MiniLM-L12-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L12-v2` | 33 | 0.817 ± 0.001 | 0.814 ± 0.001 | 0.875 ± 0.002 | ONNX int8 | 3.2 | 82.0 | 34 |
+| all-MiniLM-L6-v2-10-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.808 ± 0.003 | 0.801 ± 0.002 | 0.860 ± 0.004 | ONNX int8 | 1.1 | 38.1 | 23 |
+| all-MiniLM-L6-v2-20-epochs | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-MiniLM-L6-v2` | 23 | 0.813 ± 0.002 | 0.811 ± 0.003 | 0.868 ± 0.003 | ONNX int8 | 1.2 | 36.3 | 23 |
 | all-mpnet-base-v2 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `sentence-transformers/all-mpnet-base-v2` | 110 | 0.814 ± 0.004 | 0.805 ± 0.007 | 0.863 ± 0.001 | ONNX int8 | 7.3 | 220.4 | 110 |
 | bge-small-en-v1.5 | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `BAAI/bge-small-en-v1.5` | 33 | 0.739 ± 0.008 | 0.706 ± 0.011 | 0.815 ± 0.004 | ONNX int8 | 2.7 | 75.2 | 34 |
 | static | 8-shot, 4-bit `Qwen/Qwen2.5-7B-Instruct` | 0.783 | `static` | 8 | 0.803 ± 0.001 | 0.801 ± 0.001 | 0.824 ± 0.001 | ONNX int8 | 0.1 | 5.1 | 8 |

tinydistill-0.3.1/examples/financial-tweets/financial-tweets-7b-20-epochs.json

@@ -0,0 +1,146 @@
+{
+  "task": "financial-tweets",
+  "teacher": "Qwen/Qwen2.5-7B-Instruct",
+  "student": "sentence-transformers/all-MiniLM-L6-v2",
+  "data": {
+    "train": "hf:zeroshot/twitter-financial-news-sentiment:train",
+    "test": "hf:zeroshot/twitter-financial-news-sentiment:validation",
+    "n_train": 9543,
+    "n_test": 2388
+  },
+  "training": {
+    "epochs": 20,
+    "lr": 0.0001,
+    "batch_size": 32,
+    "gold_per_class": 0,
+    "gold_file": null,
+    "n_gold_file": 0,
+    "n_gold": 0,
+    "upper_bound": false,
+    "gold_only": false,
+    "seeds": 3,
+    "n_gold_upper_bound": null,
+    "label_texts_scored": 0,
+    "few_shot": 0,
+    "n_examples": 0,
+    "ce_weight": 0.0,
+    "student_device": "cuda",
+    "train_seconds": 820.4,
+    "teacher_device": "cuda:0",
+    "label_seconds": 6.2,
+    "load_4bit": true,
+    "shortlist": null,
+    "shortlist_model": null
+  },
+  "environment": {
+    "cpu": "x86_64",
+    "cpu_threads": 2,
+    "torch": "2.15.0+cu132",
+    "gpu": "NVIDIA GeForce RTX 4090",
+    "omp_num_threads": "2"
+  },
+  "models": [
+    {
+      "role": "teacher",
+      "name": "teacher, zero-shot, 4-bit",
+      "device": "cuda:0",
+      "model": "Qwen/Qwen2.5-7B-Instruct",
+      "accuracy": 0.8320770519262981,
+      "macro_f1": 0.7875756953889134,
+      "size_mb": null
+    },
+    {
+      "role": "student",
+      "name": "student, PyTorch",
+      "device": "cpu",
+      "accuracy": 0.824818537130095,
+      "accuracy_std": 0.004398646358623353,
+      "accuracy_seeds": [
+        0.8249581239530989,
+        0.8203517587939698,
+        0.8291457286432161
+      ],
+      "macro_f1": 0.7645078986696947,
+      "macro_f1_std": 0.0037534820261736976,
+      "macro_f1_seeds": [
+        0.7618092859102075,
+        0.7629200730807385,
+        0.7687943370181382
+      ],
+      "agreement": 0.8710217755443885,
+      "agreement_std": 0.002931323283082083,
+      "agreement_seeds": [
+        0.8697654941373534,
+        0.86892797319933,
+        0.8743718592964824
+      ],
+      "params_m": 22.714371,
+      "size_mb": 90.869492,
+      "latency_ms_b1": 4.909115499685868,
+      "latency_ms_b32": 150.38988000014797,
+      "throughput_per_s": 212.78027484275216
+    },
+    {
+      "role": "student",
+      "name": "student, ONNX fp32",
+      "device": "cpu",
+      "accuracy": 0.8249581239530989,
+      "macro_f1": 0.7618092859102075,
+      "agreement": 0.8697654941373534,
+      "size_mb": 90.990363,
+      "latency_ms_b1": 3.9684855019004317,
+      "latency_ms_b32": 148.22442249896994,
+      "throughput_per_s": 215.88884922268716
+    },
+    {
+      "role": "student",
+      "name": "student, ONNX int8",
+      "device": "cpu",
+      "accuracy": 0.8203517587939698,
+      "macro_f1": 0.7589981023943616,
+      "agreement": 0.8710217755443886,
+      "size_mb": 23.043866,
+      "latency_ms_b1": 1.3503949994628783,
+      "latency_ms_b32": 74.80496900097933,
+      "throughput_per_s": 427.77906905597496
+    }
+  ],
+  "cascade": [
+    {
+      "share_to_teacher": 0.0,
+      "threshold": 0.3775518238544464,
+      "accuracy": 0.8203517587939698,
+      "macro_f1": 0.7589981023943616
+    },
+    {
+      "share_to_teacher": 0.10008375209380234,
+      "threshold": 0.7880504727363586,
+      "accuracy": 0.8345896147403685,
+      "macro_f1": 0.7804097140755729
+    },
+    {
+      "share_to_teacher": 0.20016750418760468,
+      "threshold": 0.953411340713501,
+      "accuracy": 0.8375209380234506,
+      "macro_f1": 0.7885267426679218
+    },
+    {
+      "share_to_teacher": 0.2998324958123953,
+      "threshold": 0.9881662130355835,
+      "accuracy": 0.8358458961474037,
+      "macro_f1": 0.7879874137344648
+    },
+    {
+      "share_to_teacher": 0.5,
+      "threshold": 0.9987922310829163,
+      "accuracy": 0.833752093802345,
+      "macro_f1": 0.7876367442676434
+    },
+    {
+      "share_to_teacher": 1.0,
+      "threshold": 1.0,
+      "accuracy": 0.8320770519262981,
+      "macro_f1": 0.7875756953889134
+    }
+  ]
+}

tinydistill-0.3.1/examples/financial-tweets/financial-tweets-7b-20-epochs.md

@@ -0,0 +1,22 @@
+| Model | Device | Accuracy | Macro-F1 | Agreement with teacher | Latency, batch 1 (ms) | Latency, batch 32 (ms) | Throughput (texts/s) | Size (MB) | Speed-up, batch 1 |
+|---|---|---|---|---|---|---|---|---|---|
+| teacher, zero-shot, 4-bit | cuda:0 | 0.832 | 0.788 | n/a | n/a | n/a | n/a | n/a | n/a |
+| student, PyTorch | cpu | 0.825 ± 0.004 | 0.765 ± 0.004 | 0.871 ± 0.003 | 4.9 | 150.4 | 212.8 | 91 | n/a |
+| student, ONNX fp32 | cpu | 0.825 | 0.762 | 0.870 | 4.0 | 148.2 | 215.9 | 91 | n/a |
+| student, ONNX int8 | cpu | 0.820 | 0.759 | 0.871 | 1.4 | 74.8 | 427.8 | 23 | n/a |
+
+Task `financial-tweets`, 2388 gold-labelled test texts. Teacher: `Qwen/Qwen2.5-7B-Instruct`. Student: `sentence-transformers/all-MiniLM-L6-v2`. The student learns only from the teacher's soft labels on 9543 unlabelled training texts. Values with ± are the mean and standard deviation over 3 training seeds; the ONNX rows are the exported first-seed model.
+CPU rows use 2 threads. Latency is the median over repeated runs. Size is the teacher's weights as loaded and the students' model files. The teacher packs several texts into one forward pass; it is timed at batch 32 only when that takes under three minutes, otherwise its throughput is 1 / batch-1 latency. 
+
+### Cascade: student first, unsure texts to the teacher
+
+| Share sent to the teacher | Student confidence threshold | Accuracy | Macro-F1 | Mean latency per text (ms) |
+|---|---|---|---|---|
+| 0% | 0.378 | 0.820 | 0.759 | n/a |
+| 10% | 0.788 | 0.835 | 0.780 | n/a |
+| 20% | 0.953 | 0.838 | 0.789 | n/a |
+| 30% | 0.988 | 0.836 | 0.788 | n/a |
+| 50% | 0.999 | 0.834 | 0.788 | n/a |
+| 100% | 1.000 | 0.832 | 0.788 | n/a |
+
+The student answers every text and passes its least confident ones to the teacher. Thresholds are set on the test texts to hit each share; in deployment, set them on a validation split. Latency is the student's batch-1 latency plus the share times the batch-1 latency of the teacher.

{tinydistill-0.3.0 → tinydistill-0.3.1}/examples/financial-tweets/run.sh

@@ -12,7 +12,7 @@ export OMP_NUM_THREADS=${OMP_NUM_THREADS:-2} MKL_NUM_THREADS=${MKL_NUM_THREADS:-
 data=hf:zeroshot/twitter-financial-news-sentiment
 task=examples/financial-tweets/task.yaml
 base=("$task" --test "$data:validation" --cache-dir runs/cache --device cuda --threads "$OMP_NUM_THREADS"
-      --int8 --seeds 3)
+      --int8 --seeds 3 --epochs 5)  # 5 epochs: the default before 0.3.1
 q7=Qwen/Qwen2.5-7B-Instruct
 
 real=(--data "$data:train" --upper-bound)  # the synthetic run has no gold labels, so no upper bound there
@@ -22,6 +22,9 @@ tinydistill run "${base[@]}" "${real[@]}" --few-shot 8 --gold-per-class 8 --gold
   --out runs/financial-tweets-1.5b-8shot --teacher-devices cpu,cuda
 tinydistill run "${base[@]}" "${real[@]}" --teacher "$q7" --load-4bit --out runs/financial-tweets-7b \
   --teacher-devices cuda
+# the same student trained for 20 epochs: does training longer help on this task too?
+tinydistill run "${base[@]}" --data "$data:train" --teacher "$q7" --load-4bit --epochs 20 \
+  --out runs/financial-tweets-7b-20-epochs --skip-teacher-bench
 tinydistill run "${base[@]}" "${real[@]}" --load-4bit --teacher-devices cuda \
   --teacher "$q7,mistralai/Mistral-7B-Instruct-v0.3,microsoft/Phi-3.5-mini-instruct" \
   --out runs/financial-tweets-ensemble
@@ -30,8 +33,8 @@ tinydistill synth "$task" --per-label 500 --teacher "$q7" --load-4bit --device c
 tinydistill run "${base[@]}" --data runs/financial-tweets-synth.jsonl --teacher "$q7" --load-4bit \
   --out runs/financial-tweets-synthetic --skip-teacher-bench
 
-for run in financial-tweets-1.5b financial-tweets-1.5b-8shot financial-tweets-7b financial-tweets-ensemble \
-           financial-tweets-synthetic; do
+for run in financial-tweets-1.5b financial-tweets-1.5b-8shot financial-tweets-7b financial-tweets-7b-20-epochs \
+           financial-tweets-ensemble financial-tweets-synthetic; do
   cp "runs/$run/results.json" "examples/financial-tweets/$run.json"
   cp "runs/$run/results.md" "examples/financial-tweets/$run.md"
 done

{tinydistill-0.3.0 → tinydistill-0.3.1}/examples/go-emotions/run.sh

@@ -10,7 +10,7 @@ data=hf:google-research-datasets/go_emotions:simplified
 
 tinydistill run examples/go-emotions/task.yaml --data "$data:train" --limit 10000 --test "$data:test" \
   --cache-dir runs/cache --device cuda --threads "$OMP_NUM_THREADS" --int8 --seeds 3 --upper-bound \
-  --gold-per-class 8 --gold-only --out runs/go-emotions-1.5b --teacher-devices cpu,cuda
+  --gold-per-class 8 --gold-only --epochs 5 --out runs/go-emotions-1.5b --teacher-devices cpu,cuda
 
 cp runs/go-emotions-1.5b/results.json examples/go-emotions/go-emotions-1.5b.json
 cp runs/go-emotions-1.5b/results.md examples/go-emotions/go-emotions-1.5b.md

{tinydistill-0.3.0 → tinydistill-0.3.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tinydistill"
-version = "0.3.0"
+version = "0.3.1"
 description = "Distil an LLM classification prompt into a small, fast model that runs on a CPU."
 readme = "README.md"
 license = "MIT"

{tinydistill-0.3.0 → tinydistill-0.3.1}/src/tinydistill/__init__.py

@@ -29,4 +29,4 @@ __all__ = [
     "read_examples",
     "train_student",
 ]
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{tinydistill-0.3.0 → tinydistill-0.3.1}/src/tinydistill/cli.py

@@ -53,7 +53,15 @@ def cmd_label(a: argparse.Namespace) -> None:
 
     from .data import count_rows, read_examples
     from .progress import Progress
-    from .teacher import ScoreCache, combine_scores, iter_unique, load_teacher
+    from .teacher import (
+        ScoreCache,
+        combine_scores,
+        iter_unique,
+        load_teacher,
+        pick_device,
+        quick_key,
+        text_hash,
+    )
 
     task = load_task(a.task)
     run = Path(a.out)
@@ -85,7 +93,25 @@ def cmd_label(a: argparse.Namespace) -> None:
 
     caches, fingerprints, device = [], [], "cpu"
     seconds, scored = 0.0, 0
+    keys_file = cache_dir / "teacher-keys.json"  # quick key (tokenizer only) -> full fingerprint
+    keys = json.loads(keys_file.read_text()) if keys_file.exists() else {}
     for name in _teachers(a.teacher):  # one teacher in memory at a time
+        quick = None
+        if not a.vllm_url:
+            quick = quick_key(name, task, a.device, a.load_4bit, examples, a.shortlist, a.shortlist_model)
+            known = keys.get(quick)
+            cache = ScoreCache(cache_dir / f"teacher-{known}.jsonl") if known else None
+            if cache and all(
+                text_hash(e.text) in cache.scores
+                for _, spec, limit in splits
+                for e in read_examples(spec, task, limit, with_gold=False)
+            ):
+                log(f"{name}: every text is in the cache, so the teacher is not loaded")
+                fingerprints.append(known)
+                caches.append(cache)
+                device = pick_device(a.device)
+                device = "cuda:0" if device == "cuda" else device  # as a loaded model reports it
+                continue
         log(f"loading teacher {name}")
         scorer = load_teacher(
             name,
@@ -100,6 +126,12 @@ def cmd_label(a: argparse.Namespace) -> None:
         )
         fingerprints.append(scorer.fingerprint())
         device = str(scorer.device)
+        if quick:
+            keys = json.loads(keys_file.read_text()) if keys_file.exists() else {}
+            keys[quick] = fingerprints[-1]
+            tmp = keys_file.with_suffix(".tmp")
+            tmp.write_text(json.dumps(keys, indent=1))
+            tmp.replace(keys_file)
         cache = ScoreCache(cache_dir / f"teacher-{fingerprints[-1]}.jsonl")
         if cache.scores:
             log(f"resuming: {len(cache.scores)} texts already scored")
@@ -883,7 +915,7 @@ def build_parser() -> argparse.ArgumentParser:
         help="Hugging Face encoder, or `static` / `static:<encoder>` for a static-embedding student "
         "(no attention: much smaller and faster; try --epochs 20 --lr 1e-3)",
     )
-    train.add_argument("--epochs", type=int, default=5)
+    train.add_argument("--epochs", type=int, default=10, help="passes over the training texts (default 10)")
     train.add_argument("--batch-size", type=int, default=32)
     train.add_argument("--lr", type=float, default=1e-4)
     train.add_argument("--max-length", type=int, default=128)

{tinydistill-0.3.0 → tinydistill-0.3.1}/src/tinydistill/export.py

@@ -54,6 +54,85 @@ def export_onnx(student_dir: Path, out_dir: Path, int8: bool = False, opset: int
     return written
 
 
+def _classifier_head(model):
+    """The student's classification head as a function of one layer's hidden states (batch, tokens, dim), or
+    None when the model has no such head (the static student)."""
+    head, base = getattr(model, "classifier", None), getattr(model, "base_model", None)
+    if head is None or base is None or not hasattr(base, "encoder"):
+        return None
+    if isinstance(head, torch.nn.Linear):  # BERT style: the pooler reads the first token
+        pooler = getattr(base, "pooler", None)
+        return (lambda h: head(pooler(h))) if pooler is not None else (lambda h: head(h[:, 0]))
+    return head  # RoBERTa/MPNet style heads take the whole sequence and read its first token themselves
+
+
+class _Explain(torch.nn.Module):
+    """Logits, plus what the classifier would answer after each layer, and how much the first token (which
+    the classifier reads) attends to every token in each layer, averaged over heads."""
+
+    def __init__(self, model, head):
+        super().__init__()
+        self.model, self.head = model, head
+
+    def forward(self, input_ids, attention_mask):
+        out = self.model.base_model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            output_hidden_states=True,
+            output_attentions=True,
+        )
+        hs = torch.stack(out.hidden_states[1:], 1)  # (batch, layers, tokens, dim)
+        b, n, t, d = hs.shape
+        # the head runs once on all layers stacked as a batch (one set of weights in the graph)
+        layers = self.head(hs.reshape(b * n, t, d)).reshape(b, n, -1)  # (batch, layers, labels)
+        attention = torch.stack([a[:, :, 0, :].mean(1) for a in out.attentions], 1)  # (batch, layers, tokens)
+        return layers[:, -1], layers, attention
+
+
+def export_explain(student_dir: Path, out_dir: Path, int8: bool = False, opset: int = 17) -> Path | None:
+    """Export ``explain.onnx`` (and ``explain.int8.onnx``) with outputs ``logits``, ``layer_logits`` and
+    ``cls_attention``, for the teaching view of `tinydistill web`. None if the model has no layers to show."""
+    from transformers import AutoModelForSequenceClassification
+
+    from .static import is_static
+
+    if is_static(student_dir):
+        return None
+    model = AutoModelForSequenceClassification.from_pretrained(
+        student_dir, attn_implementation="eager"
+    ).eval()
+    head = _classifier_head(model)
+    if head is None:
+        return None
+    _, tok = load_student(student_dir)
+    enc = tok(["an example input", "another one"], padding=True, return_tensors="pt")
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / "explain.onnx"
+    torch.onnx.export(
+        _Explain(model, head),
+        (enc["input_ids"], enc["attention_mask"]),
+        str(path),
+        input_names=["input_ids", "attention_mask"],
+        output_names=["logits", "layer_logits", "cls_attention"],
+        dynamic_axes={
+            "input_ids": {0: "batch", 1: "seq"},
+            "attention_mask": {0: "batch", 1: "seq"},
+            "logits": {0: "batch"},
+            "layer_logits": {0: "batch"},
+            "cls_attention": {0: "batch", 2: "seq"},
+        },
+        opset_version=opset,
+        dynamo=False,
+    )
+    if not int8:
+        return path
+    from onnxruntime.quantization import QuantType, quantize_dynamic
+
+    q = out_dir / "explain.int8.onnx"
+    quantize_dynamic(str(path), str(q), weight_type=QuantType.QInt8)
+    return q
+
+
 class Predictor:
     """Run an exported model: ``Predictor("runs/x/onnx").predict(["text", ...])``."""
 

{tinydistill-0.3.0 → tinydistill-0.3.1}/src/tinydistill/student.py

@@ -96,7 +96,7 @@ def train_student(
     model_name: str = DEFAULT_STUDENT,
     gold: list[int | None] | None = None,
     ce_weight: float = 0.0,
-    epochs: int = 5,
+    epochs: int = 10,
     batch_size: int = 32,
     lr: float = 1e-4,
     max_length: int = 128,

{tinydistill-0.3.0 → tinydistill-0.3.1}/src/tinydistill/teacher.py

@@ -557,6 +557,34 @@ def load_teacher(
     return LabelScorer(model, tok, task, max_tokens=max_tokens, examples=examples, shortlist=short)
 
 
+def quick_key(
+    name: str,
+    task: Task,
+    device: str,
+    load_4bit: bool,
+    examples: list[tuple[str, int]] | None = None,
+    shortlist: int | None = None,
+    shortlist_model: str | None = None,
+) -> str:
+    """What decides a local teacher's cache fingerprint, from the tokenizer alone (no model weights): the
+    rendered prompt and continuations, and how the model would be loaded. `label` maps it to the full
+    fingerprint, so a run whose texts are all cached never loads the teacher."""
+    p = _Prompt()
+    p._prepare(AutoTokenizer.from_pretrained(name), task, True, True, examples, None)
+    key = {
+        "model": name,
+        "device": pick_device(device),
+        "load_4bit": load_4bit,
+        "prefix": p.prefix_ids,
+        "suffix": p.suffix,
+        "forms": p.forms,
+        "form_label": p.form_label,
+        "form_answer": p.form_answer,
+        "shortlist": [shortlist, shortlist_model] if shortlist else None,
+    }
+    return hashlib.sha256(json.dumps(key, sort_keys=True).encode()).hexdigest()[:16]
+
+
 class ScoreCache:
     """Append-only JSONL of {"h": text hash, "s": label scores}; reopening it resumes where it stopped."""