hmft 0.2.0__tar.gz

hmft-0.2.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 hmft contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

hmft-0.2.0/PKG-INFO

@@ -0,0 +1,252 @@
+Metadata-Version: 2.4
+Name: hmft
+Version: 0.2.0
+Summary: Estimate how long an LLM call will take before you send it, plus a live ETA while it streams. Fully client-side.
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://hmft.dev
+Project-URL: Repository, https://github.com/hmft-dev/hmft
+Project-URL: Changelog, https://github.com/hmft-dev/hmft/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/hmft-dev/hmft/issues
+Keywords: llm,latency,estimate,eta,streaming,tokens,tokenizer,openai,anthropic,benchmark
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tiktoken>=0.7
+Requires-Dist: httpx>=0.25
+Provides-Extra: openai
+Requires-Dist: openai>=1.30; extra == "openai"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30; extra == "anthropic"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: openai>=1.30; extra == "dev"
+Requires-Dist: anthropic>=0.30; extra == "dev"
+Dynamic: license-file
+
+# hmft
+
+Estimate how long an LLM call will take **before you send it**, and get a live ETA while it
+streams. Runs entirely on your machine: no prompt or response text ever leaves the process.
+
+```bash
+pip install hmft                      # from PyPI
+pip install "hmft[openai,anthropic]"  # with the SDK wrappers
+pip install -e ".[dev]"               # from a clone, with the test extras
+```
+
+## Quickstart
+
+No API key, no network call: the estimate is computed locally from the prompt.
+
+```python
+import hmft
+
+est = hmft.estimate(
+    [{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
+    model="gpt-4.1-mini", provider="openai",
+)
+print(est)                                      # ~2.3s (p90 4.6s): ttft 0.69s, ~220 tokens [measured]
+print(est.total_s.p50, est.total_s.p90)         # 2.27 4.58
+print(est.output_tokens.p50, est.output_tokens.p90)  # 220 352
+```
+
+Every estimate is a **band, not a point**: `p50` is the typical case, `p90` is the slow-but-likely
+case you should plan around.
+
+## Accuracy on held-out prompts
+
+186 rows from prompt sets A to D, four models, re-scored under the current code with
+`scripts/compare_runs.py`. "Coverage" is the share of rows whose actual landed at or under
+the estimated p90; the target is about 90%.
+
+**What "held out" means here, precisely.** These 186 rows are held out from **v2** constant
+fitting: `compare_runs.py` marks sets A to D held-out and fits only on the later sets (E, F).
+They are *not* untouched by v1. Set A is the v1 calibration set, and four v1.1 constants were
+fitted or corrected using rows that are in this table — `TOKENS_PER_LIST_ITEM` (set B
+actuals), `TOKENS_PER_ONE_LINE_ITEM_P50/P90` (A, B and C), the spoken-duration cue
+`490 + 59 × minutes` (C and D), and the `code_large` band (the `long_code` actuals in A, B
+and C). Read the table below as a standing regression score, not as a clean out-of-sample
+result. The cleanest out-of-sample evidence is set E, measured on different subjects, where
+the duration cue covered 29 of 30 rows.
+
+| model | held-out rows | token p90 coverage | time p90 coverage |
+|---|---|---|---|
+| gpt-4.1-mini | 46 | 100% | 98% |
+| claude-haiku-4-5 | 47 | 96% | 100% |
+| claude-sonnet-4-5 | 47 | 100% | 94% |
+| gpt-5@low | 46 | 100% † | 100% † |
+| **all non-reasoning** | **140** | **99%** | **97%** |
+
+† **gpt-5@low passes neither figure, despite the numbers, because all three of its table
+constants are wrong.** The row assumes 1,312 hidden reasoning tokens, where the split runs
+measured medians of 320 (set E) and 96 (set F); a 9.5 s first token, where the runs measured
+per-run medians of 4.1 to 7.2 s; and 123 tokens/s decode, where the runs measured 34.9 to
+108.6 tokens/s (per-run medians 58 to 81). So hmft over-estimates time for gpt-5@low (set F
+median actual/estimate 0.55) and under-estimates visible output on long-form prompts (set E
+1.27 and 1.34). The 100% coverage above is those two errors cancelling, not a pass. The
+rebuild ships in 0.3.0. See `RESULTS.md`.
+
+All four time-coverage breaches across the 140 non-reasoning rows were caused by time to
+first token, not by the length heuristics; the token estimate was inside p90 in every one.
+
+## Supported models
+
+| model | provider | status | basis |
+|---|---|---|---|
+| `gpt-4.1-mini` | openai | measured | 10 benchmark samples; 46 held-out rows |
+| `claude-haiku-4-5` | anthropic | measured | 10 benchmark samples; 47 held-out rows |
+| `claude-sonnet-4-5` | anthropic | measured | 40 benchmark samples; 47 held-out rows |
+| `gpt-5@low` | openai | experimental | 10 benchmark samples; hidden tokens, first token and decode rate all mis-set (above): over-estimates time, under-estimates tokens |
+| `gpt-5@medium` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `gpt-5-mini` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `gpt-4.1` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `openai/gpt-4.1-mini`, `openai/gpt-4.1`, `openai/gpt-5-mini`, `anthropic/claude-sonnet-4.5`, `anthropic/claude-haiku-4.5` | openrouter | experimental | 6 to 10 benchmark samples each; no held-out rows |
+| anything else | any | not measured | `basis="unmeasured"`: token band only, no timing |
+
+**measured** means the throughput row comes from a real benchmark run *and* the model has
+held-out coverage above. **experimental** means the throughput row is real but the estimate
+has not been scored against held-out prompts, or was scored and did not pass. **not
+measured** means hmft refuses to invent a number: you get an output-token band and
+`basis="unmeasured"`, never a fabricated time.
+
+A bare `gpt-5` call with no effort set maps to the `gpt-5@medium` row, the provider default,
+and `basis_key` says so. OpenRouter has no provider-wide fallback row, because it routes
+across model families; an unbenchmarked slug is `unmeasured`.
+
+## Privacy
+
+**No prompt text and no response text is ever stored, logged, printed or transmitted.**
+Nothing is uploaded anywhere, by any code path, in v1. There is no server.
+
+After each wrapped call hmft appends one JSON line to `~/.hmft/telemetry.jsonl`, created
+`0600`, so you can score your own accuracy offline. The row is an allow-list of integers,
+floats and short enum words — exactly these 24 fields and nothing else:
+
+`ts`, `hmft_version`, `model`, `provider`, `prompt_tokens`, `output_tokens`, `ttft_ms`,
+`total_ms`, `est_output_p50`, `est_output_p90`, `est_ttft_p50_s`, `est_ttft_p90_s`,
+`est_total_p50_s`, `est_total_p90_s`, `basis`, `reasoning_effort`, `effort_source`,
+`reasoning_tokens`, `finish_reason`, `visible_chunks`, `first_second_visible_chunks`,
+`last_visible_ms`, `first_event_ms`, `max_gap_ms`.
+
+No prompt, no response, no request ids, no user ids. `reasoning_effort`, `effort_source`,
+`basis` and `finish_reason` are validated against fixed sets, so free text is dropped rather
+than written. The write is enforced by an assertion against the allow-list in
+`hmft/telemetry.py`, and `tests/test_telemetry.py` checks it.
+
+Opt out with one environment variable:
+
+```bash
+HMFT_TELEMETRY=off                     # write nothing
+HMFT_TELEMETRY=/path/to/file.jsonl     # or move it somewhere else
+```
+
+## The name
+
+HMFT is short for **"how much fucking time"** — the question you actually ask when a model has
+been streaming for forty seconds and you have no idea whether it is nearly done or has barely
+started. The package, the import and the docs are all just `hmft`; the long form lives here and
+nowhere else.
+
+## How estimates work
+
+```
+total_time ≈ TTFT(prompt_tokens, model) + output_tokens / tokens_per_second(model)
+```
+
+1. **Output length comes from cues in the prompt** (`hmft/length.py`), in priority order:
+   an explicit count ("in one sentence", "about 500 words", "8 questions", "one per line"),
+   a stated speaking duration ("a 10-minute keynote" → `490 + 59 × minutes` tokens, validated
+   2 to 45 minutes), a forced format (JSON mode, forced tool call, `max_tokens`), and finally
+   a task type guessed from keywords (code, list, JSON, yes/no, translate, summarise, essay,
+   explain, chat), each with its own base band.
+2. **TTFT and tokens/second are measured constants**, per model and provider, in
+   `hmft/data/throughput.json`. They are only ever written by `scripts/benchmark.py` from real
+   runs — never estimated, never hand-edited. Unmeasured rows stay `null` and the estimator
+   reports `basis="unmeasured"` instead of guessing.
+3. **The band comes from both.** The p50 time uses the throughput `p50`; the p90 time uses the
+   throughput **`p10`**, because for a rate the slow side is the low percentile. The length
+   heuristic contributes its own p50/p90, and the two compose into `total_s`.
+
+Measured throughput, as of the `updated_at` in the table:
+
+| model | TTFT p50 / p90 | tokens/s p50 / p10 | samples |
+|---|---|---|---|
+| gpt-4.1-mini | 0.69 s / 1.622 s | 139.2 / 119.1 | 10 |
+| gpt-4.1 | 0.76 s / 2.229 s | 111.3 / 90.5 | 10 |
+| claude-haiku-4-5 | 0.67 s / 0.686 s | 83.4 / 79.5 | 10 |
+| claude-sonnet-4-5 | 0.86 s / 1.355 s | 36.8 / 34.6 | 40 |
+| gpt-5@low | 9.5 s / 17.52 s | 123.0 / 69.4 | 10 |
+
+On reasoning models the TTFT column absorbs hidden thinking time, which is why it is large and
+why those rows are experimental.
+
+## Streaming ETA
+
+```python
+from openai import OpenAI
+import hmft
+
+client = hmft.wrap_openai(OpenAI())
+stream = client.chat.completions.create(
+    model="gpt-4.1-mini", stream=True,
+    messages=[{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
+)
+print(stream.estimate)             # pre-call band
+for chunk in stream:
+    print(stream.eta(), end="\r")  # live: blends the observed rate in after a few tokens
+print(stream.result)               # actual vs estimated
+```
+
+`hmft.wrap_anthropic(Anthropic())` is the same shape around `client.messages.stream(...)`.
+Wrappers are sync-only in v1; async is a v1.x item.
+
+## Filling the throughput table
+
+```bash
+export OPENAI_API_KEY=...       # and/or ANTHROPIC_API_KEY, OPENROUTER_API_KEY
+python scripts/benchmark.py --all --dry-run    # print the plan, make no calls
+python scripts/benchmark.py --provider anthropic --models claude-haiku-4-5 --runs 3
+```
+
+Each run sends two synthetic prompts and records only token counts and timings.
+`name@effort` (for example `gpt-5@low`) sets `reasoning_effort` and gets its own row.
+
+## Known limits
+
+- Output-length heuristics are rules, not a model. Expect the p50 to be off by 2x on
+  open-ended prompts; the p90 band is what to plan around.
+- Reasoning models spend most of their wall time on hidden tokens the heuristics cannot see.
+- Non-OpenAI token counts use the o200k tokenizer as a proxy, typically within 10 to 20%.
+- Image and tool-result content blocks are not counted.
+- Talk-duration cues are validated 2 to 45 minutes; above 45 is extrapolation. The p90 held on
+  29 of 30 measured talks, but the p50 is biased per model: gpt-4.1-mini writes far less than
+  the estimate on long talks, Claude models somewhat more.
+- Provider latency drifts hour to hour; `measured_at` tells you how stale a row is.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+python -m pytest
+```
+
+All tests use synthetic prompts and fake clients, and the suite makes no network calls —
+`tests/test_no_network.py` enforces that. See `CONTRIBUTING.md` for the held-out rule and the
+version-bump rule. `BASELINE.md` is the frozen v1 reference and `RESULTS.md` records every
+measurement since.
+
+## License
+
+Apache-2.0. See `LICENSE`.

hmft-0.2.0/README.md

@@ -0,0 +1,215 @@
+# hmft
+
+Estimate how long an LLM call will take **before you send it**, and get a live ETA while it
+streams. Runs entirely on your machine: no prompt or response text ever leaves the process.
+
+```bash
+pip install hmft                      # from PyPI
+pip install "hmft[openai,anthropic]"  # with the SDK wrappers
+pip install -e ".[dev]"               # from a clone, with the test extras
+```
+
+## Quickstart
+
+No API key, no network call: the estimate is computed locally from the prompt.
+
+```python
+import hmft
+
+est = hmft.estimate(
+    [{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
+    model="gpt-4.1-mini", provider="openai",
+)
+print(est)                                      # ~2.3s (p90 4.6s): ttft 0.69s, ~220 tokens [measured]
+print(est.total_s.p50, est.total_s.p90)         # 2.27 4.58
+print(est.output_tokens.p50, est.output_tokens.p90)  # 220 352
+```
+
+Every estimate is a **band, not a point**: `p50` is the typical case, `p90` is the slow-but-likely
+case you should plan around.
+
+## Accuracy on held-out prompts
+
+186 rows from prompt sets A to D, four models, re-scored under the current code with
+`scripts/compare_runs.py`. "Coverage" is the share of rows whose actual landed at or under
+the estimated p90; the target is about 90%.
+
+**What "held out" means here, precisely.** These 186 rows are held out from **v2** constant
+fitting: `compare_runs.py` marks sets A to D held-out and fits only on the later sets (E, F).
+They are *not* untouched by v1. Set A is the v1 calibration set, and four v1.1 constants were
+fitted or corrected using rows that are in this table — `TOKENS_PER_LIST_ITEM` (set B
+actuals), `TOKENS_PER_ONE_LINE_ITEM_P50/P90` (A, B and C), the spoken-duration cue
+`490 + 59 × minutes` (C and D), and the `code_large` band (the `long_code` actuals in A, B
+and C). Read the table below as a standing regression score, not as a clean out-of-sample
+result. The cleanest out-of-sample evidence is set E, measured on different subjects, where
+the duration cue covered 29 of 30 rows.
+
+| model | held-out rows | token p90 coverage | time p90 coverage |
+|---|---|---|---|
+| gpt-4.1-mini | 46 | 100% | 98% |
+| claude-haiku-4-5 | 47 | 96% | 100% |
+| claude-sonnet-4-5 | 47 | 100% | 94% |
+| gpt-5@low | 46 | 100% † | 100% † |
+| **all non-reasoning** | **140** | **99%** | **97%** |
+
+† **gpt-5@low passes neither figure, despite the numbers, because all three of its table
+constants are wrong.** The row assumes 1,312 hidden reasoning tokens, where the split runs
+measured medians of 320 (set E) and 96 (set F); a 9.5 s first token, where the runs measured
+per-run medians of 4.1 to 7.2 s; and 123 tokens/s decode, where the runs measured 34.9 to
+108.6 tokens/s (per-run medians 58 to 81). So hmft over-estimates time for gpt-5@low (set F
+median actual/estimate 0.55) and under-estimates visible output on long-form prompts (set E
+1.27 and 1.34). The 100% coverage above is those two errors cancelling, not a pass. The
+rebuild ships in 0.3.0. See `RESULTS.md`.
+
+All four time-coverage breaches across the 140 non-reasoning rows were caused by time to
+first token, not by the length heuristics; the token estimate was inside p90 in every one.
+
+## Supported models
+
+| model | provider | status | basis |
+|---|---|---|---|
+| `gpt-4.1-mini` | openai | measured | 10 benchmark samples; 46 held-out rows |
+| `claude-haiku-4-5` | anthropic | measured | 10 benchmark samples; 47 held-out rows |
+| `claude-sonnet-4-5` | anthropic | measured | 40 benchmark samples; 47 held-out rows |
+| `gpt-5@low` | openai | experimental | 10 benchmark samples; hidden tokens, first token and decode rate all mis-set (above): over-estimates time, under-estimates tokens |
+| `gpt-5@medium` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `gpt-5-mini` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `gpt-4.1` | openai | experimental | 10 benchmark samples; no held-out rows |
+| `openai/gpt-4.1-mini`, `openai/gpt-4.1`, `openai/gpt-5-mini`, `anthropic/claude-sonnet-4.5`, `anthropic/claude-haiku-4.5` | openrouter | experimental | 6 to 10 benchmark samples each; no held-out rows |
+| anything else | any | not measured | `basis="unmeasured"`: token band only, no timing |
+
+**measured** means the throughput row comes from a real benchmark run *and* the model has
+held-out coverage above. **experimental** means the throughput row is real but the estimate
+has not been scored against held-out prompts, or was scored and did not pass. **not
+measured** means hmft refuses to invent a number: you get an output-token band and
+`basis="unmeasured"`, never a fabricated time.
+
+A bare `gpt-5` call with no effort set maps to the `gpt-5@medium` row, the provider default,
+and `basis_key` says so. OpenRouter has no provider-wide fallback row, because it routes
+across model families; an unbenchmarked slug is `unmeasured`.
+
+## Privacy
+
+**No prompt text and no response text is ever stored, logged, printed or transmitted.**
+Nothing is uploaded anywhere, by any code path, in v1. There is no server.
+
+After each wrapped call hmft appends one JSON line to `~/.hmft/telemetry.jsonl`, created
+`0600`, so you can score your own accuracy offline. The row is an allow-list of integers,
+floats and short enum words — exactly these 24 fields and nothing else:
+
+`ts`, `hmft_version`, `model`, `provider`, `prompt_tokens`, `output_tokens`, `ttft_ms`,
+`total_ms`, `est_output_p50`, `est_output_p90`, `est_ttft_p50_s`, `est_ttft_p90_s`,
+`est_total_p50_s`, `est_total_p90_s`, `basis`, `reasoning_effort`, `effort_source`,
+`reasoning_tokens`, `finish_reason`, `visible_chunks`, `first_second_visible_chunks`,
+`last_visible_ms`, `first_event_ms`, `max_gap_ms`.
+
+No prompt, no response, no request ids, no user ids. `reasoning_effort`, `effort_source`,
+`basis` and `finish_reason` are validated against fixed sets, so free text is dropped rather
+than written. The write is enforced by an assertion against the allow-list in
+`hmft/telemetry.py`, and `tests/test_telemetry.py` checks it.
+
+Opt out with one environment variable:
+
+```bash
+HMFT_TELEMETRY=off                     # write nothing
+HMFT_TELEMETRY=/path/to/file.jsonl     # or move it somewhere else
+```
+
+## The name
+
+HMFT is short for **"how much fucking time"** — the question you actually ask when a model has
+been streaming for forty seconds and you have no idea whether it is nearly done or has barely
+started. The package, the import and the docs are all just `hmft`; the long form lives here and
+nowhere else.
+
+## How estimates work
+
+```
+total_time ≈ TTFT(prompt_tokens, model) + output_tokens / tokens_per_second(model)
+```
+
+1. **Output length comes from cues in the prompt** (`hmft/length.py`), in priority order:
+   an explicit count ("in one sentence", "about 500 words", "8 questions", "one per line"),
+   a stated speaking duration ("a 10-minute keynote" → `490 + 59 × minutes` tokens, validated
+   2 to 45 minutes), a forced format (JSON mode, forced tool call, `max_tokens`), and finally
+   a task type guessed from keywords (code, list, JSON, yes/no, translate, summarise, essay,
+   explain, chat), each with its own base band.
+2. **TTFT and tokens/second are measured constants**, per model and provider, in
+   `hmft/data/throughput.json`. They are only ever written by `scripts/benchmark.py` from real
+   runs — never estimated, never hand-edited. Unmeasured rows stay `null` and the estimator
+   reports `basis="unmeasured"` instead of guessing.
+3. **The band comes from both.** The p50 time uses the throughput `p50`; the p90 time uses the
+   throughput **`p10`**, because for a rate the slow side is the low percentile. The length
+   heuristic contributes its own p50/p90, and the two compose into `total_s`.
+
+Measured throughput, as of the `updated_at` in the table:
+
+| model | TTFT p50 / p90 | tokens/s p50 / p10 | samples |
+|---|---|---|---|
+| gpt-4.1-mini | 0.69 s / 1.622 s | 139.2 / 119.1 | 10 |
+| gpt-4.1 | 0.76 s / 2.229 s | 111.3 / 90.5 | 10 |
+| claude-haiku-4-5 | 0.67 s / 0.686 s | 83.4 / 79.5 | 10 |
+| claude-sonnet-4-5 | 0.86 s / 1.355 s | 36.8 / 34.6 | 40 |
+| gpt-5@low | 9.5 s / 17.52 s | 123.0 / 69.4 | 10 |
+
+On reasoning models the TTFT column absorbs hidden thinking time, which is why it is large and
+why those rows are experimental.
+
+## Streaming ETA
+
+```python
+from openai import OpenAI
+import hmft
+
+client = hmft.wrap_openai(OpenAI())
+stream = client.chat.completions.create(
+    model="gpt-4.1-mini", stream=True,
+    messages=[{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
+)
+print(stream.estimate)             # pre-call band
+for chunk in stream:
+    print(stream.eta(), end="\r")  # live: blends the observed rate in after a few tokens
+print(stream.result)               # actual vs estimated
+```
+
+`hmft.wrap_anthropic(Anthropic())` is the same shape around `client.messages.stream(...)`.
+Wrappers are sync-only in v1; async is a v1.x item.
+
+## Filling the throughput table
+
+```bash
+export OPENAI_API_KEY=...       # and/or ANTHROPIC_API_KEY, OPENROUTER_API_KEY
+python scripts/benchmark.py --all --dry-run    # print the plan, make no calls
+python scripts/benchmark.py --provider anthropic --models claude-haiku-4-5 --runs 3
+```
+
+Each run sends two synthetic prompts and records only token counts and timings.
+`name@effort` (for example `gpt-5@low`) sets `reasoning_effort` and gets its own row.
+
+## Known limits
+
+- Output-length heuristics are rules, not a model. Expect the p50 to be off by 2x on
+  open-ended prompts; the p90 band is what to plan around.
+- Reasoning models spend most of their wall time on hidden tokens the heuristics cannot see.
+- Non-OpenAI token counts use the o200k tokenizer as a proxy, typically within 10 to 20%.
+- Image and tool-result content blocks are not counted.
+- Talk-duration cues are validated 2 to 45 minutes; above 45 is extrapolation. The p90 held on
+  29 of 30 measured talks, but the p50 is biased per model: gpt-4.1-mini writes far less than
+  the estimate on long talks, Claude models somewhat more.
+- Provider latency drifts hour to hour; `measured_at` tells you how stale a row is.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+python -m pytest
+```
+
+All tests use synthetic prompts and fake clients, and the suite makes no network calls —
+`tests/test_no_network.py` enforces that. See `CONTRIBUTING.md` for the held-out rule and the
+version-bump rule. `BASELINE.md` is the frozen v1 reference and `RESULTS.md` records every
+measurement since.
+
+## License
+
+Apache-2.0. See `LICENSE`.