evo2-flash 0.1.0__tar.gz

evo2_flash-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

evo2_flash-0.1.0/NOTICE

@@ -0,0 +1,6 @@
+evo2_flash
+
+This product includes the Evo 2 model configuration files in
+evo2_flash/evo2_configs/, taken unchanged from the evo2 package
+(https://github.com/ArcInstitute/evo2), which is licensed under the
+Apache License, Version 2.0.

evo2_flash-0.1.0/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.4
+Name: evo2-flash
+Version: 0.1.0
+Summary: Evo 2 / StripedHyena2 on vLLM: an out-of-tree plugin with a drop-in evo2.Evo2 API
+Author: Garyk Brixi, Yukun Alex Hao
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/evo-design/evo2_flash
+Requires-Python: <3.13,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: torch
+Requires-Dist: transformers
+Requires-Dist: safetensors
+Requires-Dist: numpy
+Requires-Dist: pyyaml
+Requires-Dist: huggingface_hub
+Requires-Dist: filelock
+Dynamic: license-file
+
+# evo2_flash
+
+Evo 2 (StripedHyena2) on vLLM, with a drop-in replacement for `evo2.Evo2`. Evo2_flash supports more accessible genomic ICL across diverse biological systems.
+
+## Requirements and install
+
+- vLLM **0.26.0**, the `+cu129` wheel (the PyPI default is cu130), with torch 2.11 and
+  Python 3.12. vLLM is not a declared dependency, so install it first.
+- One GPU; the numbers below are from H100/H200. The fp8 input projections that the 1B,
+  7B, 20B and 40B checkpoints ship with run as fp8 GEMMs on sm_89+ (L4, L40S, H100,
+  H200). Older GPUs (A100, RTX 3090, A40) need the fallback described under
+  [GPUs without FP8 tensor cores](#gpus-without-fp8-tensor-cores). 40B needs an H200,
+  or fp8 weights on an H100.
+
+```bash
+# vLLM 0.26.0, CUDA 12.9 build: runs on CUDA 12 and CUDA 13 drivers.
+# (PyPI's default vllm build is CUDA 13 and cannot use the GPU on a CUDA 12 driver.)
+pip install "https://github.com/vllm-project/vllm/releases/download/v0.26.0/vllm-0.26.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
+    --extra-index-url https://download.pytorch.org/whl/cu129
+pip install evo2-flash   # registers the plugin via the vllm.general_plugins entry point
+export VLLM_ENABLE_V1_MULTIPROCESSING=0   # Evo2 sets this itself; the API needs the engine in-process
+```
+
+`dtype="bfloat16"` is mandatory: Evo 2's residual stream reaches ~1e12, and fp16 gives NaN.
+
+### Check the install
+
+```bash
+python -m evo2_flash.test.test_evo2 --model_name evo2_7b
+```
+
+This is Evo 2's own correctness test (`python -m evo2.test.test_evo2`), run through
+evo2_flash. It scores four sequences and passes, with exit code 0, when the mean loss is
+within 1e-3 of the value the Evo 2 authors published. On one H100 the delta is 2.92e-4
+for `evo2_7b` and 4.96e-4 for `evo2_1b_base`. Each run takes about a minute once the
+checkpoint is converted. `--local_path` points it at a converted directory, and
+`--kv_cache_dtype fp8` checks the fp8 KV cache, with a tolerance of 5e-3 (H100,
+`evo2_7b`: 9.31e-4).
+
+## Quick start
+
+```python
+# from evo2 import Evo2
+from evo2_flash import Evo2
+
+model = Evo2("evo2_7b")
+scores = model.score_sequences(seqs)        # mean log-likelihood per sequence
+out = model.generate(prompts, n_tokens=500, temperature=1.0, top_k=4)
+out.sequences, out.logprobs_mean
+logits, emb = model(input_ids, return_embeddings=True, layer_names=["blocks.28.mlp.l3"])
+```
+
+Pass a whole prompt list in one call. vLLM batches ragged lengths continuously, and a
+repeated prompt's prefill is shared through the prefix cache.
+
+Lower level, with plain `vllm.LLM`. The tokenizer is byte-level, so token ids are UTF-8
+bytes:
+
+```python
+import evo2_flash; evo2_flash.register()    # not needed once pip-installed
+from vllm import LLM, SamplingParams
+from vllm.inputs import TokensPrompt
+
+llm = LLM(model="/path/to/evo2_7b_hf", skip_tokenizer_init=True, dtype="bfloat16",
+          enable_prefix_caching=True, max_num_seqs=128)
+outs = llm.generate([TokensPrompt(prompt_token_ids=list(b"ACGT..."))],
+                    SamplingParams(max_tokens=500, temperature=1.0, top_k=4,
+                                   ignore_eos=True, detokenize=False))   # EOD is id 0
+bytes(outs[0].outputs[0].token_ids).decode("ascii", "replace")
+```
+
+`evo2_flash.api.engine_kwargs(ckpt)` returns the validated engine configuration that
+`Evo2` uses. On a GPU without FP8 tensor cores, pass the checkpoint through
+`evo2_flash.api.checkpoint_for_this_gpu(path)` first. `Evo2` does this itself, but a
+plain `LLM` does not.
+
+## Drop-in replacement for evo2.Evo2
+
+| evo2 | evo2_flash | difference |
+|---|---|---|
+| `Evo2(name, local_path=None, use_kernels=False)` | same | `local_path` may also be a converted dir; `use_kernels` ignored |
+| `model.tokenizer` (`CharLevelTokenizer(512)`) | same semantics | `tokenize` returns `int`s, not `np.uint8` |
+| `score_sequences(seqs, batch_size, prepend_bos, reduce_method, average_reverse_complement)` | same | one vLLM call; log-softmax in fp32 (evo2: bf16) |
+| `generate(prompt_seqs, n_tokens, temperature, top_k, top_p, batched, cached_generation, verbose, force_prompt_threshold)` | same, + `seed=`, `prime_prefix_cache=`, CFG arguments | one vLLM call; `temperature=0` is greedy; `logits` lazy |
+| `forward(input_ids, return_embeddings, layer_names)` / `model(...)` | same | vortex names mapped (table below); outputs on the GPU |
+| `model.model` (vortex `StripedHyena`) | `model.model` (`StripedHyena2ForCausalLM`) | plugin module names; use `layer_names` for vortex names |
+
+Semantic differences, all deliberate or unavoidable:
+
+- **Scoring.** All sequences go into one vLLM call, so `batch_size` has no effect. Scores
+  match evo2's `batch_size=1`. evo2 with `batch_size>1`, ragged lengths and
+  `prepend_bos=False` also scores one spurious pad position for each shorter sequence.
+  `score_sequences` requests prompt logprobs, which turns off the prefix-cache read.
+- **Sampling.** Matches vortex `sample()`, and `top_k=1` is argmax at any temperature.
+  `top_k<=0` means the whole vocabulary, and `top_p<=0` or `>=1` turns top-p off.
+  `temperature=0` with `top_k!=1` is greedy here, whereas vortex divides by zero. With
+  `seed=s`, request i uses seed `s+i` and reproduces exactly.
+- **Length / EOS.** Every prompt gets exactly `n_tokens`, and EOD does not stop
+  generation, as in evo2. `logprobs_mean` is the mean log-probability of the sampled
+  tokens under the raw, untempered distribution. `batched`, `cached_generation` and
+  `force_prompt_threshold` are accepted but do nothing.
+- **`GenerationOutput.logits`** has vortex's layout. It is computed on first access by
+  one extra prefill over prompt + completion, so the values are prefill logits. They
+  agree with decode-time logits to bf16 precision. Extra fields: `token_ids` and
+  `num_cached_tokens`.
+- **forward().** Each row of `input_ids` is an independent causal sequence, and pad ids
+  are ordinary tokens. Logits come back bf16 `(B, L, 512)`. CUDA graphs are switched off
+  for these steps only, because a replayed graph does not run Python hooks.
+
+`layer_names` takes vortex module names (native names such as `model.layers.3.mixer` are
+also accepted):
+
+| vortex name | plugin module (hooked tensor) |
+|---|---|
+| `embedding_layer` | `model.embed_tokens` |
+| `blocks.N` | `model.layers.N` (residual stream after block N) |
+| `blocks.N.pre_norm`, `blocks.N.post_norm` | `model.layers.N.{pre,post}_norm` |
+| `blocks.N.mlp`, `blocks.N.mlp.l3` | `model.layers.N.mlp` / `.mlp.l3` (MLP output, before the residual add) |
+| `blocks.N.mlp.l1`, `blocks.N.mlp.l2` | `model.layers.N.mlp.l1` / `.l2` (before the gate) |
+| `blocks.N.projections` (Hyena) | `model.layers.N.mixer.in_proj` (3H, fp8 GEMM output) |
+| `blocks.N.filter` (Hyena) | **input** of `model.layers.N.mixer.out_proj` (gated Hyena output) |
+| `blocks.N.out_filter_dense` (Hyena) | `model.layers.N.mixer.out_proj` (mixer output, before the residual add) |
+| `blocks.N.inner_mha_cls`, `.inner_mha_cls.out_proj` (attention) | `model.layers.N.mixer` / `.mixer.o_proj` |
+| `norm` | `model.norm` |
+| `unembed` | the returned logits |
+
+The following names raise `ValueError`:
+- `inner_mha_cls.Wqkv`, because the converter permutes its rows
+- rotary, inner-attention and HyenaCascade internals
+- a Hyena name used on an attention block, and the reverse
+
+
+## Options
+
+| `Evo2(...)` argument | effect |
+|---|---|
+| `kv_cache_dtype="fp8"` | 2x KV capacity. Lossy: moves the upstream gate by 1.2-1.8e-3 (7B/20B/40B) |
+| `quantization="fp8"` (engine override) | fp8 weights: 2.2e-3 (40B) / 2.95e-3 (7B) on the gate; fits 40B on one 80 GB GPU |
+| `speculative=<head dir>` | a trained EAGLE/EAGLE-3 draft head, K=3, probabilistic drafting: 1.3-1.7x at 7B, 1.3-2.0x at 20B |
+| `cfg=True` | installs the classifier-free-guidance logits processor, enabling `generate(..., guidance=w)` |
+| `max_model_len`, `gpu_memory_utilization` (0.85), `cache_dir` | engine sizing and the checkpoint cache |
+
+Classifier-free guidance computes `guided = l_cond + w * (l_cond - l_base)`. The baseline
+is `baseline_prompt` (`cfg_type="uncond"`) or `negative_prompts[i]` (`cfg_type="negative"`),
+and `dna_only=True` restricts sampling to A/C/G/T. CFG and speculative decoding are mutually
+exclusive, because vLLM rejects custom logits processors under speculation.
+
+fp8-surgery checkpoints run without fp8 GEMMs. In these checkpoints block 0 is served
+from an exact token table, and optionally some Hyena layers emulate Hopper fp8
+accumulation (`hyena_token_table_layers`, `hyena_fp8_emulated_layers`,
+`hyena_fp8_emulated_channels` in the config). They load like any other converted
+checkpoint. The 1B's table variant is built by
+`evo2_flash.convert.build_token_table_checkpoint`; the others come from
+`scripts/fp8_surgery.py` in the source repository.
+
+Other `vllm.LLM` arguments pass through with a warning. `dtype`, `tokenizer`,
+`skip_tokenizer_init` and `model` cannot be overridden.
+
+## Checkpoints
+
+`Evo2("evo2_7b")` looks for `$EVO2_VLLM_CACHE/evo2_7b_hf`, defaulting to
+`$HF_HOME/evo2_vllm`. If that directory is missing, it finds the vortex `.pt` the way evo2
+does, downloading the HF snapshot if absent, and converts it once with
+`evo2_flash.convert.convert_checkpoint`. The vortex configs ship in
+`evo2_flash/evo2_configs/`, so evo2 and vortex need not be installed. 40B's `.pt.part0/.part1`
+shards are read in place.
+
+You can also pass a converted directory, `Evo2("/path/to/evo2_7b_hf")`, or a finetuned
+vortex `.pt`, `Evo2("evo2_7b", local_path="ft.pt")`. Checkpoints converted by
+`vllm_evo2` load unchanged.
+
+Names: `evo2_7b`, `evo2_7b_262k`, `evo2_7b_base`, `evo2_7b_microviridae`, `evo2_1b_base`,
+`evo2_20b`, `evo2_40b`, `evo2_40b_base` (`evo2_flash.MODEL_NAMES`).
+
+## GPUs without FP8 tensor cores
+
+On sm < 8.9 (A100, RTX 3090, A40), vLLM serves the fp8 in_proj as weight-only Marlin.
+That works for the 7B, which scores 2.5-3.8e-4 on the gate on an A100 and an RTX 3090.
+The stock 1B, however, depends on Hopper's FP8 rounding in block 0 and loses all signal
+there: gate loss 0.805 against the authors' 0.502. `Evo2(...)` handles this through
+`checkpoint_for_this_gpu`:
+
+- **1B and 20B:** the published checkpoints carry a `surgery/` sidecar. On these GPUs
+  `Evo2(...)` builds the FP8-free form from it once into the checkpoint cache, and it
+  passes the strict gate. `fp8_mode="surgery"` forces that form on any GPU.
+- **A 1B converted locally** (no sidecar): swapped for its token-table variant, built
+  once (about 20 s).
+- **A 20B without a sidecar, and the 40B:** raise with instructions.
+- **every 7B, and any surgery checkpoint:** used unchanged, as on sm_89+.
+
+`EVO2_FP8_FALLBACK=off` keeps the stock checkpoint (diagnostics). On 24 GB cards such
+as the L4, CFG sizes its admission waves to what the KV cache holds.
+
+## Limitations
+
+- One GPU per model. Tensor parallelism is not supported, because Hyena channels are
+  interleaved per triple, and pipeline parallelism is not supported either.
+- Trained draft heads exist for 7B and 20B only.
+
+## Third-party code
+
+- `evo2_flash/triton_hcl.py`: the fused modal-filter kernel, vendored from
+  [Zymrael/vortex](https://github.com/Zymrael/vortex) (Apache-2.0).
+- `evo2_flash/evo2_configs/*.yml`: the model configs from evo2 0.6.0 (Apache-2.0).
+- `evo2_flash/test/data/prompts.csv`: evo2's test sequences (Apache-2.0).

evo2_flash-0.1.0/README.md

@@ -0,0 +1,205 @@
+# evo2_flash
+
+Evo 2 (StripedHyena2) on vLLM, with a drop-in replacement for `evo2.Evo2`. Evo2_flash supports more accessible genomic ICL across diverse biological systems.
+
+## Requirements and install
+
+- vLLM **0.26.0**, the `+cu129` wheel (the PyPI default is cu130), with torch 2.11 and
+  Python 3.12. vLLM is not a declared dependency, so install it first.
+- One GPU; the numbers below are from H100/H200. The fp8 input projections that the 1B,
+  7B, 20B and 40B checkpoints ship with run as fp8 GEMMs on sm_89+ (L4, L40S, H100,
+  H200). Older GPUs (A100, RTX 3090, A40) need the fallback described under
+  [GPUs without FP8 tensor cores](#gpus-without-fp8-tensor-cores). 40B needs an H200,
+  or fp8 weights on an H100.
+
+```bash
+# vLLM 0.26.0, CUDA 12.9 build: runs on CUDA 12 and CUDA 13 drivers.
+# (PyPI's default vllm build is CUDA 13 and cannot use the GPU on a CUDA 12 driver.)
+pip install "https://github.com/vllm-project/vllm/releases/download/v0.26.0/vllm-0.26.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
+    --extra-index-url https://download.pytorch.org/whl/cu129
+pip install evo2-flash   # registers the plugin via the vllm.general_plugins entry point
+export VLLM_ENABLE_V1_MULTIPROCESSING=0   # Evo2 sets this itself; the API needs the engine in-process
+```
+
+`dtype="bfloat16"` is mandatory: Evo 2's residual stream reaches ~1e12, and fp16 gives NaN.
+
+### Check the install
+
+```bash
+python -m evo2_flash.test.test_evo2 --model_name evo2_7b
+```
+
+This is Evo 2's own correctness test (`python -m evo2.test.test_evo2`), run through
+evo2_flash. It scores four sequences and passes, with exit code 0, when the mean loss is
+within 1e-3 of the value the Evo 2 authors published. On one H100 the delta is 2.92e-4
+for `evo2_7b` and 4.96e-4 for `evo2_1b_base`. Each run takes about a minute once the
+checkpoint is converted. `--local_path` points it at a converted directory, and
+`--kv_cache_dtype fp8` checks the fp8 KV cache, with a tolerance of 5e-3 (H100,
+`evo2_7b`: 9.31e-4).
+
+## Quick start
+
+```python
+# from evo2 import Evo2
+from evo2_flash import Evo2
+
+model = Evo2("evo2_7b")
+scores = model.score_sequences(seqs)        # mean log-likelihood per sequence
+out = model.generate(prompts, n_tokens=500, temperature=1.0, top_k=4)
+out.sequences, out.logprobs_mean
+logits, emb = model(input_ids, return_embeddings=True, layer_names=["blocks.28.mlp.l3"])
+```
+
+Pass a whole prompt list in one call. vLLM batches ragged lengths continuously, and a
+repeated prompt's prefill is shared through the prefix cache.
+
+Lower level, with plain `vllm.LLM`. The tokenizer is byte-level, so token ids are UTF-8
+bytes:
+
+```python
+import evo2_flash; evo2_flash.register()    # not needed once pip-installed
+from vllm import LLM, SamplingParams
+from vllm.inputs import TokensPrompt
+
+llm = LLM(model="/path/to/evo2_7b_hf", skip_tokenizer_init=True, dtype="bfloat16",
+          enable_prefix_caching=True, max_num_seqs=128)
+outs = llm.generate([TokensPrompt(prompt_token_ids=list(b"ACGT..."))],
+                    SamplingParams(max_tokens=500, temperature=1.0, top_k=4,
+                                   ignore_eos=True, detokenize=False))   # EOD is id 0
+bytes(outs[0].outputs[0].token_ids).decode("ascii", "replace")
+```
+
+`evo2_flash.api.engine_kwargs(ckpt)` returns the validated engine configuration that
+`Evo2` uses. On a GPU without FP8 tensor cores, pass the checkpoint through
+`evo2_flash.api.checkpoint_for_this_gpu(path)` first. `Evo2` does this itself, but a
+plain `LLM` does not.
+
+## Drop-in replacement for evo2.Evo2
+
+| evo2 | evo2_flash | difference |
+|---|---|---|
+| `Evo2(name, local_path=None, use_kernels=False)` | same | `local_path` may also be a converted dir; `use_kernels` ignored |
+| `model.tokenizer` (`CharLevelTokenizer(512)`) | same semantics | `tokenize` returns `int`s, not `np.uint8` |
+| `score_sequences(seqs, batch_size, prepend_bos, reduce_method, average_reverse_complement)` | same | one vLLM call; log-softmax in fp32 (evo2: bf16) |
+| `generate(prompt_seqs, n_tokens, temperature, top_k, top_p, batched, cached_generation, verbose, force_prompt_threshold)` | same, + `seed=`, `prime_prefix_cache=`, CFG arguments | one vLLM call; `temperature=0` is greedy; `logits` lazy |
+| `forward(input_ids, return_embeddings, layer_names)` / `model(...)` | same | vortex names mapped (table below); outputs on the GPU |
+| `model.model` (vortex `StripedHyena`) | `model.model` (`StripedHyena2ForCausalLM`) | plugin module names; use `layer_names` for vortex names |
+
+Semantic differences, all deliberate or unavoidable:
+
+- **Scoring.** All sequences go into one vLLM call, so `batch_size` has no effect. Scores
+  match evo2's `batch_size=1`. evo2 with `batch_size>1`, ragged lengths and
+  `prepend_bos=False` also scores one spurious pad position for each shorter sequence.
+  `score_sequences` requests prompt logprobs, which turns off the prefix-cache read.
+- **Sampling.** Matches vortex `sample()`, and `top_k=1` is argmax at any temperature.
+  `top_k<=0` means the whole vocabulary, and `top_p<=0` or `>=1` turns top-p off.
+  `temperature=0` with `top_k!=1` is greedy here, whereas vortex divides by zero. With
+  `seed=s`, request i uses seed `s+i` and reproduces exactly.
+- **Length / EOS.** Every prompt gets exactly `n_tokens`, and EOD does not stop
+  generation, as in evo2. `logprobs_mean` is the mean log-probability of the sampled
+  tokens under the raw, untempered distribution. `batched`, `cached_generation` and
+  `force_prompt_threshold` are accepted but do nothing.
+- **`GenerationOutput.logits`** has vortex's layout. It is computed on first access by
+  one extra prefill over prompt + completion, so the values are prefill logits. They
+  agree with decode-time logits to bf16 precision. Extra fields: `token_ids` and
+  `num_cached_tokens`.
+- **forward().** Each row of `input_ids` is an independent causal sequence, and pad ids
+  are ordinary tokens. Logits come back bf16 `(B, L, 512)`. CUDA graphs are switched off
+  for these steps only, because a replayed graph does not run Python hooks.
+
+`layer_names` takes vortex module names (native names such as `model.layers.3.mixer` are
+also accepted):
+
+| vortex name | plugin module (hooked tensor) |
+|---|---|
+| `embedding_layer` | `model.embed_tokens` |
+| `blocks.N` | `model.layers.N` (residual stream after block N) |
+| `blocks.N.pre_norm`, `blocks.N.post_norm` | `model.layers.N.{pre,post}_norm` |
+| `blocks.N.mlp`, `blocks.N.mlp.l3` | `model.layers.N.mlp` / `.mlp.l3` (MLP output, before the residual add) |
+| `blocks.N.mlp.l1`, `blocks.N.mlp.l2` | `model.layers.N.mlp.l1` / `.l2` (before the gate) |
+| `blocks.N.projections` (Hyena) | `model.layers.N.mixer.in_proj` (3H, fp8 GEMM output) |
+| `blocks.N.filter` (Hyena) | **input** of `model.layers.N.mixer.out_proj` (gated Hyena output) |
+| `blocks.N.out_filter_dense` (Hyena) | `model.layers.N.mixer.out_proj` (mixer output, before the residual add) |
+| `blocks.N.inner_mha_cls`, `.inner_mha_cls.out_proj` (attention) | `model.layers.N.mixer` / `.mixer.o_proj` |
+| `norm` | `model.norm` |
+| `unembed` | the returned logits |
+
+The following names raise `ValueError`:
+- `inner_mha_cls.Wqkv`, because the converter permutes its rows
+- rotary, inner-attention and HyenaCascade internals
+- a Hyena name used on an attention block, and the reverse
+
+
+## Options
+
+| `Evo2(...)` argument | effect |
+|---|---|
+| `kv_cache_dtype="fp8"` | 2x KV capacity. Lossy: moves the upstream gate by 1.2-1.8e-3 (7B/20B/40B) |
+| `quantization="fp8"` (engine override) | fp8 weights: 2.2e-3 (40B) / 2.95e-3 (7B) on the gate; fits 40B on one 80 GB GPU |
+| `speculative=<head dir>` | a trained EAGLE/EAGLE-3 draft head, K=3, probabilistic drafting: 1.3-1.7x at 7B, 1.3-2.0x at 20B |
+| `cfg=True` | installs the classifier-free-guidance logits processor, enabling `generate(..., guidance=w)` |
+| `max_model_len`, `gpu_memory_utilization` (0.85), `cache_dir` | engine sizing and the checkpoint cache |
+
+Classifier-free guidance computes `guided = l_cond + w * (l_cond - l_base)`. The baseline
+is `baseline_prompt` (`cfg_type="uncond"`) or `negative_prompts[i]` (`cfg_type="negative"`),
+and `dna_only=True` restricts sampling to A/C/G/T. CFG and speculative decoding are mutually
+exclusive, because vLLM rejects custom logits processors under speculation.
+
+fp8-surgery checkpoints run without fp8 GEMMs. In these checkpoints block 0 is served
+from an exact token table, and optionally some Hyena layers emulate Hopper fp8
+accumulation (`hyena_token_table_layers`, `hyena_fp8_emulated_layers`,
+`hyena_fp8_emulated_channels` in the config). They load like any other converted
+checkpoint. The 1B's table variant is built by
+`evo2_flash.convert.build_token_table_checkpoint`; the others come from
+`scripts/fp8_surgery.py` in the source repository.
+
+Other `vllm.LLM` arguments pass through with a warning. `dtype`, `tokenizer`,
+`skip_tokenizer_init` and `model` cannot be overridden.
+
+## Checkpoints
+
+`Evo2("evo2_7b")` looks for `$EVO2_VLLM_CACHE/evo2_7b_hf`, defaulting to
+`$HF_HOME/evo2_vllm`. If that directory is missing, it finds the vortex `.pt` the way evo2
+does, downloading the HF snapshot if absent, and converts it once with
+`evo2_flash.convert.convert_checkpoint`. The vortex configs ship in
+`evo2_flash/evo2_configs/`, so evo2 and vortex need not be installed. 40B's `.pt.part0/.part1`
+shards are read in place.
+
+You can also pass a converted directory, `Evo2("/path/to/evo2_7b_hf")`, or a finetuned
+vortex `.pt`, `Evo2("evo2_7b", local_path="ft.pt")`. Checkpoints converted by
+`vllm_evo2` load unchanged.
+
+Names: `evo2_7b`, `evo2_7b_262k`, `evo2_7b_base`, `evo2_7b_microviridae`, `evo2_1b_base`,
+`evo2_20b`, `evo2_40b`, `evo2_40b_base` (`evo2_flash.MODEL_NAMES`).
+
+## GPUs without FP8 tensor cores
+
+On sm < 8.9 (A100, RTX 3090, A40), vLLM serves the fp8 in_proj as weight-only Marlin.
+That works for the 7B, which scores 2.5-3.8e-4 on the gate on an A100 and an RTX 3090.
+The stock 1B, however, depends on Hopper's FP8 rounding in block 0 and loses all signal
+there: gate loss 0.805 against the authors' 0.502. `Evo2(...)` handles this through
+`checkpoint_for_this_gpu`:
+
+- **1B and 20B:** the published checkpoints carry a `surgery/` sidecar. On these GPUs
+  `Evo2(...)` builds the FP8-free form from it once into the checkpoint cache, and it
+  passes the strict gate. `fp8_mode="surgery"` forces that form on any GPU.
+- **A 1B converted locally** (no sidecar): swapped for its token-table variant, built
+  once (about 20 s).
+- **A 20B without a sidecar, and the 40B:** raise with instructions.
+- **every 7B, and any surgery checkpoint:** used unchanged, as on sm_89+.
+
+`EVO2_FP8_FALLBACK=off` keeps the stock checkpoint (diagnostics). On 24 GB cards such
+as the L4, CFG sizes its admission waves to what the KV cache holds.
+
+## Limitations
+
+- One GPU per model. Tensor parallelism is not supported, because Hyena channels are
+  interleaved per triple, and pipeline parallelism is not supported either.
+- Trained draft heads exist for 7B and 20B only.
+
+## Third-party code
+
+- `evo2_flash/triton_hcl.py`: the fused modal-filter kernel, vendored from
+  [Zymrael/vortex](https://github.com/Zymrael/vortex) (Apache-2.0).
+- `evo2_flash/evo2_configs/*.yml`: the model configs from evo2 0.6.0 (Apache-2.0).
+- `evo2_flash/test/data/prompts.csv`: evo2's test sequences (Apache-2.0).