simit 0.1.0__tar.gz

simit-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

simit-0.1.0/PKG-INFO

@@ -0,0 +1,388 @@
+Metadata-Version: 2.4
+Name: simit
+Version: 0.1.0
+Summary: SIMIT-ICL: vision-language models that imagine their own in-context demonstrations at test time
+Author: monurcan
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/monurcan/simit
+Project-URL: Repository, https://github.com/monurcan/simit
+Project-URL: Issues, https://github.com/monurcan/simit/issues
+Keywords: vision-language models,in-context learning,test-time adaptation,unified multimodal models
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: src/simit/skills/assets/MERMAID_LICENSE
+Requires-Dist: torch>=2.12
+Requires-Dist: torchvision
+Requires-Dist: transformers>=4.57
+Requires-Dist: accelerate
+Requires-Dist: safetensors
+Requires-Dist: einops
+Requires-Dist: huggingface_hub
+Requires-Dist: diffusers>=0.36
+Requires-Dist: kernels
+Requires-Dist: flash-linear-attention; platform_system == "Linux"
+Requires-Dist: pillow
+Requires-Dist: numpy
+Requires-Dist: matplotlib
+Requires-Dist: networkx
+Requires-Dist: rdkit
+Requires-Dist: schemdraw
+Requires-Dist: vl-convert-python
+Requires-Dist: playwright
+Requires-Dist: optuna
+Provides-Extra: vllm
+Requires-Dist: vllm>=0.11; extra == "vllm"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: datasets; extra == "dev"
+Dynamic: license-file
+
+# SIMIT-ICL
+
+**Vision-language models that imagine their own in-context demonstrations at test time.**
+
+Given an unlabeled query `(image, question)`, SIMIT-ICL has the model synthesize a few similar
+`(image, question, answer)` examples, verifies them, keeps only the useful ones, and answers the query
+with them in context. No labels, no extra training, and one model does all the work (or a standard VLM
+plus a text-to-image tool).
+
+```python
+from simit import SIMIT
+
+model = SIMIT.from_pretrained("ByteDance-Seed/BAGEL-7B-MoT")
+
+demos  = model.imagine(image, question)          # imagined (image, question, answer) demos
+answer = model.answer(image, question, demos)    # SIMIT-ICL answer
+greedy = model.greedy(image, question)           # standard zero-shot answer, for comparison
+```
+
+How one query is processed (paper, Sec. 3 / App. G):
+
+1. **Zero-shot pass:** greedy answer + confidence `p0` (geometric-mean token probability).
+2. **Adaptive budget (ABA):** `K*(p0)` demos, 0 for queries the model is already sure about.
+3. **Triplet synthesis:** the model proposes `(description, question, answer)` triplets similar to the query.
+4. **Realization:** a router picks one of 17 *skills*. Natural images come from the model's own image
+   head (BAGEL, Lance) or an image-generator tool (standard VLMs). Charts, diagrams, molecules,
+   circuits, tables, flowcharts, etc. are written as code/specs by the model and rendered
+   deterministically, with parse/render errors fed back for repair.
+5. **Critic:** the model scores each image against its description (0-100). Failures are revised and regenerated.
+6. **Difficulty filter (DF):** keep demos whose teacher-forced answer confidence lies in `[t_low, t_high]`.
+7. **ICL answer** with the kept demos in context.
+
+## Installation
+
+```bash
+pip install simit              # BAGEL, Lance, any transformers VLM
+pip install "simit[vllm]"      # + the vLLM engine for standard VLMs (much faster than plain transformers)
+
+# or from source
+git clone https://github.com/monurcan/simit && cd simit && pip install -e .
+```
+
+Every skill dependency is a pip wheel: there is no Node.js, Mermaid CLI, cairo or manual browser
+setup. HTML and Mermaid skills render in headless Chromium through Playwright, with the following
+fallbacks:
+1. Playwright's own browser, if one is installed.
+2. A system `chromium`/`chrome`.
+3. Otherwise SIMIT downloads Playwright's headless Chromium once on first use (~100 MB, file-locked
+   so parallel processes don't race). If Playwright's downloader fails, as it does behind some HPC
+   networks, it fetches the same build with plain Python (proxy environment variables are honored).
+
+Mermaid itself ships inside the package.
+
+Requirements: Python ≥ 3.10, PyTorch ≥ 2.12 (BAGEL/Lance use `torch.nn.attention.varlen`, so no
+flash-attn build is needed), one CUDA GPU (H100/A100 80GB for the 27B example; BAGEL-7B and Lance
+fit on smaller cards).
+
+## Supported models
+
+| model | how images are made | engine |
+|---|---|---|
+| `ByteDance-Seed/BAGEL-7B-MoT` | native image head + 17 skills, critic in thinking mode (as in the paper) | built-in MoT engine |
+| `bytedance-research/Lance` | native image head; defaults to decomposed synthesis, natural images, no critic (its instruction following is weaker) | built-in MoT engine |
+| any `transformers` VLM, e.g. `Qwen/Qwen3.8-27B` | `image_generator=` tool (e.g. `black-forest-labs/FLUX.2-klein-4B`) + 17 skills; omit the tool for skills only | vLLM if installed, else transformers |
+
+```python
+# a standard VLM + a text-to-image tool
+model = SIMIT.from_pretrained("Qwen/Qwen3.8-27B", image_generator="black-forest-labs/FLUX.2-klein-4B")
+
+# force an engine
+model = SIMIT.from_pretrained("Qwen/Qwen3.8-27B", engine="transformers")   # or "vllm"
+
+# wrap a model you already loaded with the standard HF interface
+from transformers import AutoProcessor, AutoModelForMultimodalLM
+processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B")
+hf_model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B", device_map="auto")
+model = SIMIT.from_pretrained(hf_model, processor=processor, image_generator="black-forest-labs/FLUX.2-klein-4B")
+```
+
+`image_generator` can be a diffusers model id, a loaded diffusers pipeline, or any callable
+`(prompt, width, height) -> PIL.Image`. On one GPU it is loaded first and the VLM engine sizes its
+memory around it. With several GPUs it goes on the last one.
+
+## Usage
+
+```python
+from simit import SIMIT, save_demos
+
+model = SIMIT.from_pretrained("ByteDance-Seed/BAGEL-7B-MoT")
+
+# 1) fit ABA + DF on a small labeled validation set (synthesis runs once and is cached)
+model.tune(val_set, metric="vqa_accuracy", max_runtime_ratio=25)
+
+# 2) test time
+for image, question in test_set:
+    demos = model.imagine(image, question)
+    improved = model.answer(image, question, demos)
+```
+
+Main methods:
+
+| method | what it does |
+|---|---|
+| `imagine(image, question, k=None, return_details=False, time_limit=None, on_demo=None, on_progress=None)` | adaptive number of demos (ABA + DF); `k=n` forces exactly `n` verified demos; `return_details=True` returns an `Imagination` with the zero-shot answer, `p0`, the budget, every candidate and stats. `time_limit` (seconds) stops and returns the demos accepted so far; `on_demo(demo)` / `on_progress(stage)` stream progress, e.g. to a UI |
+| `zero_shot(image, question)` | the greedy answer with its confidence `p0`; `model.config.budget(zs.confidence)` is the number of demos ABA will ask for. Reused by a following `imagine` |
+| `answer(image, question, demos)` | answer with the demos in context (no demos: zero-shot) |
+| `greedy(image, question)` | the standard zero-shot greedy answer (cached and shared with `imagine`) |
+| `model(image, question)` | `imagine` then `answer` |
+| `imagine_batch(queries)` / `answer_batch(queries, demos)` | many queries at once; their model calls are batched together on the GPU (much faster than a loop) |
+| `tune(val_set, metric, n_trials=100, max_runtime_ratio=None, max_candidates=None, cache_dir=None, k_max_choices=None)` | hyperparameter search (below) |
+| `add_skill(skill)` / `remove_skill(name)` | change the skill library |
+
+`val_set` items are `(image, question, answer_or_answers)` tuples or dicts with
+`image`/`question`/`answer(s)` (and optionally their own `metric`, for validation sets that mix benchmarks). `metric` is one of `exact_match`, `vqa_accuracy`, `contains`,
+`anls`, `relaxed_accuracy`, `multiple_choice`, or any `fn(prediction, references) -> float`.
+Images can be PIL images, file paths or raw bytes.
+
+### Saving demos for fine-tuning (SIMIT-FT)
+
+```python
+from simit import save_demos, load_demos
+save_demos(demos, "imagined/query_0")      # PNGs + demos.jsonl (with LLaVA-style "conversations")
+```
+
+The JSONL has `image`, `question`, `answer`, `description`, `skill`, `confidence`,
+`verify_score` and a `conversations` field, ready for LLaVA-format fine-tuning scripts. SIMIT-FT
+itself (fine-tuning on the imagined test-set data) is out of scope for this package.
+
+### Hyperparameter tuning
+
+`tune` follows App. G. For each validation query it runs synthesis **once** and caches the
+zero-shot answer, `p0`, and `K_max` verified candidates with their confidences (to `cache_dir` too,
+if given). Each Optuna TPE trial only re-selects demos from the cache with the trial's ABA/DF rule
+and re-answers. Answers are memoized per (query, selected subset), so later trials are almost free.
+Only a few demo subsets per query are reachable under any ABA/DF setting (≤ 15 for `K_max=4`); they are
+answered in one batch up front, so trials are lookups and hundreds of them take seconds. The search
+covers `epsilon, A0, A1, B, t_low, t_high` (and `k_max` with `k_max_choices`, e.g. `[4, 6]` as in the
+paper). The paper defaults and plain `K_max`-shot ICL are evaluated first. If no configuration beats
+zero-shot on the validation set, the tuned config never imagines (zero-shot answers, no extra cost).
+The paper uses 50 validation examples per benchmark.
+
+Runtime budget, which matters because synthesis dominates runtime:
+- `max_runtime_ratio=25` caps the *estimated* batched runtime relative to zero-shot. The estimate
+  comes from three timings measured on your machine during tuning: the batched zero-shot time per
+  query, the batched synthesis time per candidate (saved as `cost.json` in `cache_dir` for later
+  runs), and the ICL answer time. In our runs the measured test-time ratio was within about ±25% of
+  the estimate, so leave some margin.
+- `max_candidates` caps the mean number of candidates synthesized per query. `1.0` with `K_max=4` is
+  the paper's operating point (ABA synthesizes ~23% of `K_max`).
+- With neither, the search maximizes accuracy alone and usually spends the whole budget.
+
+How much one candidate costs relative to a zero-shot answer depends heavily on the model and task.
+With BAGEL, a candidate needs ~1.3-1.9 generated images (critic retries), each taking ~2 s on an H100.
+A short VQA answer takes ~0.1 s.
+
+```python
+result = model.tune(val, metric="vqa_accuracy", n_trials=300, max_runtime_ratio=25, cache_dir="cache/vizwiz")
+print(result)                 # best/zero-shot score, gain, candidates/demos per query, est. runtime, params
+model.config.save("vizwiz.json")
+model.config = SIMITConfig.load("vizwiz.json")
+```
+
+## Custom skills
+
+A skill turns a text description into an image through a model-written spec. Subclass
+`simit.Skill`: the router learns the new category from `route_hint` and `examples`. If `parse` or
+`render` raises `SpecError(msg)`, `msg` is shown to the model so it can repair its spec.
+
+```python
+import simit
+from simit import SpecError
+
+class SheetMusic(simit.Skill):
+    name = "sheet_music"
+    route_hint = "musical notation: staves, notes, chords"
+    examples = [                                  # (description, spec) few-shot pairs for the spec prompt
+        ("A C major scale in quarter notes.", "X:1\nK:C\nL:1/4\nCDEF GABc|"),
+        ("A G major chord held for a whole note.", "X:1\nK:G\nL:1\n[GBd]|"),
+    ]
+
+    def parse(self, text):
+        abc = simit.extract_fenced_block(text) or text
+        if "K:" not in abc:
+            raise SpecError("the ABC spec needs a key line such as 'K:C'")
+        return abc
+
+    def render(self, abc) -> "PIL.Image.Image":
+        return my_abc_renderer(abc)               # any deterministic renderer
+
+model.add_skill(SheetMusic())
+```
+
+Optional attributes and hooks:
+- `route_examples`: example descriptions for the router (default: the descriptions in `examples`).
+- `prompt`: a full spec prompt with `{request}` to replace the one built from `examples`.
+- `max_new_tokens`, `first_temperature`, `retry_temperature`.
+- `verify_rubric = "strict" | "structured"`.
+- `render_in_subprocess`: renders run in a crash- and timeout-isolated worker pool. Skills defined in
+  `__main__` or holding unpicklable state run in a thread instead.
+- `shortcut(request)`: render without a model call.
+- `clean(text)`.
+
+The 17 built-in skills: natural, diagram, graph, molecule, circuit, vector, venn, table, scene,
+puzzle, geometry, figure (sandboxed matplotlib), mermaid, svg, vegalite, html, html_composite
+(multi-panel layouts whose panels are realized recursively).
+
+### Loading once, running per request (e.g. Hugging Face ZeroGPU)
+
+Model weights can be loaded once and wrapped in a fresh engine per request, which is what a
+ZeroGPU Space needs (it forks the process for every GPU call, so no engine threads may exist in
+the main process, and the main process must not initialize CUDA):
+
+```python
+from simit import SIMIT, SIMITConfig
+from simit.backends.bagel import BagelBackend, BagelModel
+
+weights = BagelModel.load("ByteDance-Seed/BAGEL-7B-MoT", device="cpu")   # at startup
+
+@spaces.GPU(duration=90)
+def run(image, question):
+    sim = SIMIT(BagelBackend(weights, device="cuda", use_cuda_graphs=False), config=SIMITConfig(k_max=2))
+    try:
+        demos = sim.imagine(image, question, time_limit=60)
+        return sim.greedy(image, question), sim.answer(image, question, demos)
+    finally:
+        sim.close()
+```
+
+`LanceModel.load` / `LanceBackend` work the same way. For a transformers model, pass the loaded
+model and processor to `HFBackend`. The SIMIT demo Space (three models, per-budget presets, a
+streaming UI) is built this way.
+
+### New model families
+
+`simit.backends.register_backend(name, detect, factory)` adds a backend. It subclasses
+`simit.backends.Backend` and implements `submit_generate`, `submit_score` and, for native image
+generation, `submit_image`. `pipeline_defaults` can change synthesis defaults for that model
+(see `LanceBackend`).
+
+## Configuration
+
+`SIMITConfig` (`model.config`, or `from_pretrained(..., config=...)`). `None` means the backend's default.
+
+| field | default | meaning |
+|---|---|---|
+| `k_max` | 4 | max demos per query |
+| `use_aba`, `epsilon`, `A0`, `A1`, `B` | True, 0.14, 0.35, 0.60, 0.2 | adaptive budget allocation (App. D) |
+| `use_df`, `t_low`, `t_high` | True, 0.2, 0.9 | difficulty-filter confidence band |
+| `synthesis` | `"batch"` (Lance: `"decomposed"`) | one call for all triplets vs. three short calls per triplet |
+| `diversity_prompt` | True | the paper's diversity-encouraging triplet prompt |
+| `use_skills` | True (Lance: False) | structured skill library vs. natural images only |
+| `verify`, `verify_threshold`, `verify_think` | True (Lance: False), 50, BAGEL: True | critic |
+| `attempts_per_slot`, `repair_retries`, `verify_rounds` | 4, 2, 5 | breadth-first slot-filling budget |
+| `image_size` | 400 | natively generated image side |
+| `speculative` | on for `imagine`, off for `imagine_batch`/`tune` | start all remaining candidate attempts at once (lower latency for a lone query; wasted work when the GPU is busy) |
+| `answer_max_new_tokens` | 128 | answer length (also per call: `max_new_tokens=`) |
+| `seed` | None | image-generation seed |
+
+Environment variables:
+- `SIMIT_CHROMIUM`: path to a Chromium/Chrome binary.
+- `PLAYWRIGHT_BROWSERS_PATH`: where Playwright's browsers live.
+- `SIMIT_RENDER_WORKERS`: number of render processes. The default is derived from the CPUs
+  available to the job (LSF/SLURM aware).
+
+## Performance
+
+Everything that can overlap does:
+
+- **Built-in engine for BAGEL/Lance**, running both the understanding and generation experts:
+  - continuous batching of decode, prefill and diffusion steps from all concurrent requests
+  - automatic prefix caching of shared prompt prefixes (the long router and skill few-shot prompts
+    are prefilled once)
+  - CUDA graphs for decode
+  - experts run on contiguous token slices, with fused Triton kernels for RMSNorm, RoPE and SwiGLU
+    (bit-exact with the eager ops up to reduction order)
+  - separate high-priority text and image streams, so diffusion doesn't stall decoding
+- **vLLM** for standard VLMs (transformers fallback batches concurrent requests).
+- **Pipeline:**
+  - triplets are streamed into realization as they are decoded, and synthesis stops once enough are kept
+  - breadth-first slot filling: every triplet gets one cheap attempt before failures get repairs
+  - the online ABA/DF rule stops as soon as `K*` demos pass. A lone `imagine` call runs the
+    remaining candidate attempts in parallel (`speculative`), since an idle GPU makes that nearly free
+  - renders run in a process pool (warm matplotlib zygote, one shared headless browser)
+- **`imagine_batch`/`tune`** run many queries concurrently so their calls share GPU batches.
+
+Measured on one H100 with `tests/bench.py`: LMMs-Eval-Lite slices, tuned on 60 validation queries
+(300 trials), tested on 60 different queries. Runtime is the batched wall time relative to batched
+zero-shot answering. With 60 test queries, differences of a few points are within noise.
+
+Tuned with `max_runtime_ratio=25`:
+
+| model | task | test zero-shot → SIMIT-ICL | runtime (estimated) |
+|---|---|---|---|
+| BAGEL-7B-MoT | VizWiz | 0.494 → **0.556** (+12.4%) | 25.0× (22.0×) |
+| BAGEL-7B-MoT | OK-VQA | 0.650 → 0.650 (nothing beat zero-shot on validation) | 1.0× |
+| Lance | VizWiz | 0.106 → 0.089 | 27.0× (21.3×) |
+| Lance | OK-VQA | 0.483 → 0.489 (+1.1%) | 17.4× (23.3×) |
+
+Tuned with `max_candidates=1.0`:
+
+| model | task | test zero-shot → SIMIT-ICL | runtime |
+|---|---|---|---|
+| BAGEL-7B-MoT | VizWiz | 0.494 → **0.567** (+14.6%) | 29.3× |
+| BAGEL-7B-MoT | OK-VQA | 0.650 → 0.661 (+1.7%) | 64.6× |
+| BAGEL-7B-MoT | ChartQA | 0.817 → 0.817 (zero-shot chosen) | 1.0× |
+| BAGEL-7B-MoT | AI2D | 0.950 → 0.933 | 12.1× |
+| Qwen3.8-27B + FLUX.2-klein-4B (vLLM) | VizWiz | 0.522 → 0.522 | 12.0× |
+| Qwen3.8-27B + FLUX.2-klein-4B (vLLM) | OK-VQA / ChartQA | unchanged (zero-shot chosen) | 0.9× |
+| Qwen3.8-27B + FLUX.2-klein-4B (vLLM) | AI2D | 0.900 → 0.900 | 49.7× |
+
+What the numbers show:
+- BAGEL's VizWiz gain reproduces the paper's direction.
+- Lance matches the research implementation's VizWiz behavior. Its zero-shot answers are mostly
+  "No", which gives a very low baseline.
+- Qwen3.8-27B gains little from imagined demos on these tasks.
+- The OK-VQA and AI2D rows at 50-65× show why a runtime cap beats a candidate cap. One BAGEL
+  candidate needs ~1.3-1.9 generated images at ~2 s each (50 steps, CFG), while a short answer
+  takes ~0.1 s.
+
+Single-query latency, i.e. what one demo request feels like:
+- **BAGEL:** 13.9 s on average on VizWiz (16 s for queries that get demos), with speculative attempts.
+  Greedy takes 0.29 s.
+- **Qwen3.8-27B + FLUX:** ~20 s for `imagine(k=2)`.
+- **Lance:** ~1-2 s.
+
+The BAGEL/Lance engine matches the research implementation: 17/18 (BAGEL) and 15/18 (Lance)
+identical greedy answers on reference samples, with the rest near-ties.
+
+## Differences from the research code
+
+- The tuner caches `K_max` candidates and memoizes answers per demo subset (as in App. G), with an
+  optional synthesis budget (`max_candidates`).
+- Fixed a cleanup bug that stripped valid CSS after CJK characters in HTML specs.
+- Duplicate questions within a query's demos are dropped, and a `[...]` copied from the triplet
+  template around a whole field (`Answer: [3]`) is removed, as are chat preambles ("Sure, here's
+  the first one: ...") in decomposed questions.
+- Triplet synthesis is streamed and stops early.
+- When the critic's reasoning hits the token limit before its `SCORE:` line, a short follow-up asks
+  for the score instead of discarding the candidate.
+- Prompts (triplet, router, skills, critic, revision, ICL format) are the research prompts verbatim;
+  the triplet prompt is question-free, as in the code.
+
+## License
+
+Apache-2.0. `simit/backends/mot/wan_vae.py` is adapted from vllm-omni (Apache-2.0). Mermaid
+(MIT) is bundled under `simit/skills/assets/`.