renderers 0.1.11.dev1__tar.gz → 0.1.11.dev3__tar.gz

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: renderers
-Version: 0.1.11.dev1
+Version: 0.1.11.dev3
 Summary: Chat template renderers — deterministic message-to-token conversion for LLM training
 License-Expression: Apache-2.0
 License-File: LICENSE
@@ -11,7 +11,11 @@ Requires-Dist: openai-harmony>=0.0.4
 Requires-Dist: openai>=1.108.1
 Requires-Dist: prime-pydantic-config>=0.3.0.dev83
 Requires-Dist: tiktoken
-Requires-Dist: transformers>=4.50.0
+Provides-Extra: multimodal
+Requires-Dist: pillow>=12.2.0; extra == 'multimodal'
+Requires-Dist: transformers>=4.50.0; extra == 'multimodal'
+Provides-Extra: transformers
+Requires-Dist: transformers>=4.50.0; extra == 'transformers'
 Description-Content-Type: text/markdown
 
 # renderers
@@ -26,13 +30,30 @@ Standalone on PyPI, and portable across training and inference stacks (transform
 uv add renderers
 ```
 
+The base install supports text renderers with a bring-your-own tokenizer. Add
+the Hugging Face integration for the tokenizer-loading helpers, or the complete
+media stack for image/audio rendering:
+
+```bash
+uv add 'renderers[transformers]'
+uv add 'renderers[multimodal]'
+```
+
+A BYO tokenizer must expose `encode`, `decode`, `convert_tokens_to_ids`, and
+token IDs such as `eos_token_id`. Character offsets are optional: tokenizers
+supporting `return_offsets_mapping=True` also receive precise per-token
+`is_content` attribution; without offsets, renderers return `is_content=[]`.
+`DefaultRenderer` additionally requires `apply_chat_template`. This includes
+text-only Inkling training: `InklingRenderer` loads its Transformers processor
+only when image or audio content is actually rendered.
+
 ## At a glance
 
 ```python
-from transformers import AutoTokenizer
 from renderers import create_renderer
+from renderers.base import load_tokenizer
 
-tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
+tok = load_tokenizer("Qwen/Qwen3-8B")              # renderers[transformers]
 r = create_renderer(tok)                            # → Qwen3Renderer (auto-resolved)
 
 prompt_ids = r.render_ids(
@@ -92,17 +113,10 @@ r = create_renderer(tok)                # AutoRendererConfig is the implicit def
 
 Auto-detect matches `tokenizer.name_or_path` against `MODEL_RENDERER_MAP` by **exact match**. Prefix matching is intentionally off — same architecture can ship different chat templates (base vs instruct, fine-tune renames). Fine-tunes must pass an explicit typed config (e.g. `Qwen3RendererConfig()`). Unknown text-only names fall back to `DefaultRenderer`, unless `AutoRendererConfig(thinking_retention=...)` was set; the default renderer cannot implement that bridge policy.
 
-### Pools
-
-```python
-from renderers import create_renderer_pool
-
-pool = create_renderer_pool("Qwen/Qwen3-8B", size=16)
-with pool.checkout() as r:
-    ids = r.render_ids(messages)
-```
-
-Each slot owns its own tokenizer copy. Construction fans out across a thread pool so a 32-slot pool doesn't serially eat ~10–15s of `from_pretrained` calls at startup.
+Without the `transformers` extra, exact-match registered models still
+auto-resolve. For an unknown name, renderers cannot safely probe `AutoConfig`
+to distinguish a text model from an unknown VLM; pass an explicit typed config
+such as `DefaultRendererConfig()` for a known text-only model.
 
 ## Why use a renderer
 
@@ -125,7 +139,7 @@ Each break fragments a rollout into multiple training samples — every fragment
 
 ## Typed renderer configs
 
-Each renderer accepts a typed pydantic config at construction. Some fields mirror chat-template kwargs; others configure renderer-only behavior such as image caching, parsers, or Harmony preamble construction. `create_renderer` and `create_renderer_pool` take one positional `config` argument and an optional keyword-only `chat_template_kwargs` mapping:
+Each renderer accepts a typed pydantic config at construction. Some fields mirror chat-template kwargs; others configure renderer-only behavior such as image caching, parsers, or Harmony preamble construction. `create_renderer` takes one positional `config` argument and an optional keyword-only `chat_template_kwargs` mapping:
 
 ```python
 from renderers import (
@@ -175,7 +189,7 @@ Fallback for unsupported text-only models. Wraps `apply_chat_template` and accep
 
 ## Roadmap
 
-- **VLM expansion.** `ImagePart` support exists for Qwen3-VL, Qwen3.5-family, Gemma 4, and Kimi K2.5 / K2.6 multimodal templates. Remaining work: audio/video support, broader VLM coverage, and more RL validation. Gemma 4 image preprocessing requires a Transformers release that provides `Gemma4Processor`.
+- **VLM expansion.** `ImagePart` support exists for Qwen3-VL, Qwen3.5-family, Gemma 4, and Kimi K2.5 / K2.6 multimodal templates. Install `renderers[multimodal]` for Pillow and the Hugging Face processors. Remaining work: audio/video support, broader VLM coverage, and more RL validation. Gemma 4 image preprocessing requires a Transformers release that provides `Gemma4Processor`.
 - **Patched chat templates.** Some shipped templates re-tokenize history or normalize JSON in ways that break token identity. Plan: a `use_patched` opt-in per renderer that renders the same surface form while avoiding known-bad patterns. (Auto-stripping thinking from past turns is *not* one of these — that's intended template behaviour the renderer reproduces; use `thinking_retention` to override it.)
 
 ## Testing

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/README.md

@@ -10,13 +10,30 @@ Standalone on PyPI, and portable across training and inference stacks (transform
 uv add renderers
 ```
 
+The base install supports text renderers with a bring-your-own tokenizer. Add
+the Hugging Face integration for the tokenizer-loading helpers, or the complete
+media stack for image/audio rendering:
+
+```bash
+uv add 'renderers[transformers]'
+uv add 'renderers[multimodal]'
+```
+
+A BYO tokenizer must expose `encode`, `decode`, `convert_tokens_to_ids`, and
+token IDs such as `eos_token_id`. Character offsets are optional: tokenizers
+supporting `return_offsets_mapping=True` also receive precise per-token
+`is_content` attribution; without offsets, renderers return `is_content=[]`.
+`DefaultRenderer` additionally requires `apply_chat_template`. This includes
+text-only Inkling training: `InklingRenderer` loads its Transformers processor
+only when image or audio content is actually rendered.
+
 ## At a glance
 
 ```python
-from transformers import AutoTokenizer
 from renderers import create_renderer
+from renderers.base import load_tokenizer
 
-tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
+tok = load_tokenizer("Qwen/Qwen3-8B")              # renderers[transformers]
 r = create_renderer(tok)                            # → Qwen3Renderer (auto-resolved)
 
 prompt_ids = r.render_ids(
@@ -76,17 +93,10 @@ r = create_renderer(tok)                # AutoRendererConfig is the implicit def
 
 Auto-detect matches `tokenizer.name_or_path` against `MODEL_RENDERER_MAP` by **exact match**. Prefix matching is intentionally off — same architecture can ship different chat templates (base vs instruct, fine-tune renames). Fine-tunes must pass an explicit typed config (e.g. `Qwen3RendererConfig()`). Unknown text-only names fall back to `DefaultRenderer`, unless `AutoRendererConfig(thinking_retention=...)` was set; the default renderer cannot implement that bridge policy.
 
-### Pools
-
-```python
-from renderers import create_renderer_pool
-
-pool = create_renderer_pool("Qwen/Qwen3-8B", size=16)
-with pool.checkout() as r:
-    ids = r.render_ids(messages)
-```
-
-Each slot owns its own tokenizer copy. Construction fans out across a thread pool so a 32-slot pool doesn't serially eat ~10–15s of `from_pretrained` calls at startup.
+Without the `transformers` extra, exact-match registered models still
+auto-resolve. For an unknown name, renderers cannot safely probe `AutoConfig`
+to distinguish a text model from an unknown VLM; pass an explicit typed config
+such as `DefaultRendererConfig()` for a known text-only model.
 
 ## Why use a renderer
 
@@ -109,7 +119,7 @@ Each break fragments a rollout into multiple training samples — every fragment
 
 ## Typed renderer configs
 
-Each renderer accepts a typed pydantic config at construction. Some fields mirror chat-template kwargs; others configure renderer-only behavior such as image caching, parsers, or Harmony preamble construction. `create_renderer` and `create_renderer_pool` take one positional `config` argument and an optional keyword-only `chat_template_kwargs` mapping:
+Each renderer accepts a typed pydantic config at construction. Some fields mirror chat-template kwargs; others configure renderer-only behavior such as image caching, parsers, or Harmony preamble construction. `create_renderer` takes one positional `config` argument and an optional keyword-only `chat_template_kwargs` mapping:
 
 ```python
 from renderers import (
@@ -159,7 +169,7 @@ Fallback for unsupported text-only models. Wraps `apply_chat_template` and accep
 
 ## Roadmap
 
-- **VLM expansion.** `ImagePart` support exists for Qwen3-VL, Qwen3.5-family, Gemma 4, and Kimi K2.5 / K2.6 multimodal templates. Remaining work: audio/video support, broader VLM coverage, and more RL validation. Gemma 4 image preprocessing requires a Transformers release that provides `Gemma4Processor`.
+- **VLM expansion.** `ImagePart` support exists for Qwen3-VL, Qwen3.5-family, Gemma 4, and Kimi K2.5 / K2.6 multimodal templates. Install `renderers[multimodal]` for Pillow and the Hugging Face processors. Remaining work: audio/video support, broader VLM coverage, and more RL validation. Gemma 4 image preprocessing requires a Transformers release that provides `Gemma4Processor`.
 - **Patched chat templates.** Some shipped templates re-tokenize history or normalize JSON in ways that break token identity. Plan: a `use_patched` opt-in per renderer that renders the same surface form while avoiding known-bad patterns. (Auto-stripping thinking from past turns is *not* one of these — that's intended template behaviour the renderer reproduces; use `thinking_retention` to override it.)
 
 ## Testing

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/docs/renderer-config.md

@@ -1,8 +1,7 @@
 # Renderer config
 
-`renderers.RendererConfig` is the typed input to `create_renderer` and
-`create_renderer_pool`. It pins the renderer choice and its config at
-construction time.
+`renderers.RendererConfig` is the typed input to `create_renderer`. It pins the
+renderer choice and its config at construction time.
 
 ```python
 from renderers import create_renderer, Qwen35RendererConfig
@@ -77,10 +76,6 @@ r = create_renderer(
     tokenizer,
     chat_template_kwargs={"enable_thinking": False},
 )
-pool = create_renderer_pool(
-    "Qwen/Qwen3-8B",
-    chat_template_kwargs={"enable_thinking": False},
-)
 ```
 
 Renderers resolves auto configs before applying `chat_template_kwargs`, so the

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/pyproject.toml

@@ -19,11 +19,6 @@ dependencies = [
     "openai>=1.108.1",
     "tiktoken",
     "jinja2",
-    # Keep this floor compatible with prime-rl's transformers pin. Inkling's
-    # tokenizer and text-only renderer work on older releases; image/audio
-    # processing fails lazily with an upgrade message when InklingProcessor is
-    # unavailable (native support starts in transformers 5.14).
-    "transformers>=4.50.0",
     # Used by GptOssRenderer to render and parse harmony tokens. Vendoring
     # OpenAI's reference implementation keeps us byte-identical with vLLM
     # (which also uses it) and saves us mirroring a 330-line Jinja template.
@@ -42,6 +37,25 @@ dependencies = [
     "prime-pydantic-config>=0.3.0.dev83",
 ]
 
+[project.optional-dependencies]
+# Tokenizer loading uses Hugging Face. Text-only renderers can instead be
+# constructed with a compatible BYO tokenizer and do not import this
+# dependency. Character offsets are optional.
+transformers = [
+    # Keep this floor compatible with prime-rl's transformers pin. Inkling's
+    # tokenizer and text-only renderer work on older releases; image/audio
+    # processing fails lazily with an upgrade message when InklingProcessor is
+    # unavailable (native support starts in transformers 5.14).
+    "transformers>=4.50.0",
+]
+
+# Image/audio renderers also need Pillow to resolve media inputs. Keep this as
+# a separate convenience extra so tokenizer-only users do not pull it in.
+multimodal = [
+    "pillow>=12.2.0",
+    "transformers>=4.50.0",
+]
+
 [tool.hatch.version]
 source = "vcs"
 # Tags look like ``renderers-v0.1.8`` (prefix matches the publish.yml
@@ -90,6 +104,8 @@ dev = [
     "torch>=2.11.0",
     "torchvision>=0.26.0",
     "ty>=0.0.1a29,<0.0.22",
+    # Optional for consumers, but required by tokenizer/parity/VLM tests.
+    "transformers>=4.50.0",
 ]
 
 [tool.uv]

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/renderers/__init__.py

@@ -7,6 +7,7 @@ except ImportError:
     __version__ = "0+unknown"
 
 from renderers.base import (
+    ChatTemplateTokenizer,
     Content,
     ContentPart,
     ImagePart,
@@ -14,6 +15,7 @@ from renderers.base import (
     Message,
     MultiModalData,
     MultimodalRenderer,
+    OffsetTokenizer,
     ParsedResponse,
     ParsedToolCall,
     PlaceholderRange,
@@ -21,9 +23,9 @@ from renderers.base import (
     RenderedTokens,
     RenderedTrainingSample,
     Renderer,
-    RendererPool,
     TextPart,
     ThinkingPart,
+    Tokenizer,
     ToolCall,
     ToolCallFunction,
     ToolCallParseStatus,
@@ -33,7 +35,6 @@ from renderers.base import (
     build_training_sample,
     build_trajectory_step,
     create_renderer,
-    create_renderer_pool,
     extract_message_tool_names,
     is_multimodal,
     reject_assistant_in_extension,
@@ -74,17 +75,13 @@ from renderers.configs import (
     RendererConfig,
 )
 
-# Concrete renderer classes are lazy-loaded so that consumers needing
-# only the config layer (``RendererConfig`` discriminated union) don't
-# pay the ``transformers`` import cost. Each renderer module does
-# ``from transformers.tokenization_utils import PreTrainedTokenizer``
-# at module level, so eager imports here would drag ``transformers``
-# into every downstream ``import renderers``. ``__getattr__`` (PEP 562)
-# resolves the names on first attribute access, so ``from renderers
-# import DefaultRenderer`` and ``renderers.DefaultRenderer`` both work
-# transparently. ``create_renderer`` doesn't depend on these eager
-# imports — ``renderers.base._populate_registry`` lazy-imports the
-# concrete classes itself when a renderer is instantiated.
+# Concrete renderer classes are lazy-loaded so that consumers needing only the
+# config layer (``RendererConfig`` discriminated union) don't import every
+# renderer module. Renderer tokenizer annotations use the local ``Tokenizer``
+# protocols, so resolving a text renderer remains safe when the optional
+# ``transformers`` dependency is absent. ``__getattr__`` (PEP 562) resolves the
+# names on first attribute access, while ``renderers.base._populate_registry``
+# handles lazy registration for ``create_renderer``.
 _LAZY_RENDERERS: dict[str, str] = {
     "DeepSeekR1Renderer": "renderers.deepseek_r1",
     "DeepSeekV3Renderer": "renderers.deepseek_v3",
@@ -134,6 +131,7 @@ def __dir__() -> list[str]:
 __all__ = [
     "AutoRendererConfig",
     "BaseRendererConfig",
+    "ChatTemplateTokenizer",
     "Content",
     "ContentPart",
     "DeepSeekR1Renderer",
@@ -184,6 +182,7 @@ __all__ = [
     "Nemotron3RendererConfig",
     "Nemotron3UltraRenderer",
     "Nemotron3UltraRendererConfig",
+    "OffsetTokenizer",
     "OverlongPromptError",
     "ParsedResponse",
     "ParsedToolCall",
@@ -205,9 +204,9 @@ __all__ = [
     "RenderedTrainingSample",
     "Renderer",
     "RendererConfig",
-    "RendererPool",
     "TextPart",
     "ThinkingPart",
+    "Tokenizer",
     "ToolCall",
     "ToolCallFunction",
     "ToolCallParseStatus",
@@ -219,7 +218,6 @@ __all__ = [
     "build_trajectory_step",
     "config_from_name",
     "create_renderer",
-    "create_renderer_pool",
     "extract_message_tool_names",
     "is_multimodal",
     "reject_assistant_in_extension",

{renderers-0.1.11.dev1 → renderers-0.1.11.dev3}/renderers/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.11.dev1'
-__version_tuple__ = version_tuple = (0, 1, 11, 'dev1')
+__version__ = version = '0.1.11.dev3'
+__version_tuple__ = version_tuple = (0, 1, 11, 'dev3')
 
 __commit_id__ = commit_id = None