renderers 0.1.12.dev18__tar.gz → 0.1.12.dev19__tar.gz

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: renderers
-Version: 0.1.12.dev18
+Version: 0.1.12.dev19
 Summary: Chat template renderers — deterministic message-to-token conversion for LLM training
 License-Expression: Apache-2.0
 License-File: LICENSE
@@ -120,6 +120,8 @@ Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `qwen3.6`, `qwen3.
 
 ```python
 class Renderer(Protocol):
+    @property
+    def is_prefix_stable(self) -> bool: ...
     def render(messages, *, tools=None, add_generation_prompt=False) -> RenderedTokens: ...
     def render_ids(messages, *, tools=None, add_generation_prompt=False) -> list[int]: ...
     def parse_response(token_ids) -> ParsedResponse: ...
@@ -132,6 +134,35 @@ class Renderer(Protocol):
 - `ParsedResponse` is `(content, reasoning_content, tool_calls)`. It scans token ids for special-token boundaries (e.g. id `151657` for `<tool_call>` on Qwen3) — a literal `"<tool_call>"` in user content tokenizes to ordinary text ids and never matches.
 - Round-trip: rendering `[user, assistant(content, reasoning, tool_calls)]`, slicing the assistant completion, and feeding it through `parse_response` returns an equivalent structured message. Tested per-renderer in `tests/test_roundtrip.py`.
 
+### Prefix stability
+
+`renderer.is_prefix_stable` describes the constructed renderer after config and
+`chat_template_kwargs` resolution. A `True` value guarantees that full renders
+of assistant-terminated conversations stay token prefixes when more
+user/tool/assistant messages are appended, with a fixed system/developer
+preamble, tools, and `add_generation_prompt=False`. `False` means this is not
+guaranteed; opaque `DefaultRenderer` templates report `False`.
+
+The flag follows the template's emit rules, independently of the
+`thinking_retention` bridge override. For example, GLM-5 with
+`clear_thinking=False` is stable, while the default strips earlier reasoning.
+Disabling thinking generation alone does not guarantee stability for datasets
+that already contain reasoning. Other causes include GPT-OSS's final-turn
+terminator and Nemotron's moving effort hint. DeepSeek V4 reports `False`
+because its task/continuation message controls can rewrite the prefix; Inkling
+reports `False` because reused tool-call IDs can change earlier tool-response
+names, even though it retains reasoning.
+
+SFT consumers can use this flag to warn that a single full-conversation render
+may omit earlier reasoning. `build_training_sample` still returns one sample
+per conversation, without expanding it into one sample per assistant turn.
+The guarantee assumes unchanged message contents and deterministic tokenization
+and media preprocessing. Stability does not guarantee that the template emits
+every input field (some always omit reasoning), or that rendering reproduces sampled tokens. Continue
+using `bridge_to_next_turn` for sampled trajectories. Custom renderers should
+expose the same property; consumers supporting older/custom implementations
+can conservatively use `getattr(renderer, "is_prefix_stable", False)`.
+
 ### `bridge_to_next_turn` (the core contract)
 
 Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return ids for the next turn's prompt such that the result starts with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continues with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render.
@@ -276,6 +307,30 @@ uv run pytest
 
 Round-trip parity (render → parse → original) and token-level parity against each model's independent reference encoder are tested per renderer. Most references use `apply_chat_template`; DeepSeek V4 uses its shipped Python encoder, and GPT-OSS uses Harmony.
 
+Prefix-stability audit witnesses live in
+[`tests/fixtures/prefix_stability.json`](tests/fixtures/prefix_stability.json).
+Each case records a renderer/config, a conversation ending in an assistant,
+messages to append, and the reason the original tokens stop being a prefix.
+Shared scenarios keep the corpus small; paired stable settings guard the
+configuration switches. `DefaultRenderer` is treated as unknown, without
+inventing a witness for its opaque template.
+
+This audit is opt-in and is not collected by the regular test suite or CI.
+Run it manually when changing renderer emission rules or the stability
+classification:
+
+```bash
+uv run pytest tests/audit_prefix_stability.py -q
+```
+
+The tests render both inputs and compare the original token IDs with the
+same-length prefix of the extended render. They print the first differing
+tokens and a decoded text diff on failure. These are executable witnesses,
+not snapshots of token IDs that could pass without exercising the renderer.
+Add a named case for each new instability mechanism or configuration branch.
+A witness disproves stability; finite positive controls do not prove stability
+for every possible conversation or third-party tokenizer/processor.
+
 ## License
 
 Licensed under the [Apache License, Version 2.0](LICENSE).

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/README.md

@@ -100,6 +100,8 @@ Hand-coded renderers ship for `qwen3`, `qwen3-vl`, `qwen3.5`, `qwen3.6`, `qwen3.
 
 ```python
 class Renderer(Protocol):
+    @property
+    def is_prefix_stable(self) -> bool: ...
     def render(messages, *, tools=None, add_generation_prompt=False) -> RenderedTokens: ...
     def render_ids(messages, *, tools=None, add_generation_prompt=False) -> list[int]: ...
     def parse_response(token_ids) -> ParsedResponse: ...
@@ -112,6 +114,35 @@ class Renderer(Protocol):
 - `ParsedResponse` is `(content, reasoning_content, tool_calls)`. It scans token ids for special-token boundaries (e.g. id `151657` for `<tool_call>` on Qwen3) — a literal `"<tool_call>"` in user content tokenizes to ordinary text ids and never matches.
 - Round-trip: rendering `[user, assistant(content, reasoning, tool_calls)]`, slicing the assistant completion, and feeding it through `parse_response` returns an equivalent structured message. Tested per-renderer in `tests/test_roundtrip.py`.
 
+### Prefix stability
+
+`renderer.is_prefix_stable` describes the constructed renderer after config and
+`chat_template_kwargs` resolution. A `True` value guarantees that full renders
+of assistant-terminated conversations stay token prefixes when more
+user/tool/assistant messages are appended, with a fixed system/developer
+preamble, tools, and `add_generation_prompt=False`. `False` means this is not
+guaranteed; opaque `DefaultRenderer` templates report `False`.
+
+The flag follows the template's emit rules, independently of the
+`thinking_retention` bridge override. For example, GLM-5 with
+`clear_thinking=False` is stable, while the default strips earlier reasoning.
+Disabling thinking generation alone does not guarantee stability for datasets
+that already contain reasoning. Other causes include GPT-OSS's final-turn
+terminator and Nemotron's moving effort hint. DeepSeek V4 reports `False`
+because its task/continuation message controls can rewrite the prefix; Inkling
+reports `False` because reused tool-call IDs can change earlier tool-response
+names, even though it retains reasoning.
+
+SFT consumers can use this flag to warn that a single full-conversation render
+may omit earlier reasoning. `build_training_sample` still returns one sample
+per conversation, without expanding it into one sample per assistant turn.
+The guarantee assumes unchanged message contents and deterministic tokenization
+and media preprocessing. Stability does not guarantee that the template emits
+every input field (some always omit reasoning), or that rendering reproduces sampled tokens. Continue
+using `bridge_to_next_turn` for sampled trajectories. Custom renderers should
+expose the same property; consumers supporting older/custom implementations
+can conservatively use `getattr(renderer, "is_prefix_stable", False)`.
+
 ### `bridge_to_next_turn` (the core contract)
 
 Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return ids for the next turn's prompt such that the result starts with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continues with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render.
@@ -256,6 +287,30 @@ uv run pytest
 
 Round-trip parity (render → parse → original) and token-level parity against each model's independent reference encoder are tested per renderer. Most references use `apply_chat_template`; DeepSeek V4 uses its shipped Python encoder, and GPT-OSS uses Harmony.
 
+Prefix-stability audit witnesses live in
+[`tests/fixtures/prefix_stability.json`](tests/fixtures/prefix_stability.json).
+Each case records a renderer/config, a conversation ending in an assistant,
+messages to append, and the reason the original tokens stop being a prefix.
+Shared scenarios keep the corpus small; paired stable settings guard the
+configuration switches. `DefaultRenderer` is treated as unknown, without
+inventing a witness for its opaque template.
+
+This audit is opt-in and is not collected by the regular test suite or CI.
+Run it manually when changing renderer emission rules or the stability
+classification:
+
+```bash
+uv run pytest tests/audit_prefix_stability.py -q
+```
+
+The tests render both inputs and compare the original token IDs with the
+same-length prefix of the extended render. They print the first differing
+tokens and a decoded text diff on failure. These are executable witnesses,
+not snapshots of token IDs that could pass without exercising the renderer.
+Add a named case for each new instability mechanism or configuration branch.
+A witness disproves stability; finite positive controls do not prove stability
+for every possible conversation or third-party tokenizer/processor.
+
 ## License
 
 Licensed under the [Apache License, Version 2.0](LICENSE).

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.12.dev18'
-__version_tuple__ = version_tuple = (0, 1, 12, 'dev18')
+__version__ = version = '0.1.12.dev19'
+__version_tuple__ = version_tuple = (0, 1, 12, 'dev19')
 
 __commit_id__ = commit_id = None

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/base.py

@@ -722,6 +722,26 @@ class ChatTemplateTokenizer(Tokenizer, Protocol):
 class Renderer(Protocol):
     """Owns message ↔ token conversion for a specific model family."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether extending a conversation preserves its rendered token prefix.
+
+        For valid conversations ending in an assistant message, ``True`` means
+        that appending user/tool/assistant messages leaves the existing token
+        IDs unchanged when both renders use ``add_generation_prompt=False``.
+        The system/developer preamble, tools, and renderer configuration must
+        stay fixed. This describes full renders, not sampled completions or
+        the ability to bridge them; keep using ``bridge_to_next_turn`` for that.
+
+        The declaration is conservative over message contents (including
+        supplied reasoning even when thinking generation is disabled) and
+        tools. ``False`` means stability is not guaranteed, including opaque
+        templates. It does not mean every conversation changes. Conversely,
+        ``True`` does not promise to emit fields the template always ignores.
+        Custom renderers should declare this property from their emit rules.
+        """
+        ...
+
     def render(
         self,
         messages: list[Message],

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/deepseek_v3.py

@@ -56,6 +56,11 @@ class DeepSeekV3Renderer:
     stored for protocol uniformity.
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     #: Default typed config; the R1 subclass overrides this.
     _config_cls: type = DeepSeekV3RendererConfig
     _implied_thinking_retention = "all"

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/deepseek_v4.py

@@ -294,6 +294,12 @@ def _prepare_messages(messages: list[Message]) -> list[_LogicalMessage]:
 class DeepSeekV4Renderer:
     """Renderer for ``deepseek-ai/DeepSeek-V4-Flash-0731``."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Task transitions and unterminated assistant continuations can rewrite the prefix.
+        return False
+
     _implied_thinking_retention = "tool_cycle"
 
     def __init__(

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/default.py

@@ -97,6 +97,12 @@ class DefaultRenderer:
     :class:`renderers.DefaultRendererConfig`).
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # An opaque Jinja template cannot promise prefix stability.
+        return False
+
     def __init__(
         self,
         tokenizer: ChatTemplateTokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/gemma4.py

@@ -431,6 +431,12 @@ class _ArgumentParser:
 class Gemma4Renderer:
     """Deterministic renderer for the canonical Gemma 4 instruction models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # preserve_thinking only retains tool-call reasoning, not final answers.
+        return False
+
     _config_cls = Gemma4RendererConfig
 
     def __init__(

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/glm45.py

@@ -55,6 +55,12 @@ _TOOLS_FOOTER = (
 class GLM45Renderer:
     """Deterministic message → token renderer for GLM-4.5 Air models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Historical reasoning is stripped regardless of bridge policy.
+        return False
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/glm5.py

@@ -57,6 +57,14 @@ _TOOLS_FOOTER = (
 class GLM5Renderer:
     """Deterministic message → token renderer for GLM-5 models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # GLM-5.1 also changes empty thinking wrappers across user boundaries.
+        return not self.config.clear_thinking and not (
+            self.empty_think_on_last_assistant and self.config.enable_thinking
+        )
+
     # GLM-5.1 flips this on: even when the most-recent assistant has no
     # reasoning content, the template wraps it with ``<think></think>``
     # instead of just emitting ``</think>`` as a separator. Subclassed in

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/gpt_oss.py

@@ -121,6 +121,12 @@ def _arguments_to_str(arguments: Any) -> str:
 class GptOssRenderer:
     """Deterministic message → token renderer for OpenAI gpt-oss (harmony)."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # The final assistant terminator changes even when auto_drop_analysis=False.
+        return False
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/hy3.py

@@ -76,6 +76,14 @@ _TOOL_RESPONSE_END = f"</tool_response{_HYTK}>"
 class Hy3Renderer:
     """Deterministic message → token renderer for Tencent Hy3 models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Require retention regardless of whether tools are supplied.
+        return not self.config.raw_last_assistant and (
+            self.config.is_training or self.config.preserved_thinking is True
+        )
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/inkling.py

@@ -173,6 +173,12 @@ class InklingRenderer:
     the :class:`~renderers.base.MultimodalRenderer` protocol.
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Reused tool-call IDs can change names resolved for earlier tool responses.
+        return False
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/kimi_k2.py

@@ -47,6 +47,11 @@ class KimiK2Renderer:
     but have no effect on the byte-level output.
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/kimi_k25.py

@@ -593,6 +593,12 @@ class KimiK25Renderer:
     The tokenizer should be ``moonshotai/Kimi-K2-Instruct`` (same as K2).
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Completing a tool cycle strips its earlier reasoning, even with thinking=False.
+        return False
+
     supports_process_multimodal = True
 
     def __init__(

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/laguna_xs2.py

@@ -120,6 +120,11 @@ _TOOLS_HEADER_XS21 = (
 
 
 class LagunaXS2Renderer:
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/llama_3.py

@@ -91,6 +91,11 @@ _TOOLS_IN_USER_INTRO = (
 class Llama3Renderer:
     """Deterministic message → token renderer for Llama-3.x Instruct models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/minimax_m2.py

@@ -58,6 +58,12 @@ _TOOLS_INSTRUCTIONS = (
 class MiniMaxM2Renderer:
     """Deterministic message → token renderer for MiniMax M2 / M2.5 models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Only assistants after the last user retain reasoning.
+        return False
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/nemotron3.py

@@ -118,6 +118,12 @@ class Nemotron3Renderer:
     their own discriminator and differ only by the class-level hooks here.
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # An effort hint moves to the newest user, independently of thinking retention.
+        return not self.config.truncate_history_thinking and not self._effort_hint
+
     # Variant hooks (overridden by ``Nemotron3UltraRenderer``): the default
     # config to build when none is passed, and whether to use Ultra's
     # reasoning-block glue.

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/prime_qwen3.py

@@ -193,6 +193,11 @@ class _TokenBuilder:
 class PrimeQwen3Renderer:
     """Renderer for PrimeIntellect/Qwen3-0.6B and Qwen3-1.7B."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/qwen3.py

@@ -59,6 +59,12 @@ _TOOLS_FOOTER = (
 class Qwen3Renderer:
     """Deterministic message → token renderer for Qwen3 models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Even with thinking disabled, supplied reasoning is stripped from history.
+        return False
+
     def __init__(
         self,
         tokenizer: Tokenizer,

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/qwen35.py

@@ -126,6 +126,12 @@ def _default_enable_thinking(tokenizer) -> bool:
 class Qwen35Renderer:
     """Deterministic message → token renderer for Qwen3.5 models."""
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        # Qwen3.6/3.8 can retain every thinking block through their template knob.
+        return bool(getattr(self.config, "preserve_thinking", False))
+
     supports_process_multimodal = True
     _config_cls: type = Qwen35RendererConfig
 

{renderers-0.1.12.dev18 → renderers-0.1.12.dev19}/renderers/qwen3_vl.py

@@ -312,6 +312,11 @@ class Qwen3VLRenderer:
     ``thinking_retention`` still controls whether the bridge is attempted.
     """
 
+    @property
+    def is_prefix_stable(self) -> bool:
+        """Whether full renders preserve completed conversation prefixes."""
+        return True
+
     supports_process_multimodal = True
 
     def __init__(

renderers-0.1.12.dev19/tests/audit_prefix_stability.py

@@ -0,0 +1,81 @@
+"""Manual audit: uv run pytest tests/audit_prefix_stability.py -q."""
+
+import json
+from difflib import unified_diff
+from pathlib import Path
+
+import pytest
+from conftest import _load
+from renderers import DefaultRenderer, create_renderer
+from renderers.base import RENDERER_REGISTRY, _populate_registry
+from renderers.configs import _config_class_for
+
+_CORPUS = json.loads(
+    (Path(__file__).parent / "fixtures" / "prefix_stability.json").read_text()
+)
+
+
+def _prefix_diff(tokenizer, before, after):
+    candidate = after[: len(before)]
+    if before == candidate:
+        return "No differing token: this witness now preserves the original prefix."
+    index = next(
+        (i for i, (left, right) in enumerate(zip(before, candidate)) if left != right),
+        min(len(before), len(candidate)),
+    )
+    diff = "\n".join(
+        unified_diff(
+            tokenizer.decode(before).splitlines(),
+            tokenizer.decode(candidate).splitlines(),
+            fromfile="original render",
+            tofile="same-length prefix after append",
+            lineterm="",
+        )
+    )
+    return (
+        f"First differing token: {index}\n"
+        f"Before: {before[index : index + 8]}\n"
+        f"After:  {candidate[index : index + 8]}\n{diff}"
+    )
+
+
+@pytest.mark.parametrize("case", _CORPUS["cases"], ids=lambda case: case["id"])
+def test_prefix_stability_witness(case):
+    config = dict(case["config"])
+    name = config.pop("name")
+    config_cls = _config_class_for(name)
+    tokenizer, _ = _load(case["model"], name)
+    renderer = create_renderer(tokenizer, config_cls(**config))
+    scenario = _CORPUS["scenarios"][case["scenario"]]
+    messages = scenario["messages"]
+    assert messages[-1]["role"] == "assistant" and scenario["append"]
+    before = renderer.render_ids(messages, tools=scenario.get("tools"))
+    after = renderer.render_ids(
+        messages + scenario["append"], tools=scenario.get("tools")
+    )
+    extends = after[: len(before)] == before
+    assert renderer.is_prefix_stable is case["stable"], case["reason"]
+    assert extends is case["stable"], (
+        f"{case['reason']}\n{_prefix_diff(tokenizer, before, after)}"
+    )
+
+
+def test_corpus_covers_builtin_renderers():
+    _populate_registry()
+    assert {case["config"]["name"] for case in _CORPUS["cases"]} | {"default"} == set(
+        RENDERER_REGISTRY
+    )
+    assert len({case["id"] for case in _CORPUS["cases"]}) == len(_CORPUS["cases"])
+    # An opaque template is unknown; inventing a counterexample would be wrong.
+    assert DefaultRenderer(object()).is_prefix_stable is False
+
+
+def test_stability_uses_auto_resolved_template_kwargs():
+    tokenizer, _ = _load("Qwen/Qwen3.8-27B", "qwen3.8")
+    assert create_renderer(tokenizer).is_prefix_stable is True
+    assert (
+        create_renderer(
+            tokenizer, chat_template_kwargs={"preserve_thinking": False}
+        ).is_prefix_stable
+        is False
+    )