renderers 0.1.12.dev3__tar.gz → 0.1.12.dev5__tar.gz

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/.github/workflows/test.yml

@@ -21,7 +21,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
 
     steps:
       - uses: actions/checkout@v4

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.5
 Name: renderers
-Version: 0.1.12.dev3
+Version: 0.1.12.dev5
 Summary: Chat template renderers — deterministic message-to-token conversion for LLM training
 License-Expression: Apache-2.0
 License-File: LICENSE
-Requires-Python: <3.14,>=3.10
+Requires-Python: <3.15,>=3.10
 Requires-Dist: jinja2
 Requires-Dist: numpy
 Requires-Dist: openai-harmony>=0.0.4
@@ -63,10 +63,39 @@ prompt_ids = r.render_ids(
 # Feed prompt_ids to a Token-In, Token-Out endpoint.
 # It returns completion_ids sampled by the model.
 
-parsed = r.parse_response(completion_ids)
+parsed = r.parse_response(completion_ids, prompt_ids=prompt_ids)
 # ParsedResponse(content=..., reasoning_content=..., tool_calls=...)
 ```
 
+Pass the exact sampling `prompt_ids` into `parse_response` to identify reasoning
+opened by the generation prompt. Without its closing marker, the sampled text
+stays in `reasoning_content`, with empty `content` and no executable tool calls,
+even if the engine reports `stop`. This applies to every supported reasoning
+format, including Harmony analysis and Gemma's post-tool thought channel.
+Omitting `prompt_ids`, passing `None`, and passing `[]` all parse the completion
+as self-contained: reasoning needs its own initial opener or channel header.
+Generation settings never supply missing parsing context. If the prompt supplied
+the opener, pass that prompt; `generate` does this automatically.
+
+For reasoning-first formats, only an initial `<think>` (after any assistant
+header) or reasoning already opened in the prompt starts reasoning. After its
+first closing marker, the rest is content. Later think markers remain literal
+content: `Before<think>example</think>After` is entirely content. Harmony,
+Gemma, and Inkling follow their explicit channel or message-segment grammar.
+
+`ParsedResponse.reasoning_complete` is `False` while a reasoning channel is open,
+`True` when it is closed (or the format has no reasoning channel), and `None` when
+its state cannot be determined. It does not certify tool syntax, normal engine
+termination, nonempty final output, or task success. The generate client forwards
+this narrow signal and passes prompt context automatically. Custom renderers used
+with `generate` should accept `parse_response(..., tools=None, prompt_ids=None)`;
+formats that do not need prompt context can ignore it.
+
+Summary consumers should require normal termination (`finish_reason="stop"`),
+nonempty final content, and no tool calls. A length-limited final answer remains
+truncated even when its reasoning channel is closed. Keep rejected attempts in
+usage/training accounting, and replace conversation history only after acceptance.
+
 For the next turn, extend the previous sampled stream instead of re-rendering history:
 
 ```python
@@ -99,9 +128,10 @@ class Renderer(Protocol):
 Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return ids for the next turn's prompt such that the result starts with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continues with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render.
 
 Each hand-coded bridge:
-1. Anchors at the previous turn's canonical close token. On clean stops it's already in `prev_completion_ids`. On truncation, the renderer synthesizes the close as non-loss prompt context.
-2. Refuses assistant content in `new_messages` — re-rendering sampled tokens would replace them with canonical template bytes.
-3. Renders only the new messages in the framing the model family expects.
+1. Checks the previous turn's reasoning state. If reasoning is unfinished and no stop was sampled, append its closing marker as non-loss prompt context before the canonical turn close. If a stop was already sampled inside reasoning, return `None`: inserting a close before it would violate the exact-prefix contract. Parsing never repairs the sampled output.
+2. Anchors at the previous turn's canonical close token. On clean stops it's already in `prev_completion_ids`. On truncation, the renderer synthesizes the close as non-loss prompt context.
+3. Refuses assistant content in `new_messages` — re-rendering sampled tokens would replace them with canonical template bytes.
+4. Renders only the new messages in the framing the model family expects.
 
 `DefaultRenderer.bridge_to_next_turn` returns `None` unconditionally — the template's close is unknown, so the contract can't be proven.
 

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/README.md

@@ -43,10 +43,39 @@ prompt_ids = r.render_ids(
 # Feed prompt_ids to a Token-In, Token-Out endpoint.
 # It returns completion_ids sampled by the model.
 
-parsed = r.parse_response(completion_ids)
+parsed = r.parse_response(completion_ids, prompt_ids=prompt_ids)
 # ParsedResponse(content=..., reasoning_content=..., tool_calls=...)
 ```
 
+Pass the exact sampling `prompt_ids` into `parse_response` to identify reasoning
+opened by the generation prompt. Without its closing marker, the sampled text
+stays in `reasoning_content`, with empty `content` and no executable tool calls,
+even if the engine reports `stop`. This applies to every supported reasoning
+format, including Harmony analysis and Gemma's post-tool thought channel.
+Omitting `prompt_ids`, passing `None`, and passing `[]` all parse the completion
+as self-contained: reasoning needs its own initial opener or channel header.
+Generation settings never supply missing parsing context. If the prompt supplied
+the opener, pass that prompt; `generate` does this automatically.
+
+For reasoning-first formats, only an initial `<think>` (after any assistant
+header) or reasoning already opened in the prompt starts reasoning. After its
+first closing marker, the rest is content. Later think markers remain literal
+content: `Before<think>example</think>After` is entirely content. Harmony,
+Gemma, and Inkling follow their explicit channel or message-segment grammar.
+
+`ParsedResponse.reasoning_complete` is `False` while a reasoning channel is open,
+`True` when it is closed (or the format has no reasoning channel), and `None` when
+its state cannot be determined. It does not certify tool syntax, normal engine
+termination, nonempty final output, or task success. The generate client forwards
+this narrow signal and passes prompt context automatically. Custom renderers used
+with `generate` should accept `parse_response(..., tools=None, prompt_ids=None)`;
+formats that do not need prompt context can ignore it.
+
+Summary consumers should require normal termination (`finish_reason="stop"`),
+nonempty final content, and no tool calls. A length-limited final answer remains
+truncated even when its reasoning channel is closed. Keep rejected attempts in
+usage/training accounting, and replace conversation history only after acceptance.
+
 For the next turn, extend the previous sampled stream instead of re-rendering history:
 
 ```python
@@ -79,9 +108,10 @@ class Renderer(Protocol):
 Given `(prev_prompt_ids, prev_completion_ids)` and new environment messages, return ids for the next turn's prompt such that the result starts with `prev_prompt_ids + prev_completion_ids` byte-for-byte and continues with the new messages plus the next assistant opener. If that cannot be proven safe, return `None` and the caller falls back to a full render.
 
 Each hand-coded bridge:
-1. Anchors at the previous turn's canonical close token. On clean stops it's already in `prev_completion_ids`. On truncation, the renderer synthesizes the close as non-loss prompt context.
-2. Refuses assistant content in `new_messages` — re-rendering sampled tokens would replace them with canonical template bytes.
-3. Renders only the new messages in the framing the model family expects.
+1. Checks the previous turn's reasoning state. If reasoning is unfinished and no stop was sampled, append its closing marker as non-loss prompt context before the canonical turn close. If a stop was already sampled inside reasoning, return `None`: inserting a close before it would violate the exact-prefix contract. Parsing never repairs the sampled output.
+2. Anchors at the previous turn's canonical close token. On clean stops it's already in `prev_completion_ids`. On truncation, the renderer synthesizes the close as non-loss prompt context.
+3. Refuses assistant content in `new_messages` — re-rendering sampled tokens would replace them with canonical template bytes.
+4. Renders only the new messages in the framing the model family expects.
 
 `DefaultRenderer.bridge_to_next_turn` returns `None` unconditionally — the template's close is unknown, so the contract can't be proven.
 

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/examples/sglang/multiturn_generate_sglang.py

@@ -138,7 +138,7 @@ def main() -> None:
         )
         output1 = engine.generate(input_ids=prompt_ids, sampling_params=sampling)
         completion1 = completion_ids(output1, prompt_ids)
-        parsed1 = renderer.parse_response(completion1)
+        parsed1 = renderer.parse_response(completion1, prompt_ids=prompt_ids)
         print_parsed(label, "turn 1", parsed1)
 
         assistant = {"role": "assistant", "content": parsed1.content}
@@ -197,7 +197,11 @@ def main() -> None:
 
         output2 = engine.generate(input_ids=bridged_ids, sampling_params=sampling)
         completion2 = completion_ids(output2, bridged_ids)
-        print_parsed(label, "turn 2", renderer.parse_response(completion2))
+        print_parsed(
+            label,
+            "turn 2",
+            renderer.parse_response(completion2, prompt_ids=bridged_ids),
+        )
 
         engine.shutdown()
 

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/examples/sglang/online_multiturn_sglang.py

@@ -161,7 +161,7 @@ async def run_one(
         max_new_tokens=max_new_tokens,
     )
     completion1 = completion_ids(output1, prompt_ids)
-    parsed1 = renderer.parse_response(completion1)
+    parsed1 = renderer.parse_response(completion1, prompt_ids=prompt_ids)
     print_parsed(label, "turn 1", parsed1)
 
     assistant: dict[str, Any] = {"role": "assistant", "content": parsed1.content}
@@ -226,7 +226,9 @@ async def run_one(
         max_new_tokens=max_new_tokens,
     )
     completion2 = completion_ids(output2, bridged_ids)
-    print_parsed(label, "turn 2", renderer.parse_response(completion2))
+    print_parsed(
+        label, "turn 2", renderer.parse_response(completion2, prompt_ids=bridged_ids)
+    )
 
 
 async def main() -> None:

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/examples/tinker/multiturn_generate_tinker.py

@@ -128,7 +128,7 @@ async def main() -> None:
             sampling_params=sampling_params,
         )
         completion1 = list(output1.sequences[0].tokens)
-        parsed1 = renderer.parse_response(completion1)
+        parsed1 = renderer.parse_response(completion1, prompt_ids=prompt_ids)
         print_parsed(label, "turn 1", parsed1)
 
         assistant = {"role": "assistant", "content": parsed1.content}
@@ -191,7 +191,11 @@ async def main() -> None:
             sampling_params=sampling_params,
         )
         completion2 = list(output2.sequences[0].tokens)
-        print_parsed(label, "turn 2", renderer.parse_response(completion2))
+        print_parsed(
+            label,
+            "turn 2",
+            renderer.parse_response(completion2, prompt_ids=bridged_ids),
+        )
 
 
 if __name__ == "__main__":

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/examples/transformers/multiturn_generate_transformers.py

@@ -135,7 +135,7 @@ def main() -> None:
             pad_token_id=pad_token_id,
         )[0]
         completion1 = output1[input_ids.shape[-1] :].tolist()
-        parsed1 = renderer.parse_response(completion1)
+        parsed1 = renderer.parse_response(completion1, prompt_ids=prompt_ids)
         print_parsed(label, "turn 1", parsed1)
 
         assistant = {"role": "assistant", "content": parsed1.content}
@@ -203,7 +203,11 @@ def main() -> None:
             pad_token_id=pad_token_id,
         )[0]
         completion2 = output2[bridged_input_ids.shape[-1] :].tolist()
-        print_parsed(label, "turn 2", renderer.parse_response(completion2))
+        print_parsed(
+            label,
+            "turn 2",
+            renderer.parse_response(completion2, prompt_ids=bridged_ids),
+        )
 
         del hf_model
         gc.collect()

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/examples/vllm/multiturn_generate_vllm.py

@@ -131,7 +131,7 @@ def main() -> None:
             use_tqdm=False,
         )[0]
         completion1 = list(output1.outputs[0].token_ids)
-        parsed1 = renderer.parse_response(completion1)
+        parsed1 = renderer.parse_response(completion1, prompt_ids=prompt_ids)
         print_parsed(label, "turn 1", parsed1)
 
         assistant = {"role": "assistant", "content": parsed1.content}
@@ -194,7 +194,11 @@ def main() -> None:
             use_tqdm=False,
         )[0]
         completion2 = list(output2.outputs[0].token_ids)
-        print_parsed(label, "turn 2", renderer.parse_response(completion2))
+        print_parsed(
+            label,
+            "turn 2",
+            renderer.parse_response(completion2, prompt_ids=bridged_ids),
+        )
 
         del llm
         gc.collect()

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/pyproject.toml

@@ -13,7 +13,7 @@ description = "Chat template renderers — deterministic message-to-token conver
 readme = "README.md"
 license = "Apache-2.0"
 license-files = ["LICENSE"]
-requires-python = ">=3.10,<3.14"
+requires-python = ">=3.10,<3.15"
 dependencies = [
     "numpy",
     "openai>=1.108.1",

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.12.dev3'
-__version_tuple__ = version_tuple = (0, 1, 12, 'dev3')
+__version__ = version = '0.1.12.dev5'
+__version_tuple__ = version_tuple = (0, 1, 12, 'dev5')
 
 __commit_id__ = commit_id = None

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/base.py

@@ -632,6 +632,13 @@ class ParsedResponse:
     content: str
     reasoning_content: str | None = None
     tool_calls: list[ParsedToolCall] = field(default_factory=list)
+    reasoning_complete: bool | None = True
+    """Whether the reasoning channel is closed at the end of the completion.
+
+    False means generation ended inside reasoning; this is unrelated to the
+    engine finish reason, tool-call validity, or whether final text is useful.
+    None means the parser cannot determine the state for its reasoning format.
+    """
 
 
 @dataclass
@@ -743,9 +750,16 @@ class Renderer(Protocol):
         token_ids: list[int],
         *,
         tools: list[ToolSpec] | None = None,
+        prompt_ids: list[int] | None = None,
     ) -> ParsedResponse:
         """Parse completion tokens back into a structured message.
 
+        Pass the exact ``prompt_ids`` used for sampling whenever available.
+        This identifies reasoning prefilled by the prompt, including Gemma's
+        post-tool thought channel and partial assistant prefixes. Omitting it,
+        passing None, and passing [] all parse a self-contained completion;
+        generation settings do not imply an initial reasoning channel.
+
         ``tools`` is the same list passed to ``render`` for this turn.
         XML-style formats (Qwen3.5, GLM, MiniMax, Laguna) render argument
         values verbatim inside ``<arg_value>`` tags with no quoting, so

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/client.py

@@ -367,7 +367,11 @@ async def generate(
 
     completion_logprobs = _parse_completion_logprobs(choice, completion_ids)
 
-    parsed = renderer.parse_response(completion_ids, tools=tools)
+    parsed = renderer.parse_response(
+        completion_ids,
+        prompt_ids=list(effective_prompt_ids or prompt_ids),
+        tools=tools,
+    )
 
     routed_experts = choice.get("routed_experts")
     # vLLM's native kept-set sampling masks (``--return-sampling-mask``):
@@ -401,6 +405,7 @@ async def generate(
         "reasoning_content": parsed.reasoning_content,
         "tool_calls": parsed.tool_calls,
         "finish_reason": finish_reason,
+        "reasoning_complete": parsed.reasoning_complete,
         "routed_experts": routed_experts,
         "sampling_mask": sampling_mask,
         # The mm sidecar consumed on the request side, surfaced back so

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/deepseek_v3.py

@@ -14,6 +14,8 @@ from __future__ import annotations
 
 import json
 
+from renderers.reasoning import scan_reasoning, prompt_ends_in_reasoning
+
 from renderers.base import (
     Message,
     ParsedResponse,
@@ -283,6 +285,7 @@ class DeepSeekV3Renderer:
         token_ids: list[int],
         *,
         tools: list[ToolSpec] | None = None,  # noqa: ARG002 — args land in a ```json fence, schema not needed
+        prompt_ids: list[int] | None = None,
     ) -> ParsedResponse:
         return parse_deepseek_v3(
             self._tokenizer,
@@ -293,6 +296,12 @@ class DeepSeekV3Renderer:
             tool_call_begin_id=self._tool_call_begin,
             tool_call_end_id=self._tool_call_end,
             tool_sep_id=self._tool_sep,
+            prefilled_thinking=prompt_ends_in_reasoning(
+                self._tokenizer,
+                prompt_ids,
+                stop_ids=set(self.get_stop_token_ids()),
+                assistant_prefix="<｜Assistant｜>",
+            ),
         )
 
     def get_stop_token_ids(self) -> list[int]:
@@ -312,6 +321,23 @@ class DeepSeekV3Renderer:
             or reject_assistant_in_extension(new_messages)
         ):
             return None
+
+        boundary = scan_reasoning(
+            self._tokenizer,
+            previous_completion_ids,
+            prompt_ids=previous_prompt_ids,
+            stop_ids=set(self.get_stop_token_ids()),
+            tool_start_id=self._tool_calls_begin,
+            assistant_prefix="<｜Assistant｜>",
+        )
+        if boundary.is_open:
+            if any(t in self.get_stop_token_ids() for t in previous_completion_ids):
+                return None
+            previous_completion_ids = [
+                *previous_completion_ids,
+                *list(self._tokenizer.encode("</think>", add_special_tokens=False)),
+            ]
+
         if should_rerender_for_thinking_retention(
             self.effective_thinking_retention,
             new_messages,

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/deepseek_v4.py

@@ -22,6 +22,8 @@ import json
 from dataclasses import dataclass, field
 from typing import Any, Mapping
 
+from renderers.reasoning import scan_reasoning, prompt_ends_in_reasoning
+
 from renderers.base import (
     Message,
     ParsedResponse,
@@ -783,13 +785,20 @@ class DeepSeekV4Renderer:
         token_ids: list[int],
         *,
         tools: list[ToolSpec] | None = None,  # noqa: ARG002
+        prompt_ids: list[int] | None = None,
     ) -> ParsedResponse:
         return parse_deepseek_v4(
             self._tokenizer,
             token_ids,
             stop_ids={self._eos},
-            thinking_enabled=self.config.enable_thinking,
+            thinking_enabled=prompt_ends_in_reasoning(
+                self._tokenizer,
+                prompt_ids,
+                stop_ids=set(self.get_stop_token_ids()),
+                assistant_prefix="<｜Assistant｜>",
+            ),
             think_end_id=self._think_end,
+            think_start_id=self._think_start,
             dsml_id=self._dsml,
         )
 
@@ -810,6 +819,20 @@ class DeepSeekV4Renderer:
             or reject_assistant_in_extension(new_messages)
         ):
             return None
+
+        boundary = scan_reasoning(
+            self._tokenizer,
+            previous_completion_ids,
+            prompt_ids=previous_prompt_ids,
+            stop_ids=set(self.get_stop_token_ids()),
+            tool_start_id=self._dsml,
+            assistant_prefix="<｜Assistant｜>",
+        )
+        if boundary.is_open:
+            if any(t in self.get_stop_token_ids() for t in previous_completion_ids):
+                return None
+            previous_completion_ids = [*previous_completion_ids, self._think_end]
+
         if should_rerender_for_thinking_retention(
             self.effective_thinking_retention,
             new_messages,

{renderers-0.1.12.dev3 → renderers-0.1.12.dev5}/renderers/default.py

@@ -11,6 +11,8 @@ from __future__ import annotations
 import json
 from typing import Any
 
+from renderers.reasoning import scan_reasoning, prompt_ends_in_reasoning
+
 from renderers.base import (
     ChatTemplateTokenizer,
     Message,
@@ -20,8 +22,14 @@ from renderers.base import (
     extract_message_tool_names,
     resolve_thinking_retention,
 )
+from renderers.parsing import (
+    _reasoning_end_token_index,
+    _strip_stop_tokens,
+)
+
 from renderers.configs import DefaultRendererConfig
 from renderers.parsers import (
+    ThinkTextReasoningParser,
     get_reasoning_parser,
     get_tool_parser,
 )
@@ -176,50 +184,66 @@ class DefaultRenderer:
         token_ids: list[int],
         *,
         tools: list[ToolSpec] | None = None,  # noqa: ARG002 — DefaultRenderer relies on configured tool_parser, schema not consulted here
+        prompt_ids: list[int] | None = None,
     ) -> ParsedResponse:
+        ids = _strip_stop_tokens(token_ids, set(self.get_stop_token_ids()))
+        boundary = scan_reasoning(
+            self._tokenizer,
+            ids,
+            prefilled=prompt_ends_in_reasoning(
+                self._tokenizer,
+                prompt_ids,
+                stop_ids=set(self.get_stop_token_ids()),
+            ),
+        )
+        if boundary.is_open:
+            return ParsedResponse(
+                content="", reasoning_content=boundary.text, reasoning_complete=False
+            )
+        reasoning_end = (
+            _reasoning_end_token_index(self._tokenizer, ids)
+            if boundary.text is not None
+            else 0
+        )
         # 1. Extract tool calls while we still have token ids (most formats
         #    use special-token delimiters, so id-level matching is reliable).
         if self._tool_parser is not None:
-            content_ids, tool_calls = self._tool_parser.extract(list(token_ids))
+            content_ids, tool_calls = self._tool_parser.extract(ids[reasoning_end:])
+            content_ids = [*ids[:reasoning_end], *content_ids]
+            for tc in tool_calls:
+                if tc.token_span is not None:
+                    start, end = tc.token_span
+                    tc.token_span = (start + reasoning_end, end + reasoning_end)
         else:
-            content_ids = list(token_ids)
+            content_ids = ids
             tool_calls = []
 
         # 2. Decode (keep special tokens so a downstream reasoning parser can
         #    still see things like <think>/</think> when they're tokens).
         text = self._tokenizer.decode(content_ids, skip_special_tokens=False)
 
-        # 3. Extract reasoning from the decoded text. Falls back to a built-in
-        #    <think>...</think> sniff so unconfigured users get the same behavior
-        #    as before. Preserve whitespace at the <think>/</think> boundary —
-        #    the chat template round-trips it verbatim (e.g. GLM emits
-        #    `{{ '\\n<think>' + reasoning_content + '</think>' }}` then
-        #    `{{- content }}` with no separator), so a leading `\\n` on content
-        #    or trailing `\\n` on reasoning_content must stay in the parsed
-        #    fields for re-render to be byte-identical. Stripping here causes
-        #    the re-rendered assistant message to shift by one BPE token after
-        #    `</think>`, cascading through downstream tokenization and breaking
-        #    the "extension property" in trajectory step tokenization.
-        if self._reasoning_parser is not None:
-            reasoning_content, text = self._reasoning_parser.extract(text)
+        # Use the same initial boundary for extraction and tool scanning.
+        # Preserve whitespace inside reasoning and after its close: templates
+        # may emit these fields back-to-back, so stripping shifts token offsets.
+        if self._reasoning_parser is None or isinstance(
+            self._reasoning_parser, ThinkTextReasoningParser
+        ):
+            reasoning_content = boundary.text
+            if boundary.text is not None:
+                text = text.partition("</think>")[2]
         else:
-            reasoning_content = None
-            if "</think>" in text:
-                before, after = text.split("</think>", 1)
-                if "<think>" in before:
-                    reasoning_content = before.split("<think>", 1)[-1]
-                else:
-                    reasoning_content = before
-                text = after
-
-        # Strip any remaining special tokens from the final content (we kept
-        # them around for the reasoning parser above).
+            reasoning_content, text = self._reasoning_parser.extract(text)
+
         text = _strip_special_tokens(self._tokenizer, text)
 
         return ParsedResponse(
             content=text,
             reasoning_content=reasoning_content if reasoning_content else None,
             tool_calls=tool_calls,
+            reasoning_complete=True
+            if isinstance(self._reasoning_parser, ThinkTextReasoningParser)
+            or reasoning_content is not None
+            else None,
         )
 
     def get_stop_token_ids(self) -> list[int]:
@@ -254,9 +278,9 @@ def _resolve_parser(value, tokenizer, factory):
 
 
 def _strip_special_tokens(tokenizer, text: str) -> str:
-    """Remove any special-token substrings that slipped into decoded text."""
+    """Remove framing tokens while preserving literal think markers in content."""
     specials = getattr(tokenizer, "all_special_tokens", None) or []
     for token in specials:
-        if token and token in text:
+        if token and token not in {"<think>", "</think>"} and token in text:
             text = text.replace(token, "")
     return text