msgflux 0.2.0__tar.gz → 0.3.0__tar.gz

{msgflux-0.2.0 → msgflux-0.3.0}/.pre-commit-config.yaml

@@ -7,12 +7,12 @@ repos:
       - id: gitleaks
 
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.11.6 # uv version.
+    rev: 0.11.7 # uv version.
     hooks:
       - id: uv-lock
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.10
+    rev: v0.15.11
     hooks:
       - id: ruff
         args: [--fix]

{msgflux-0.2.0 → msgflux-0.3.0}/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## [0.3.0] - 2026-04-20
+
+- Features: added OpenAI prompt cache retention support (#52), token logprobs metadata for chat completions (#53), and the `LLMAsVerifier` workflow with docs and examples (#54).
+- Fixes: mapped OpenRouter `reasoning_max_tokens` to `extra_body.reasoning.max_tokens` and enforced the `reasoning_effort` conflict rule (#57).
+- Docs: updated chat completion and agent reasoning docs to reflect the OpenRouter-only reasoning budget behavior (#56, #57).
+- Maintenance: updated pre-commit hooks (#55).
+
 ## [0.2.0] - 2026-04-17
 
 - Features: added `provider/model-id` string shorthand support to `Agent`, `Speaker`, and `Transcriber` (#26, #27, #28), plus runtime tool controls for `Agent` (#30).

{msgflux-0.2.0 → msgflux-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: msgflux
-Version: 0.2.0
+Version: 0.3.0
 Summary: Messages and Dynamic AI Systems in Python powered by pretrained models
 Project-URL: Homepage, https://github.com/msgflux/msgflux
 Project-URL: Documentation, https://github.com/msgflux/msgflux

{msgflux-0.2.0 → msgflux-0.3.0}/docs/anatomy/index.md

@@ -41,3 +41,5 @@ Use this section when you need to answer questions like:
   `dict[K, V]` is encoded for strict structured outputs and restored at runtime.
 - [ReAct Provider Schemas](react-provider-schemas.md): how ReAct uses tool
   schemas to build a dynamic provider-facing response format.
+- [LLM-as-a-Verifier](llm-as-a-verifier.md): how verifier prompts, concurrent
+  execution, score-token parsing, and weighted aggregation fit together.

msgflux-0.3.0/docs/anatomy/llm-as-a-verifier.md

@@ -0,0 +1,317 @@
+# LLM-as-a-Verifier
+
+`src/msgflux/generation/verifiers/llm_as_a_verifier.py` is the first verifier
+module in msgFlux.
+
+At a high level, it turns a verification task into a reusable runtime with four
+main jobs:
+
+- build a verifier prompt from task, criteria, candidates, and context
+- request discrete score tokens from the model
+- estimate a stable score from `logprobs` instead of trusting only the chosen
+  token
+- aggregate repeated attempts and multiple criteria into a final verdict
+
+It is not tied to `Agent`. It is a callable verification layer that can be used
+from plain generation code, tool workflows, candidate reranking, or future
+optimizers.
+
+## Mental Model
+
+The verifier pipeline looks like this:
+
+```text
+task + candidates + criteria
+  -> prompt builder
+  -> request batch
+  -> concurrent model calls
+  -> per-attempt evidence extraction
+     -> text token
+     -> logprob distribution
+  -> criterion aggregation
+  -> final verifier result
+```
+
+The important part is that the verifier does not ask the model for an open-ended
+score. It asks for a small discrete token from a known scale, then uses the
+token distribution to estimate the final normalized score.
+
+## Why The Output Is A Score Token
+
+The verifier uses a discrete score scale such as `A..T`.
+
+That choice is deliberate:
+
+- it keeps the scoring space bounded
+- it makes the provider response easier to parse
+- it makes `logprobs` directly useful
+- it lets the runtime compute an expected score instead of a brittle hard label
+
+Letters are preferred over digits for the default scale because they behave
+more like single symbolic labels as granularity grows.
+
+That matters for `logprobs` extraction:
+
+- `A..T` is a cleaner discrete label set than `1..20`
+- multi-digit numeric scales are more likely to split across tokens
+- numeric labels are more likely to attach to punctuation or formatting noise
+- letter labels are easier to rank, normalize, and recover from imperfect tag
+  formatting
+
+So the default runtime chooses a letter-based scale not for presentation, but
+for tokenization stability and better score-distribution extraction.
+
+So the model is not asked to emit a floating-point score. It is asked to emit a
+single token inside a known tag:
+
+```text
+<score>A</score>
+```
+
+or, in pairwise mode:
+
+```text
+<score_A>A</score_A>
+<score_B>H</score_B>
+```
+
+`ScoreScale.letter(granularity=20)` is the public way to ask for 20 score
+levels explicitly. That is also the default runtime scale.
+
+## Prompt Construction
+
+`default_prompt_builder(...)` assembles the verifier prompt from
+`VerificationPromptInput`.
+
+That input contains:
+
+- `task`
+- `criterion`
+- `candidates`
+- `score_scale`
+- optional `context`
+- optional verifier-level instructions
+
+The default builder always explains:
+
+- the criterion being judged
+- the valid score-token range
+- the exact tag format expected at the end
+
+This is why the verifier can support custom prompting without changing the rest
+of the execution pipeline. The builder is only responsible for prompt text; the
+runtime remains responsible for scoring and aggregation.
+
+## Trajectory Formatting Helpers
+
+Benchmark-style presets work best when the candidate contains the full execution
+evidence, not only the final answer.
+
+The public API therefore exposes:
+
+- `format_terminal_trajectory(...)`
+- `format_swe_bench_trajectory(...)`
+
+These helpers deliberately format only candidate evidence:
+
+- summaries
+- command/output steps
+- metadata such as expected output
+- final patch text for SWE-style runs
+- final answer or final narration
+
+The task or issue description stays outside the helper and should still be
+passed through the verifier's `task` argument. That keeps the verifier API
+clean: `task` carries the evaluation target, while the formatted candidate
+string carries the evidence being judged.
+
+## Request Batching And Concurrency
+
+The verifier builds a `VerificationRequest` for each independent
+criterion/repetition pair.
+
+That means the number of model calls in a single verification is:
+
+```text
+len(criteria) * n_verifications
+```
+
+Those requests are independent, so the verifier fans them out concurrently:
+
+- `__call__` uses `F.map_gather(...)`
+- `acall` uses `F.amap_gather(...)`
+
+This matters because a verifier with multiple criteria and repeated attempts can
+otherwise accumulate latency linearly.
+
+The runtime still keeps the final result deterministic:
+
+- requests are built in criterion order, then repetition order
+- gather responses preserve input order
+- attempts are regrouped per criterion after execution
+
+So the work is concurrent, but the aggregation order remains stable.
+
+## Why `select_best(...)` Is Separate
+
+`__call__` and `acall` support only one or two candidates.
+
+That boundary is intentional:
+
+- one candidate -> verification
+- two candidates -> pairwise comparison
+- more than two candidates -> tournament selection
+
+`select_best(...)` and `aselect_best(...)` implement the tournament layer on top
+of the base verifier. They run pairwise matches, accumulate wins, and use
+average score as a tiebreaker.
+
+The tournament layer can also fan out its pairwise matches concurrently. That
+means a multi-candidate selection now has two concurrency layers:
+
+- concurrent attempts inside each verifier call
+- concurrent pairwise matches across the tournament
+
+This keeps the core verifier simple instead of making `__call__` return
+different shapes for different candidate counts.
+
+## Scoring From Logprobs
+
+The central scoring path is:
+
+```text
+model response
+  -> locate score token
+  -> collect chosen token + top_logprobs alternatives
+  -> convert logprobs to probabilities
+  -> compute expected value on the discrete score scale
+  -> normalize to [0, 1]
+```
+
+This is why `LLMAsVerifier` always requests:
+
+- `logprobs=True`
+- `top_logprobs=<scale size or override>`
+
+If the provider returns a good token distribution, the verifier uses that
+distribution directly.
+
+If not, and `strict_logprobs=False`, it falls back to parsing the score from the
+response text.
+
+## Why The Parser Has Heuristics
+
+Provider tokenization around XML-like tags is not always clean.
+
+The verifier therefore uses a layered extraction strategy:
+
+1. parse the text tag if possible
+2. try to find the matching score entry directly in `logprobs`
+3. fall back to finding the expected score token near the closing tag
+4. if strict mode is disabled, fall back to text-only extraction
+
+This is also why the parser tolerates malformed openings such as:
+
+```text
+<20A</score>
+```
+
+The runtime treats the text and token stream as evidence that must be reconciled,
+not as perfectly formatted output.
+
+## Aggregation Model
+
+There are two aggregation stages.
+
+### Attempt Aggregation
+
+Each criterion can run multiple repetitions.
+
+Those repeated attempts are averaged into a single score per criterion and
+candidate.
+
+### Criterion Aggregation
+
+Each criterion carries a weight.
+
+The final score per candidate is the weighted average of all criterion scores.
+
+That is what makes the verifier reusable across different presets: the
+execution path stays the same while the criteria set changes.
+
+## Verdict Semantics
+
+The final verdict depends on candidate count.
+
+For a single candidate:
+
+- `pass`
+- `fail`
+- `uncertain`
+
+For two candidates:
+
+- winner label
+- `tie`
+
+This is why `VerifierResult` exposes both:
+
+- `verdict`
+- optional `winner`
+
+## Verbose Mode
+
+When `verbose=True`, the verifier stores enough information to debug prompt and
+scoring behavior without rerunning the call.
+
+Per attempt, it keeps:
+
+- final prompt text
+- raw response text
+- per-candidate evidence
+- extracted token probabilities
+
+That data is exposed both in:
+
+- `criteria_results[*].attempts[*]`
+- `result.metadata["raw_outputs"]`
+
+Tournament mode mirrors that same information in each match entry of
+`TournamentResult.metadata["raw_outputs"]`.
+
+## Presets
+
+The preset constructors are thin wrappers around the same runtime:
+
+- `trajectory_analysis`
+- `answer_reranking`
+- `grounded_answer_verification`
+- `patch_selection`
+- `terminal_bench`
+- `swe_bench_verified`
+- `tool_trace_verification`
+- `synthetic_data_filtering`
+
+Each preset only supplies:
+
+- a default criteria set
+- optional default instructions or ground-truth notes
+
+The runtime path stays identical.
+
+This is important for maintenance because it means new verifier presets should
+usually be data/config additions, not new execution engines.
+
+## Main Design Choices
+
+The current design makes a few explicit tradeoffs:
+
+- verifier core is independent from `Agent`
+- prompt building is pluggable, scoring is centralized
+- discrete score tokens are preferred over free-form numeric scores
+- `logprobs` are first-class evidence
+- tournament selection is built on top of pairwise verification
+- verbose debugging is part of the runtime, not an external wrapper
+
+These choices make the verifier easier to reuse for reranking, trajectory
+selection, patch selection, and future optimizer integrations.

{msgflux-0.2.0 → msgflux-0.3.0}/docs/learn/models/chat_completion.md

@@ -59,17 +59,20 @@ Chat completion models are stateless - they don't maintain conversation history
         reasoning_effort="medium",     # "minimal", "low", "medium", "high"
         enable_thinking=True,          # Enable extended model reasoning
         return_reasoning=True,         # Include reasoning content in response
-        reasoning_max_tokens=4096,     # Max tokens reserved for reasoning/thinking
+        reasoning_max_tokens=4096,     # OpenRouter only: max tokens reserved for reasoning/thinking
         reasoning_in_tool_call=True,   # Preserve reasoning context across tool calls
         # --- Output ---
         modalities=["text"],           # ["text"], ["audio"] or ["text", "audio"]
         audio={"voice": "alloy", "format": "mp3"},  # Audio output config
         verbosity="medium",            # Response verbosity: "low", "medium", "high"
+        logprobs=True,                 # Include token logprobs in metadata
+        top_logprobs=2,                # Return 2 alternatives per token
         parallel_tool_calls=True,      # Allow model to call multiple tools in parallel
         validate_typed_parser_output=False,  # Validate typed parser output with schema
         verbose=False,                 # Print raw output before transformation
         # --- Search ---
         web_search_options={},         # Web search config (OpenAI / OpenRouter only)
+        prompt_cache_retention="24h",  # OpenAI only: "in_memory" or "24h"
         # --- Infrastructure ---
         base_url="https://api.openai.com/v1",  # Override provider API endpoint
         context_length=128000,         # Override maximum context window
@@ -837,7 +840,7 @@ msgFlux exposes five parameters that control reasoning behaviour at model initia
 | Parameter | Description | Default |
 |---|---|---|
 | `reasoning_effort` | How much reasoning to do. One of `"minimal"`, `"low"`, `"medium"`, `"high"`. | — |
-| `reasoning_max_tokens` | Hard cap (in tokens) on the internal thinking budget. | — |
+| `reasoning_max_tokens` | OpenRouter-only hard cap (in tokens) for the internal thinking budget. Maps to `extra_body.reasoning.max_tokens` and cannot be combined with `reasoning_effort`. | — |
 | `return_reasoning` | Store the reasoning trace in `response.reasoning`. When `False`, reasoning is discarded even if the provider returns it. | `True` |
 | `enable_thinking` | Activate extended model reasoning. | `False` |
 | `reasoning_in_tool_call` | Preserve reasoning context across tool calls so the model keeps its chain of thought intact. When enabled, the `ToolCallAggregator` embeds the reasoning in `<think>` tags inside the assistant message history, allowing the model to see its previous reasoning when processing tool results. | `False` |
@@ -1007,7 +1010,7 @@ Providers that keep reasoning internal (like OpenAI) still report how many token
 
 ### 12.6 **Controlling the Reasoning Budget**
 
-`reasoning_max_tokens` caps how many tokens the model can use for internal thinking. Use it to bound latency and cost while still enabling reasoning:
+`reasoning_max_tokens` is an OpenRouter-only parameter in msgFlux. It maps to `extra_body={"reasoning": {"max_tokens": ...}}` on the OpenRouter API and cannot be combined with `reasoning_effort`.
 
 ???+ example
 
@@ -1015,17 +1018,17 @@ Providers that keep reasoning internal (like OpenAI) still report how many token
     import msgflux as mf
 
     model = mf.Model.chat_completion(
-        "groq/openai/gpt-oss-120b",
-        reasoning_effort="high",
-        reasoning_max_tokens=512,   # Cap the thinking budget
+        "openrouter/anthropic/claude-sonnet-4.5",
+        reasoning_max_tokens=2000,
     )
 
-    response = model("Solve: if 3x + 7 = 22, what is x?")
-    print(response.consume_reasoning())   # Kept short by the token cap
+    response = model("What's the weather like in Boston? Then recommend what to wear.")
+    print(response.consume_reasoning())
     print(response.consume())
-    # x = 5
     ```
 
+OpenRouter's own API examples pass the reasoning budget inside `extra_body={"reasoning": {"max_tokens": 2000}}`; msgFlux translates the model-level `reasoning_max_tokens` parameter to that shape for you.
+
 ### 12.7 **Streaming with Reasoning**
 
 Streaming introduces a dual-queue architecture. Content and reasoning flow through independent queues, allowing consumers to process them in parallel or sequentially.
@@ -1383,7 +1386,61 @@ if "tool_call" in model_response.response_type:
 
 The Agent reads `model_response.reasoning` to pass it downstream. If the Agent's `config["reasoning_in_response"]` is `True`, the final output is wrapped as `dotdict(answer=raw_response, reasoning=reasoning)` — this is an explicit opt-in at the Agent level, not a silent model-level behaviour.
 
-## 13. **Response Metadata**
+## 13. **Token Logprobs**
+
+`logprobs` and `top_logprobs` are forwarded for `openai/...` chat completion models. Providers that inherit from `OpenAIChatCompletion` but are not OpenAI do not receive these fields.
+
+Use `logprobs=True` to request token log probabilities. Set `top_logprobs` to the number of alternative tokens you want returned for each generated token.
+
+| Parameter | Description |
+|---|---|
+| `logprobs` | Enables token-level logprob data in the response metadata. |
+| `top_logprobs` | Number of alternative tokens returned per generated token. Use with `logprobs=True`. |
+
+The returned payload is exposed in `response.metadata.logprobs` and follows OpenAI's native shape. It includes the `content` list with token entries and nested `top_logprobs` alternatives.
+
+???+ example
+
+    ```python
+    import msgflux as mf
+
+    model = mf.Model.chat_completion(
+        "openai/gpt-4.1-mini",
+        logprobs=True,
+        top_logprobs=2,
+    )
+
+    response = model("Hello!")
+    print(response.metadata.logprobs["content"][0]["token"])
+    print(response.metadata.logprobs["content"][0]["top_logprobs"][0]["token"])
+    ```
+
+## 14. **OpenAI Prompt Caching**
+
+`prompt_cache_retention` is an OpenAI-only initialization parameter. msgFlux forwards it only for `openai/...` chat completion models. OpenAI-compatible providers that inherit from `OpenAIChatCompletion` do not use it.
+
+Use it when you want OpenAI to keep cached prefixes in memory or retain them for longer:
+
+| Value | Behaviour |
+|---|---|
+| `"in_memory"` | Default. Keeps the cache in volatile memory for short-lived reuse. |
+| `"24h"` | Extended retention. Keeps cached prefixes available for longer. |
+
+???+ example
+
+    ```python
+    import msgflux as mf
+
+    model = mf.Model.chat_completion(
+        "openai/gpt-5.1",
+        prompt_cache_retention="24h",
+    )
+
+    response = model("Summarize the attached policy.")
+    print(response.consume())
+    ```
+
+## 15. **Response Metadata**
 
 All responses include metadata with usage information:
 
@@ -1419,7 +1476,7 @@ All responses include metadata with usage information:
         print(f"Request cost: ${cost:.4f}")
     ```
 
-## 14. **Error Handling**
+## 16. **Error Handling**
 
 Handle common errors gracefully:
 
@@ -1441,7 +1498,7 @@ Handle common errors gracefully:
         print(f"API error: {e}")
     ```
 
-## 15. **Model Profiles**
+## 17. **Model Profiles**
 
 Model profiles provide metadata about capabilities, pricing, and limits from [models.dev](https://models.dev).
 
@@ -1507,11 +1564,11 @@ Every initialized model exposes a `.profile` property that returns this metadata
             print(f"Estimated cost: ${cost:.4f}")
         ```
 
-## 16. **Adding a Custom Provider**
+## 18. **Adding a Custom Provider**
 
 If the service you want to use exposes an **OpenAI-compatible API**, you can add it as a provider by subclassing `OpenAIChatCompletion`. The process has two stages depending on how compatible the endpoint is.
 
-### 16.1 **Stage 1 — URL and API key only**
+### 18.1 **Stage 1 — URL and API key only**
 
 When the target API is fully OpenAI-compatible and only requires a different base URL and authentication key, the entire subclass is a small configuration mixin plus the `@register_model` decorator.
 
@@ -1555,7 +1612,7 @@ After registering, the model is available through the standard factory. The stri
     print(response.consume())
     ```
 
-### 16.2 **Stage 2 — Adapting parameters**
+### 18.2 **Stage 2 — Adapting parameters**
 
 Some providers are mostly OpenAI-compatible but have small differences: renamed fields, required extra headers, or unsupported parameters. Override `_adapt_params` to transform the parameter dict before it reaches the API.
 
@@ -1598,12 +1655,22 @@ The built-in OpenRouter provider is a real-world example:
             if params["tool_choice"] is None:
                 params["tool_choice"] = "auto" if params["tools"] else "none"
 
-            # 3. Map reasoning_effort to the provider format
+            # 3. Map reasoning parameters to the provider format
             reasoning_effort = params.pop("reasoning_effort", None)
+            reasoning_max_tokens = params.pop("reasoning_max_tokens", None)
+            if reasoning_effort is not None and reasoning_max_tokens is not None:
+                raise ValueError(
+                    "`reasoning_max_tokens` cannot be used together with "
+                    "`reasoning_effort` for OpenRouter."
+                )
             if reasoning_effort is not None:
                 extra_body = params.get("extra_body", {})
                 extra_body["reasoning"] = {"effort": reasoning_effort}
                 params["extra_body"] = extra_body
+            if reasoning_max_tokens is not None:
+                extra_body = params.get("extra_body", {})
+                extra_body["reasoning"] = {"max_tokens": reasoning_max_tokens}
+                params["extra_body"] = extra_body
 
             # 4. Add required headers
             params["extra_headers"] = {
@@ -1619,11 +1686,11 @@ Common adaptations inside `_adapt_params`:
 |---|---|
 | Provider uses `max_completion_tokens` instead of `max_tokens` | `params["max_completion_tokens"] = params.pop("max_tokens")` |
 | Provider rejects `tool_choice=None` | Set it explicitly to `"auto"` or `"none"` |
-| Provider uses a different field for reasoning | `params.pop("reasoning_effort")` and remap into `extra_body` |
+| Provider uses a different field for reasoning | `params.pop("reasoning_effort")` or `params.pop("reasoning_max_tokens")` and remap into `extra_body` |
 | Provider requires extra headers | Add keys to `params["extra_headers"]` |
 | Provider accepts non-standard extensions | Add keys to `params["extra_body"]` |
 
-### 16.3 **Stage 3 — Using a different client**
+### 18.3 **Stage 3 — Using a different client**
 
 The two previous stages assume the service is reached through the `openai` Python package. If you want to use a completely different HTTP client or SDK — one that is **not** the `openai` package but still exposes a compatible interface — override `_initialize` instead.
 

{msgflux-0.2.0 → msgflux-0.3.0}/docs/learn/nn/agent/reasoning.md

@@ -8,9 +8,9 @@ Modern language models can "think before answering" — generating an internal c
 |---|---|---|
 | **What it is** | The model's native thinking capability (e.g. `reasoning_effort="high"`) | A `generation_schema` that forces structured thinking (CoT, ReAct, SelfConsistency) |
 | **Where reasoning lives** | `response.reasoning` — a first-class field on the response object | Inside `response.consume()` — as a field of the structured output (e.g. `result.reasoning`) |
-| **Configured via** | Model params: `reasoning_effort`, `return_reasoning`, `reasoning_max_tokens` | Agent param: `generation_schema=ChainOfThought` |
+| **Configured via** | Model params: `reasoning_effort`, `return_reasoning`, `reasoning_max_tokens` (OpenRouter only) | Agent param: `generation_schema=ChainOfThought` |
 | **Works with any model** | No — requires a reasoning-capable model (Groq gpt-oss, OpenAI o-series, etc.) | Yes — any model that supports structured output |
-| **Controllable budget** | Yes — `reasoning_effort` and `reasoning_max_tokens` | No — the model decides how much to write in the schema field |
+| **Controllable budget** | Yes — `reasoning_effort`; OpenRouter also supports `reasoning_max_tokens` | No — the model decides how much to write in the schema field |
 | **Can combine with tools** | Yes — `reasoning_in_tool_call=True` preserves the chain across calls | Yes — ReAct is specifically designed for tool use |
 
 !!! tip
@@ -28,10 +28,9 @@ Model-level reasoning is configured at model initialization through parameters f
 import msgflux as mf
 
 model = mf.Model.chat_completion(
-    "groq/openai/gpt-oss-120b",
-    reasoning_effort="low",       # How much to think: "minimal", "low", "medium", "high"
+    "openrouter/anthropic/claude-sonnet-4.5",
     return_reasoning=True,        # Store the trace in response.reasoning (default: True)
-    reasoning_max_tokens=1024,    # Cap the thinking budget in tokens
+    reasoning_max_tokens=1024,    # OpenRouter only: cap the thinking budget in tokens
     reasoning_in_tool_call=True,  # Preserve reasoning across tool call rounds
 )
 ```
@@ -600,7 +599,7 @@ This is useful for debugging the relationship between the model's thinking and i
 |---|---|---|---|
 | `reasoning_effort` | `str` | — | `"minimal"`, `"low"`, `"medium"`, `"high"` |
 | `return_reasoning` | `bool` | `True` | Store reasoning in `response.reasoning` |
-| `reasoning_max_tokens` | `int` | — | Cap reasoning token budget |
+| `reasoning_max_tokens` | `int` | — | OpenRouter-only cap on reasoning token budget |
 | `reasoning_in_tool_call` | `bool` | `False` | Embed reasoning in `<think>` tags across tool call rounds |
 | `enable_thinking` | `bool` | `False` | Provider-level switch (e.g. Anthropic) |