msgflux 0.2.0__tar.gz → 0.4.1__tar.gz

{msgflux-0.2.0 → msgflux-0.4.1}/.pre-commit-config.yaml

@@ -7,12 +7,12 @@ repos:
       - id: gitleaks
 
   - repo: https://github.com/astral-sh/uv-pre-commit
-    rev: 0.11.6 # uv version.
+    rev: 0.11.7 # uv version.
     hooks:
       - id: uv-lock
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.10
+    rev: v0.15.11
     hooks:
       - id: ruff
         args: [--fix]

msgflux-0.4.1/CHANGELOG.md

@@ -0,0 +1,35 @@
+# Changelog
+
+## [0.4.1] - 2026-04-22
+
+- Fixes: restored uv.lock file for CI compatibility (#78).
+
+## [0.4.0] - 2026-04-22
+
+- Features: added new web retrievers (Arxiv, Brave, Exa, LinkUp, SerpAPI, Tavily) and model integrations (Brave, Exa), plus configurable WebSearch tool (#71).
+- Web Retriever: added support for Wikipedia highlighting in web retriever section (#73).
+- Providers: added OpenAI-compatible `extra_body` support for chat completions (#72).
+- Maintenance: removed uv.lock file (#74).
+
+## [0.3.0] - 2026-04-20
+
+- Features: added OpenAI prompt cache retention support (#52), token logprobs metadata for chat completions (#53), and the `LLMAsVerifier` workflow with docs and examples (#54).
+- Fixes: mapped OpenRouter `reasoning_max_tokens` to `extra_body.reasoning.max_tokens` and enforced the `reasoning_effort` conflict rule (#57).
+- Docs: updated chat completion and agent reasoning docs to reflect the OpenRouter-only reasoning budget behavior (#56, #57).
+- Maintenance: updated pre-commit hooks (#55).
+
+## [0.2.0] - 2026-04-17
+
+- Features: added `provider/model-id` string shorthand support to `Agent`, `Speaker`, and `Transcriber` (#26, #27, #28), plus runtime tool controls for `Agent` (#30).
+- Fixes: fixed registry naming for generic callables (#29).
+- Maintenance: updated release automation and cleaned up the release script (#36, #49).
+
+## [0.1.1] - 2026-04-15
+
+- Fixed message copying in the OpenAI provider (#47).
+- Fixed isolation of injected messages for agent tools (#46).
+- Updated CI and dependency tooling to keep release checks passing (#45, #44, #38, #37, #35, #34, #33, #32).
+
+## [0.1.0] - 2026-04-09
+
+## [0.1.0a31] - 2026-04-09

{msgflux-0.2.0 → msgflux-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: msgflux
-Version: 0.2.0
+Version: 0.4.1
 Summary: Messages and Dynamic AI Systems in Python powered by pretrained models
 Project-URL: Homepage, https://github.com/msgflux/msgflux
 Project-URL: Documentation, https://github.com/msgflux/msgflux

{msgflux-0.2.0 → msgflux-0.4.1}/docs/anatomy/index.md

@@ -41,3 +41,5 @@ Use this section when you need to answer questions like:
   `dict[K, V]` is encoded for strict structured outputs and restored at runtime.
 - [ReAct Provider Schemas](react-provider-schemas.md): how ReAct uses tool
   schemas to build a dynamic provider-facing response format.
+- [LLM-as-a-Verifier](llm-as-a-verifier.md): how verifier prompts, concurrent
+  execution, score-token parsing, and weighted aggregation fit together.

msgflux-0.4.1/docs/anatomy/llm-as-a-verifier.md

@@ -0,0 +1,317 @@
+# LLM-as-a-Verifier
+
+`src/msgflux/generation/verifiers/llm_as_a_verifier.py` is the first verifier
+module in msgFlux.
+
+At a high level, it turns a verification task into a reusable runtime with four
+main jobs:
+
+- build a verifier prompt from task, criteria, candidates, and context
+- request discrete score tokens from the model
+- estimate a stable score from `logprobs` instead of trusting only the chosen
+  token
+- aggregate repeated attempts and multiple criteria into a final verdict
+
+It is not tied to `Agent`. It is a callable verification layer that can be used
+from plain generation code, tool workflows, candidate reranking, or future
+optimizers.
+
+## Mental Model
+
+The verifier pipeline looks like this:
+
+```text
+task + candidates + criteria
+  -> prompt builder
+  -> request batch
+  -> concurrent model calls
+  -> per-attempt evidence extraction
+     -> text token
+     -> logprob distribution
+  -> criterion aggregation
+  -> final verifier result
+```
+
+The important part is that the verifier does not ask the model for an open-ended
+score. It asks for a small discrete token from a known scale, then uses the
+token distribution to estimate the final normalized score.
+
+## Why The Output Is A Score Token
+
+The verifier uses a discrete score scale such as `A..T`.
+
+That choice is deliberate:
+
+- it keeps the scoring space bounded
+- it makes the provider response easier to parse
+- it makes `logprobs` directly useful
+- it lets the runtime compute an expected score instead of a brittle hard label
+
+Letters are preferred over digits for the default scale because they behave
+more like single symbolic labels as granularity grows.
+
+That matters for `logprobs` extraction:
+
+- `A..T` is a cleaner discrete label set than `1..20`
+- multi-digit numeric scales are more likely to split across tokens
+- numeric labels are more likely to attach to punctuation or formatting noise
+- letter labels are easier to rank, normalize, and recover from imperfect tag
+  formatting
+
+So the default runtime chooses a letter-based scale not for presentation, but
+for tokenization stability and better score-distribution extraction.
+
+So the model is not asked to emit a floating-point score. It is asked to emit a
+single token inside a known tag:
+
+```text
+<score>A</score>
+```
+
+or, in pairwise mode:
+
+```text
+<score_A>A</score_A>
+<score_B>H</score_B>
+```
+
+`ScoreScale.letter(granularity=20)` is the public way to ask for 20 score
+levels explicitly. That is also the default runtime scale.
+
+## Prompt Construction
+
+`default_prompt_builder(...)` assembles the verifier prompt from
+`VerificationPromptInput`.
+
+That input contains:
+
+- `task`
+- `criterion`
+- `candidates`
+- `score_scale`
+- optional `context`
+- optional verifier-level instructions
+
+The default builder always explains:
+
+- the criterion being judged
+- the valid score-token range
+- the exact tag format expected at the end
+
+This is why the verifier can support custom prompting without changing the rest
+of the execution pipeline. The builder is only responsible for prompt text; the
+runtime remains responsible for scoring and aggregation.
+
+## Trajectory Formatting Helpers
+
+Benchmark-style presets work best when the candidate contains the full execution
+evidence, not only the final answer.
+
+The public API therefore exposes:
+
+- `format_terminal_trajectory(...)`
+- `format_swe_bench_trajectory(...)`
+
+These helpers deliberately format only candidate evidence:
+
+- summaries
+- command/output steps
+- metadata such as expected output
+- final patch text for SWE-style runs
+- final answer or final narration
+
+The task or issue description stays outside the helper and should still be
+passed through the verifier's `task` argument. That keeps the verifier API
+clean: `task` carries the evaluation target, while the formatted candidate
+string carries the evidence being judged.
+
+## Request Batching And Concurrency
+
+The verifier builds a `VerificationRequest` for each independent
+criterion/repetition pair.
+
+That means the number of model calls in a single verification is:
+
+```text
+len(criteria) * n_verifications
+```
+
+Those requests are independent, so the verifier fans them out concurrently:
+
+- `__call__` uses `F.map_gather(...)`
+- `acall` uses `F.amap_gather(...)`
+
+This matters because a verifier with multiple criteria and repeated attempts can
+otherwise accumulate latency linearly.
+
+The runtime still keeps the final result deterministic:
+
+- requests are built in criterion order, then repetition order
+- gather responses preserve input order
+- attempts are regrouped per criterion after execution
+
+So the work is concurrent, but the aggregation order remains stable.
+
+## Why `select_best(...)` Is Separate
+
+`__call__` and `acall` support only one or two candidates.
+
+That boundary is intentional:
+
+- one candidate -> verification
+- two candidates -> pairwise comparison
+- more than two candidates -> tournament selection
+
+`select_best(...)` and `aselect_best(...)` implement the tournament layer on top
+of the base verifier. They run pairwise matches, accumulate wins, and use
+average score as a tiebreaker.
+
+The tournament layer can also fan out its pairwise matches concurrently. That
+means a multi-candidate selection now has two concurrency layers:
+
+- concurrent attempts inside each verifier call
+- concurrent pairwise matches across the tournament
+
+This keeps the core verifier simple instead of making `__call__` return
+different shapes for different candidate counts.
+
+## Scoring From Logprobs
+
+The central scoring path is:
+
+```text
+model response
+  -> locate score token
+  -> collect chosen token + top_logprobs alternatives
+  -> convert logprobs to probabilities
+  -> compute expected value on the discrete score scale
+  -> normalize to [0, 1]
+```
+
+This is why `LLMAsVerifier` always requests:
+
+- `logprobs=True`
+- `top_logprobs=<scale size or override>`
+
+If the provider returns a good token distribution, the verifier uses that
+distribution directly.
+
+If not, and `strict_logprobs=False`, it falls back to parsing the score from the
+response text.
+
+## Why The Parser Has Heuristics
+
+Provider tokenization around XML-like tags is not always clean.
+
+The verifier therefore uses a layered extraction strategy:
+
+1. parse the text tag if possible
+2. try to find the matching score entry directly in `logprobs`
+3. fall back to finding the expected score token near the closing tag
+4. if strict mode is disabled, fall back to text-only extraction
+
+This is also why the parser tolerates malformed openings such as:
+
+```text
+<20A</score>
+```
+
+The runtime treats the text and token stream as evidence that must be reconciled,
+not as perfectly formatted output.
+
+## Aggregation Model
+
+There are two aggregation stages.
+
+### Attempt Aggregation
+
+Each criterion can run multiple repetitions.
+
+Those repeated attempts are averaged into a single score per criterion and
+candidate.
+
+### Criterion Aggregation
+
+Each criterion carries a weight.
+
+The final score per candidate is the weighted average of all criterion scores.
+
+That is what makes the verifier reusable across different presets: the
+execution path stays the same while the criteria set changes.
+
+## Verdict Semantics
+
+The final verdict depends on candidate count.
+
+For a single candidate:
+
+- `pass`
+- `fail`
+- `uncertain`
+
+For two candidates:
+
+- winner label
+- `tie`
+
+This is why `VerifierResult` exposes both:
+
+- `verdict`
+- optional `winner`
+
+## Verbose Mode
+
+When `verbose=True`, the verifier stores enough information to debug prompt and
+scoring behavior without rerunning the call.
+
+Per attempt, it keeps:
+
+- final prompt text
+- raw response text
+- per-candidate evidence
+- extracted token probabilities
+
+That data is exposed both in:
+
+- `criteria_results[*].attempts[*]`
+- `result.metadata["raw_outputs"]`
+
+Tournament mode mirrors that same information in each match entry of
+`TournamentResult.metadata["raw_outputs"]`.
+
+## Presets
+
+The preset constructors are thin wrappers around the same runtime:
+
+- `trajectory_analysis`
+- `answer_reranking`
+- `grounded_answer_verification`
+- `patch_selection`
+- `terminal_bench`
+- `swe_bench_verified`
+- `tool_trace_verification`
+- `synthetic_data_filtering`
+
+Each preset only supplies:
+
+- a default criteria set
+- optional default instructions or ground-truth notes
+
+The runtime path stays identical.
+
+This is important for maintenance because it means new verifier presets should
+usually be data/config additions, not new execution engines.
+
+## Main Design Choices
+
+The current design makes a few explicit tradeoffs:
+
+- verifier core is independent from `Agent`
+- prompt building is pluggable, scoring is centralized
+- discrete score tokens are preferred over free-form numeric scores
+- `logprobs` are first-class evidence
+- tournament selection is built on top of pairwise verification
+- verbose debugging is part of the runtime, not an external wrapper
+
+These choices make the verifier easier to reuse for reranking, trajectory
+selection, patch selection, and future optimizer integrations.

{msgflux-0.2.0 → msgflux-0.4.1}/docs/dependency-management.md

@@ -10,16 +10,17 @@ hide:
 
 ### Chat Completion
 
-| Provider    | Dependency        | Auth Env             |
-|-------------|-------------------|----------------------|
-| Cerebras    | `msgflux[openai]` | `CEREBRAS_API_KEY`   |
-| Groq        | `msgflux[openai]` | `GROQ_API_KEY`       |
-| Ollama      | `msgflux[openai]` |                      |
-| OpenAI      | `msgflux[openai]` | `OPENAI_API_KEY`     |
-| OpenRouter  | `msgflux[openai]` | `OPENROUTER_API_KEY` |
-| SambaNova   | `msgflux[openai]` | `SAMBANOVA_API_KEY`  |
-| Together    | `msgflux[openai]` | `TOGETHER_API_KEY`   |
-| vLLM        | `msgflux[openai]` |                      |
+| Provider    | Dependency        | Auth Env               |
+|-------------|-------------------|------------------------|
+| Brave       | `msgflux[openai]` | `BRAVE_SEARCH_API_KEY` |
+| Cerebras    | `msgflux[openai]` | `CEREBRAS_API_KEY`     |
+| Groq        | `msgflux[openai]` | `GROQ_API_KEY`         |
+| Ollama      | `msgflux[openai]` |                        |
+| OpenAI      | `msgflux[openai]` | `OPENAI_API_KEY`       |
+| OpenRouter  | `msgflux[openai]` | `OPENROUTER_API_KEY`   |
+| SambaNova   | `msgflux[openai]` | `SAMBANOVA_API_KEY`    |
+| Together    | `msgflux[openai]` | `TOGETHER_API_KEY`     |
+| vLLM        | `msgflux[openai]` |                        |
 
 ### Image Embedder
 
@@ -144,6 +145,13 @@ hide:
 
 ### Web Search
 
-| Provider    | Dependency    |
-|-------------|---------------|
-| Wikipedia   | `wikipedia`   |
+| Provider    | Installation                     | Auth Env                         |
+|-------------|----------------------------------|-----------------------------------|
+| Trafilatura | `httpx`, `trafilatura`           |                                   |
+| SerpApi     | `httpx`                          | `SERPAPI_KEY`                     |
+| Linkup      | `linkup-sdk`                     | `LINKUP_API_KEY`                  |
+| Tavily      | `tavily-python`                  | `TAVILY_API_KEY`                  |
+| Exa         | `exa-py`                         | `EXA_API_KEY`                     |
+| Brave       | `brave-search-python-client`     | `BRAVE_SEARCH_API_KEY`            |
+| arXiv       | `arxiv`                          |                                   |
+| Wikipedia   | `wikipedia`                      |                                   |

{msgflux-0.2.0 → msgflux-0.4.1}/docs/index.md

@@ -34,6 +34,8 @@ hide:
 
 msgFlux is an open-source framework for building dynamic AI systems with **composable modules**. It treats prompts, signatures, tools, and message flow as explicit program structure instead of ad-hoc glue. Architecture, data flow, and prompting remain separate layers, so systems can evolve by changing contracts, modules, or routes without forcing everything to change together.
 
+For web search, msgFlux includes both retriever-backed and model-backed options. The built-in `WebSearch` tool can be pointed at a retriever backend like `retriever/wikipedia` or a model backend like `model/openai/gpt-4o-search-preview`, returns `data` plus `annotations` when the backend supplies citations, and can read its engine plus JSON params from `MSGFLUX_TOOL_WEB_SEARCH_*` environment variables.
+
 ## **AI Systems *not* ML Systems**
 
 **ML systems** are systems *for* AI - they train, evaluate, and deploy models. **AI systems** are systems *with* AI - software where pretrained models operate as components inside a broader application. In that setting, the model is not the whole product; it is one building block among many. You are not optimizing weights - you are designing behavior, interfaces, and flow around a model. This is the space msgFlux occupies.