thinkless 0.3.0__tar.gz → 0.4.1__tar.gz

{thinkless-0.3.0 → thinkless-0.4.1}/CHANGELOG.md

@@ -6,6 +6,46 @@ All notable changes are recorded here. The format follows
 
 ## Unreleased
 
+## 0.4.1 - 2026-09-25
+
+Project and packaging updates; no change to the library's behavior.
+
+### Changed
+
+- Package metadata: development status Beta, more keywords, and links to
+  Discussions, the decision benchmark leaderboard, release notes and funding.
+- The README links to good first issues and Discussions and shows the docs,
+  leaderboard and OpenSSF Scorecard badges.
+- The submission guide says submitters regenerate the leaderboard page in
+  their pull request; the release guide explains that release tags are
+  permanent.
+
+### Security
+
+- Every GitHub Action in the workflows is pinned to a commit.
+- An OpenSSF Scorecard workflow checks the repository's supply chain weekly.
+
+## 0.4.0 - 2026-09-25
+
+The decision benchmark.
+
+### Added
+
+- The decision benchmark 1.0 (`thinkless bench decisions`): eight public
+  tasks across the four question kinds (banking77, clinc150 with out of
+  scope, MASSIVE in five languages, MultiWOZ 2.2 conversations, jailbreaks,
+  Civil Comments toxicity, HelpSteer2 helpfulness ratings and WNUT 2017
+  entities), with frozen rows and published hashes, calibration rows from a
+  separate split, result files that hold every prediction, `verify` that
+  recomputes every metric, a generated leaderboard and open submissions.
+  Baselines for GLiNER 2.5, Laya, Qwen3-1.7B and Qwen 3.7 Flash.
+
+### Changed
+
+- A list of chat messages now renders as `role: text` lines for providers
+  that read text, instead of one `key: value` line per field.
+- README links to the documentation point at the documentation site.
+
 ## 0.3.0 - 2026-09-25
 
 Shadow mode, framework adapters, limits, a decision server and MCP tools.

{thinkless-0.3.0 → thinkless-0.4.1}/PKG-INFO

@@ -1,18 +1,24 @@
 Metadata-Version: 2.5
 Name: thinkless
-Version: 0.3.0
+Version: 0.4.1
 Summary: The decision plane for AI agents. Rules and small calibrated models make the routine decisions, the LLM is reserved for the steps that need thought, and every step is traced.
 Project-URL: Homepage, https://github.com/inboxpraveen/ThinkLess
 Project-URL: Documentation, https://inboxpraveen.github.io/ThinkLess/
 Project-URL: Repository, https://github.com/inboxpraveen/ThinkLess
 Project-URL: Issues, https://github.com/inboxpraveen/ThinkLess/issues
+Project-URL: Discussions, https://github.com/inboxpraveen/ThinkLess/discussions
 Project-URL: Changelog, https://github.com/inboxpraveen/ThinkLess/blob/main/CHANGELOG.md
+Project-URL: Release notes, https://github.com/inboxpraveen/ThinkLess/releases
+Project-URL: Decision benchmark, https://inboxpraveen.github.io/ThinkLess/decision-benchmark/leaderboard/
+Project-URL: Funding, https://github.com/sponsors/inboxpraveen
 Author: Praveen Kumar
 License-Expression: Apache-2.0
 License-File: LICENSE
-Keywords: agents,classification,decision-models,gliner,jev,laya,llm,observability,routing,system-one,tracing
-Classifier: Development Status :: 3 - Alpha
+Keywords: agents,ai-agents,benchmark,calibration,classification,decision-models,gliner,guardrails,jev,langgraph,laya,llm,llm-routing,mcp,observability,openai-agents,system-one,tracing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pydantic :: 2
 Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
 Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
@@ -109,10 +115,13 @@ Description-Content-Type: text/markdown
 
 <p align="center">
   <a href="https://pypi.org/project/thinkless/"><img alt="PyPI" src="https://img.shields.io/pypi/v/thinkless.svg"></a>
+  <a href="https://pypi.org/project/thinkless/"><img alt="Python 3.10 to 3.13" src="https://img.shields.io/pypi/pyversions/thinkless.svg"></a>
   <a href="https://github.com/inboxpraveen/ThinkLess/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/inboxpraveen/ThinkLess/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://inboxpraveen.github.io/ThinkLess/"><img alt="Docs" src="https://img.shields.io/badge/docs-inboxpraveen.github.io-blue.svg"></a>
+  <a href="https://inboxpraveen.github.io/ThinkLess/decision-benchmark/leaderboard/"><img alt="Decision benchmark" src="https://img.shields.io/badge/decision%20benchmark-leaderboard-brightgreen.svg"></a>
+  <a href="https://scorecard.dev/viewer/?uri=github.com/inboxpraveen/ThinkLess"><img alt="OpenSSF Scorecard" src="https://api.scorecard.dev/projects/github.com/inboxpraveen/ThinkLess/badge"></a>
+  <a href="https://github.com/inboxpraveen/ThinkLess/discussions"><img alt="Discussions" src="https://img.shields.io/github/discussions/inboxpraveen/ThinkLess.svg"></a>
   <a href="https://github.com/inboxpraveen/ThinkLess/blob/main/LICENSE"><img alt="License: Apache 2.0" src="https://img.shields.io/badge/license-Apache%202.0-blue.svg"></a>
-  <img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-blue.svg">
-  <a href="https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md"><img alt="Benchmarks" src="https://img.shields.io/badge/benchmarks-reproducible-brightgreen.svg"></a>
 </p>
 
 ThinkLess is an open-source decision plane for AI agents. The routine
@@ -159,7 +168,7 @@ if intent.is_("refund_duplicate_charge"):       # only true when the answer is c
 Same agent, same 53 labeled support tickets, two decision planes: `llm` sends
 every decision to the LLM, `hybrid` asks rules, GLiNER and Laya first. Hosted
 models ran through OpenRouter, and the dollar figures are what OpenRouter
-billed ([full results](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md)):
+billed ([full results](https://inboxpraveen.github.io/ThinkLess/benchmarks/)):
 
 | Fallback LLM | Success, `llm` | Success, `hybrid` | Billed per 1k tickets, `llm` | Billed per 1k tickets, `hybrid` | Decision time |
 |---|---:|---:|---:|---:|---:|
@@ -185,7 +194,7 @@ billed ([full results](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/
   validated on a separate calibration set first. The baseline is the
   strongest form of the LLM design: all six triage questions in one prompt.
 - The support set is 53 synthetic tickets: evidence of the mechanism and its
-  failure modes, not a leaderboard. The [benchmarks page](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md)
+  failure modes, not a leaderboard. The [benchmarks page](https://inboxpraveen.github.io/ThinkLess/benchmarks/)
   lists the weak areas found and what was done about each.
 
 ## Quick start
@@ -263,11 +272,11 @@ intent = router(engine, INTENT, {"order_status": "tracking", "refund": "refunds"
 builder.add_conditional_edges(START, intent, intent.destinations)
 ```
 
-- [LangGraph](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/langgraph.md):
+- [LangGraph](https://inboxpraveen.github.io/ThinkLess/integrations/langgraph/):
   routers, decision nodes and gated tools.
-- [OpenAI Agents SDK](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/openai-agents.md):
+- [OpenAI Agents SDK](https://inboxpraveen.github.io/ThinkLess/integrations/openai-agents/):
   input guardrails, tool guardrails and routing to a specialist agent.
-- [Any framework](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/any-framework.md):
+- [Any framework](https://inboxpraveen.github.io/ThinkLess/integrations/any-framework/):
   `route` and `gate` for Pydantic AI, a hand-written loop, or anything else.
 
 Before switching anything, measure it. Shadow mode runs ThinkLess next to your
@@ -288,10 +297,10 @@ def classify(message: str) -> str:
 thinkless shadow report shadow/intent.jsonl --volume 2000000
 ```
 
-The [migration guide](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/migration.md)
-walks through it, and the [FAQ](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/faq.md)
+The [migration guide](https://inboxpraveen.github.io/ThinkLess/guides/migration/)
+walks through it, and the [FAQ](https://inboxpraveen.github.io/ThinkLess/faq/)
 covers the common questions. To share one engine across services, run it as
-a [decision server or MCP tools](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/serving.md).
+a [decision server or MCP tools](https://inboxpraveen.github.io/ThinkLess/guides/serving/).
 
 ## Try the demo agent
 
@@ -314,31 +323,31 @@ thinkless bench intents --dataset banking77
 **Typed questions.** `Choice`, `Score`, `YesNo` and `Extract` declare what you
 want to know and which answers are allowed. They follow the System One wire
 format, so the same question works on every provider.
-[Questions](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/questions.md)
+[Questions](https://inboxpraveen.github.io/ThinkLess/concepts/questions/)
 
 **A confidence cascade.** Providers are tried cheapest first. Each gets one
 call with every open question it supports; answers that clear their threshold
 close, the rest move on. Unresolved questions come back `uncertain` with the
 best answer seen, and `decision.is_()` never matches an uncertain answer.
-[The cascade](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/cascade.md)
+[The cascade](https://inboxpraveen.github.io/ThinkLess/concepts/cascade/)
 
 **One meaning of confidence.** Providers disagree about what "confidence"
 means (Laya and TypeSafe compute it differently), so a threshold of 0.8 would
 mean different things per backend. ThinkLess derives a single normalized
 confidence from each provider's probabilities.
-[Confidence](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/confidence.md)
+[Confidence](https://inboxpraveen.github.io/ThinkLess/concepts/confidence/)
 
 **Thresholds from data.** `thinkless calibrate` sweeps thresholds on labeled
 examples and recommends the lowest one that meets your accuracy target, per
 question and per provider (`thresholds={"intent@gliner": 0.9}`).
-[Calibration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/calibration.md)
+[Calibration](https://inboxpraveen.github.io/ThinkLess/guides/calibration/)
 
 **Traces for everything.** Decisions, provider attempts, generations, tool
 calls and rule checks are spans with W3C-compatible ids. Write JSONL audit
 files, export to any OpenTelemetry backend with GenAI semantic conventions, or
 open the self-contained HTML viewer. Content capture can be switched off
 without losing structure, timings or confidences.
-[Tracing](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/tracing.md)
+[Tracing](https://inboxpraveen.github.io/ThinkLess/concepts/tracing/)
 
 ## Providers and backends
 
@@ -358,57 +367,64 @@ without losing structure, timings or confidences.
 | `OpenAICompatibleLLM` | OpenAI, Ollama, vLLM, LM Studio, llama.cpp, Groq |
 | `AnthropicLLM` | Claude, with server-side refusal fallbacks |
 
-Writing your own provider is one class. [Providers](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/providers.md),
-[LLM backends](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/llm-backends.md)
+Writing your own provider is one class. [Providers](https://inboxpraveen.github.io/ThinkLess/guides/providers/),
+[LLM backends](https://inboxpraveen.github.io/ThinkLess/guides/llm-backends/)
 
 ## Documentation
 
-- [Getting started](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/getting-started.md),
-  [installation](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/installation.md)
-- Use it in your agent: [overview](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/index.md),
-  [LangGraph](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/langgraph.md),
-  [OpenAI Agents SDK](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/openai-agents.md),
-  [any framework](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/any-framework.md),
-  [migration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/migration.md),
-  [shadow mode](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/shadow-mode.md),
-  [FAQ](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/faq.md)
-- Concepts: [the four planes](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/planes.md),
-  [questions](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/questions.md),
-  [confidence](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/confidence.md),
-  [the cascade](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/cascade.md),
-  [tracing](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/tracing.md)
-- Guides: [the support agent](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/support-agent.md),
-  [calibration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/calibration.md),
-  [providers](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/providers.md),
-  [LLM backends](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/llm-backends.md),
-  [production](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/production.md),
-  [serving](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/serving.md),
-  [running a pilot](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/pilot.md),
-  [troubleshooting](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/troubleshooting.md)
-- [Benchmarks](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md), [CLI reference](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/reference/cli.md),
-  [Python API](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/reference/api.md), [roadmap](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/roadmap.md)
+- [Getting started](https://inboxpraveen.github.io/ThinkLess/getting-started/),
+  [installation](https://inboxpraveen.github.io/ThinkLess/guides/installation/)
+- Use it in your agent: [overview](https://inboxpraveen.github.io/ThinkLess/integrations/),
+  [LangGraph](https://inboxpraveen.github.io/ThinkLess/integrations/langgraph/),
+  [OpenAI Agents SDK](https://inboxpraveen.github.io/ThinkLess/integrations/openai-agents/),
+  [any framework](https://inboxpraveen.github.io/ThinkLess/integrations/any-framework/),
+  [migration](https://inboxpraveen.github.io/ThinkLess/guides/migration/),
+  [shadow mode](https://inboxpraveen.github.io/ThinkLess/guides/shadow-mode/),
+  [FAQ](https://inboxpraveen.github.io/ThinkLess/faq/)
+- Concepts: [the four planes](https://inboxpraveen.github.io/ThinkLess/concepts/planes/),
+  [questions](https://inboxpraveen.github.io/ThinkLess/concepts/questions/),
+  [confidence](https://inboxpraveen.github.io/ThinkLess/concepts/confidence/),
+  [the cascade](https://inboxpraveen.github.io/ThinkLess/concepts/cascade/),
+  [tracing](https://inboxpraveen.github.io/ThinkLess/concepts/tracing/)
+- Guides: [the support agent](https://inboxpraveen.github.io/ThinkLess/guides/support-agent/),
+  [calibration](https://inboxpraveen.github.io/ThinkLess/guides/calibration/),
+  [providers](https://inboxpraveen.github.io/ThinkLess/guides/providers/),
+  [LLM backends](https://inboxpraveen.github.io/ThinkLess/guides/llm-backends/),
+  [production](https://inboxpraveen.github.io/ThinkLess/guides/production/),
+  [serving](https://inboxpraveen.github.io/ThinkLess/guides/serving/),
+  [running a pilot](https://inboxpraveen.github.io/ThinkLess/guides/pilot/),
+  [troubleshooting](https://inboxpraveen.github.io/ThinkLess/guides/troubleshooting/)
+- [Benchmarks](https://inboxpraveen.github.io/ThinkLess/benchmarks/), [CLI reference](https://inboxpraveen.github.io/ThinkLess/reference/cli/),
+  [Python API](https://inboxpraveen.github.io/ThinkLess/reference/api/), [roadmap](https://inboxpraveen.github.io/ThinkLess/roadmap/)
 
 ## Status
 
 ThinkLess is pre-1.0: the core API (questions, the engine, decisions and
 traces) is meant to stay stable, and providers, adapters and benchmarks will
-grow. Next up are a neutral benchmark for decision models, cross-request
-batching in the server, calibrated LLM confidence from log probabilities, and
-live Jev runs. See the [roadmap](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/roadmap.md).
+grow. The [decision benchmark](https://inboxpraveen.github.io/ThinkLess/decision-benchmark/)
+compares decision models on eight public tasks and is open to submissions.
+Next up are more models on it, cross-request batching in the server, and
+calibrated LLM confidence from log probabilities. See the [roadmap](https://inboxpraveen.github.io/ThinkLess/roadmap/).
 
 ## Contributing
 
 Issues, providers, benchmark runs on other hardware and models, and new demo
-scenarios are all welcome. Start with [CONTRIBUTING.md](https://github.com/inboxpraveen/ThinkLess/blob/main/CONTRIBUTING.md). The
-bar for changes that affect accuracy, latency or cost is evidence: a before
-and after from `thinkless bench` or `thinkless calibrate`.
+scenarios are all welcome. Start with [CONTRIBUTING.md](https://github.com/inboxpraveen/ThinkLess/blob/main/CONTRIBUTING.md),
+pick a [good first issue](https://github.com/inboxpraveen/ThinkLess/labels/good%20first%20issue),
+or ask in [Discussions](https://github.com/inboxpraveen/ThinkLess/discussions).
+The bar for changes that affect accuracy, latency or cost is evidence: a
+before and after from `thinkless bench` or `thinkless calibrate`.
 
 ## Sponsoring
 
 If ThinkLess saves your team time or money, consider
 [sponsoring its development](https://github.com/sponsors/inboxpraveen).
-Sponsorship funds frontier-model benchmark runs, more providers and more
-demos.
+
+Model vendors can sponsor runs of their models on the
+[decision benchmark](https://inboxpraveen.github.io/ThinkLess/decision-benchmark/).
+Sponsored runs are done by the maintainers and published as verified,
+whatever the result: sponsorship buys runs, not rankings. Sponsorship also
+funds more providers and more demos.
 
 ## Citing
 

{thinkless-0.3.0 → thinkless-0.4.1}/README.md

@@ -4,10 +4,13 @@
 
 <p align="center">
   <a href="https://pypi.org/project/thinkless/"><img alt="PyPI" src="https://img.shields.io/pypi/v/thinkless.svg"></a>
+  <a href="https://pypi.org/project/thinkless/"><img alt="Python 3.10 to 3.13" src="https://img.shields.io/pypi/pyversions/thinkless.svg"></a>
   <a href="https://github.com/inboxpraveen/ThinkLess/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/inboxpraveen/ThinkLess/actions/workflows/ci.yml/badge.svg"></a>
+  <a href="https://inboxpraveen.github.io/ThinkLess/"><img alt="Docs" src="https://img.shields.io/badge/docs-inboxpraveen.github.io-blue.svg"></a>
+  <a href="https://inboxpraveen.github.io/ThinkLess/decision-benchmark/leaderboard/"><img alt="Decision benchmark" src="https://img.shields.io/badge/decision%20benchmark-leaderboard-brightgreen.svg"></a>
+  <a href="https://scorecard.dev/viewer/?uri=github.com/inboxpraveen/ThinkLess"><img alt="OpenSSF Scorecard" src="https://api.scorecard.dev/projects/github.com/inboxpraveen/ThinkLess/badge"></a>
+  <a href="https://github.com/inboxpraveen/ThinkLess/discussions"><img alt="Discussions" src="https://img.shields.io/github/discussions/inboxpraveen/ThinkLess.svg"></a>
   <a href="https://github.com/inboxpraveen/ThinkLess/blob/main/LICENSE"><img alt="License: Apache 2.0" src="https://img.shields.io/badge/license-Apache%202.0-blue.svg"></a>
-  <img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-blue.svg">
-  <a href="https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md"><img alt="Benchmarks" src="https://img.shields.io/badge/benchmarks-reproducible-brightgreen.svg"></a>
 </p>
 
 ThinkLess is an open-source decision plane for AI agents. The routine
@@ -54,7 +57,7 @@ if intent.is_("refund_duplicate_charge"):       # only true when the answer is c
 Same agent, same 53 labeled support tickets, two decision planes: `llm` sends
 every decision to the LLM, `hybrid` asks rules, GLiNER and Laya first. Hosted
 models ran through OpenRouter, and the dollar figures are what OpenRouter
-billed ([full results](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md)):
+billed ([full results](https://inboxpraveen.github.io/ThinkLess/benchmarks/)):
 
 | Fallback LLM | Success, `llm` | Success, `hybrid` | Billed per 1k tickets, `llm` | Billed per 1k tickets, `hybrid` | Decision time |
 |---|---:|---:|---:|---:|---:|
@@ -80,7 +83,7 @@ billed ([full results](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/
   validated on a separate calibration set first. The baseline is the
   strongest form of the LLM design: all six triage questions in one prompt.
 - The support set is 53 synthetic tickets: evidence of the mechanism and its
-  failure modes, not a leaderboard. The [benchmarks page](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md)
+  failure modes, not a leaderboard. The [benchmarks page](https://inboxpraveen.github.io/ThinkLess/benchmarks/)
   lists the weak areas found and what was done about each.
 
 ## Quick start
@@ -158,11 +161,11 @@ intent = router(engine, INTENT, {"order_status": "tracking", "refund": "refunds"
 builder.add_conditional_edges(START, intent, intent.destinations)
 ```
 
-- [LangGraph](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/langgraph.md):
+- [LangGraph](https://inboxpraveen.github.io/ThinkLess/integrations/langgraph/):
   routers, decision nodes and gated tools.
-- [OpenAI Agents SDK](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/openai-agents.md):
+- [OpenAI Agents SDK](https://inboxpraveen.github.io/ThinkLess/integrations/openai-agents/):
   input guardrails, tool guardrails and routing to a specialist agent.
-- [Any framework](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/any-framework.md):
+- [Any framework](https://inboxpraveen.github.io/ThinkLess/integrations/any-framework/):
   `route` and `gate` for Pydantic AI, a hand-written loop, or anything else.
 
 Before switching anything, measure it. Shadow mode runs ThinkLess next to your
@@ -183,10 +186,10 @@ def classify(message: str) -> str:
 thinkless shadow report shadow/intent.jsonl --volume 2000000
 ```
 
-The [migration guide](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/migration.md)
-walks through it, and the [FAQ](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/faq.md)
+The [migration guide](https://inboxpraveen.github.io/ThinkLess/guides/migration/)
+walks through it, and the [FAQ](https://inboxpraveen.github.io/ThinkLess/faq/)
 covers the common questions. To share one engine across services, run it as
-a [decision server or MCP tools](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/serving.md).
+a [decision server or MCP tools](https://inboxpraveen.github.io/ThinkLess/guides/serving/).
 
 ## Try the demo agent
 
@@ -209,31 +212,31 @@ thinkless bench intents --dataset banking77
 **Typed questions.** `Choice`, `Score`, `YesNo` and `Extract` declare what you
 want to know and which answers are allowed. They follow the System One wire
 format, so the same question works on every provider.
-[Questions](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/questions.md)
+[Questions](https://inboxpraveen.github.io/ThinkLess/concepts/questions/)
 
 **A confidence cascade.** Providers are tried cheapest first. Each gets one
 call with every open question it supports; answers that clear their threshold
 close, the rest move on. Unresolved questions come back `uncertain` with the
 best answer seen, and `decision.is_()` never matches an uncertain answer.
-[The cascade](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/cascade.md)
+[The cascade](https://inboxpraveen.github.io/ThinkLess/concepts/cascade/)
 
 **One meaning of confidence.** Providers disagree about what "confidence"
 means (Laya and TypeSafe compute it differently), so a threshold of 0.8 would
 mean different things per backend. ThinkLess derives a single normalized
 confidence from each provider's probabilities.
-[Confidence](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/confidence.md)
+[Confidence](https://inboxpraveen.github.io/ThinkLess/concepts/confidence/)
 
 **Thresholds from data.** `thinkless calibrate` sweeps thresholds on labeled
 examples and recommends the lowest one that meets your accuracy target, per
 question and per provider (`thresholds={"intent@gliner": 0.9}`).
-[Calibration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/calibration.md)
+[Calibration](https://inboxpraveen.github.io/ThinkLess/guides/calibration/)
 
 **Traces for everything.** Decisions, provider attempts, generations, tool
 calls and rule checks are spans with W3C-compatible ids. Write JSONL audit
 files, export to any OpenTelemetry backend with GenAI semantic conventions, or
 open the self-contained HTML viewer. Content capture can be switched off
 without losing structure, timings or confidences.
-[Tracing](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/tracing.md)
+[Tracing](https://inboxpraveen.github.io/ThinkLess/concepts/tracing/)
 
 ## Providers and backends
 
@@ -253,57 +256,64 @@ without losing structure, timings or confidences.
 | `OpenAICompatibleLLM` | OpenAI, Ollama, vLLM, LM Studio, llama.cpp, Groq |
 | `AnthropicLLM` | Claude, with server-side refusal fallbacks |
 
-Writing your own provider is one class. [Providers](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/providers.md),
-[LLM backends](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/llm-backends.md)
+Writing your own provider is one class. [Providers](https://inboxpraveen.github.io/ThinkLess/guides/providers/),
+[LLM backends](https://inboxpraveen.github.io/ThinkLess/guides/llm-backends/)
 
 ## Documentation
 
-- [Getting started](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/getting-started.md),
-  [installation](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/installation.md)
-- Use it in your agent: [overview](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/index.md),
-  [LangGraph](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/langgraph.md),
-  [OpenAI Agents SDK](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/openai-agents.md),
-  [any framework](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/integrations/any-framework.md),
-  [migration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/migration.md),
-  [shadow mode](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/shadow-mode.md),
-  [FAQ](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/faq.md)
-- Concepts: [the four planes](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/planes.md),
-  [questions](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/questions.md),
-  [confidence](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/confidence.md),
-  [the cascade](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/cascade.md),
-  [tracing](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/concepts/tracing.md)
-- Guides: [the support agent](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/support-agent.md),
-  [calibration](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/calibration.md),
-  [providers](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/providers.md),
-  [LLM backends](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/llm-backends.md),
-  [production](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/production.md),
-  [serving](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/serving.md),
-  [running a pilot](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/pilot.md),
-  [troubleshooting](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/guides/troubleshooting.md)
-- [Benchmarks](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/benchmarks.md), [CLI reference](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/reference/cli.md),
-  [Python API](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/reference/api.md), [roadmap](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/roadmap.md)
+- [Getting started](https://inboxpraveen.github.io/ThinkLess/getting-started/),
+  [installation](https://inboxpraveen.github.io/ThinkLess/guides/installation/)
+- Use it in your agent: [overview](https://inboxpraveen.github.io/ThinkLess/integrations/),
+  [LangGraph](https://inboxpraveen.github.io/ThinkLess/integrations/langgraph/),
+  [OpenAI Agents SDK](https://inboxpraveen.github.io/ThinkLess/integrations/openai-agents/),
+  [any framework](https://inboxpraveen.github.io/ThinkLess/integrations/any-framework/),
+  [migration](https://inboxpraveen.github.io/ThinkLess/guides/migration/),
+  [shadow mode](https://inboxpraveen.github.io/ThinkLess/guides/shadow-mode/),
+  [FAQ](https://inboxpraveen.github.io/ThinkLess/faq/)
+- Concepts: [the four planes](https://inboxpraveen.github.io/ThinkLess/concepts/planes/),
+  [questions](https://inboxpraveen.github.io/ThinkLess/concepts/questions/),
+  [confidence](https://inboxpraveen.github.io/ThinkLess/concepts/confidence/),
+  [the cascade](https://inboxpraveen.github.io/ThinkLess/concepts/cascade/),
+  [tracing](https://inboxpraveen.github.io/ThinkLess/concepts/tracing/)
+- Guides: [the support agent](https://inboxpraveen.github.io/ThinkLess/guides/support-agent/),
+  [calibration](https://inboxpraveen.github.io/ThinkLess/guides/calibration/),
+  [providers](https://inboxpraveen.github.io/ThinkLess/guides/providers/),
+  [LLM backends](https://inboxpraveen.github.io/ThinkLess/guides/llm-backends/),
+  [production](https://inboxpraveen.github.io/ThinkLess/guides/production/),
+  [serving](https://inboxpraveen.github.io/ThinkLess/guides/serving/),
+  [running a pilot](https://inboxpraveen.github.io/ThinkLess/guides/pilot/),
+  [troubleshooting](https://inboxpraveen.github.io/ThinkLess/guides/troubleshooting/)
+- [Benchmarks](https://inboxpraveen.github.io/ThinkLess/benchmarks/), [CLI reference](https://inboxpraveen.github.io/ThinkLess/reference/cli/),
+  [Python API](https://inboxpraveen.github.io/ThinkLess/reference/api/), [roadmap](https://inboxpraveen.github.io/ThinkLess/roadmap/)
 
 ## Status
 
 ThinkLess is pre-1.0: the core API (questions, the engine, decisions and
 traces) is meant to stay stable, and providers, adapters and benchmarks will
-grow. Next up are a neutral benchmark for decision models, cross-request
-batching in the server, calibrated LLM confidence from log probabilities, and
-live Jev runs. See the [roadmap](https://github.com/inboxpraveen/ThinkLess/blob/main/docs/roadmap.md).
+grow. The [decision benchmark](https://inboxpraveen.github.io/ThinkLess/decision-benchmark/)
+compares decision models on eight public tasks and is open to submissions.
+Next up are more models on it, cross-request batching in the server, and
+calibrated LLM confidence from log probabilities. See the [roadmap](https://inboxpraveen.github.io/ThinkLess/roadmap/).
 
 ## Contributing
 
 Issues, providers, benchmark runs on other hardware and models, and new demo
-scenarios are all welcome. Start with [CONTRIBUTING.md](https://github.com/inboxpraveen/ThinkLess/blob/main/CONTRIBUTING.md). The
-bar for changes that affect accuracy, latency or cost is evidence: a before
-and after from `thinkless bench` or `thinkless calibrate`.
+scenarios are all welcome. Start with [CONTRIBUTING.md](https://github.com/inboxpraveen/ThinkLess/blob/main/CONTRIBUTING.md),
+pick a [good first issue](https://github.com/inboxpraveen/ThinkLess/labels/good%20first%20issue),
+or ask in [Discussions](https://github.com/inboxpraveen/ThinkLess/discussions).
+The bar for changes that affect accuracy, latency or cost is evidence: a
+before and after from `thinkless bench` or `thinkless calibrate`.
 
 ## Sponsoring
 
 If ThinkLess saves your team time or money, consider
 [sponsoring its development](https://github.com/sponsors/inboxpraveen).
-Sponsorship funds frontier-model benchmark runs, more providers and more
-demos.
+
+Model vendors can sponsor runs of their models on the
+[decision benchmark](https://inboxpraveen.github.io/ThinkLess/decision-benchmark/).
+Sponsored runs are done by the maintainers and published as verified,
+whatever the result: sponsorship buys runs, not rankings. Sponsorship also
+funds more providers and more demos.
 
 ## Citing
 

thinkless-0.4.1/benchmarks/decisions/README.md

@@ -0,0 +1,18 @@
+# Decision benchmark
+
+A provider-neutral benchmark for the models that make an agent's routine
+decisions. The method, the leaderboard and the submission guide are in the
+documentation: https://inboxpraveen.github.io/ThinkLess/decision-benchmark/
+
+| Path | |
+|---|---|
+| `tasks/<task>/test.jsonl`, `calibration.jsonl` | the frozen rows of benchmark 1.0 |
+| `tasks/NOTICE.md` | sources, revisions, licenses and changes for every task |
+| `build.py` | how the rows were sampled; maintainers only, and never rerun within a version |
+| `results/verified/` | runs done by the maintainers |
+| `results/submitted/` | runs submitted by others, verified by CI |
+
+```bash
+thinkless bench decisions run --provider gliner --name "GLiNER 2.5 base"
+thinkless bench decisions verify GLiNER-2.5-base.json
+```

{thinkless-0.3.0 → thinkless-0.4.1}/pyproject.toml

@@ -13,20 +13,29 @@ license-files = ["LICENSE"]
 authors = [{ name = "Praveen Kumar" }]
 keywords = [
   "agents",
+  "ai-agents",
   "llm",
+  "llm-routing",
   "decision-models",
-  "system-one",
-  "routing",
+  "guardrails",
+  "langgraph",
+  "openai-agents",
+  "mcp",
+  "benchmark",
+  "calibration",
   "classification",
   "observability",
   "tracing",
+  "system-one",
   "jev",
   "laya",
   "gliner",
 ]
 classifiers = [
-  "Development Status :: 3 - Alpha",
+  "Development Status :: 4 - Beta",
+  "Framework :: Pydantic :: 2",
   "Intended Audience :: Developers",
+  "Intended Audience :: Information Technology",
   "Intended Audience :: Science/Research",
   "License :: OSI Approved :: Apache Software License",
   "Operating System :: OS Independent",
@@ -97,7 +106,11 @@ Homepage = "https://github.com/inboxpraveen/ThinkLess"
 Documentation = "https://inboxpraveen.github.io/ThinkLess/"
 Repository = "https://github.com/inboxpraveen/ThinkLess"
 Issues = "https://github.com/inboxpraveen/ThinkLess/issues"
+Discussions = "https://github.com/inboxpraveen/ThinkLess/discussions"
 Changelog = "https://github.com/inboxpraveen/ThinkLess/blob/main/CHANGELOG.md"
+"Release notes" = "https://github.com/inboxpraveen/ThinkLess/releases"
+"Decision benchmark" = "https://inboxpraveen.github.io/ThinkLess/decision-benchmark/leaderboard/"
+Funding = "https://github.com/sponsors/inboxpraveen"
 
 [project.scripts]
 thinkless = "thinkless.cli.main:app"

thinkless-0.4.1/src/thinkless/_version.py

@@ -0,0 +1 @@
+__version__ = "0.4.1"

thinkless-0.4.1/src/thinkless/bench/decisions/__init__.py

@@ -0,0 +1,35 @@
+"""The decision benchmark: a neutral test of decision models on public data.
+
+Eight tasks cover the four question kinds on public datasets: support and
+assistant intents, out-of-scope detection, five languages, a multi-turn
+conversation, jailbreaks, toxicity, rating an assistant's reply, and entity
+extraction. Every provider gets the same question and the same frozen rows.
+Result files hold raw predictions, and every metric is recomputed from them
+by :func:`verify_result`, so a result can be checked without trusting who
+produced it.
+"""
+
+from .leaderboard import collect, leaderboard_markdown
+from .run import BenchmarkResult, Submission, TaskResult, build_submission_provider, run_benchmark
+from .score import TaskMetrics, score_task
+from .tasks import TaskSpec, benchmark_version, load_rows, load_tasks, question_for
+from .verify import Verification, verify_result
+
+__all__ = [
+    "BenchmarkResult",
+    "Submission",
+    "TaskMetrics",
+    "TaskResult",
+    "TaskSpec",
+    "Verification",
+    "benchmark_version",
+    "build_submission_provider",
+    "collect",
+    "leaderboard_markdown",
+    "load_rows",
+    "load_tasks",
+    "question_for",
+    "run_benchmark",
+    "score_task",
+    "verify_result",
+]