stated-confidence 0.1.0__tar.gz

stated_confidence-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.coverage
+node_modules/
+.DS_Store

stated_confidence-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 John Lockwood
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

stated_confidence-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.5
+Name: stated-confidence
+Version: 0.1.0
+Summary: Make LLM agents state the confidence level and the evidential basis behind each claim.
+Project-URL: Homepage, https://github.com/johnwlockwood/stated-confidence
+Author-email: John Lockwood <johnwlockwood@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,confidence,grounding,hallucination,llm,pydantic,pydantic-ai
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: pydantic>=2.5
+Provides-Extra: pydantic-ai
+Requires-Dist: pydantic-ai-slim>=1.0; extra == 'pydantic-ai'
+Description-Content-Type: text/markdown
+
+# stated-confidence
+
+Make LLM agents say how sure they are, and what that certainty rests on.
+
+Ask a model to attach a confidence **level** and an evidential **basis** to
+each claim it produces and it fabricates less. The level alone is common
+practice; the basis is what does the work. Choosing between "documented
+fact", "expert opinion", and "educated guess" forces a provenance check the
+model would otherwise skip, and a claim it cannot back gets labelled as a
+guess or quietly dropped instead of dressed up as a figure.
+
+This package is that schema, the vocabulary behind it, prompt guidance for
+filling it, and the consistency checks that send hedging-without-meaning-it
+back to the model. It was extracted from a production knowledge-graph
+generator where it was observed to sharply reduce invented facts and
+figures. The effect is observed, not yet measured; an evaluation harness is
+planned.
+
+## Install
+
+```bash
+pip install stated-confidence            # pydantic only
+pip install "stated-confidence[pydantic-ai]"
+```
+
+## The shape
+
+```python
+from stated_confidence import Confidence, DataAnalysisBasis
+
+
+class Finding(BaseModel):
+    text: str
+    confidence: Confidence[DataAnalysisBasis]
+```
+
+A `Confidence` has:
+
+| field       | meaning                                                                   |
+| ----------- | ------------------------------------------------------------------------- |
+| `level`     | `HIGH`, `MEDIUM`, `LOW`, or `SPECULATIVE`. Never a number.                |
+| `basis`     | The kind of evidence, from a basis set chosen for the domain.             |
+| `reasoning` | One or two sentences on why that level and basis apply.                   |
+| `evidence`  | What concretely backs the claim: sources, ids, column names, tool results. |
+| `type`      | Optional: `FACTUAL`, `CONSENSUS`, `THEORETICAL`, `SPECULATIVE`.           |
+| `label`     | Optional localized label for display.                                     |
+
+`SPECULATIVE` is not the bottom of the scale. It is the level for an idea
+the model is offering rather than a claim it is making: an extension a
+source suggests, a hypothesis, a what-if. A generator that has somewhere
+honest to put an idea does not dress it up as a fact; take that room away
+and the ideas come back as fabricated "facts" or disappear altogether.
+
+The level scale is fixed. The **basis set** is the extension point. Three
+ship:
+
+- `GeneralKnowledgeBasis` for claims about the world: documented fact,
+  scientific consensus, historical record, industry standard, expert
+  opinion, theoretical framework, fictional universe, speculation, educated
+  guess.
+- `DataAnalysisBasis` for agents reasoning over data they retrieved: query
+  result, computed, schema inference, general knowledge, assumption.
+- `DecisionBasis` for recommendations: policy, measured outcome, precedent,
+  trade-off analysis, expert judgment, assumption.
+
+Define your own by subclassing `BasisEnum` with a `__terms__` table and a
+`__fallback__` value.
+
+## Prompting
+
+```python
+from stated_confidence import confidence_instructions
+
+system_prompt += confidence_instructions(DataAnalysisBasis, localized=True)
+```
+
+## Pydantic AI
+
+```python
+from pydantic_ai import Agent
+from stated_confidence.pydantic_ai import confidence_output_validator
+
+agent = Agent("openai:gpt-5", output_type=Report)
+agent.output_validator(confidence_output_validator())
+```
+
+Every `Confidence` anywhere in the output is checked. A `HIGH` level on a
+basis of `ASSUMPTION`, a `SPECULATIVE` level on a basis of `DOCUMENTED_FACT`,
+or a non-`HIGH` level with empty reasoning comes back to the model as a
+retry listing what to fix. Pass extra `(ctx, output) -> list[str]` callables for rules that
+need run context, such as requiring that evidence names a column the
+agent's query actually returned.
+
+## Lenient parsing, strict prompting
+
+Models drift. `"High"` parses. `0.9` parses (as `HIGH`). An invented basis
+value parses as the set's fallback and is logged at `WARNING` on the
+`stated_confidence` logger so the set can grow from what models actually
+say.
+
+## Display
+
+`describe(value)` returns a label and description for any level, type, or
+basis value; `tone(level)` maps to `positive`, `neutral`, `caution`, or
+`warning` with no CSS framework attached. `taxonomy.json` at the package
+root is the same vocabulary as data, for other languages to consume.

stated_confidence-0.1.0/README.md

@@ -0,0 +1,106 @@
+# stated-confidence
+
+Make LLM agents say how sure they are, and what that certainty rests on.
+
+Ask a model to attach a confidence **level** and an evidential **basis** to
+each claim it produces and it fabricates less. The level alone is common
+practice; the basis is what does the work. Choosing between "documented
+fact", "expert opinion", and "educated guess" forces a provenance check the
+model would otherwise skip, and a claim it cannot back gets labelled as a
+guess or quietly dropped instead of dressed up as a figure.
+
+This package is that schema, the vocabulary behind it, prompt guidance for
+filling it, and the consistency checks that send hedging-without-meaning-it
+back to the model. It was extracted from a production knowledge-graph
+generator where it was observed to sharply reduce invented facts and
+figures. The effect is observed, not yet measured; an evaluation harness is
+planned.
+
+## Install
+
+```bash
+pip install stated-confidence            # pydantic only
+pip install "stated-confidence[pydantic-ai]"
+```
+
+## The shape
+
+```python
+from stated_confidence import Confidence, DataAnalysisBasis
+
+
+class Finding(BaseModel):
+    text: str
+    confidence: Confidence[DataAnalysisBasis]
+```
+
+A `Confidence` has:
+
+| field       | meaning                                                                   |
+| ----------- | ------------------------------------------------------------------------- |
+| `level`     | `HIGH`, `MEDIUM`, `LOW`, or `SPECULATIVE`. Never a number.                |
+| `basis`     | The kind of evidence, from a basis set chosen for the domain.             |
+| `reasoning` | One or two sentences on why that level and basis apply.                   |
+| `evidence`  | What concretely backs the claim: sources, ids, column names, tool results. |
+| `type`      | Optional: `FACTUAL`, `CONSENSUS`, `THEORETICAL`, `SPECULATIVE`.           |
+| `label`     | Optional localized label for display.                                     |
+
+`SPECULATIVE` is not the bottom of the scale. It is the level for an idea
+the model is offering rather than a claim it is making: an extension a
+source suggests, a hypothesis, a what-if. A generator that has somewhere
+honest to put an idea does not dress it up as a fact; take that room away
+and the ideas come back as fabricated "facts" or disappear altogether.
+
+The level scale is fixed. The **basis set** is the extension point. Three
+ship:
+
+- `GeneralKnowledgeBasis` for claims about the world: documented fact,
+  scientific consensus, historical record, industry standard, expert
+  opinion, theoretical framework, fictional universe, speculation, educated
+  guess.
+- `DataAnalysisBasis` for agents reasoning over data they retrieved: query
+  result, computed, schema inference, general knowledge, assumption.
+- `DecisionBasis` for recommendations: policy, measured outcome, precedent,
+  trade-off analysis, expert judgment, assumption.
+
+Define your own by subclassing `BasisEnum` with a `__terms__` table and a
+`__fallback__` value.
+
+## Prompting
+
+```python
+from stated_confidence import confidence_instructions
+
+system_prompt += confidence_instructions(DataAnalysisBasis, localized=True)
+```
+
+## Pydantic AI
+
+```python
+from pydantic_ai import Agent
+from stated_confidence.pydantic_ai import confidence_output_validator
+
+agent = Agent("openai:gpt-5", output_type=Report)
+agent.output_validator(confidence_output_validator())
+```
+
+Every `Confidence` anywhere in the output is checked. A `HIGH` level on a
+basis of `ASSUMPTION`, a `SPECULATIVE` level on a basis of `DOCUMENTED_FACT`,
+or a non-`HIGH` level with empty reasoning comes back to the model as a
+retry listing what to fix. Pass extra `(ctx, output) -> list[str]` callables for rules that
+need run context, such as requiring that evidence names a column the
+agent's query actually returned.
+
+## Lenient parsing, strict prompting
+
+Models drift. `"High"` parses. `0.9` parses (as `HIGH`). An invented basis
+value parses as the set's fallback and is logged at `WARNING` on the
+`stated_confidence` logger so the set can grow from what models actually
+say.
+
+## Display
+
+`describe(value)` returns a label and description for any level, type, or
+basis value; `tone(level)` maps to `positive`, `neutral`, `caution`, or
+`warning` with no CSS framework attached. `taxonomy.json` at the package
+root is the same vocabulary as data, for other languages to consume.

stated_confidence-0.1.0/pyproject.toml

@@ -0,0 +1,60 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "stated-confidence"
+version = "0.1.0"
+description = "Make LLM agents state the confidence level and the evidential basis behind each claim."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "John Lockwood", email = "johnwlockwood@gmail.com" }]
+requires-python = ">=3.12"
+keywords = ["llm", "agents", "pydantic", "pydantic-ai", "hallucination", "confidence", "grounding"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Typing :: Typed",
+]
+dependencies = ["pydantic>=2.5"]
+
+[project.optional-dependencies]
+pydantic-ai = ["pydantic-ai-slim>=1.0"]
+
+[project.urls]
+Homepage = "https://github.com/johnwlockwood/stated-confidence"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/stated_confidence"]
+
+[tool.hatch.build.targets.sdist]
+include = ["src/stated_confidence", "tests", "taxonomy.json", "README.md", "LICENSE"]
+
+[dependency-groups]
+dev = [
+    "pytest>=8.4",
+    "pytest-asyncio>=1.0",
+    "pydantic-ai-slim>=1.0",
+    "ruff>=0.12",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+asyncio_mode = "auto"
+
+[tool.ruff]
+line-length = 79
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "B", "C4", "UP"]
+ignore = ["E501"]
+
+[tool.ruff.format]
+quote-style = "double"

stated_confidence-0.1.0/src/stated_confidence/__init__.py

@@ -0,0 +1,41 @@
+"""stated-confidence: make agents say how sure they are, and on what basis.
+
+An LLM asked to attach a confidence *level* and an evidential *basis* to
+each claim it makes tends to fabricate less: naming the kind of evidence
+forces a provenance check the model would otherwise skip. This package is
+that schema, the vocabulary behind it, prompt guidance for filling it, and
+the consistency checks that send hedging-without-meaning-it back to the
+model.
+"""
+
+from .basis import (
+    BasisEnum,
+    DataAnalysisBasis,
+    DecisionBasis,
+    GeneralKnowledgeBasis,
+)
+from .checks import check_all, check_consistency, iter_confidences
+from .guidance import confidence_instructions
+from .levels import ConfidenceLevel, ConfidenceType
+from .model import Confidence
+from .presentation import SCALE_EXPLANATION, Tone, describe, tone
+from .terms import Term
+
+__all__ = [
+    "SCALE_EXPLANATION",
+    "BasisEnum",
+    "Confidence",
+    "ConfidenceLevel",
+    "ConfidenceType",
+    "DataAnalysisBasis",
+    "DecisionBasis",
+    "GeneralKnowledgeBasis",
+    "Term",
+    "Tone",
+    "check_all",
+    "check_consistency",
+    "confidence_instructions",
+    "describe",
+    "iter_confidences",
+    "tone",
+]

stated_confidence-0.1.0/src/stated_confidence/basis.py

@@ -0,0 +1,260 @@
+"""Basis vocabularies: *what kind of evidence* stands behind a claim.
+
+The level scale is fixed; the basis set is the extension point. A domain
+declares its own by subclassing ``BasisEnum`` and filling ``__terms__``
+and ``__fallback__``. Three sets ship: general knowledge, data analysis,
+and decisions.
+"""
+
+import logging
+from enum import StrEnum
+from typing import ClassVar
+
+from .levels import normalize_token
+from .terms import Term
+
+logger = logging.getLogger("stated_confidence")
+
+
+class BasisEnum(StrEnum):
+    """Base class for basis vocabularies.
+
+    ``__terms__`` maps each value to its ``Term``; ``__fallback__`` is the
+    value substituted for anything unrecognized. Models occasionally invent
+    plausible values (one emitted ``INDUSTRY_STANDARD`` before it existed),
+    and one bad string must not fail a whole response. The substitution is
+    logged at WARNING on the ``stated_confidence`` logger so a basis set
+    can be grown deliberately from what models actually say.
+    """
+
+    __terms__: ClassVar[dict[str, Term]] = {}
+    __fallback__: ClassVar[str] = ""
+    __registry__: ClassVar[list[type["BasisEnum"]]] = []
+
+    def __init_subclass__(cls, **kwargs: object) -> None:
+        super().__init_subclass__(**kwargs)
+        BasisEnum.__registry__.append(cls)
+
+    @classmethod
+    def registered(cls) -> "list[type[BasisEnum]]":
+        """Every basis set defined so far, shipped and user-defined."""
+        return list(cls.__registry__)
+
+    @classmethod
+    def _missing_(cls, value: object) -> "BasisEnum | None":
+        if not isinstance(value, str):
+            return None
+        token = normalize_token(value)
+        for member in cls:
+            if member.value == token:
+                return member
+        if not cls.__fallback__:
+            return None
+        logger.warning(
+            "Unrecognized %s %r; using %s",
+            cls.__name__,
+            value,
+            cls.__fallback__,
+        )
+        return cls(cls.__fallback__)
+
+    @classmethod
+    def fallback(cls) -> "BasisEnum":
+        return cls(cls.__fallback__)
+
+    @property
+    def term(self) -> Term:
+        return type(self).__terms__[self.value]
+
+    @property
+    def strength(self) -> str | None:
+        return self.term.strength
+
+    @classmethod
+    def weakest(cls) -> list["BasisEnum"]:
+        return [member for member in cls if member.strength == "weak"]
+
+
+class GeneralKnowledgeBasis(BasisEnum):
+    """For claims about the world: facts, history, science, fiction."""
+
+    DOCUMENTED_FACT = "DOCUMENTED_FACT"
+    SCIENTIFIC_CONSENSUS = "SCIENTIFIC_CONSENSUS"
+    HISTORICAL_RECORD = "HISTORICAL_RECORD"
+    INDUSTRY_STANDARD = "INDUSTRY_STANDARD"
+    EXPERT_OPINION = "EXPERT_OPINION"
+    THEORETICAL_FRAMEWORK = "THEORETICAL_FRAMEWORK"
+    FICTIONAL_UNIVERSE = "FICTIONAL_UNIVERSE"
+    SPECULATION = "SPECULATION"
+    EDUCATED_GUESS = "EDUCATED_GUESS"
+
+    __fallback__ = "EXPERT_OPINION"
+    __terms__ = {
+        "DOCUMENTED_FACT": Term(
+            label="Documented fact",
+            description="Stated directly in recorded, citable sources.",
+            guidance="recorded evidence you could cite",
+            strength="strong",
+        ),
+        "SCIENTIFIC_CONSENSUS": Term(
+            label="Scientific consensus",
+            description="Supported by peer-reviewed scientific agreement.",
+            guidance="peer-reviewed agreement",
+            strength="strong",
+        ),
+        "HISTORICAL_RECORD": Term(
+            label="Historical record",
+            description="Part of the documented historical record.",
+            guidance="documented history",
+            strength="strong",
+        ),
+        "INDUSTRY_STANDARD": Term(
+            label="Industry standard",
+            description=(
+                "Established standards or widespread industry practice."
+            ),
+            guidance=(
+                "established standards and widespread engineering practice"
+            ),
+            strength="strong",
+        ),
+        "EXPERT_OPINION": Term(
+            label="Expert opinion",
+            description="Based on the judgment of subject-matter experts.",
+            guidance="the judgment of subject-matter experts",
+            strength="moderate",
+        ),
+        "THEORETICAL_FRAMEWORK": Term(
+            label="Theoretical framework",
+            description="Follows from an established theory or model.",
+            guidance="an established theory or model",
+            strength="moderate",
+        ),
+        "FICTIONAL_UNIVERSE": Term(
+            label="Fictional universe",
+            description="True within a fictional or creative work.",
+            guidance="creative content that is true within its fiction",
+            strength="moderate",
+        ),
+        "SPECULATION": Term(
+            label="Speculation",
+            description="An unverified theory or possibility.",
+            guidance="unverified theories",
+            strength="weak",
+        ),
+        "EDUCATED_GUESS": Term(
+            label="Educated guess",
+            description="A plausible inference made without direct evidence.",
+            guidance="plausible inference with no direct evidence",
+            strength="weak",
+        ),
+    }
+
+
+class DataAnalysisBasis(BasisEnum):
+    """For claims an agent makes about data it retrieved or computed."""
+
+    QUERY_RESULT = "QUERY_RESULT"
+    COMPUTED = "COMPUTED"
+    SCHEMA_INFERENCE = "SCHEMA_INFERENCE"
+    GENERAL_KNOWLEDGE = "GENERAL_KNOWLEDGE"
+    ASSUMPTION = "ASSUMPTION"
+
+    __fallback__ = "ASSUMPTION"
+    __terms__ = {
+        "QUERY_RESULT": Term(
+            label="Query result",
+            description="Read directly from the rows the query returned.",
+            guidance="a value read directly from returned rows",
+            strength="strong",
+        ),
+        "COMPUTED": Term(
+            label="Computed",
+            description=(
+                "Arithmetic over the returned rows, such as a ratio, "
+                "difference, or trend."
+            ),
+            guidance=(
+                "arithmetic you did over returned rows "
+                "(a ratio, difference, or trend)"
+            ),
+            strength="strong",
+        ),
+        "SCHEMA_INFERENCE": Term(
+            label="Schema inference",
+            description=(
+                "Inferred from table or column names and the schema, "
+                "not verified against data."
+            ),
+            guidance=(
+                "something inferred from table or column names "
+                "rather than checked against data"
+            ),
+            strength="moderate",
+        ),
+        "GENERAL_KNOWLEDGE": Term(
+            label="General knowledge",
+            description=(
+                "Background knowledge about the domain, not from this data."
+            ),
+            guidance="background knowledge about the domain, not this data",
+            strength="moderate",
+        ),
+        "ASSUMPTION": Term(
+            label="Assumption",
+            description="A premise that was not checked against the data.",
+            guidance="a premise you did not check against the data",
+            strength="weak",
+        ),
+    }
+
+
+class DecisionBasis(BasisEnum):
+    """For recommendations and decisions."""
+
+    POLICY = "POLICY"
+    MEASURED_OUTCOME = "MEASURED_OUTCOME"
+    PRECEDENT = "PRECEDENT"
+    TRADEOFF_ANALYSIS = "TRADEOFF_ANALYSIS"
+    EXPERT_JUDGMENT = "EXPERT_JUDGMENT"
+    ASSUMPTION = "ASSUMPTION"
+
+    __fallback__ = "ASSUMPTION"
+    __terms__ = {
+        "POLICY": Term(
+            label="Policy",
+            description="Required by a stated policy, rule, or constraint.",
+            guidance="a stated policy, rule, or hard constraint",
+            strength="strong",
+        ),
+        "MEASURED_OUTCOME": Term(
+            label="Measured outcome",
+            description="Observed results from measurement or experiment.",
+            guidance="results actually measured or observed",
+            strength="strong",
+        ),
+        "PRECEDENT": Term(
+            label="Precedent",
+            description="What was done in comparable past cases.",
+            guidance="what comparable past cases did",
+            strength="moderate",
+        ),
+        "TRADEOFF_ANALYSIS": Term(
+            label="Trade-off analysis",
+            description="Weighing the options against stated criteria.",
+            guidance="weighing the options against stated criteria",
+            strength="moderate",
+        ),
+        "EXPERT_JUDGMENT": Term(
+            label="Expert judgment",
+            description="Based on the judgment of subject-matter experts.",
+            guidance="the judgment of subject-matter experts",
+            strength="moderate",
+        ),
+        "ASSUMPTION": Term(
+            label="Assumption",
+            description="A premise that was taken as given, not verified.",
+            guidance="a premise taken as given without verification",
+            strength="weak",
+        ),
+    }