boothpy 0.4.2__tar.gz → 0.4.4__tar.gz

{boothpy-0.4.2 → boothpy-0.4.4}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: boothpy
-Version: 0.4.2
-Summary: A lightweight checkpoint layer for LLM outputs — Path B (self-reported confidence with reconsideration retries, sync and async) plus Path A's evidence-agreement gate.
+Version: 0.4.4
+Summary: A lightweight checkpoint library for LLM outputs.
 Author: Vedant Brahmbhatt
 License: MIT License
         
@@ -57,7 +57,7 @@ The name comes from the idea of a **ticket booth, toll booth, or parking/payment
 
 ## Current Status
 
-**v0.4.2**
+**v0.4.4**
 
 BOOTH currently provides:
 
@@ -69,7 +69,7 @@ BOOTH currently provides:
 * synchronous and asynchronous LLM checkpoint functions
 * evidence-agreement checking against evidence supplied by the caller
 * configurable confidence and evidence thresholds
-* structured result objects, including `result.method` to identify which mechanism produced a result
+* structured result objects, including `result.method` to identify which mechanism produced a result, and `result.parsed` to expose the model's raw JSON response
 * attempt history
 * explicit `VERIFIED`, `REPAIRED`, `AMBIGUOUS`, `UNCERTAIN`, and `BLOCKED` statuses
 
@@ -125,6 +125,8 @@ If BOOTH cannot obtain an acceptable result, it returns `UNCERTAIN`.
 
 BOOTH also provides `check_with_evidence()` for applications that already have evidence from their own RAG, search, database, or tool pipeline.
 
+**On robustness to malformed model output:** every field BOOTH extracts from the model's JSON response is validated against its expected type before being trusted, not just parsed and used as-is. A model returning the JSON string `"false"` instead of the boolean `false` for `ambiguous`, or the boolean `true` instead of a number for `confidence`, is rejected and treated as an unparseable attempt rather than silently coerced into the wrong value — the same discipline BOOTH has always applied to an out-of-range confidence score, now applied consistently across every coerced field. See the [Changelog](CHANGELOG.md)'s `v0.4.4` entry for the specific cases this covers.
+
 ---
 
 ## Features
@@ -253,7 +255,7 @@ result = booth.check(
 )
 ```
 
-`validator` receives the attempt's answer and returns either:
+`validator` receives the attempt's answer and returns one of:
 
 * `True` / `False` — a plain pass/fail. `False` produces a generic failure message.
 * `(bool, str)` — pass/fail plus a specific reason, shown to the model verbatim on the retry prompt:
@@ -267,14 +269,19 @@ def validate_amount(answer: str):
 result = booth.check(call_llm, prompt, validator=validate_amount)
 ```
 
+* `(bool, None)`, or the equivalent as a **list** rather than a tuple (`[bool, str]` / `[bool, None]`) — accepted with the identical contract as the tuple form above (0.4.4+). `return True, None` is a natural way to write "passed, no message needed," and `[passed, message]` is an easy habit to fall into; both previously triggered an "invalid return type" rejection even though the intent was clear. A `False` with no message gets a generic fallback message, same as bare `False`.
+* `numpy.bool_` (and other duck-typed booleans) are accepted anywhere a plain `bool` is (0.4.4+), recognized by type identity rather than by importing `numpy` — BOOTH stays zero-dependency.
+
 **Ordering:** an attempt is only run through `validator` if it parsed successfully **and** was not flagged ambiguous — a question that's ambiguous as asked isn't something a validator should be judging, and there is nothing to validate if the response never parsed. A validation failure is checked *before* the confidence gate: an answer that fails your validator does not get a chance to pass purely on high self-reported confidence.
 
-An exception raised inside `validator`, or a return value that isn't `bool` or `(bool, str)`, is treated as a failed validation — it never propagates out of `check()`/`acheck()`.
+An exception raised inside `validator`, or a return value that isn't one of the accepted shapes above, is treated as a failed validation — it never propagates out of `check()`/`acheck()`.
 
 `validator=None` (the default) is a true no-op: every code path this parameter introduces is unreachable if you never pass it, so existing calls are unaffected.
 
 `validator` must be **synchronous**, for both `check()` and `acheck()`. If your validation logic needs to await something (an API call, a DB lookup), resolve it yourself first and pass a plain sync closure in.
 
+The type alias for a validator function, `ValidatorFn`, is importable from the top-level package (`from booth import ValidatorFn`) for type-hinting your own validator functions.
+
 ---
 
 ### `result.method`
@@ -293,7 +300,7 @@ result.method
 #                    attempt that did parse and did pass validation
 ```
 
-This is most useful for `UNCERTAIN` results, where it distinguishes three genuinely different problems that call for different fixes:
+This is most useful for `UNCERTAIN` results, where it distinguishes genuinely different problems that call for different fixes:
 
 ```python
 if result.status == booth.UNCERTAIN:
@@ -309,6 +316,40 @@ if result.status == booth.UNCERTAIN:
 
 ---
 
+### `result.parsed`
+
+Exposes the model's **raw JSON response object**, exactly as returned — before any of BOOTH's own coercion (`str(answer)`, `float(confidence)`, forcing `interpretations` into a list of strings, and so on).
+
+```python
+result = booth.check(call_llm, "What is the order ID for this request?")
+
+result.answer         # BOOTH's normalized answer, e.g. "ORD-4471"
+result.parsed          # the raw dict the model returned, unmodified
+```
+
+This is a **transparency layer, not a second validation layer** — BOOTH already decided `answer`/`confidence`/`ambiguous` from this same object; `parsed` exists so you can see exactly what the model said, including fields BOOTH itself doesn't use:
+
+```python
+def call_llm(prompt: str) -> str:
+    # imagine the model, prompted to also include a source field,
+    # returns:
+    # {"answer": "30 days", "confidence": "0.95", "ambiguous": false,
+    #  "source_document": "policy.pdf"}
+    ...
+
+result = booth.check(call_llm, prompt)
+
+result.confidence             # 0.95 (float — BOOTH coerced it)
+result.parsed["confidence"]   # "0.95" (str — the raw value, untouched)
+result.parsed["source_document"]  # "policy.pdf" — a field BOOTH never asked for or uses
+```
+
+That type divergence between `result.confidence` and `result.parsed["confidence"]` is deliberate, not a bug — `parsed` is meant to show you precisely what came back, even where BOOTH normalized it for its own use.
+
+`parsed` reflects the attempt that determined the result: the winning attempt for `VERIFIED`/`REPAIRED`/`AMBIGUOUS`, the last *successfully parsed* attempt for `UNCERTAIN`, or `None` if no attempt ever parsed. `check_with_evidence()` results always have `parsed=None` — there is no LLM JSON parse involved on that path at all.
+
+---
+
 ### Synchronous and asynchronous APIs
 
 BOOTH provides both:
@@ -360,7 +401,7 @@ async def main():
 asyncio.run(main())
 ```
 
-Both APIs use the same decision logic, including `validator`. The difference is how the supplied LLM function is called.
+Both APIs use the same decision logic, including `validator` and `parsed`. The difference is how the supplied LLM function is called.
 
 ---
 
@@ -416,7 +457,7 @@ A score of `0.87` passes. A score of `0.62` does not.
 
 Boolean comparison results are treated as strict pass/fail values. `evidence_threshold` is not applied to boolean results.
 
-`check_with_evidence()` has no `validator` concept — it is a standalone, single-purpose comparison gate, untouched by the `validator` addition in this release.
+`check_with_evidence()` has no `validator` concept, and its results always have `parsed=None` — it is a standalone, single-purpose comparison gate, untouched by either the `validator` or `parsed` additions.
 
 ---
 
@@ -484,7 +525,7 @@ Optional callback invoked after each attempt.
 
 #### `validator` (keyword-only, 0.4.2+)
 
-Optional `Callable[[str], bool | tuple[bool, str]]`. Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See [Custom validation with `validator`](#custom-validation-with-validator) above for the full contract. Must be synchronous. Default `None` — a true no-op.
+Optional `ValidatorFn` — `Callable[[str], bool | tuple[bool, str]]`. Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See [Custom validation with `validator`](#custom-validation-with-validator) above for the full accepted-shape contract (widened in 0.4.4 to also accept `(bool, None)`, list forms, and `numpy.bool_`). Must be synchronous. Default `None` — a true no-op.
 
 ---
 
@@ -530,6 +571,7 @@ Checks an answer against caller-supplied evidence. It:
 * performs no retries
 * does not modify a previous `BoothResult`
 * has no `validator` parameter — it is a standalone comparison gate
+* always returns `parsed=None` — there is no LLM JSON response involved
 * uses the caller's `compare_fn`
 
 #### `answer`
@@ -568,6 +610,7 @@ result.ambiguous
 result.interpretations
 result.all_parse_failed
 result.method
+result.parsed
 ```
 
 ### `answer`
@@ -588,7 +631,7 @@ The comparison score produced by `check_with_evidence()`. `None` for normal `che
 
 ### `attempts`
 
-The full history of LLM attempts made by `check()` or `acheck()`, each including per-attempt `passed_validation` / `validation_error` (always `True` / `None` if no `validator` was supplied). Evidence checks do not make attempts, so their attempt list is empty.
+The full history of LLM attempts made by `check()` or `acheck()`, each including per-attempt `passed_validation` / `validation_error` (always `True` / `None` if no `validator` was supplied) and `parsed` (the raw JSON object for that specific attempt, `None` if it failed to parse). Evidence checks do not make attempts, so their attempt list is empty.
 
 ### `n_attempts`
 
@@ -614,6 +657,10 @@ The interpretations reported when the model marks a question as ambiguous.
 
 Which mechanism produced the result — `"ambiguity"`, `"evidence"`, `"parse_failure"`, `"validation"`, or `"confidence"`. See [`result.method`](#resultmethod) above.
 
+### `parsed` (0.4.3+)
+
+The model's raw, uncoerced JSON response object. See [`result.parsed`](#resultparsed) above.
+
 ---
 
 ## Result Statuses
@@ -666,6 +713,7 @@ BOOTH currently does **not**:
 * provide calibrated confidence probabilities
 * guarantee that retrieved evidence is correct, complete, relevant, or current
 * guarantee that a custom `validator` is itself correct — a validator can pass a wrong answer or reject a correct one, same as any other application-supplied rule
+* validate or enforce a schema on `result.parsed` — it is exposed as-is, entirely unvalidated
 * replace application-specific validation or safety systems (though `validator` gives you a documented hook to plug your own logic into BOOTH's retry loop rather than reimplementing that loop yourself)
 
 BOOTH is a **checkpoint library**, not an LLM framework, search engine, RAG framework, or autonomous verification system.
@@ -686,6 +734,10 @@ Ambiguity detection depends on the model recognizing the ambiguity. BOOTH can de
 
 `validator` is exactly as reliable as the logic you give it. BOOTH enforces that a validator's decision is respected consistently in the retry loop — it does not, and cannot, check whether the validator's own logic is actually correct for your use case.
 
+### `result.parsed` is unvalidated
+
+`result.parsed` is the model's raw JSON object, exposed as-is. BOOTH does not validate its shape, enforce a schema on it, or guarantee any field beyond the five it extracts for itself (`answer`, `confidence`, `ambiguous`, `interpretations`, `chosen_interpretation`) is present or well-typed. Reading extra fields from `parsed` is entirely the caller's responsibility, including handling missing keys or unexpected types.
+
 ### Evidence quality
 
 Evidence checking is only as useful as the evidence and comparison function supplied by the application. If the evidence is wrong, incomplete, outdated, or unrelated, BOOTH does not independently detect that. Likewise, a weak `compare_fn` can produce a misleading result. This includes the case where retrieved evidence is baked into the model's own prompt as RAG context — a wrong document can make the model's answer both more *confident* and more evidence-*consistent*, without becoming more correct.
@@ -742,6 +794,7 @@ Future BOOTH development may explore:
 * richer composition of multiple checkpoint results
 * better evaluation and calibration tooling
 * additional integrations with retrieval and tool systems
+* an optional, lightweight structured-schema gate on top of `result.parsed` — deferred deliberately until there's a clear, minimal shape for it, rather than reaching for a general schema/validation dependency
 
 These are future directions, not capabilities currently guaranteed by the library.
 
@@ -775,9 +828,9 @@ Run the test suite:
 pytest
 ```
 
-The test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, and evidence checking. CI runs the full suite on push/PR across Python 3.9–3.12.
+The test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, evidence checking, raw-response exposure via `result.parsed`, and a dedicated regression suite for the `v0.4.4` bugfix batch. CI runs the full suite on push/PR across Python 3.9–3.12.
 
-The validator tests include cases for: `validator=None` full-regression parity, boolean and `(bool, str)` returns, exceptions raised inside a validator, invalid return types, validator never running on ambiguous or unparseable attempts, fail-then-pass producing `REPAIRED`, the distinct validation-failure retry prompt, mixed multi-attempt histories and how `result.method` resolves them, and `check()`/`acheck()` parity under `validator`.
+The `v0.4.4` regression tests specifically verify: a JSON string `"false"` for `ambiguous` is not misread as `True`; a JSON boolean for `confidence` is rejected rather than silently accepted as `1.0`/`0.0`; `ValidatorFn` is importable from the top-level package; a falsy-but-present `chosen_interpretation` (`0`, `""`) is preserved rather than discarded to `None`; `numpy.bool_`-shaped validator returns are accepted; `(bool, None)` and list-shaped validator returns are accepted while genuinely malformed shapes are still rejected; confidence exactly equal to `threshold` passes on both `check()` and `acheck()`; list-wrapped JSON (`[{...}]`) still recovers via the parsing fallback; and the previously-unverified nested-brace edge case's actual behavior is now locked in by a test rather than assumed.
 
 ---
 
@@ -787,9 +840,11 @@ The validator tests include cases for: `validator=None` full-regression parity,
 2. **Make uncertainty explicit.** When an output does not meet the configured acceptance condition, return a structured status instead of silently passing it through.
 3. **Treat ambiguity, validation, and confidence as separate, ordered checks.** A confident, validator-passing answer can still be ambiguous; a confident answer can still fail a caller's own validation rule before confidence is ever consulted.
 4. **Reconsider instead of blindly resampling.** Retries give the model an opportunity to examine its previous response — and the reason it failed (parse failure, validation failure, or low confidence) determines what the model is actually shown.
-5. **Keep evidence retrieval outside BOOTH.** Applications remain free to use their own RAG, search, database, or tool infrastructure.
-6. **Do not pretend agreement is truth.** Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim.
-7. **Stay provider-agnostic.** BOOTH works with different LLM providers because the application supplies the model-calling function.
+5. **Expose, don't reinterpret.** `result.parsed` is a transparency layer over data BOOTH already has, not a second parser or validator — it shows the model's raw response rather than deciding what it should mean.
+6. **Reject invalid data, don't silently coerce it.** An out-of-range confidence, a boolean masquerading as a confidence score, or an unrecognized value for `ambiguous` are all treated as reasons to reject the attempt — never guessed at or silently reinterpreted into something that happens not to crash. Applied consistently across every coerced field as of `v0.4.4`.
+7. **Keep evidence retrieval outside BOOTH.** Applications remain free to use their own RAG, search, database, or tool infrastructure.
+8. **Do not pretend agreement is truth.** Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim.
+9. **Stay provider-agnostic.** BOOTH works with different LLM providers because the application supplies the model-calling function.
 
 ---
 

{boothpy-0.4.2 → boothpy-0.4.4}/README.md

@@ -12,7 +12,7 @@ The name comes from the idea of a **ticket booth, toll booth, or parking/payment
 
 ## Current Status
 
-**v0.4.2**
+**v0.4.4**
 
 BOOTH currently provides:
 
@@ -24,7 +24,7 @@ BOOTH currently provides:
 * synchronous and asynchronous LLM checkpoint functions
 * evidence-agreement checking against evidence supplied by the caller
 * configurable confidence and evidence thresholds
-* structured result objects, including `result.method` to identify which mechanism produced a result
+* structured result objects, including `result.method` to identify which mechanism produced a result, and `result.parsed` to expose the model's raw JSON response
 * attempt history
 * explicit `VERIFIED`, `REPAIRED`, `AMBIGUOUS`, `UNCERTAIN`, and `BLOCKED` statuses
 
@@ -80,6 +80,8 @@ If BOOTH cannot obtain an acceptable result, it returns `UNCERTAIN`.
 
 BOOTH also provides `check_with_evidence()` for applications that already have evidence from their own RAG, search, database, or tool pipeline.
 
+**On robustness to malformed model output:** every field BOOTH extracts from the model's JSON response is validated against its expected type before being trusted, not just parsed and used as-is. A model returning the JSON string `"false"` instead of the boolean `false` for `ambiguous`, or the boolean `true` instead of a number for `confidence`, is rejected and treated as an unparseable attempt rather than silently coerced into the wrong value — the same discipline BOOTH has always applied to an out-of-range confidence score, now applied consistently across every coerced field. See the [Changelog](CHANGELOG.md)'s `v0.4.4` entry for the specific cases this covers.
+
 ---
 
 ## Features
@@ -208,7 +210,7 @@ result = booth.check(
 )
 ```
 
-`validator` receives the attempt's answer and returns either:
+`validator` receives the attempt's answer and returns one of:
 
 * `True` / `False` — a plain pass/fail. `False` produces a generic failure message.
 * `(bool, str)` — pass/fail plus a specific reason, shown to the model verbatim on the retry prompt:
@@ -222,14 +224,19 @@ def validate_amount(answer: str):
 result = booth.check(call_llm, prompt, validator=validate_amount)
 ```
 
+* `(bool, None)`, or the equivalent as a **list** rather than a tuple (`[bool, str]` / `[bool, None]`) — accepted with the identical contract as the tuple form above (0.4.4+). `return True, None` is a natural way to write "passed, no message needed," and `[passed, message]` is an easy habit to fall into; both previously triggered an "invalid return type" rejection even though the intent was clear. A `False` with no message gets a generic fallback message, same as bare `False`.
+* `numpy.bool_` (and other duck-typed booleans) are accepted anywhere a plain `bool` is (0.4.4+), recognized by type identity rather than by importing `numpy` — BOOTH stays zero-dependency.
+
 **Ordering:** an attempt is only run through `validator` if it parsed successfully **and** was not flagged ambiguous — a question that's ambiguous as asked isn't something a validator should be judging, and there is nothing to validate if the response never parsed. A validation failure is checked *before* the confidence gate: an answer that fails your validator does not get a chance to pass purely on high self-reported confidence.
 
-An exception raised inside `validator`, or a return value that isn't `bool` or `(bool, str)`, is treated as a failed validation — it never propagates out of `check()`/`acheck()`.
+An exception raised inside `validator`, or a return value that isn't one of the accepted shapes above, is treated as a failed validation — it never propagates out of `check()`/`acheck()`.
 
 `validator=None` (the default) is a true no-op: every code path this parameter introduces is unreachable if you never pass it, so existing calls are unaffected.
 
 `validator` must be **synchronous**, for both `check()` and `acheck()`. If your validation logic needs to await something (an API call, a DB lookup), resolve it yourself first and pass a plain sync closure in.
 
+The type alias for a validator function, `ValidatorFn`, is importable from the top-level package (`from booth import ValidatorFn`) for type-hinting your own validator functions.
+
 ---
 
 ### `result.method`
@@ -248,7 +255,7 @@ result.method
 #                    attempt that did parse and did pass validation
 ```
 
-This is most useful for `UNCERTAIN` results, where it distinguishes three genuinely different problems that call for different fixes:
+This is most useful for `UNCERTAIN` results, where it distinguishes genuinely different problems that call for different fixes:
 
 ```python
 if result.status == booth.UNCERTAIN:
@@ -264,6 +271,40 @@ if result.status == booth.UNCERTAIN:
 
 ---
 
+### `result.parsed`
+
+Exposes the model's **raw JSON response object**, exactly as returned — before any of BOOTH's own coercion (`str(answer)`, `float(confidence)`, forcing `interpretations` into a list of strings, and so on).
+
+```python
+result = booth.check(call_llm, "What is the order ID for this request?")
+
+result.answer         # BOOTH's normalized answer, e.g. "ORD-4471"
+result.parsed          # the raw dict the model returned, unmodified
+```
+
+This is a **transparency layer, not a second validation layer** — BOOTH already decided `answer`/`confidence`/`ambiguous` from this same object; `parsed` exists so you can see exactly what the model said, including fields BOOTH itself doesn't use:
+
+```python
+def call_llm(prompt: str) -> str:
+    # imagine the model, prompted to also include a source field,
+    # returns:
+    # {"answer": "30 days", "confidence": "0.95", "ambiguous": false,
+    #  "source_document": "policy.pdf"}
+    ...
+
+result = booth.check(call_llm, prompt)
+
+result.confidence             # 0.95 (float — BOOTH coerced it)
+result.parsed["confidence"]   # "0.95" (str — the raw value, untouched)
+result.parsed["source_document"]  # "policy.pdf" — a field BOOTH never asked for or uses
+```
+
+That type divergence between `result.confidence` and `result.parsed["confidence"]` is deliberate, not a bug — `parsed` is meant to show you precisely what came back, even where BOOTH normalized it for its own use.
+
+`parsed` reflects the attempt that determined the result: the winning attempt for `VERIFIED`/`REPAIRED`/`AMBIGUOUS`, the last *successfully parsed* attempt for `UNCERTAIN`, or `None` if no attempt ever parsed. `check_with_evidence()` results always have `parsed=None` — there is no LLM JSON parse involved on that path at all.
+
+---
+
 ### Synchronous and asynchronous APIs
 
 BOOTH provides both:
@@ -315,7 +356,7 @@ async def main():
 asyncio.run(main())
 ```
 
-Both APIs use the same decision logic, including `validator`. The difference is how the supplied LLM function is called.
+Both APIs use the same decision logic, including `validator` and `parsed`. The difference is how the supplied LLM function is called.
 
 ---
 
@@ -371,7 +412,7 @@ A score of `0.87` passes. A score of `0.62` does not.
 
 Boolean comparison results are treated as strict pass/fail values. `evidence_threshold` is not applied to boolean results.
 
-`check_with_evidence()` has no `validator` concept — it is a standalone, single-purpose comparison gate, untouched by the `validator` addition in this release.
+`check_with_evidence()` has no `validator` concept, and its results always have `parsed=None` — it is a standalone, single-purpose comparison gate, untouched by either the `validator` or `parsed` additions.
 
 ---
 
@@ -439,7 +480,7 @@ Optional callback invoked after each attempt.
 
 #### `validator` (keyword-only, 0.4.2+)
 
-Optional `Callable[[str], bool | tuple[bool, str]]`. Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See [Custom validation with `validator`](#custom-validation-with-validator) above for the full contract. Must be synchronous. Default `None` — a true no-op.
+Optional `ValidatorFn` — `Callable[[str], bool | tuple[bool, str]]`. Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See [Custom validation with `validator`](#custom-validation-with-validator) above for the full accepted-shape contract (widened in 0.4.4 to also accept `(bool, None)`, list forms, and `numpy.bool_`). Must be synchronous. Default `None` — a true no-op.
 
 ---
 
@@ -485,6 +526,7 @@ Checks an answer against caller-supplied evidence. It:
 * performs no retries
 * does not modify a previous `BoothResult`
 * has no `validator` parameter — it is a standalone comparison gate
+* always returns `parsed=None` — there is no LLM JSON response involved
 * uses the caller's `compare_fn`
 
 #### `answer`
@@ -523,6 +565,7 @@ result.ambiguous
 result.interpretations
 result.all_parse_failed
 result.method
+result.parsed
 ```
 
 ### `answer`
@@ -543,7 +586,7 @@ The comparison score produced by `check_with_evidence()`. `None` for normal `che
 
 ### `attempts`
 
-The full history of LLM attempts made by `check()` or `acheck()`, each including per-attempt `passed_validation` / `validation_error` (always `True` / `None` if no `validator` was supplied). Evidence checks do not make attempts, so their attempt list is empty.
+The full history of LLM attempts made by `check()` or `acheck()`, each including per-attempt `passed_validation` / `validation_error` (always `True` / `None` if no `validator` was supplied) and `parsed` (the raw JSON object for that specific attempt, `None` if it failed to parse). Evidence checks do not make attempts, so their attempt list is empty.
 
 ### `n_attempts`
 
@@ -569,6 +612,10 @@ The interpretations reported when the model marks a question as ambiguous.
 
 Which mechanism produced the result — `"ambiguity"`, `"evidence"`, `"parse_failure"`, `"validation"`, or `"confidence"`. See [`result.method`](#resultmethod) above.
 
+### `parsed` (0.4.3+)
+
+The model's raw, uncoerced JSON response object. See [`result.parsed`](#resultparsed) above.
+
 ---
 
 ## Result Statuses
@@ -621,6 +668,7 @@ BOOTH currently does **not**:
 * provide calibrated confidence probabilities
 * guarantee that retrieved evidence is correct, complete, relevant, or current
 * guarantee that a custom `validator` is itself correct — a validator can pass a wrong answer or reject a correct one, same as any other application-supplied rule
+* validate or enforce a schema on `result.parsed` — it is exposed as-is, entirely unvalidated
 * replace application-specific validation or safety systems (though `validator` gives you a documented hook to plug your own logic into BOOTH's retry loop rather than reimplementing that loop yourself)
 
 BOOTH is a **checkpoint library**, not an LLM framework, search engine, RAG framework, or autonomous verification system.
@@ -641,6 +689,10 @@ Ambiguity detection depends on the model recognizing the ambiguity. BOOTH can de
 
 `validator` is exactly as reliable as the logic you give it. BOOTH enforces that a validator's decision is respected consistently in the retry loop — it does not, and cannot, check whether the validator's own logic is actually correct for your use case.
 
+### `result.parsed` is unvalidated
+
+`result.parsed` is the model's raw JSON object, exposed as-is. BOOTH does not validate its shape, enforce a schema on it, or guarantee any field beyond the five it extracts for itself (`answer`, `confidence`, `ambiguous`, `interpretations`, `chosen_interpretation`) is present or well-typed. Reading extra fields from `parsed` is entirely the caller's responsibility, including handling missing keys or unexpected types.
+
 ### Evidence quality
 
 Evidence checking is only as useful as the evidence and comparison function supplied by the application. If the evidence is wrong, incomplete, outdated, or unrelated, BOOTH does not independently detect that. Likewise, a weak `compare_fn` can produce a misleading result. This includes the case where retrieved evidence is baked into the model's own prompt as RAG context — a wrong document can make the model's answer both more *confident* and more evidence-*consistent*, without becoming more correct.
@@ -697,6 +749,7 @@ Future BOOTH development may explore:
 * richer composition of multiple checkpoint results
 * better evaluation and calibration tooling
 * additional integrations with retrieval and tool systems
+* an optional, lightweight structured-schema gate on top of `result.parsed` — deferred deliberately until there's a clear, minimal shape for it, rather than reaching for a general schema/validation dependency
 
 These are future directions, not capabilities currently guaranteed by the library.
 
@@ -730,9 +783,9 @@ Run the test suite:
 pytest
 ```
 
-The test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, and evidence checking. CI runs the full suite on push/PR across Python 3.9–3.12.
+The test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, evidence checking, raw-response exposure via `result.parsed`, and a dedicated regression suite for the `v0.4.4` bugfix batch. CI runs the full suite on push/PR across Python 3.9–3.12.
 
-The validator tests include cases for: `validator=None` full-regression parity, boolean and `(bool, str)` returns, exceptions raised inside a validator, invalid return types, validator never running on ambiguous or unparseable attempts, fail-then-pass producing `REPAIRED`, the distinct validation-failure retry prompt, mixed multi-attempt histories and how `result.method` resolves them, and `check()`/`acheck()` parity under `validator`.
+The `v0.4.4` regression tests specifically verify: a JSON string `"false"` for `ambiguous` is not misread as `True`; a JSON boolean for `confidence` is rejected rather than silently accepted as `1.0`/`0.0`; `ValidatorFn` is importable from the top-level package; a falsy-but-present `chosen_interpretation` (`0`, `""`) is preserved rather than discarded to `None`; `numpy.bool_`-shaped validator returns are accepted; `(bool, None)` and list-shaped validator returns are accepted while genuinely malformed shapes are still rejected; confidence exactly equal to `threshold` passes on both `check()` and `acheck()`; list-wrapped JSON (`[{...}]`) still recovers via the parsing fallback; and the previously-unverified nested-brace edge case's actual behavior is now locked in by a test rather than assumed.
 
 ---
 
@@ -742,9 +795,11 @@ The validator tests include cases for: `validator=None` full-regression parity,
 2. **Make uncertainty explicit.** When an output does not meet the configured acceptance condition, return a structured status instead of silently passing it through.
 3. **Treat ambiguity, validation, and confidence as separate, ordered checks.** A confident, validator-passing answer can still be ambiguous; a confident answer can still fail a caller's own validation rule before confidence is ever consulted.
 4. **Reconsider instead of blindly resampling.** Retries give the model an opportunity to examine its previous response — and the reason it failed (parse failure, validation failure, or low confidence) determines what the model is actually shown.
-5. **Keep evidence retrieval outside BOOTH.** Applications remain free to use their own RAG, search, database, or tool infrastructure.
-6. **Do not pretend agreement is truth.** Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim.
-7. **Stay provider-agnostic.** BOOTH works with different LLM providers because the application supplies the model-calling function.
+5. **Expose, don't reinterpret.** `result.parsed` is a transparency layer over data BOOTH already has, not a second parser or validator — it shows the model's raw response rather than deciding what it should mean.
+6. **Reject invalid data, don't silently coerce it.** An out-of-range confidence, a boolean masquerading as a confidence score, or an unrecognized value for `ambiguous` are all treated as reasons to reject the attempt — never guessed at or silently reinterpreted into something that happens not to crash. Applied consistently across every coerced field as of `v0.4.4`.
+7. **Keep evidence retrieval outside BOOTH.** Applications remain free to use their own RAG, search, database, or tool infrastructure.
+8. **Do not pretend agreement is truth.** Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim.
+9. **Stay provider-agnostic.** BOOTH works with different LLM providers because the application supplies the model-calling function.
 
 ---
 

{boothpy-0.4.2 → boothpy-0.4.4}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "boothpy"
-version = "0.4.2"
-description = "A lightweight checkpoint layer for LLM outputs — Path B (self-reported confidence with reconsideration retries, sync and async) plus Path A's evidence-agreement gate."
+version = "0.4.4"
+description = "A lightweight checkpoint library for LLM outputs."
 readme = "README.md"
 requires-python = ">=3.9"
 license = { file = "LICENSE" }

{boothpy-0.4.2 → boothpy-0.4.4}/src/booth/__init__.py

@@ -5,6 +5,7 @@ from booth.core import (
     acheck,
     check_with_evidence,
     CompareFn,
+    ValidatorFn,
     VERIFIED,
     REPAIRED,
     AMBIGUOUS,
@@ -14,7 +15,7 @@ from booth.core import (
     DEFAULT_MAX_RETRIES,
 )
 
-__version__ = "0.4.2"
+__version__ = "0.4.4"
 
 __all__ = [
     "Attempt",
@@ -23,6 +24,7 @@ __all__ = [
     "acheck",
     "check_with_evidence",
     "CompareFn",
+    "ValidatorFn",
     "VERIFIED",
     "REPAIRED",
     "AMBIGUOUS",