jevdecision 0.1.0__tar.gz

jevdecision-0.1.0/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+All notable changes are documented here, following [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/).
+This project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-10-13
+
+### Added
+- Installable standard-library-only package, annotated public API, MIT license, source/sdist examples, and `jev-decision` / `jev-decision-eval` commands.
+- Typed per-question refusals, refusal counts/rates, and score Brier, log-loss and ECE metrics.
+- Versioned decision/evaluation/provenance records with producer metadata and canonical JSON.
+
+### Changed
+- Live evaluation durably appends responses and case results in a sibling staging directory; completed output is published atomically. Authentication failures stop with exit 2.
+- Decision CLI returns exit 1 for a result containing refusals while retaining sibling answers.
+
+### Fixed
+- Correctly rounded floating-point accumulation gives identical bundled replay metrics on Python 3.11 and 3.13.
+- Examples ship once in the source/sdist, rather than being duplicated in the installed package.
+- Missing score probabilities stay unknown; only complete distributions can derive a score. Missing metric inputs fail requirements.
+- Probability values must be finite and within [0,1]; complete distributions must sum to 1 within rounding tolerance.
+- HTTP protocol failures become retryable `APIError`; HTTP 408 is retryable.
+
+### Security
+- Remote provider URLs require HTTPS; HTTP is limited to loopback hosts. URL credentials are rejected and authenticated POST redirects are never followed.

jevdecision-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bicheng Wang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jevdecision-0.1.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include CHANGELOG.md
+recursive-include examples *.json *.jsonl

jevdecision-0.1.0/PKG-INFO

@@ -0,0 +1,364 @@
+Metadata-Version: 2.4
+Name: jevdecision
+Version: 0.1.0
+Summary: Typed AI-assisted decisions and replayable offline evaluation records
+Author: Bicheng Wang
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/BichengWang/ai-decision-system/tree/main/ai-workflows/jev-decision
+Project-URL: Source, https://github.com/BichengWang/ai-decision-system
+Project-URL: Issues, https://github.com/BichengWang/ai-decision-system/issues
+Project-URL: Changelog, https://github.com/BichengWang/ai-decision-system/blob/main/ai-workflows/jev-decision/CHANGELOG.md
+Keywords: decisions,evaluation,replay,probability,gating
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# Jev decision model (`jevdecision`)
+
+Replayable decision records for AI-assisted go/no-go decisions. A decision model answers bounded
+questions about a state with **typed answers and reported probabilities** instead of
+generated text: a yes/no probability, one label from a fixed set, or a position
+on an ordered scale. These can inform gating agent tool calls, routing tickets,
+and triage; missing choice uncertainty never becomes permission to route.
+
+[Jev](https://docs.typesafe.ai/api) (TypeSafe's System One model) is the default
+backend. The [OpenAI Decisions API](https://developers.openai.com/api/docs/guides/decisions)
+is a drop-in alternative: questions are defined once and answers come back in the
+same types from either provider. The package uses only the Python standard
+library, so it has no dependencies and no lockfile.
+
+## Install
+
+Requires Python 3.11 or later; runtime dependencies: none. Examples live in the clone
+and source distribution under `examples/`; they are not installed in the wheel.
+
+```bash
+# Once 0.1.0 is published:
+pip install jevdecision
+```
+
+The distribution includes `jev-decision` and `jev-decision-eval`; existing
+`python -m jevdecision` and `python -m jevdecision.evaluation` commands remain supported.
+Use the `examples/` directory in a clone or unpacked source distribution.
+
+## Question types
+
+| Type | Jev request | OpenAI request | Answer |
+|---|---|---|---|
+| `Noul` (alias `Predicate`) | `noul` | `predicate` | `NoulAnswer.probability`: probability of yes |
+| `Choice` | `choice`, `criteria` = label → description | `choice`, `choices` = `[{value, description}]` | `ChoiceAnswer.choice`, `.probabilities` per label, `.confidence` |
+| `Score` | `score`, `criteria` = ordered levels | `score`, `levels` = `[{label}]` | `ScoreAnswer.score` (probability-weighted level index), `.level`, `.probabilities` per level |
+
+Questions are validated before anything is sent: names are unique identifiers,
+instructions are non-empty, a choice has 1 to 255 options, and a score has 2 to
+10 distinct levels. Responses are checked against the questions they answer.
+Every question must have a matching answer or an explicit `type: "refusal"` result; a choice must be one of the
+options, and probabilities and scores must lie in range. Probabilities must be finite and within [0,1]. Complete distributions must sum
+to 1 within `1e-6 * number_of_options + 1e-3`; partial collections retain unknown
+entries and are not treated as complete distributions. Invalid responses raise
+`ResponseError`, so the application never acts on a malformed answer.
+
+## Usage
+
+```python
+from jevdecision import Choice, DecisionModel, Noul, Score
+
+model = DecisionModel.from_env()   # DECISION_BACKEND=jev (default) or openai
+
+decision = model.decide(
+    "Help! My payouts have been failing for 3 days.",
+    [
+        Noul("refund", "Is the customer asking for a refund?"),
+        Choice("department", "Which team should handle this?", {
+            "billing": "Payments, invoicing, refunds",
+            "technical": "Bugs, outages, integrations",
+            "sales": "Pricing, upgrades, new accounts",
+        }),
+        Score("frustration", "How frustrated is the customer?", ["Calm", "Frustrated", "Very angry"]),
+    ],
+)
+decision["department"].choice        # "technical"
+decision["frustration"].level        # "Frustrated"
+
+# Single-question helpers
+model.gate(state, refund_question, threshold=0.8)                  # bool
+model.route(state, department_question, min_confidence=0.7,
+            fallback="human_review")                               # label or fallback
+```
+
+`route` uses the provider's `confidence` when it returns one, otherwise the
+reported probability of the chosen label. If neither is supplied, it returns
+`fallback` (default `None`), even with `min_confidence=0.0`. A reported confidence
+still takes precedence over a reported choice probability; neither is a calibrated
+safety guarantee or a replacement for application policy.
+
+### Refusals and score uncertainty
+
+An explicit per-question refusal becomes `RefusalAnswer` with `.name` and
+`.type == "refusal"`. `ask`/`decide` retain valid sibling answers. `gate` returns
+`False` for a refusal, even at threshold zero; `route` returns its fallback.
+The OpenAI guide documents this marker; TypeSafe's reference does not document
+refusal fields, so no reason or other provider-specific field is assumed.
+
+Score probabilities are a tuple in level order, with `None` for missing entries.
+`.probability` is the reported probability of the nearest level, or `None`.
+A supplied score remains usable without probabilities; deriving a missing score
+requires a complete distribution. Missing values are never zero-filled.
+
+### Choice uncertainty contract
+
+`ask` and `decide` retain a valid choice even when its uncertainty is unknown:
+
+- `.probabilities` contains only labels whose probabilities were actually supplied,
+  in question-option order. Missing, null, or empty probability collections yield `{}`.
+- `.probability` is `None` when the chosen label has no reported probability.
+  A supplied `confidence` does not manufacture a choice probability.
+- `.ranked()` ranks only reported probabilities. `to_dict()` preserves missing
+  entries instead of fabricating a complete distribution.
+- A reported `0.0` remains zero, distinct from an absent value.
+
+**Safety correction:** earlier code fabricated probability `1.0` for a choice
+without probabilities and filled omitted labels with `0.0`. Callers must use
+`.probability` or `.probabilities.get(label)` and handle `None`; do not rely on
+every label being present. To inspect an unscored recommendation, use `ask` or
+`decide`, not a confidence-gated route with an invented probability.
+
+## Configuration
+
+| Variable | Backend | Meaning |
+|---|---|---|
+| `DECISION_BACKEND` | both | `jev` (default) or `openai` |
+| `TYPESAFE_API_KEY` | `jev` | Bearer token for `https://api.typesafe.ai/v1/systemone` |
+| `JEV_MODEL` | `jev` | Model route (default `jev-latest`) |
+| `OPENAI_API_KEY` | `openai` | Bearer token for `https://api.openai.com/v1/decisions` |
+| `OPENAI_DECISIONS_MODEL` | `openai` | Model (default `gpt-6-luna`) |
+
+Every backend also takes `api_key`, `model`, `base_url`, `path`, `timeout`,
+`max_retries`, `backoff`, `max_retry_wait`, and `transport` arguments.
+`max_retries` must be a non-negative integer; zero disables retries. `backoff`
+must be a finite non-negative number of seconds; zero permits immediate retries
+unless `Retry-After` requests a wait. Invalid settings raise `DecisionError`
+before any request is sent. Use `base_url` and `path` for a gateway
+that serves Jev elsewhere, for example `path="/v1/decisions"` with
+`model="typesafe/jev"`. Request timeout (408), rate-limit (429), server (5xx), connection and HTTP protocol errors are
+retried with exponential backoff (2 retries by default). When the provider sends a
+`Retry-After` header (seconds or an HTTP date), the client waits at least that long
+before retrying. A provider that asks for more than `max_retry_wait` seconds (60 by
+default) fails the request at once with `APIError`, whose `retry_after` attribute
+carries the requested wait, instead of blocking the caller. Other HTTP errors are
+raised as `APIError` immediately. `transport` is any callable
+`(url, headers, body_bytes, timeout) -> dict`, which tests use to replace the
+network.
+
+Remote `base_url` values require HTTPS. HTTP is accepted only for `localhost`
+or a literal loopback IP address (IPv4 or IPv6). Credentials, query strings and
+fragments in the base URL are rejected with `QuestionError`. Authenticated POSTs
+never follow redirects: every 3xx response raises a non-retryable `APIError`.
+Custom transports must enforce equivalent credential protection themselves.
+
+The structured state can be text, a JSON object, or a list of non-empty strings. Jev
+receives it unchanged. The OpenAI backend sends text, so it serializes an object
+as JSON with sorted keys and joins a list with blank lines.
+
+## Command line
+
+From this directory, with Python 3.11 or later:
+
+```bash
+python3 -m unittest discover -s tests
+python3 -m jevdecision --questions examples/support-ticket.json \
+    --state "Help! My payouts have been failing for 3 days." --dry-run
+python3 -m jevdecision --questions examples/support-ticket.json \
+    --state-file ticket.json --backend openai
+```
+
+The questions file uses Jev's native question map
+([`examples/support-ticket.json`](examples/support-ticket.json)). `predicate` is
+accepted as an alias of `noul`. A `.json` state file is sent as structured
+state, and any other file is sent as text. `--dry-run` prints the provider request
+without sending it, so it needs no API key; the state is validated first either
+way. The decision is printed as JSON. Exit codes: 0 for success, 1 for a result containing refusals or a provider
+or response error, 2 for invalid input (questions, state, or a `--timeout` that is
+not a positive number of seconds).
+
+From the repository root, `make jev-test` runs the tests and
+`make jev-dry-run` prints the example request for both backends.
+
+## Evaluating a decision model
+
+Measure the model on labeled cases before software acts on its answers.
+`jevdecision.evaluation` scores every question against the expected answers and
+can gate the result on thresholds you set.
+
+A case file is JSON Lines with a state and the expected answer to some or all
+questions: `true` or `false` for a `noul`, a label for a `choice`, and a level
+(or its index) for a `score`:
+
+```json
+{"id": "t02", "state": "I was charged twice for March. Please refund the duplicate charge.",
+ "labels": {"refund": true, "department": "billing", "frustration": "Frustrated"}}
+```
+
+| Type | Metrics |
+|---|---|
+| `noul` | `accuracy` (at probability 0.5), `brier`, `log_loss`, `ece`, `base_rate`, `mean_probability` |
+| `choice` | `accuracy`, `brier` (summed over labels), `log_loss`, `ece` (probability of the chosen label against correctness), a confusion matrix |
+| `score` | `accuracy` (nearest level), `within_one`, `mae` (in level steps), `brier`, `log_loss`, `ece` (nearest level probability) |
+
+Every question also reports `n`, the number of cases that label it, and
+`accuracy_lower` and `accuracy_upper`, the 95% Wilson score interval for
+`accuracy`. With 12 cases, 11 correct answers give an accuracy of 0.917 but an
+interval of [0.646, 0.985], so a gate on `accuracy_lower` asks for enough cases as
+well as a high hit rate. `ece` is the
+expected calibration error over ten equal-width probability bins. Log loss clips
+probabilities to [1e-6, 1 - 1e-6].
+
+Every question also reports `refusals` and `refusal_rate` among its labeled,
+parsed results. Refusals count in `n`, accuracy and its Wilson interval as
+incorrect, and make probability metrics unmeasured. Score MAE and within-one
+are also unmeasured if any result refuses. Request/parse errors remain separate.
+
+Missing choice probabilities do not exclude otherwise valid answers from `n`,
+accuracy, its interval, or the confusion matrix. Probability metrics require the
+inputs they measure on **every scored case**, not a selectively observed subset:
+
+- Brier score requires a reported probability for every option.
+- Log loss requires the expected label's reported probability.
+- ECE requires the chosen label's reported probability; provider `confidence`
+  is not substituted for that probability.
+
+If any scored case lacks a metric's required inputs, that metric is `null` in
+JSON and `unmeasured` in the report. A requirement on it fails. Per-case records
+retain the prediction and a null chosen probability, and raw responses remain
+available for replay. Re-evaluating older responses with missing probabilities
+may therefore fail gates that previously scored fabricated values. Complete
+probability responses retain their previous scores. Score Brier requires every
+level probability, log loss requires the expected level probability, and ECE
+requires the nearest level probability on every case. Missing inputs yield
+`null` and fail metric requirements, as for choices.
+
+A requirement file supplies explicit question metric bounds, each a `min` or
+`max`. The overall JSON field is `status` (not a boolean `pass`):
+
+- `PASS`: at least one valid question metric bound was supplied, and all checks pass.
+- `FAIL`: an explicit metric bound or the error-rate check fails.
+- `UNGATED`: all checks pass but there are no question metric bounds. Omitting
+  `--require`, `{}`, `{"max_error_rate": 0}`, and `{"questions": {}}` are all
+  ungated, not model-quality validation.
+
+`max_error_rate` (default 0) bounds the share of cases whose request or response
+failed; it alone is not a question gate. A failed case counts as an error, never
+as a wrong answer. A bound on an unmeasured metric fails. A valid question bound
+means a finite numeric `min`/`max` on a supported metric; it does not mean the
+threshold is scientifically adequate. For example, an `n`-only gate checks sample
+count, not accuracy. Select meaningful thresholds for the decision being made.
+Malformed specs (including JSON `null`, `false`, arrays, empty per-question metric
+objects and empty bounds) are invalid input, not ungated runs. In the Python
+`evaluate()` API, `None` means no requirement spec. Individual requirement rows
+retain boolean `pass` values; the overall report does not.
+
+```json
+{"max_error_rate": 0.0,
+ "questions": {"refund": {"n": {"min": 10}, "accuracy_lower": {"min": 0.8}, "brier": {"max": 0.1}}}}
+```
+
+```bash
+# Replay supplied responses: no API key; origin remains unverified
+python3 -m jevdecision.evaluation --questions examples/support-ticket.json \
+    --cases examples/support-ticket-cases.jsonl \
+    --responses examples/support-ticket-responses.jsonl \
+    --require examples/support-ticket-requirements.json --out eval-run
+
+# Ask the provider live (needs its API key); no question bounds means UNGATED
+python3 -m jevdecision.evaluation --questions examples/support-ticket.json \
+    --cases examples/support-ticket-cases.jsonl --backend openai --out live-run
+```
+
+`--out` must be new or empty. Files are prepared in a sibling `.NAME-*` staging
+directory, then published with `os.replace` after all summaries are complete.
+Live runs print the staging path and append/fsync each completed response and
+case result before requesting the next case, including malformed JSON responses
+that parsed as JSON but failed answer validation. Interrupted runs retain these
+JSONL files for inspection and replay; the target remains available for a rerun.
+A 401/403 stops immediately with exit 2 and retains progress without a final report.
+There is no automatic resume; replay the saved responses with the original cases
+(missing responses count as errors), or start a new live run.
+
+Each run writes `evaluation.json` (canonical JSON),
+a readable `EVALUATION.md`, `responses.jsonl` with the supplied replay responses
+or responses obtained through the live backend, and `cases.jsonl` with one row
+per case: each labeled question's expected and predicted answer (with probability
+or score) and whether it was correct, or the case's error. `EVALUATION.md` lists
+the first 20 incorrect answers. Exported responses can be passed as `--responses`
+to re-score without calling a provider. The bundled replay produces byte-identical
+`evaluation.json`, `provenance.json`,
+`responses.jsonl` and `cases.jsonl` on Python 3.11 and 3.13; live-to-replay
+metrics remain the same, but the report's `source` changes.
+
+Decision records use `schema_version: 1`; evaluation and provenance records use
+`schema_version: 2`. They carry `produced_by: {"name": "jevdecision", "version": "0.1.0"}`.
+Raw provider response replay remains compatible with existing unversioned fixtures.
+JSON records use sorted keys, compact separators and UTF-8 without ASCII escaping;
+SHA-256 digests are lowercase hexadecimal. JSONL rows use the same encoding.
+
+CLI runs also write a separate deterministic `provenance.json`:
+
+- `inputs` records SHA-256 of the exact question, case, requirement and response
+  file bytes read for the run, with `null` for absent optional inputs.
+- `response_mode` is `live` or `replay`. Replay origin is explicitly **unverified**:
+  files may be hand-written, and their `model` fields do not authenticate provider output.
+- `outputs` records SHA-256 of `evaluation.json`, `EVALUATION.md`, `responses.jsonl`
+  and `cases.jsonl`. The manifest does not hash itself, include timestamps, paths,
+  headers or credentials, or enter the metric records. Whitespace-only input
+  changes can therefore change provenance without changing `evaluation.json`.
+
+Hashes identify bytes, not authenticity, reviewer independence or model quality.
+Raw responses and labeled cases may contain sensitive material; inspect them
+before sharing. `--backend` names the response format to replay (`jev` by default).
+The CLI and Markdown display the same overall status. Exit codes: **0** for PASS,
+**1** for FAIL or a provider that cannot be used, **2** for invalid input/unreadable files or live 401/403 (authentication failures
+retain staged progress; invalid input produces no final artifacts), and **3** for UNGATED.
+
+The example responses are hand-written in Jev's response format to show the
+workflow. They are not output from Jev or any other model, and the example
+report says nothing about any model's quality.
+
+From the repository root, `make jev-eval` runs the example into a temporary directory.
+
+## Limitations
+
+- Both provider APIs are new. The Jev request and response shapes follow
+  TypeSafe's published examples. The OpenAI Decisions API is in public beta, so
+  the parser accepts both its object and its list answer layouts. Recheck the
+  field mapping against the provider references before you depend on it in
+  production.
+- Image inputs and per-request options beyond the model are not modeled yet.
+- Unit tests use recorded response shapes and never call a provider.
+- Only accuracy carries a confidence interval. Brier, log loss, calibration
+  error and MAE are point estimates, which are noisy with a few dozen cases, so set
+  `n` minimums that match the thresholds you rely on.
+- A live evaluation sends one request per case, sequentially.
+- Float sums use `math.fsum` for correctly rounded accumulation. The bundled replay
+  has identical bytes and hashes on Python 3.11 and 3.13; this check is not a
+  guarantee for every platform, math library or future algorithm version.
+
+## Local software verification
+
+Release checks cover refusals, unknown probabilities, malformed distributions,
+HTTP retries and redirect protection, interrupted live progress, authentication
+stops and atomic publication. Tests use synthetic provider responses and a local
+HTTP fixture. Fresh wheel installs exercise both backend dry runs and the gated
+12-case replay example; exported responses reproduce the evaluation bytes across
+Python 3.11 and 3.13. These checks are not live-provider calls, calibration results,
+or deployment-safety evidence.