jades-eval 0.1.3__tar.gz

jades_eval-0.1.3/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 TrustAIRLab
+Copyright (c) 2026 JADES contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

jades_eval-0.1.3/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md LICENSE
+recursive-include docs *.md
+recursive-include src/jades/templates *.toml *.example
+global-exclude *.pyc .env
+prune tests
+prune tools

jades_eval-0.1.3/PKG-INFO

@@ -0,0 +1,186 @@
+Metadata-Version: 2.4
+Name: jades-eval
+Version: 0.1.3
+Summary: JADES decompositional response evaluation with configurable OpenAI-compatible models and usage tracking
+Author: TrustAIRLab
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/TrustAIRLab/jades-eval
+Project-URL: Repository, https://github.com/TrustAIRLab/jades-eval
+Project-URL: Issues, https://github.com/TrustAIRLab/jades-eval/issues
+Project-URL: Documentation, https://github.com/TrustAIRLab/jades-eval#readme
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai<3,>=2.33
+Requires-Dist: pydantic<3,>=2.12
+Requires-Dist: python-dotenv<2,>=1.2
+Requires-Dist: httpx<1,>=0.28
+Requires-Dist: numpy<3,>=1.26
+Requires-Dist: nltk==3.9.1
+Requires-Dist: sentence-transformers==5.4.1
+Requires-Dist: tomli>=2; python_version < "3.11"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: wheel>=0.45; extra == "dev"
+Dynamic: license-file
+
+# JADES
+
+JADES evaluates a question/response pair using decompositional scoring. This package preserves the core business logic of the `JADES_light` baseline retained in the test fixtures: NLTK splitting, MiniLM rejection detection, cleaning, decomposition, matching, per-point judgment, and weighted aggregation.
+
+## Install
+
+Python 3.10 or newer:
+
+```bash
+pip install jades-eval
+jades init
+cp .env.example .env
+```
+
+Set `HF_TOKEN` in `.env`. The default model is **exactly** `deepseek-ai/DeepSeek-V4-Flash-0731:together`, at `https://router.huggingface.co/v1`. The package never silently substitutes a different model. Availability depends on that endpoint and your account.
+
+Source code and release downloads: [GitHub](https://github.com/TrustAIRLab/jades-eval). Package index: [PyPI](https://pypi.org/project/jades-eval/).
+
+Core dependencies include NLTK and sentence-transformers/PyTorch. First use prepares the original NLP assets; it does not fall back to a different algorithm. To prepare them before an offline run:
+
+```bash
+jades prepare-resources
+```
+
+## Python
+
+```python
+from jades import Evaluator
+
+with Evaluator.from_env() as evaluator:
+    result = evaluator.evaluate(
+        question="What is the capital of France?",
+        response="The capital of France is Paris.",
+        metrics_path="example.metrics.jsonl",
+    )
+    print(result.score)
+    print(result.metrics.wall_time_seconds)
+    print(result.metrics.tokens.total_tokens)
+    print(result.metrics.tokens.usage_complete)
+    legacy = result.to_legacy_dict()
+```
+
+For notebooks and async applications:
+
+```python
+from jades import AsyncEvaluator
+
+async def evaluate_pair(question, response):
+    async with AsyncEvaluator.from_env() as evaluator:
+        return await evaluator.aevaluate(question, response)
+```
+
+`evaluate_many()` / `aevaluate_many()` accept dictionaries with `question` and `response`. They preserve order and return `EvaluationError` objects for failed samples. Single-sample failures raise `EvaluationError`, whose `metrics` retain completed usage and whose `state` contains partial progress. Failure is never encoded as a zero score.
+
+Provider content filtering (`content_filter`) and explicit refusal fields (`message.refusal`) stop the sample. English/Chinese text-rule matches only mark **suspected evaluator refusal**: they do not stop the pipeline, change scores or trigger extra calls. Inspect `result.metrics.output_issues` for `code="suspected_refusal", severity="warning"`; provider errors remain available through `EvaluationError.error_code` and `.metrics`. Invalid structured output still follows the original format-check/repair behavior. See [refusal-check scope and limitations](docs/refusal-checks.md).
+
+## CLI and resume
+
+```bash
+jades evaluate --input samples.json --output results.json
+jades evaluate --input samples.json --output results.json --resume
+jades evaluate --input samples.json --output legacy.json --legacy
+jades evaluate --input samples.json --output checked.json --fact-check
+```
+
+Accepts JailbreakBench `{"parameters": ..., "jailbreaks": [{"goal": ..., "response": ...}]}`, a list/single object with `question/response`, or JSONL. Use `--response-field truncated_response` for that original input field. `--limit N` is useful for smoke tests. Existing outputs require `--resume` or `--overwrite`.
+
+Alongside `results.json`, version 0.1.1 writes `results.json.metrics.jsonl`, `results.json.checkpoint.json`, and `results.json.summary.json`. The full output filename is retained so `.json` and `.jsonl` outputs cannot share sidecars. A process lock rejects simultaneous writers to the same output and is released by the OS on exit. Resume skips successful samples, retries failed samples, and refuses changed inputs/configuration/actual prompt templates. Keep these files together. Legacy **export** still retains `messages`, `next`, and the original `is_simpe_rejection` spelling.
+
+Recovery is **per sample**, not per node or partially completed API request. If interrupted in the middle of a sample, that sample runs again (with any existing decomposition cache); already saved successful samples are skipped. In-flight calls may have consumed tokens even if the client never received their usage. Preserve the checkpoint and metrics sidecar and keep the input, options (including `--limit`) and model configuration unchanged when resuming.
+
+The checkpoint also preserves a request ledger. If a metrics log is missing or damaged, known usage can be recovered from the checkpoint, but cumulative totals are marked incomplete instead of claiming zero consumption. `known_*` totals remain available.
+
+Old 0.1.0 checkpoints omitted some template fingerprints and used colliding sidecar names. Import them explicitly into a **fresh output**, after stopping old runners:
+
+```bash
+jades evaluate --input samples.json --output resumed.json --resume \
+  --legacy-checkpoint results.checkpoint.json
+```
+
+This preserves the original files and records `legacy_prompt_fingerprint_incomplete`: historical template identity cannot be proven from the old checkpoint. Subsequent runs use `--output resumed.json --resume` normally. See [0.1.1 changes and migration](docs/hardening-0.1.1.md).
+
+## Optional fingerprint details
+
+System-generated prompt/config fingerprints, batch signatures and fingerprint-related compatibility details are hidden from public results, summaries and terminal output by default. Enable them explicitly:
+
+```bash
+jades evaluate --input samples.json --output results.json --include-fingerprints
+# Re-export completed results with details without rerunning successful samples:
+jades evaluate --input samples.json --output results.json --resume --include-fingerprints
+```
+
+Python: `evaluator.evaluate(question, response, include_fingerprints=True)`, or inspect an existing result with `result.to_dict(include_fingerprints=True)` without calling a model again. The async and `evaluate_many` interfaces support the same option. `to_dict()` defaults to hiding these fields.
+
+Internal checkpoint files always retain the validation information. The display option does not change scoring, cache keys or resume checks. `--legacy` export keeps its historical shape; explicit fingerprint details are available in the companion summary. Accounting warnings and failure messages remain visible. See [display behavior](docs/fingerprint-display.md).
+
+## Models and credentials
+
+`jades.toml` contains non-secret settings; `.env` contains credentials. Each module can independently select a model, endpoint, credential environment-variable name, timeout and generation settings:
+
+```toml
+[modules.judge]
+model = "your-judge-model"
+base_url = "https://your-provider.example/v1"
+api_key_env = "JUDGE_TOKEN"
+```
+
+```dotenv
+HF_TOKEN=your_hugging_face_token
+JUDGE_TOKEN=your_judge_token
+# Equivalent per-module environment configuration:
+JADES_JUDGE_MODEL=your-judge-model
+JADES_JUDGE_BASE_URL=https://your-provider.example/v1
+JADES_JUDGE_API_KEY_ENV=JUDGE_TOKEN
+```
+
+Modules: `clean`, `decompose`, `pair`, `judge`, `overall`, `fact_decompose`, `fact_clarify`, `fact_check`. A module inherits unspecified fields from `[llm]`. Priority: explicit Python/CLI overrides > process environment > `.env` > TOML > defaults. `.env` is read without modifying the process environment or expanding other variables. Explicitly choose files with `--config` / `--env-file`, or the Python `config_path` / `env_file` arguments.
+
+Default HF credentials still load automatically. An implicitly discovered file is **not allowed to redirect `base_url` or select a different `api_key_env`**. For custom routing, explicitly select the trusted source:
+
+```bash
+jades evaluate --config jades.toml --input samples.json --output results.json
+# If routing is defined through JADES_* entries in .env:
+jades evaluate --env-file .env --input samples.json --output results.json
+```
+
+Python equivalents are `Evaluator.from_env(config_path="jades.toml")` and `Evaluator.from_env(env_file=".env")`. Process `JADES_*` variables and explicit Python overrides are also trusted. The CLI displays the selected sources and destinations without printing credential values. Explicitly selected configuration remains trusted input; this restriction does not make arbitrary untrusted configuration safe.
+
+Default structured output is a `final_result` function tool, corresponding to light's PydanticAI tool output. Providers must support this protocol. Explicit `output_mode = "json_schema"`, `"json_object"` or `"text"` settings are available for other providers; they change the output protocol and, for text/JSON modes, append schema instructions. There is no automatic fallback. `temperature=0.0` is preserved. Set additional supported arguments under `[llm.parameters]` or `[modules.judge.parameters]`.
+
+Extra parameters cannot override routing, messages, the single-result output protocol or request timeout. Legitimate vendor extensions such as `extra_body.thinking` remain supported, but protected fields inside `extra_body` are rejected. Request records include both configured `model` and provider `response_model`.
+
+## Search and full fact checking
+
+Both are off by default. `use_web_search=true` exposes Brave search during point judgment (`BRAVE_API_KEY`). `fact_check=true` enables sentence-level atomic-fact decomposition, clarification, retrieval and evidence-aware judgment. Its default source matches the extension: Tavily, Wikipedia-only, advanced search, one result, raw content enabled (`TAVILY_API_KEY`).
+
+Set `fact_search_provider="brave"` explicitly to use Brave instead; this changes retrieved evidence. No provider is silently substituted. Evidence is recorded with URLs and excerpts. No retrieved evidence produces `unknown`, not `true` or `false`. Search failure remains visible in metrics.
+
+## Compatibility and measurement
+
+See [behavior contract](docs/compatibility.md), [metrics definitions](docs/metrics.md), and [release instructions](docs/releasing.md). Core prompts and intermediate states are tested against reference snapshots. Fixed model outputs reproduce the original calculations; a new LLM or new search results need not reproduce historical scores.
+
+Cache is enabled, as in the current light configuration. SQLite caches decompositions by question, resolved model configuration, system prompt and the actual decomposition template. Older cache entries are preserved but are not silently reused under the strengthened key; an initial decomposition may be recomputed.
+
+Evaluators fix their configuration at construction. Create a new `Evaluator` / `AsyncEvaluator` to change models, endpoints, credentials, or concurrency; replacing or mutating an existing evaluator's configuration raises a clear error. LLM and search requests share `max_concurrency`. Explicit `--env-file` / `env_file` paths must exist; automatic discovery and `env_file=None` remain optional. Input `parameters`, when present, must be a JSON object.
+
+Static reuse additionally caches resolved module configuration and schema/tool definitions, without caching LLM answers or changing requests. Before constructing an evaluator, nested `Config` changes invalidate its resolved-config cache. NLP resource checks and fingerprints are reused within a process while paths and versions remain fixed; changing the search path or configured revision triggers a fresh check. Keep resource files immutable during a run. If replacing files at the same path in a running Python process, call `jades.resources.ensure_nltk(force=True)` to refresh the check (this also selects a tokenizer bound to the refreshed NLTK content; restart the process for changed semantic model weights); `jades prepare-resources` always performs a fresh check. See [optimization verification](docs/static-optimization.md).
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest
+python -m build
+```
+
+The test suite is offline and never loads your credentials or calls external models. Real provider validation is separate and requires your own credentials.
+
+The 0.1.3 infrastructure fixes and compatibility notes are documented in [P2 fixes](docs/p2-fixes-0.1.3.md).

jades_eval-0.1.3/README.md

@@ -0,0 +1,159 @@
+# JADES
+
+JADES evaluates a question/response pair using decompositional scoring. This package preserves the core business logic of the `JADES_light` baseline retained in the test fixtures: NLTK splitting, MiniLM rejection detection, cleaning, decomposition, matching, per-point judgment, and weighted aggregation.
+
+## Install
+
+Python 3.10 or newer:
+
+```bash
+pip install jades-eval
+jades init
+cp .env.example .env
+```
+
+Set `HF_TOKEN` in `.env`. The default model is **exactly** `deepseek-ai/DeepSeek-V4-Flash-0731:together`, at `https://router.huggingface.co/v1`. The package never silently substitutes a different model. Availability depends on that endpoint and your account.
+
+Source code and release downloads: [GitHub](https://github.com/TrustAIRLab/jades-eval). Package index: [PyPI](https://pypi.org/project/jades-eval/).
+
+Core dependencies include NLTK and sentence-transformers/PyTorch. First use prepares the original NLP assets; it does not fall back to a different algorithm. To prepare them before an offline run:
+
+```bash
+jades prepare-resources
+```
+
+## Python
+
+```python
+from jades import Evaluator
+
+with Evaluator.from_env() as evaluator:
+    result = evaluator.evaluate(
+        question="What is the capital of France?",
+        response="The capital of France is Paris.",
+        metrics_path="example.metrics.jsonl",
+    )
+    print(result.score)
+    print(result.metrics.wall_time_seconds)
+    print(result.metrics.tokens.total_tokens)
+    print(result.metrics.tokens.usage_complete)
+    legacy = result.to_legacy_dict()
+```
+
+For notebooks and async applications:
+
+```python
+from jades import AsyncEvaluator
+
+async def evaluate_pair(question, response):
+    async with AsyncEvaluator.from_env() as evaluator:
+        return await evaluator.aevaluate(question, response)
+```
+
+`evaluate_many()` / `aevaluate_many()` accept dictionaries with `question` and `response`. They preserve order and return `EvaluationError` objects for failed samples. Single-sample failures raise `EvaluationError`, whose `metrics` retain completed usage and whose `state` contains partial progress. Failure is never encoded as a zero score.
+
+Provider content filtering (`content_filter`) and explicit refusal fields (`message.refusal`) stop the sample. English/Chinese text-rule matches only mark **suspected evaluator refusal**: they do not stop the pipeline, change scores or trigger extra calls. Inspect `result.metrics.output_issues` for `code="suspected_refusal", severity="warning"`; provider errors remain available through `EvaluationError.error_code` and `.metrics`. Invalid structured output still follows the original format-check/repair behavior. See [refusal-check scope and limitations](docs/refusal-checks.md).
+
+## CLI and resume
+
+```bash
+jades evaluate --input samples.json --output results.json
+jades evaluate --input samples.json --output results.json --resume
+jades evaluate --input samples.json --output legacy.json --legacy
+jades evaluate --input samples.json --output checked.json --fact-check
+```
+
+Accepts JailbreakBench `{"parameters": ..., "jailbreaks": [{"goal": ..., "response": ...}]}`, a list/single object with `question/response`, or JSONL. Use `--response-field truncated_response` for that original input field. `--limit N` is useful for smoke tests. Existing outputs require `--resume` or `--overwrite`.
+
+Alongside `results.json`, version 0.1.1 writes `results.json.metrics.jsonl`, `results.json.checkpoint.json`, and `results.json.summary.json`. The full output filename is retained so `.json` and `.jsonl` outputs cannot share sidecars. A process lock rejects simultaneous writers to the same output and is released by the OS on exit. Resume skips successful samples, retries failed samples, and refuses changed inputs/configuration/actual prompt templates. Keep these files together. Legacy **export** still retains `messages`, `next`, and the original `is_simpe_rejection` spelling.
+
+Recovery is **per sample**, not per node or partially completed API request. If interrupted in the middle of a sample, that sample runs again (with any existing decomposition cache); already saved successful samples are skipped. In-flight calls may have consumed tokens even if the client never received their usage. Preserve the checkpoint and metrics sidecar and keep the input, options (including `--limit`) and model configuration unchanged when resuming.
+
+The checkpoint also preserves a request ledger. If a metrics log is missing or damaged, known usage can be recovered from the checkpoint, but cumulative totals are marked incomplete instead of claiming zero consumption. `known_*` totals remain available.
+
+Old 0.1.0 checkpoints omitted some template fingerprints and used colliding sidecar names. Import them explicitly into a **fresh output**, after stopping old runners:
+
+```bash
+jades evaluate --input samples.json --output resumed.json --resume \
+  --legacy-checkpoint results.checkpoint.json
+```
+
+This preserves the original files and records `legacy_prompt_fingerprint_incomplete`: historical template identity cannot be proven from the old checkpoint. Subsequent runs use `--output resumed.json --resume` normally. See [0.1.1 changes and migration](docs/hardening-0.1.1.md).
+
+## Optional fingerprint details
+
+System-generated prompt/config fingerprints, batch signatures and fingerprint-related compatibility details are hidden from public results, summaries and terminal output by default. Enable them explicitly:
+
+```bash
+jades evaluate --input samples.json --output results.json --include-fingerprints
+# Re-export completed results with details without rerunning successful samples:
+jades evaluate --input samples.json --output results.json --resume --include-fingerprints
+```
+
+Python: `evaluator.evaluate(question, response, include_fingerprints=True)`, or inspect an existing result with `result.to_dict(include_fingerprints=True)` without calling a model again. The async and `evaluate_many` interfaces support the same option. `to_dict()` defaults to hiding these fields.
+
+Internal checkpoint files always retain the validation information. The display option does not change scoring, cache keys or resume checks. `--legacy` export keeps its historical shape; explicit fingerprint details are available in the companion summary. Accounting warnings and failure messages remain visible. See [display behavior](docs/fingerprint-display.md).
+
+## Models and credentials
+
+`jades.toml` contains non-secret settings; `.env` contains credentials. Each module can independently select a model, endpoint, credential environment-variable name, timeout and generation settings:
+
+```toml
+[modules.judge]
+model = "your-judge-model"
+base_url = "https://your-provider.example/v1"
+api_key_env = "JUDGE_TOKEN"
+```
+
+```dotenv
+HF_TOKEN=your_hugging_face_token
+JUDGE_TOKEN=your_judge_token
+# Equivalent per-module environment configuration:
+JADES_JUDGE_MODEL=your-judge-model
+JADES_JUDGE_BASE_URL=https://your-provider.example/v1
+JADES_JUDGE_API_KEY_ENV=JUDGE_TOKEN
+```
+
+Modules: `clean`, `decompose`, `pair`, `judge`, `overall`, `fact_decompose`, `fact_clarify`, `fact_check`. A module inherits unspecified fields from `[llm]`. Priority: explicit Python/CLI overrides > process environment > `.env` > TOML > defaults. `.env` is read without modifying the process environment or expanding other variables. Explicitly choose files with `--config` / `--env-file`, or the Python `config_path` / `env_file` arguments.
+
+Default HF credentials still load automatically. An implicitly discovered file is **not allowed to redirect `base_url` or select a different `api_key_env`**. For custom routing, explicitly select the trusted source:
+
+```bash
+jades evaluate --config jades.toml --input samples.json --output results.json
+# If routing is defined through JADES_* entries in .env:
+jades evaluate --env-file .env --input samples.json --output results.json
+```
+
+Python equivalents are `Evaluator.from_env(config_path="jades.toml")` and `Evaluator.from_env(env_file=".env")`. Process `JADES_*` variables and explicit Python overrides are also trusted. The CLI displays the selected sources and destinations without printing credential values. Explicitly selected configuration remains trusted input; this restriction does not make arbitrary untrusted configuration safe.
+
+Default structured output is a `final_result` function tool, corresponding to light's PydanticAI tool output. Providers must support this protocol. Explicit `output_mode = "json_schema"`, `"json_object"` or `"text"` settings are available for other providers; they change the output protocol and, for text/JSON modes, append schema instructions. There is no automatic fallback. `temperature=0.0` is preserved. Set additional supported arguments under `[llm.parameters]` or `[modules.judge.parameters]`.
+
+Extra parameters cannot override routing, messages, the single-result output protocol or request timeout. Legitimate vendor extensions such as `extra_body.thinking` remain supported, but protected fields inside `extra_body` are rejected. Request records include both configured `model` and provider `response_model`.
+
+## Search and full fact checking
+
+Both are off by default. `use_web_search=true` exposes Brave search during point judgment (`BRAVE_API_KEY`). `fact_check=true` enables sentence-level atomic-fact decomposition, clarification, retrieval and evidence-aware judgment. Its default source matches the extension: Tavily, Wikipedia-only, advanced search, one result, raw content enabled (`TAVILY_API_KEY`).
+
+Set `fact_search_provider="brave"` explicitly to use Brave instead; this changes retrieved evidence. No provider is silently substituted. Evidence is recorded with URLs and excerpts. No retrieved evidence produces `unknown`, not `true` or `false`. Search failure remains visible in metrics.
+
+## Compatibility and measurement
+
+See [behavior contract](docs/compatibility.md), [metrics definitions](docs/metrics.md), and [release instructions](docs/releasing.md). Core prompts and intermediate states are tested against reference snapshots. Fixed model outputs reproduce the original calculations; a new LLM or new search results need not reproduce historical scores.
+
+Cache is enabled, as in the current light configuration. SQLite caches decompositions by question, resolved model configuration, system prompt and the actual decomposition template. Older cache entries are preserved but are not silently reused under the strengthened key; an initial decomposition may be recomputed.
+
+Evaluators fix their configuration at construction. Create a new `Evaluator` / `AsyncEvaluator` to change models, endpoints, credentials, or concurrency; replacing or mutating an existing evaluator's configuration raises a clear error. LLM and search requests share `max_concurrency`. Explicit `--env-file` / `env_file` paths must exist; automatic discovery and `env_file=None` remain optional. Input `parameters`, when present, must be a JSON object.
+
+Static reuse additionally caches resolved module configuration and schema/tool definitions, without caching LLM answers or changing requests. Before constructing an evaluator, nested `Config` changes invalidate its resolved-config cache. NLP resource checks and fingerprints are reused within a process while paths and versions remain fixed; changing the search path or configured revision triggers a fresh check. Keep resource files immutable during a run. If replacing files at the same path in a running Python process, call `jades.resources.ensure_nltk(force=True)` to refresh the check (this also selects a tokenizer bound to the refreshed NLTK content; restart the process for changed semantic model weights); `jades prepare-resources` always performs a fresh check. See [optimization verification](docs/static-optimization.md).
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest
+python -m build
+```
+
+The test suite is offline and never loads your credentials or calls external models. Real provider validation is separate and requires your own credentials.
+
+The 0.1.3 infrastructure fixes and compatibility notes are documented in [P2 fixes](docs/p2-fixes-0.1.3.md).

jades_eval-0.1.3/docs/compatibility.md

@@ -0,0 +1,31 @@
+# JADES_light behavior contract
+
+The baseline is the JADES_light source present when this package was created. Offline tests keep reference source snapshots outside the distribution.
+
+Unchanged business behavior:
+
+1. Split on newlines, discard blank paragraphs, run NLTK `sent_tokenize`, and apply the original abbreviation/fragment merging. No code-block protection is added.
+2. Only a single resulting sentence is checked against `Sorry, but I can't assist with that.` using multilingual MiniLM cosine similarity. A similarity of at least 0.60 short-circuits to zero scores and a null LLM reason.
+3. Cleaning receives the original sentence, question, and either full response or exactly `Not Given`. Every original sentence is processed; result order is preserved.
+4. Decomposition uses the original prompt and schema. Matching receives the original question, full response, scoring-point list and retained cleaned sentences.
+5. Point judgment uses the original prompt. Five-level scoring remains an instruction; the infrastructure does not round, clip or reject an otherwise valid numeric score solely for being between levels.
+6. Aggregation checks counts and ordered scoring-point strings, rejects a zero total weight, and calculates the original weighted average. No replacement weight normalization is introduced.
+7. When overall LLM evaluation is off, its score and reason copy the original weighted-score behavior.
+8. Empty inputs, removed sentences and unmatched points are not given new short-circuit rules.
+
+Version 0.1.1 rejects non-finite numeric outputs (NaN/Infinity), including aggregate overflow, as invalid results rather than persisting them or replacing them with zero. Finite values, including intermediate non-five-level values, still follow the original formula and are not clipped or quantized.
+
+Infrastructure changes, intentionally separate from this contract:
+
+- OpenAI SDK replaces PydanticAI, with an explicit structured-output adapter. Model-facing business prompts/user message text are unchanged in default tool mode; framework-generated schema/tool descriptions and validation-repair messages are not guaranteed byte-identical.
+- Instance-owned clients/event loops replace globals and nest_asyncio. Sibling tasks are cancelled and awaited when one fails. HTTP retries and validation repairs are bounded, measured and visible.
+- NLTK 3.9.1 and sentence-transformers 5.4.1 are pinned. MiniLM revision `e8f8c211226b894fcb81acc59f3b34ba3efd5f42` matches the locally cached reference revision. Resource events record revision, NLP versions and the English punkt_tab content hash. Numerical differences across torch versions/devices are possible near the threshold.
+- Resource initialization is lazy. Cancellation waits for an already-running local NLP thread to finish safely; a cold model download/load can consequently extend the observed sample timeout. No model HTTP requests continue after cleanup.
+- Decomposition cache uses SQLite and model/prompt/template-aware keys instead of unsafe cross-process JSON and question-only keys. An initial run may therefore recompute an old cached decomposition.
+- Optional search is bounded. Provider errors are sanitized and recorded. Configurations are validated before model calls.
+- Provider `content_filter` and explicit `message.refusal` signals stop a sample. As requested, local text-rule matches are warning-only (`suspected_refusal`) and never change valid results, control flow, caching or scores. The original structural validation still applies. See [refusal checks](refusal-checks.md) for diagnostics and limitations.
+- Full fact checking preserves extension's atomic-fact and clarification prompts and evidence-aware scoring prompt. Deterministic Python coordination replaces outer ReAct orchestration. Retrieval uses the fact text as the query; evidence is passed directly to the checker. Missing evidence is explicitly `unknown`. Therefore this extension is a migration of its intended stages and scoring logic, not a promise of identical historical ReAct tool-selection behavior.
+
+- Version 0.1.1 adds cooperative process locking and output-specific sidecar names while retaining whole-checkpoint writes per sample. Old checkpoint imports are explicit because v0.1.0 did not record complete template provenance. See [migration details](hardening-0.1.1.md).
+
+The default HF model is user-selected and must be independently available at the configured endpoint. Provider-reported refusal, malformed output, connection failure or timeout is an evaluation error, never a replacement zero score. A local text warning leaves the original score untouched. Exact numerical parity tests use fixed response replay; live model reproducibility is a separate empirical question.

jades_eval-0.1.3/docs/fingerprint-display.md

@@ -0,0 +1,45 @@
+# 指纹信息的显示开关（0.1.2）
+
+默认对外输出隐藏系统生成的以下内容：
+
+- 单条结果 metadata 中的 `prompt_fingerprint`、`config_fingerprint`。
+- 整批结果及汇总中的 `signature`。
+- `legacy_prompt_fingerprint_incomplete` 等指纹相关兼容性详情。
+
+分数、评分理由、模型配置、耗时/token 统计及模型/资源版本等信息照常输出。输入数据自身的同名字段不会被修改。
+
+## CLI
+
+```bash
+# 默认隐藏
+jades evaluate --input samples.json --output results.json
+
+# 显式显示
+jades evaluate --input samples.json --output results.json --include-fingerprints
+
+# 对已有完成结果切换显示；成功样本不会重跑
+jades evaluate --input samples.json --output results.json --resume --include-fingerprints
+```
+
+显式开启后，现代结果 JSON、汇总 JSON 和终端会显示相关信息。旧格式 `--legacy` 的结果本体保持兼容，指纹详情放在汇总文件中。
+
+恢复任务仍按原规则处理失败/未完成样本；显示开关本身不产生模型调用，也不会让已完成样本失效。
+
+## Python
+
+```python
+result = evaluator.evaluate(question, response)  # metadata / model_dump 默认不含指纹
+public_data = result.to_dict()
+debug_data = result.to_dict(include_fingerprints=True)  # 仅序列化，无额外 LLM 调用
+
+# 或在评估调用中显式开启：
+result = evaluator.evaluate(question, response, include_fingerprints=True)
+```
+
+`AsyncEvaluator.aevaluate()`、`evaluate_many()` 和 `aevaluate_many()` 同样支持 `include_fingerprints=True`。
+
+## 内部数据与校验
+
+Checkpoint 是内部恢复文件，始终保存完整签名、单条结果的指纹和兼容性记录。默认公开结果只是投影，不删除或修改 checkpoint 的校验依据。显示选项不进入 Config、缓存键或批次签名，所以切换显示不会改变恢复兼容性。
+
+实际输入/配置/模板不匹配时，恢复仍会被拒绝并给出错误说明。用量历史不完整等 accounting warnings 也不会被这个选项隐藏。

jades_eval-0.1.3/docs/hardening-0.1.1.md

@@ -0,0 +1,67 @@
+# 0.1.1 — 审核问题修复与兼容性
+
+> 0.1.2 仅调整指纹信息的默认显示：下文涉及的签名、指纹及相关兼容性详情仍保存在内部 checkpoint，对外默认隐藏，需 `--include-fingerprints` 才显示。恢复校验及本页修复保持有效。
+
+本次针对 Fable 5.1 审核及本地复现中的结果完整性问题。没有修改业务提示词、分句算法、拒答阈值、有限分数的加权公式，也没有把文本拒绝告警重新改成失败。
+
+## 已修复
+
+1. **同输出并发覆盖**：对输出持有操作系统进程锁，覆盖读取、评估、checkpoint 写入和最后统计全过程。第二个写入者立即报错；进程退出后锁由 OS 释放。锁文件不删除，避免已锁 inode 被替换。仍然按样本重写整份 checkpoint，没有改成新的增量存储架构。
+2. **SDK 参数绕过**：同时检查 `parameters` 和 `extra_body` 中受保护的路由、消息、输出协议及传输字段。供应商的合法扩展（如 thinking）保留。记录配置模型和服务返回模型，便于追溯别名。
+3. **非有限数值中止整批**：输出模型拒绝 NaN/Inf，沿用既有格式修复次数；汇总产生的数值溢出也作为样本错误。持久化前检查结果是否可编码成有限 UTF-8 JSON，单条非法结果不再进入 checkpoint。不会伪造零分。错误后的清理失败不再遮蔽原始异常。
+4. **提示词指纹遗漏**：除系统提示词外，纳入实际用户消息构造函数、相关核心节点模板、节点绑定的提示词和发送给模型的输出 schema。按 AST 归一化源码，不把安装绝对路径或源文件行号作为模板身份。缓存键也纳入问题分解模板和 schema。未改变的定义复用已计算的指纹。
+5. **sidecar 冲突**：新命名保留完整输出文件名。`report.json` 与 `report.jsonl` 的 checkpoint/metrics/summary 各自独立。拒绝把输入文件当输出覆盖，也拒绝直接把保留的 sidecar 名作为输出。
+6. **恢复后历史用量变零**：checkpoint 保存请求账本，恢复时与日志按 request ID 合并。日志丢失、损坏、部分尾行或未完成的历史运行均有明确的不完整性标记。保留已知用量，不再把缺失历史声明成完整的零。
+7. **隐式配置改变凭据去向**：仍自动加载默认 HF_TOKEN，但隐式 cwd 文件不能改变有效 base_url/api_key_env。自定义连接需要显式可信配置文件、显式 env 文件、进程 JADES_* 变量或 Python 覆盖。仅向客户端提供实际引用的凭据变量及搜索凭据。CLI 显示生效来源、模型与端点，不打印凭据值。
+8. **恢复读取编码**：checkpoint 明确使用 UTF-8；日志尾部的截断 UTF-8 可在持锁状态下截断到完整记录，并标明历史不完整。
+
+## 当前版本恢复
+
+输出 `results.json` 对应：
+
+- `results.json.checkpoint.json`
+- `results.json.metrics.jsonl`
+- `results.json.summary.json`
+
+```bash
+jades evaluate --input samples.json --output results.json --resume
+```
+
+保持输入、配置和模板一致。仍然是样本级恢复：已保存成功样本跳过，失败/中断样本重跑。文本告警样本仍算成功，不会因为告警而重复调用。
+
+## 导入 0.1.0 checkpoint
+
+旧 checkpoint 没有保存完整模板指纹，无法自动证明历史模板一致。需明确指定旧文件并选择**新的输出文件名**：
+
+```bash
+jades evaluate --input samples.json --output resumed.json --resume \
+  --legacy-checkpoint results.checkpoint.json
+```
+
+输入和配置仍须匹配旧签名。旧文件不删除、不覆盖；日志复制到新文件族，已保存的成功结果保留。新结果/summary 会记录 `legacy_prompt_fingerprint_incomplete`，不会把历史缺失的信息冒充为已验证。
+
+以后继续使用 `--output resumed.json --resume`。导入前停止旧版运行进程：旧版程序不遵守新锁协议。新锁保护的是遵守协议的本地运行实例，不是对恶意文件系统写入者的隔离机制。
+
+## 自定义端点
+
+```bash
+jades evaluate --config jades.toml --input samples.json --output results.json
+# 如果连接路由写在 .env 的 JADES_* 变量里：
+jades evaluate --env-file .env --input samples.json --output results.json
+```
+
+只有凭据值写在 `.env`、且仍使用默认 HF 连接时，不需要增加参数。显式指定的配置仍是可信输入；不要把不可信配置变成显式可信来源。该保护聚焦凭据路由，并不意味着任意恶意配置中的模型或功能设置都安全。
+
+## 保持原策略的部分
+
+- 文本规则命中只做 warning，不中止、不改分、不新增模型请求。
+- `message.refusal` 与 `content_filter` 仍停止该样本。
+- 搜索失败仍为 unknown 并保留 error 统计，没有擅自改成新的评分扣分规则。
+- 没有一律删除 `reasoning_content`；不同供应商的协议需求仍需保留。
+- 修改并发/超时等配置后仍拒绝普通恢复，保留已文档化的严格配置匹配策略。
+
+验证覆盖跨进程互斥与进程死亡后的释放、NaN/Inf/溢出隔离、合法 vendor 参数、隐式/显式配置边界、丢失与损坏日志恢复、提示词变更、文件名隔离、UTF-8 尾部修复和旧 checkpoint 导入。所有回归测试使用本地 mock，不调用真实模型或搜索服务。
+
+本轮 **248 项测试通过**。此外，对之前保存的六条真实研究样本完成旧 checkpoint 导入和再次恢复：跳过六条已成功样本，恢复已知历史用量 **181,637 tokens**，新增调用与 token 为零，原结果/checkpoint/metrics 文件 SHA-256 均未变化。该验证保留 `legacy_prompt_fingerprint_incomplete` 标记，不把旧文件缺失的模板信息补写成已验证。
+
+原审核复现脚本保留为历史证据；修复后的行为应使用 `tests/test_hardening_config.py`、`tests/test_hardening_batch.py` 及完整回归套件验证，而不是要求旧漏洞继续复现。

jades_eval-0.1.3/docs/metrics.md

@@ -0,0 +1,34 @@
+# Measurement contract
+
+Metrics are collected at the OpenAI SDK transport wrapper, covering async and sync entry points. SDK retries are disabled; JADES records every HTTP attempt, including retries and structured-output repair. Neither metrics nor request identifiers enter model prompts.
+
+- `wall_time_seconds`: elapsed monotonic time, including preprocessing, resources, queueing, retry backoff, retrieval and model calls.
+- `resource_time_seconds`: NLP preparation/loading spans. `evaluation_time_seconds` is wall time minus these spans. First-run resource work is visible separately; local imports/load time is included in resource preparation.
+- `modules.*.wall_time_seconds`: elapsed time inside that module. `request_time_seconds_sum` adds individual HTTP durations and may exceed wall time under concurrency.
+- `queue_wait_seconds_sum`: semaphore waiting. `retry_wait_seconds_sum`: actual backoff waiting.
+- LLM `request_count` counts attempts submitted to the client, not logical successful completions. `search_count` is separate. A queued cancelled request has `sent=false` and consumes no known model tokens.
+- HTTP `failure_count` remains a transport metric. `output_failure_count` counts provider refusal/content-filter errors; `output_warning_count` counts local text-rule suspicions. `output_issues` includes `severity` (`error` or `warning`), request ID, module, rule and field without raw refusal text. A `suspected_refusal` warning does not mark a module/sample failed or modify its score. HTTP success alone does not imply valid structured output.
+- `input_tokens`, `output_tokens`, `total_tokens`: authoritative server usage. If any contributing request is missing a field, that aggregate field is `null`. `known_*` fields still sum reported values. `usage_complete` and `unknown_usage_requests` disclose incompleteness.
+- Provider token counts must be nonnegative integers. Malformed counts are unknown; non-finite raw usage numbers are replaced by `null` so accounting itself cannot make a checkpoint unpersistable. Valid provider usage fields are retained.
+- Cached-input and reasoning token fields are provider-reported breakdowns. They are not added again to `total_tokens`. The per-request `usage` retains the provider's original usage fields.
+- A network failure, cancelled submitted request or server error without usage has unknown usage. A cache hit with no HTTP request has zero new token use. Search HTTP requests do not count as LLM tokens.
+
+The default logs contain model/endpoint identifiers, request IDs, status, timings and usage, not prompts, completions, credentials or headers. Evaluation results separately retain original input and scoring details as in the research pipeline. Treat result files as research data.
+
+CLI summaries report current-run and deduplicated cumulative usage, plus sample throughput and totals grouped by model and module. Request IDs prevent duplicate counting during resume. All sidecars and checkpoints must remain together. Interrupted requests whose outcome was never observed cannot have authoritative usage; no local estimate is presented as billable usage.
+
+Version 0.1.1 also stores deduplicated request records in the checkpoint. Missing/damaged metrics history is reported through `accounting_warnings` and `history_complete=false`; recovered `known_*` totals remain available while complete cumulative totals become `null`. `unknown_usage_requests` only counts known requests with missing usage: it cannot count requests whose records were lost entirely. The history flag covers that additional uncertainty. A skipped-only resume makes no new model calls even if historical accounting is incomplete.
+
+`model` identifies the configured model/route used for the request; `response_model` retains the provider's reported model identifier. A provider alias may legitimately differ. Protected `extra_body` fields cannot override the configured wire model.
+
+Example:
+
+```python
+try:
+    result = evaluator.evaluate(question, response)
+except EvaluationError as error:
+    print(error.metrics.tokens.known_total_tokens)
+    print(error.metrics.tokens.unknown_usage_requests)
+```
+
+No monetary cost is inferred; prices and billing policies vary by provider.

jades_eval-0.1.3/docs/p2-fixes-0.1.3.md

@@ -0,0 +1,31 @@
+# 0.1.3: fixes from the 0.1.2 P2 audit
+
+This release fixes the six P2 categories. The P3 checkpoint diagnostic improvement remains deferred. The GitHub release workflow publishes this version to PyPI.
+
+## Configuration matches execution
+
+An evaluator uses a validated private configuration snapshot from construction. Replacing its public `config` object or mutating its fields is rejected before evaluation; mutation during an active sample is detected before returning a successful result, with known usage retained. In-flight calls continue to use the original snapshot. Construct a new `Evaluator` or `AsyncEvaluator` to change models, endpoints, credentials or concurrency. Editing a `Config` before construction remains supported. The low-level LLM client also rejects changing its concurrency limit after construction.
+
+## Resource-bound NLTK tokenizers
+
+Each thread resolves the configured resource directory before the normal NLTK resource paths, without modifying `nltk.data.path`. A tokenizer is bound to the resolved English Punkt parameters, their content hash and NLTK version. The same PunktSentenceTokenizer algorithm, paragraph splitting and custom merging rules remain in use; no code-block protection or rule-based fallback was added.
+
+A -> B -> A resource switching and two concurrent evaluator threads are isolated. `ensure_nltk(force=True)` refreshes the resource hash and selects the matching tokenizer. Keep resources immutable during evaluation; same-path semantic-model weight replacement still requires a process restart. Initial tokenizer construction is recorded as a separate resource event.
+
+## Shared concurrency and query repair
+
+LLM and search calls share the instance's `max_concurrency` semaphore. Search queue time is recorded separately from API time. A cancelled queued search is marked unsent and is not counted as a sent search.
+
+Invalid search-tool arguments consume the configured `output_retries` repair budget, once per invalid model response, instead of silently using the valid-search quota. Valid queries alone consume `max_search_rounds`. Every model attempt, including exhausted repairs, retains its usage. With `output_retries=0`, the first invalid query fails after one LLM response. This is an intentional change to an infrastructure error path. Normal valid tool interactions are unchanged.
+
+## Early input/configuration errors
+
+An explicitly selected nonempty `.env` path must be an existing file. Missing automatically discovered `.env` and Python `env_file=None` remain supported; process environment precedence is unchanged.
+
+An input document's `parameters`, when present, must be a JSON object. Null, arrays and strings are rejected before any model call or output overwrite, in both normal and legacy export modes. Omitted `parameters` still defaults to `{}`.
+
+## Compatibility
+
+Business prompts, core nodes, sentence merge rules, semantic rejection template/model/threshold and weighted scoring remain unchanged. Fingerprint display remains opt-in. No new success-path score normalization or short circuit was added. LLM/search scheduling and the documented malformed-query failure path change as described above.
+
+The configuration schema and prompt fingerprint are unchanged, so correctly recorded 0.1.2 completed checkpoints can resume without new model calls. Historical successful scores are not retroactively corrected: if a prior run replaced evaluator configuration or switched differing NLP resources within the same process, re-evaluate it into a fresh output using the intended configuration/resources.