contexttrace 0.8.0__tar.gz → 0.9.0__tar.gz

{contexttrace-0.8.0 → contexttrace-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: contexttrace
-Version: 0.8.0
+Version: 0.9.0
 Summary: Local-first SDK and CLI for RAG and agent reliability tracing, citation checks, and failure diagnosis.
 Author: ContextTrace contributors
 License: MIT
@@ -34,6 +34,12 @@ Requires-Dist: llama-index-core>=0.10; extra == "llamaindex"
 Provides-Extra: local
 Provides-Extra: local-ml
 Requires-Dist: sentence-transformers>=2.7; extra == "local-ml"
+Provides-Extra: nli
+Requires-Dist: torch>=2.0; extra == "nli"
+Requires-Dist: transformers>=4.41; extra == "nli"
+Provides-Extra: nli-onnx
+Requires-Dist: onnxruntime>=1.17; extra == "nli-onnx"
+Requires-Dist: transformers>=4.41; extra == "nli-onnx"
 Provides-Extra: fastapi
 Requires-Dist: fastapi>=0.110; extra == "fastapi"
 Provides-Extra: langgraph
@@ -55,6 +61,9 @@ Requires-Dist: langgraph>=0.2; extra == "all"
 Requires-Dist: llama-index-core>=0.10; extra == "all"
 Requires-Dist: opentelemetry-api>=1.24; extra == "all"
 Requires-Dist: sentence-transformers>=2.7; extra == "all"
+Requires-Dist: torch>=2.0; extra == "all"
+Requires-Dist: transformers>=4.41; extra == "all"
+Requires-Dist: onnxruntime>=1.17; extra == "all"
 Provides-Extra: test
 Requires-Dist: pytest>=8.0; extra == "test"
 
@@ -62,7 +71,7 @@ Requires-Dist: pytest>=8.0; extra == "test"
 
 **Local-first evidence-chain debugging for RAG and AI agents.**
 
-ContextTrace shows where an answer stopped being grounded:
+ContextTrace shows where an answer stopped being grounded in the evidence you gave it:
 
 ```text
 query -> retrieved context -> answer claims -> citations -> verdicts -> root cause
@@ -116,7 +125,9 @@ contexttrace verify trace.json --report
 contexttrace qa trace.json --corpus docs/ --report
 ```
 
-ContextTrace classifies each claim as `supported`, `partially_supported`, `unsupported`, `unverifiable`, or `contradicted`, then explains the likely fix.
+ContextTrace classifies each claim as `supported`, `partially_supported`, `unsupported`, `unverifiable`, or `contradicted`, then exposes separate statuses for support, truth, source freshness, citation quality, and likely fix.
+
+Important: `supported` means grounded by the selected evidence span. It does not mean independently true, current, or authoritative.
 
 ## Local Verification Modes
 
@@ -125,7 +136,8 @@ ContextTrace classifies each claim as `supported`, `partially_supported`, `unsup
 | `lexical` | Fast default checks with no optional dependencies. |
 | `semantic` | Local paraphrase and role-aware contradiction checks. |
 | `local_ml` | Offline hash-embedding similarity, optionally backed by a local SentenceTransformers model. |
-| `judge` | Higher-accuracy local LLM judging through Ollama, LM Studio, vLLM, or a local OpenAI-compatible server. |
+| `nli` | Local claim+span entailment or contradiction with a local Transformers or ONNX NLI model. |
+| `judge` | Higher-accuracy local LLM judging through Ollama, LM Studio, vLLM, or a local OpenAI-compatible server. The judge sees selected evidence spans, not the full answer prose. |
 
 Run the stronger local non-LLM verifier:
 
@@ -138,7 +150,16 @@ Optional neural local-ML support never downloads models automatically:
 
 ```bash
 pip install "contexttrace[local-ml]"
-set CONTEXTTRACE_LOCAL_ML_MODEL_PATH=C:\models\bge-small-en-v1.5
+set CONTEXTTRACE_LOCAL_ML_MODEL_PATH=C:/models/bge-small-en-v1.5
+```
+
+Run local NLI when you want mechanical claim-versus-span entailment:
+
+```bash
+pip install "contexttrace[nli]"
+set CONTEXTTRACE_NLI_MODEL_PATH=C:/models/deberta-v3-nli
+contexttrace verify trace.json --mode nli --report
+contexttrace nli-calibrate --case-set all --report
 ```
 
 Run a local judge with Ollama:
@@ -169,6 +190,10 @@ contexttrace suite run contexttrace-suite.json --endpoint http://localhost:8000/
 
 Common root causes include `retrieval_miss`, `reranking_failure`, `chunking_issue`, `corpus_gap`, `answer_overreach`, `stale_source`, `citation_mismatch`, and `should_have_abstained`.
 
+`support_status`, `truth_status`, and `source_status` stay separate so a claim can be grounded by a source while the source itself remains stale, wrong, or unassessed.
+
+Source metadata can include `source_authority`, `source_timestamp`, `source_version`, `canonical`, or `canonical_source`. ContextTrace uses those local fields to flag `grounded_but_stale`, `grounded_but_conflicted`, `grounded_by_low_authority_source`, or `supported_by_canonical_source`.
+
 ## Capture Existing Systems
 
 Capture one live endpoint response:
@@ -247,7 +272,7 @@ ContextTrace makes no network calls unless you point it at an endpoint or config
 
 ## Limits
 
-ContextTrace is a diagnostic tool, not a correctness proof. Claim extraction is rule-based, contradiction detection is conservative, and high-stakes outputs still need human review.
+ContextTrace is a diagnostic tool, not a correctness proof. It verifies grounding against provided evidence; it does not certify real-world truth. Claim extraction is rule-based, contradiction detection is conservative, and high-stakes outputs still need human review.
 
 ## Links
 

{contexttrace-0.8.0 → contexttrace-0.9.0}/README.md

@@ -2,7 +2,7 @@
 
 **Local-first evidence-chain debugging for RAG and AI agents.**
 
-ContextTrace shows where an answer stopped being grounded:
+ContextTrace shows where an answer stopped being grounded in the evidence you gave it:
 
 ```text
 query -> retrieved context -> answer claims -> citations -> verdicts -> root cause
@@ -56,7 +56,9 @@ contexttrace verify trace.json --report
 contexttrace qa trace.json --corpus docs/ --report
 ```
 
-ContextTrace classifies each claim as `supported`, `partially_supported`, `unsupported`, `unverifiable`, or `contradicted`, then explains the likely fix.
+ContextTrace classifies each claim as `supported`, `partially_supported`, `unsupported`, `unverifiable`, or `contradicted`, then exposes separate statuses for support, truth, source freshness, citation quality, and likely fix.
+
+Important: `supported` means grounded by the selected evidence span. It does not mean independently true, current, or authoritative.
 
 ## Local Verification Modes
 
@@ -65,7 +67,8 @@ ContextTrace classifies each claim as `supported`, `partially_supported`, `unsup
 | `lexical` | Fast default checks with no optional dependencies. |
 | `semantic` | Local paraphrase and role-aware contradiction checks. |
 | `local_ml` | Offline hash-embedding similarity, optionally backed by a local SentenceTransformers model. |
-| `judge` | Higher-accuracy local LLM judging through Ollama, LM Studio, vLLM, or a local OpenAI-compatible server. |
+| `nli` | Local claim+span entailment or contradiction with a local Transformers or ONNX NLI model. |
+| `judge` | Higher-accuracy local LLM judging through Ollama, LM Studio, vLLM, or a local OpenAI-compatible server. The judge sees selected evidence spans, not the full answer prose. |
 
 Run the stronger local non-LLM verifier:
 
@@ -78,7 +81,16 @@ Optional neural local-ML support never downloads models automatically:
 
 ```bash
 pip install "contexttrace[local-ml]"
-set CONTEXTTRACE_LOCAL_ML_MODEL_PATH=C:\models\bge-small-en-v1.5
+set CONTEXTTRACE_LOCAL_ML_MODEL_PATH=C:/models/bge-small-en-v1.5
+```
+
+Run local NLI when you want mechanical claim-versus-span entailment:
+
+```bash
+pip install "contexttrace[nli]"
+set CONTEXTTRACE_NLI_MODEL_PATH=C:/models/deberta-v3-nli
+contexttrace verify trace.json --mode nli --report
+contexttrace nli-calibrate --case-set all --report
 ```
 
 Run a local judge with Ollama:
@@ -109,6 +121,10 @@ contexttrace suite run contexttrace-suite.json --endpoint http://localhost:8000/
 
 Common root causes include `retrieval_miss`, `reranking_failure`, `chunking_issue`, `corpus_gap`, `answer_overreach`, `stale_source`, `citation_mismatch`, and `should_have_abstained`.
 
+`support_status`, `truth_status`, and `source_status` stay separate so a claim can be grounded by a source while the source itself remains stale, wrong, or unassessed.
+
+Source metadata can include `source_authority`, `source_timestamp`, `source_version`, `canonical`, or `canonical_source`. ContextTrace uses those local fields to flag `grounded_but_stale`, `grounded_but_conflicted`, `grounded_by_low_authority_source`, or `supported_by_canonical_source`.
+
 ## Capture Existing Systems
 
 Capture one live endpoint response:
@@ -187,7 +203,7 @@ ContextTrace makes no network calls unless you point it at an endpoint or config
 
 ## Limits
 
-ContextTrace is a diagnostic tool, not a correctness proof. Claim extraction is rule-based, contradiction detection is conservative, and high-stakes outputs still need human review.
+ContextTrace is a diagnostic tool, not a correctness proof. It verifies grounding against provided evidence; it does not certify real-world truth. Claim extraction is rule-based, contradiction detection is conservative, and high-stakes outputs still need human review.
 
 ## Links
 

contexttrace-0.9.0/contexttrace/_version.py

@@ -0,0 +1 @@
+__version__ = "0.9.0"

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/cli.py

@@ -31,6 +31,7 @@ from contexttrace.verify import (
     audit_trace,
     build_judge_provider,
     run_judge_calibration,
+    run_nli_calibration,
     compare_failures,
     compare_trace_files,
     list_verify_demos,
@@ -38,6 +39,7 @@ from contexttrace.verify import (
     load_verify_demo,
     verify_trace,
     write_judge_calibration_report,
+    write_nli_calibration_report,
 )
 from contexttrace.verify.benchmark import run_verify_benchmark, write_verify_benchmark_report
 from contexttrace.verify.audit_benchmark import run_audit_benchmark, write_audit_benchmark_report
@@ -64,13 +66,16 @@ from contexttrace.verify.trace_inspect import inspect_trace
 from contexttrace.viewer import serve_viewer
 
 
-SAMPLE_QUESTIONS = [
-    {
-        "id": "refund_policy",
-        "query": "What is the refund policy?",
-        "expected_sources": ["refund_policy.md"],
-    }
-]
+SAMPLE_QUESTIONS = [
+    {
+        "id": "refund_policy",
+        "query": "What is the refund policy?",
+        "expected_sources": ["refund_policy.md"],
+    }
+]
+
+BASIC_VERIFY_MODES = ["lexical", "semantic", "local_ml", "local-ml", "nli"]
+JUDGE_VERIFY_MODES = [*BASIC_VERIFY_MODES, "judge"]
 
 
 @click.group(context_settings={"help_option_names": ["-h", "--help"]})
@@ -244,7 +249,7 @@ def report(
 @click.option("--json", "json_output", is_flag=True, help="Print the full verification result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML verification report.")
 @click.option("--out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml", "judge"]), help="Evidence scoring mode.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(JUDGE_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--judge-provider", default=None, help="Judge provider for --mode judge, for example ollama or local_openai.")
 @click.option("--judge-base-url", default=None, help="Judge base URL. Defaults are local for ollama/local_openai/lmstudio/vllm.")
 @click.option("--judge-api-key", default=None, help="Judge API key. Optional for local providers; prefer CONTEXTTRACE_JUDGE_API_KEY in CI.")
@@ -354,7 +359,7 @@ def inspect_command(trace_json: str, json_output: bool) -> int:
 @click.option("--json", "json_output", is_flag=True, help="Print the full QA result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML evidence QA report.")
 @click.option("--out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--fail-on", multiple=True, help="Fail on high_risk, medium_risk, any_risk, unsupported, should_abstain, audit_failure, or inspect_warning.")
 def qa_command(
     trace_json: str,
@@ -429,7 +434,7 @@ def suite_group() -> None:
 @click.argument("trace_json", nargs=-1, required=True)
 @click.option("--out", default="contexttrace-suite.json", show_default=True, help="Suite JSON file to write.")
 @click.option("--name", default=None, help="Suite name.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode for baseline QA.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode for baseline QA.")
 @click.option("--corpus", "corpus_path", default=None, help="Optional local corpus directory or file for baseline retrieval/corpus audit.")
 def suite_create_command(
     trace_json: tuple[str, ...],
@@ -461,7 +466,7 @@ def suite_create_command(
 @click.argument("suite_json")
 @click.argument("trace_json", nargs=-1, required=True)
 @click.option("--out", default=None, help="Suite JSON file to write. Defaults to overwriting suite_json.")
-@click.option("--mode", default=None, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode for added baselines. Defaults to the suite mode.")
+@click.option("--mode", default=None, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode for added baselines. Defaults to the suite mode.")
 @click.option("--corpus", "corpus_path", default=None, help="Optional local corpus directory or file for baseline retrieval/corpus audit.")
 @click.option("--replace", is_flag=True, help="Replace existing cases with the same generated case IDs.")
 def suite_add_command(
@@ -600,7 +605,7 @@ def suite_prune_command(
 @click.option("--json", "json_output", is_flag=True, help="Print the full suite result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML suite report.")
 @click.option("--report-out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default=None, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode. Defaults to the suite mode.")
+@click.option("--mode", default=None, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode. Defaults to the suite mode.")
 @click.option("--fail-on", multiple=True, help="Fail on failed_case, regression, unsupported, should_abstain, high_risk, medium_risk, error, or any_failure.")
 @click.pass_context
 def suite_run_command(
@@ -711,7 +716,7 @@ def suite_report_command(results_json: str, out: Optional[str]) -> int:
 @click.option("--json", "json_output", is_flag=True, help="Print the full verification result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML verification report.")
 @click.option("--out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml", "judge"]), help="Evidence scoring mode.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(JUDGE_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--judge-provider", default=None, help="Judge provider for --mode judge, for example ollama or local_openai.")
 @click.option("--judge-base-url", default=None, help="Judge base URL. Defaults are local for ollama/local_openai/lmstudio/vllm.")
 @click.option("--judge-api-key", default=None, help="Judge API key. Optional for local providers; prefer CONTEXTTRACE_JUDGE_API_KEY in CI.")
@@ -769,7 +774,7 @@ def verify_demo_command(
 
 
 @cli.command("verify-benchmark")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml", "judge"]), help="Evidence scoring mode.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(JUDGE_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--case-set", default="contexttrace", show_default=True, type=click.Choice(["contexttrace", "external", "all"]), help="Benchmark case set to run.")
 @click.option("--json", "json_output", is_flag=True, help="Print benchmark results as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML benchmark report.")
@@ -926,8 +931,95 @@ def judge_calibrate_command(
     return 1 if result.get("failures") else 0
 
 
+@cli.command("nli-calibrate")
+@click.option("--case-set", default="all", show_default=True, type=click.Choice(["contexttrace", "external", "all"]), help="Golden benchmark case set to run.")
+@click.option("--model-path", default=None, help="Local NLI model directory or .onnx file. Defaults to CONTEXTTRACE_NLI_MODEL_PATH.")
+@click.option("--tokenizer-path", default=None, help="Optional local tokenizer directory. Defaults to model path.")
+@click.option("--backend", default="auto", show_default=True, type=click.Choice(["auto", "transformers", "onnx"]), help="Local NLI backend.")
+@click.option("--max-length", default=512, show_default=True, type=int, help="Maximum NLI input token length.")
+@click.option("--json", "json_output", is_flag=True, help="Print calibration result as JSON.")
+@click.option("--report", is_flag=True, help="Generate a local HTML calibration report.")
+@click.option("--out", default=None, help="HTML calibration report path. Implies --report when provided.")
+@click.option("--min-exact-match-rate", default=0.8, show_default=True, type=float, help="Minimum acceptable failure-label exact match rate.")
+@click.option("--min-entailment-precision", default=0.85, show_default=True, type=float, help="Minimum acceptable supported-claim precision.")
+@click.option("--min-contradiction-recall", default=0.7, show_default=True, type=float, help="Minimum acceptable contradiction recall.")
+@click.option("--max-dangerous-miss-rate", default=0.05, show_default=True, type=float, help="Maximum allowed rate of risky cases predicted as no failure.")
+@click.option("--max-p95-latency-ms", default=0.0, show_default=True, type=float, help="Maximum p95 case latency in milliseconds. Use 0 to disable.")
+def nli_calibrate_command(
+    case_set: str,
+    model_path: Optional[str],
+    tokenizer_path: Optional[str],
+    backend: str,
+    max_length: int,
+    json_output: bool,
+    report: bool,
+    out: Optional[str],
+    min_exact_match_rate: float,
+    min_entailment_precision: float,
+    min_contradiction_recall: float,
+    max_dangerous_miss_rate: float,
+    max_p95_latency_ms: float,
+) -> int:
+    """Calibrate a local NLI model against golden RAG failure cases."""
+
+    result = run_nli_calibration(
+        model_path=model_path,
+        tokenizer_path=tokenizer_path,
+        backend=backend,
+        max_length=max_length,
+        case_set=case_set,
+        min_exact_match_rate=min_exact_match_rate,
+        min_entailment_precision=min_entailment_precision,
+        min_contradiction_recall=min_contradiction_recall,
+        max_dangerous_miss_rate=max_dangerous_miss_rate,
+        max_p95_latency_ms=max_p95_latency_ms,
+    )
+    written_report = None
+    if report or out:
+        nli_name = _safe_filename("%s_%s" % (
+            (result.get("nli") or {}).get("backend") or "nli",
+            (result.get("nli") or {}).get("model") or "model",
+        ))
+        output_path = out or str(Path(".contexttrace") / "reports" / ("nli_calibration_%s.html" % nli_name))
+        written_report = write_nli_calibration_report(result, path=output_path)
+    if json_output:
+        if written_report:
+            click.echo("Report: %s" % written_report, err=True)
+        click.echo(json.dumps(result, indent=2))
+        for failure in result.get("failures") or []:
+            click.echo("NLI calibration failed: %s" % failure, err=True)
+        return 1 if result.get("failures") else 0
+
+    scorecard = result["scorecard"]
+    click.echo("Status: %s" % result["status"])
+    click.echo("NLI: %s" % json.dumps(result.get("nli") or {}, sort_keys=True))
+    click.echo("Cases: %s" % result["cases"])
+    click.echo("Exact match rate: %.3f" % float(scorecard["exact_match_rate"]))
+    click.echo("Verdict match rate: %.3f" % float(scorecard["verdict_match_rate"]))
+    click.echo("Citation match rate: %.3f" % float(scorecard["citation_match_rate"]))
+    click.echo("Entailment precision: %.3f" % float(scorecard["entailment_precision"]))
+    click.echo("Entailment recall: %.3f" % float(scorecard["entailment_recall"]))
+    click.echo("Contradiction recall: %.3f" % float(scorecard["contradiction_recall"]))
+    click.echo("Unsupported-like recall: %.3f" % float(scorecard["unsupported_like_recall"]))
+    click.echo("Dangerous miss rate: %.3f" % float(scorecard["dangerous_miss_rate"]))
+    click.echo("Unsupported marked supported: %s" % int(scorecard["unsupported_as_supported_cases"]))
+    click.echo("Case latency p50/p95 ms: %.3f / %.3f" % (
+        float(scorecard["case_latency_p50_ms"]),
+        float(scorecard["case_latency_p95_ms"]),
+    ))
+    click.echo("NLI call latency p50/p95 ms: %.3f / %.3f" % (
+        float(scorecard["nli_call_latency_p50_ms"]),
+        float(scorecard["nli_call_latency_p95_ms"]),
+    ))
+    if written_report:
+        click.echo("Report: %s" % written_report)
+    for failure in result.get("failures") or []:
+        click.echo("NLI calibration failed: %s" % failure, err=True)
+    return 1 if result.get("failures") else 0
+
+
 @cli.command("audit-benchmark")
-@click.option("--mode", default="semantic", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode.")
+@click.option("--mode", default="semantic", show_default=True, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--case-set", default="real", show_default=True, type=click.Choice(["real"]), help="Benchmark case set to run.")
 @click.option("--json", "json_output", is_flag=True, help="Print audit benchmark results as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML audit benchmark report.")
@@ -987,7 +1079,7 @@ def audit_benchmark_command(mode: str, case_set: str, json_output: bool, report:
 @click.option("--json", "json_output", is_flag=True, help="Print the full comparison result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML regression report.")
 @click.option("--out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode for raw trace inputs.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode for raw trace inputs.")
 @click.option("--fail-on", multiple=True, help="Fail on new_failure, new_unsupported, new_citation_mismatch, should_abstain_flip, support_rate_drop, new_root_cause, or any_regression.")
 def compare_command(
     baseline_json: str,
@@ -1048,7 +1140,7 @@ def compare_command(
 @click.option("--json", "json_output", is_flag=True, help="Print the full audit result as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML retrieval audit report.")
 @click.option("--out", default=None, help="HTML report path. Implies --report when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml"]), help="Evidence scoring mode.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(BASIC_VERIFY_MODES), help="Evidence scoring mode.")
 @click.option("--fail-on", multiple=True, help="Fail on retrieval_miss, reranking_failure, chunking_issue, corpus_gap, answer_overreach, stale_source, insufficient_context, or any_failure.")
 def audit_command(
     trace_json: str,
@@ -1132,7 +1224,7 @@ def capture_group() -> None:
 @click.option("--json", "json_output", is_flag=True, help="Print capture output as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML verification report. Implies --verify.")
 @click.option("--report-out", default=None, help="HTML report path. Implies --report and --verify when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml", "judge"]), help="Evidence scoring mode when verifying.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(JUDGE_VERIFY_MODES), help="Evidence scoring mode when verifying.")
 @click.option("--judge-provider", default=None, help="Judge provider for --mode judge, for example ollama or local_openai.")
 @click.option("--judge-base-url", default=None, help="Judge base URL. Defaults are local for ollama/local_openai/lmstudio/vllm.")
 @click.option("--judge-api-key", default=None, help="Judge API key. Optional for local providers; prefer CONTEXTTRACE_JUDGE_API_KEY in CI.")
@@ -1221,7 +1313,7 @@ def capture_endpoint_command(
 @click.option("--json", "json_output", is_flag=True, help="Print capture output as JSON.")
 @click.option("--report", is_flag=True, help="Generate a local HTML verification report. Implies --verify.")
 @click.option("--report-out", default=None, help="HTML report path. Implies --report and --verify when provided.")
-@click.option("--mode", default="lexical", show_default=True, type=click.Choice(["lexical", "semantic", "local_ml", "local-ml", "judge"]), help="Evidence scoring mode when verifying.")
+@click.option("--mode", default="lexical", show_default=True, type=click.Choice(JUDGE_VERIFY_MODES), help="Evidence scoring mode when verifying.")
 @click.option("--judge-provider", default=None, help="Judge provider for --mode judge, for example ollama or local_openai.")
 @click.option("--judge-base-url", default=None, help="Judge base URL. Defaults are local for ollama/local_openai/lmstudio/vllm.")
 @click.option("--judge-api-key", default=None, help="Judge API key. Optional for local providers; prefer CONTEXTTRACE_JUDGE_API_KEY in CI.")

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/verify/__init__.py

@@ -28,7 +28,37 @@ from contexttrace.verify.calibration import (
     run_judge_calibration,
     write_judge_calibration_report,
 )
+from contexttrace.verify.nli_calibration import (
+    nli_calibration_failures,
+    run_nli_calibration,
+    write_nli_calibration_report,
+)
 from contexttrace.verify.local_ml import LocalMLError, local_ml_similarity
+from contexttrace.verify.local_nli import (
+    LocalNLIError,
+    LocalNLIJudge,
+    NLIResult,
+    build_nli_provider,
+    local_nli_entailment,
+)
+from contexttrace.verify.statuses import (
+    SOURCE_FRESHNESS_UNKNOWN,
+    TRUTH_NOT_ASSESSED,
+    attach_grounding_statuses,
+    source_status,
+    status_note,
+    support_status,
+    truth_status,
+)
+from contexttrace.verify.source_trust import (
+    GROUNDED_BUT_CONFLICTED,
+    GROUNDED_BUT_STALE,
+    GROUNDED_BY_LOW_AUTHORITY_SOURCE,
+    SUPPORTED_BY_CANONICAL_SOURCE,
+    attach_source_assessments,
+    source_assessment,
+    source_status_from_assessment,
+)
 
 __all__ = [
     "CachedJudge",
@@ -37,17 +67,29 @@ __all__ = [
     "JudgeError",
     "JudgeVerdict",
     "LocalMLError",
+    "LocalNLIError",
+    "LocalNLIJudge",
+    "NLIResult",
     "OllamaJudge",
     "OpenAICompatibleJudge",
     "RAGTrace",
+    "SOURCE_FRESHNESS_UNKNOWN",
+    "GROUNDED_BUT_CONFLICTED",
+    "GROUNDED_BUT_STALE",
+    "GROUNDED_BY_LOW_AUTHORITY_SOURCE",
+    "SUPPORTED_BY_CANONICAL_SOURCE",
+    "TRUTH_NOT_ASSESSED",
     "TraceCitation",
     "TraceContext",
     "VerificationInputError",
     "audit_failures",
-    "audit_trace",
+    "audit_trace",
     "audit_trace_file",
     "audit_trace_with_corpus",
+    "attach_grounding_statuses",
+    "attach_source_assessments",
     "build_judge_provider",
+    "build_nli_provider",
     "calibration_failures",
     "compare_failures",
     "compare_trace_files",
@@ -59,11 +101,21 @@ __all__ = [
     "load_trace_file",
     "load_verify_demo",
     "local_ml_similarity",
+    "local_nli_entailment",
     "qa_failures",
     "qa_trace",
     "run_judge_calibration",
+    "run_nli_calibration",
     "run_audit_benchmark",
+    "nli_calibration_failures",
+    "source_status",
+    "source_assessment",
+    "source_status_from_assessment",
+    "status_note",
+    "support_status",
+    "truth_status",
     "verify_trace",
     "verify_trace_file",
     "write_judge_calibration_report",
+    "write_nli_calibration_report",
 ]

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/verify/benchmark.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import json
+import time
 from dataclasses import dataclass
 from html import escape
 from pathlib import Path
@@ -47,6 +48,8 @@ def run_verify_benchmark(
     mode: str = "lexical",
     case_set: str = "contexttrace",
     judge: ClaimJudge | None = None,
+    nli: ClaimJudge | None = None,
+    time_cases: bool = False,
 ) -> dict[str, Any]:
     rows = []
     labels = set()
@@ -58,7 +61,9 @@ def run_verify_benchmark(
     abstention_expected = 0
 
     for case in benchmark_cases(case_set=case_set):
-        result = verify_trace(case.trace, mode=mode, judge=judge)
+        started = time.perf_counter()
+        result = verify_trace(case.trace, mode=mode, judge=judge, nli=nli)
+        latency_ms = round((time.perf_counter() - started) * 1000, 3)
         predicted = _predicted_labels(result)
         expected_verdict_counts = dict(case.expected_verdict_counts)
         predicted_verdict_counts = {
@@ -112,6 +117,7 @@ def run_verify_benchmark(
                 "summary": result.get("summary") or {},
                 "claims": result.get("claims") or [],
                 "abstention": result.get("abstention") or {},
+                **({"latency_ms": latency_ms} if time_cases else {}),
             }
         )
 

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/verify/citations.py

@@ -82,7 +82,8 @@ def find_citation_for_claim(
 
 
 def _source_fully_supports_claim(claim_text: str, match: object, *, mode: str) -> bool:
-    fact_mode = "semantic" if mode == "semantic" else "lexical"
+    normalized_mode = str(mode or "").strip().lower().replace("-", "_")
+    fact_mode = "semantic" if normalized_mode in {"semantic", "local_ml", "nli"} else "lexical"
     fact_match = compare_facts(
         claim_text,
         str(getattr(match, "supporting_text", "") or getattr(match, "snippet", "")),

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/verify/evidence.py

@@ -209,10 +209,11 @@ def score_claim_against_context(
     spans = split_context_spans(context)
     best_score = 0.0
     best_terms: list[str] = []
-    best_snippet = context.text.strip()
-    best_start: int | None = None
-    best_end: int | None = None
-    best_hash: str | None = None
+    fallback_span = spans[0] if spans else None
+    best_snippet = fallback_span.text.strip() if fallback_span is not None else context.text.strip()
+    best_start: int | None = fallback_span.start_char if fallback_span is not None else None
+    best_end: int | None = fallback_span.end_char if fallback_span is not None else None
+    best_hash: str | None = fallback_span.span_hash if fallback_span is not None else None
     span_candidates: list[dict[str, object]] = []
     for span in spans:
         score, terms = lexical_score(claim_text, span.text, mode=mode)
@@ -395,7 +396,7 @@ def _local_ml_enabled(mode: str) -> bool:
 
 
 def _semantic_enabled(mode: str) -> bool:
-    return str(mode or "").strip().lower().replace("-", "_") in {"semantic", "local_ml"}
+    return str(mode or "").strip().lower().replace("-", "_") in {"semantic", "local_ml", "nli"}
 
 
 def _token_mode(mode: str) -> str:

{contexttrace-0.8.0 → contexttrace-0.9.0}/contexttrace/verify/judges.py

@@ -35,11 +35,13 @@ DEFAULT_OPENAI_MODEL = "gpt-4.1-mini"
 DEFAULT_JUDGE_CACHE_PATH = ".contexttrace/judge_cache.json"
 JUDGE_SYSTEM_PROMPT = (
     "You are a strict RAG evidence judge. Given a user query, one generated claim, "
-    "and retrieved evidence contexts, return only JSON with this schema: "
+    "and selected evidence spans, return only JSON with this schema: "
     "{\"verdict\":\"supported|partially_supported|unsupported|contradicted|unverifiable\","
     "\"confidence\":0.0,\"reason\":\"brief explanation\","
     "\"matched_facts\":[],\"missing_facts\":[],\"conflicting_facts\":[]}. "
     "Use supported only when the evidence directly entails every material part of the claim. "
+    "Treat supplied contexts as minimal evidence spans, not the full source document. "
+    "Do not infer support from omitted surrounding context or from fluent answer wording. "
     "Use contradicted when the evidence conflicts with the claim, including wrong entities, "
     "wrong dates, wrong numbers, negation conflicts, or reversed causal/attribution roles. "
     "Use partially_supported when some material facts are supported but others are missing. "
@@ -331,7 +333,7 @@ def judge_cache_key(
     payload: dict[str, Any],
 ) -> str:
     cache_payload = {
-        "version": 1,
+        "version": 2,
         "provider": provider,
         "model": model or "",
         "system_prompt": JUDGE_SYSTEM_PROMPT,
@@ -417,6 +419,7 @@ def build_judge_provider(
 
 def _claim_payload(*, query: str, claim: str, contexts: list[TraceContext]) -> dict[str, Any]:
     return {
+        "evidence_scope": "selected_evidence_spans_only",
         "query": query,
         "claim": claim,
         "contexts": [