underwrite 0.1.1__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {underwrite-0.1.1 → underwrite-0.2.0}/PKG-INFO +138 -150
- {underwrite-0.1.1 → underwrite-0.2.0}/README.md +137 -149
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/capabilities.v1.schema.json +1 -1
- underwrite-0.2.0/contracts/common_result_inspection.v2.schema.json +336 -0
- underwrite-0.2.0/contracts/common_result_inspection_actions.v2.schema.json +425 -0
- underwrite-0.2.0/contracts/deepeval_accounting.v3.schema.json +1232 -0
- underwrite-0.2.0/contracts/deepeval_bound_record.v2.schema.json +2587 -0
- underwrite-0.2.0/contracts/deepeval_metric_bindings.v2.schema.json +513 -0
- underwrite-0.2.0/contracts/deepeval_record.v3.schema.json +2294 -0
- underwrite-0.2.0/contracts/deepeval_trace_index.v1.schema.json +683 -0
- underwrite-0.2.0/contracts/deepeval_trace_index.v2.schema.json +703 -0
- underwrite-0.2.0/contracts/deepeval_trace_record.v1.schema.json +1749 -0
- underwrite-0.2.0/contracts/deepeval_trace_record.v2.schema.json +1765 -0
- underwrite-0.2.0/contracts/evidence_result.v2.schema.json +1063 -0
- underwrite-0.2.0/contracts/goppi_trajectory_review.v1.schema.json +1589 -0
- underwrite-0.2.0/contracts/goppi_trajectory_verifier_response.v1.schema.json +254 -0
- underwrite-0.2.0/contracts/langfuse_accounting.v2.schema.json +1849 -0
- underwrite-0.2.0/contracts/langfuse_cli_accounting.v1.schema.json +1231 -0
- underwrite-0.2.0/contracts/langfuse_cli_record.v1.schema.json +2439 -0
- underwrite-0.2.0/contracts/langfuse_cli_row_index.v1.schema.json +897 -0
- underwrite-0.2.0/contracts/langfuse_record.v2.schema.json +3057 -0
- underwrite-0.2.0/contracts/langfuse_row_index.v2.schema.json +1293 -0
- underwrite-0.2.0/contracts/langfuse_row_record.v1.schema.json +3786 -0
- underwrite-0.2.0/contracts/promptfoo_accounting.v2.schema.json +270 -0
- underwrite-0.2.0/contracts/promptfoo_record.v2.schema.json +1330 -0
- underwrite-0.2.0/contracts/promptfoo_row_index.v2.schema.json +369 -0
- underwrite-0.2.0/contracts/trajectory_record.v1.schema.json +2621 -0
- underwrite-0.2.0/contracts/trajectory_record_read.v1.schema.json +2596 -0
- underwrite-0.2.0/contracts/trajectory_verification_history.v1.schema.json +146 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/pyproject.toml +33 -1
- {underwrite-0.1.1 → underwrite-0.2.0}/pyproject.toml.orig +33 -3
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/capabilities.v1.schema.json +1 -1
- underwrite-0.2.0/src/underwrite/_contracts/common_result_inspection.v2.schema.json +336 -0
- underwrite-0.2.0/src/underwrite/_contracts/common_result_inspection_actions.v2.schema.json +425 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_accounting.v3.schema.json +1232 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_bound_record.v2.schema.json +2587 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_metric_bindings.v2.schema.json +513 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_record.v3.schema.json +2294 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_index.v1.schema.json +683 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_index.v2.schema.json +703 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_record.v1.schema.json +1749 -0
- underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_record.v2.schema.json +1765 -0
- underwrite-0.2.0/src/underwrite/_contracts/evidence_result.v2.schema.json +1063 -0
- underwrite-0.2.0/src/underwrite/_contracts/goppi_trajectory_review.v1.schema.json +1589 -0
- underwrite-0.2.0/src/underwrite/_contracts/goppi_trajectory_verifier_response.v1.schema.json +254 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_accounting.v2.schema.json +1849 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_accounting.v1.schema.json +1231 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_record.v1.schema.json +2439 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_row_index.v1.schema.json +897 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_record.v2.schema.json +3057 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_row_index.v2.schema.json +1293 -0
- underwrite-0.2.0/src/underwrite/_contracts/langfuse_row_record.v1.schema.json +3786 -0
- underwrite-0.2.0/src/underwrite/_contracts/promptfoo_accounting.v2.schema.json +270 -0
- underwrite-0.2.0/src/underwrite/_contracts/promptfoo_record.v2.schema.json +1330 -0
- underwrite-0.2.0/src/underwrite/_contracts/promptfoo_row_index.v2.schema.json +369 -0
- underwrite-0.2.0/src/underwrite/_contracts/trajectory_record.v1.schema.json +2621 -0
- underwrite-0.2.0/src/underwrite/_contracts/trajectory_record_read.v1.schema.json +2596 -0
- underwrite-0.2.0/src/underwrite/_contracts/trajectory_verification_history.v1.schema.json +146 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_deepeval.py +90 -17
- underwrite-0.2.0/src/underwrite/cli/audit_langfuse.py +171 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_promptfoo.py +19 -12
- underwrite-0.2.0/src/underwrite/cli/audit_trajectory.py +80 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/evidence.py +21 -3
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/inspect_result.py +31 -11
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/main.py +194 -23
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_deepeval.py +212 -48
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_deepeval_bound.py +66 -28
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_langfuse.py +154 -31
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_promptfoo.py +63 -27
- underwrite-0.2.0/src/underwrite/cli/record_trajectory.py +445 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/records.py +18 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/deepeval_binding.py +83 -12
- underwrite-0.2.0/src/underwrite/instrument/evidence/deepeval_compat.py +165 -0
- underwrite-0.2.0/src/underwrite/instrument/evidence/deepeval_trace.py +351 -0
- underwrite-0.2.0/src/underwrite/instrument/evidence/goppi_trajectory.py +693 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/langfuse_accounting.py +19 -1
- underwrite-0.2.0/src/underwrite/instrument/evidence/langfuse_cli.py +466 -0
- underwrite-0.2.0/src/underwrite/instrument/evidence/langfuse_compat.py +274 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/langfuse_row_index.py +14 -10
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired_binary.py +0 -6
- underwrite-0.2.0/src/underwrite/instrument/evidence/promptfoo_compat.py +145 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/accounting.py +71 -18
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/application.py +12 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/deepeval_accounting.py +14 -13
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/deepeval_binding.py +37 -13
- underwrite-0.2.0/src/underwrite/instrument/ingest/deepeval_compat.py +84 -0
- underwrite-0.2.0/src/underwrite/instrument/ingest/deepeval_trace.py +96 -0
- underwrite-0.2.0/src/underwrite/instrument/ingest/goppi_trajectory.py +408 -0
- underwrite-0.2.0/src/underwrite/instrument/ingest/goppi_verifier.py +126 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/langfuse_accounting.py +86 -16
- underwrite-0.2.0/src/underwrite/instrument/ingest/langfuse_cli.py +131 -0
- underwrite-0.2.0/src/underwrite/instrument/ingest/langfuse_compat.py +121 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/local_capture.py +113 -1
- underwrite-0.1.1/src/underwrite/cli/audit_langfuse.py +0 -104
- {underwrite-0.1.1 → underwrite-0.2.0}/LICENSE +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/NOTICE +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_decision.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_decision.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_inspection_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/baseline_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/binary_run_audit.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/binary_run_inventory.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/boundary_allowlist.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/change_candidate.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/cli_result.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/common_result_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/common_result_inspection_actions.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_claims.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_claims.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_comparison.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_comparison_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_accounting.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_bound_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_metric_bindings.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_record.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/evidence_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/evidence_result.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/exemptions.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/gate_rules.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_row_index.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_policy.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_policy.v3.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_replay.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_semantics.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/metric_definition.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/observation.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_evidence.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_input.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_policy.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_handoff_review.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_review.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_review_context.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_row_index.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/read.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/read.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_catalog.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_document.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_spec.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_verification.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_view.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_inspection_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/result_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/contracts/statistical_seed.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_decision.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_decision.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_inspection_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/baseline_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/binary_run_audit.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/binary_run_inventory.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/boundary_allowlist.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/change_candidate.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/cli_result.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/common_result_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/common_result_inspection_actions.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_claims.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_claims.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_comparison.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_comparison_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_accounting.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_bound_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_metric_bindings.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_record.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/evidence_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/evidence_result.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/exemptions.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/gate_rules.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_row_index.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_policy.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_policy.v3.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_replay.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_semantics.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/metric_definition.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/observation.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_evidence.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_input.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_policy.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_handoff_review.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_review.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_review_context.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_accounting.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_row_index.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/read.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/read.v2.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_catalog.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_document.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_spec.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_verification.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_view.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_case.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_inspection.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_inspection_error.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/result_record.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/statistical_seed.v1.schema.json +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_fileio.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_json.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/claims.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/classify.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/decision.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/incident.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/lifecycle.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/adjudication/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/accept.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_baseline.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_counts.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_run.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/boundary.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/capabilities.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/commands.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/compare_declarations.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/gate_rules.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/handoff_limits.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/handoff_review.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/inspection.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/instrument.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/legacy_input.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/local_json.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/measure.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/measurement_flow.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/pair_binary.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/prepare_event_policy.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/project_baseline.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_baseline.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_capture.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_measure.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/release.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/render.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/replay_measure.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/review_pair.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/_pin_storage.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/drift.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/pins.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/_resampling.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/aa.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/accounting.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/common_result.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/controlled.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/declarations.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/deepeval_accounting.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_baseline.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_identity.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_p7_handoff.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/judge.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired_review.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/retrieval.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/run_audit.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/selfcheck.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/split.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/_shape.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/deepeval.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/evidence_bundle.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/inspect.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/langfuse.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/openinference.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/otlp_json.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/promptfoo.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/formats.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_baseline.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_p7_handoff.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_pilot.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/inspection.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/project.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release_bindings.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release_compose.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/schema.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/transport.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/availability.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/calibration.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/directional.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/eprocess.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/event_policy.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/event_read.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/jury.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/read.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/read_input.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/stratify.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/read.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/semantics.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/__init__.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/_record_errors.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/_record_format.py +0 -0
- {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/records.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: underwrite
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.2.0
|
|
4
4
|
Summary: Local evidence normalization, measurement, and acceptance checks
|
|
5
5
|
License-Expression: Apache-2.0
|
|
6
6
|
License-File: LICENSE
|
|
@@ -21,7 +21,7 @@ Description-Content-Type: text/markdown
|
|
|
21
21
|
|
|
22
22
|
Find mismatches. Keep missing evidence visible. Know what to check next.
|
|
23
23
|
|
|
24
|
-
**Local CLI & Python library ·
|
|
24
|
+
**Local CLI & Python library · JSON results · No server or model calls**
|
|
25
25
|
|
|
26
26
|
[](https://github.com/ntts9990/underwrite/actions/workflows/ci.yml)
|
|
27
27
|
[](https://pypi.org/project/underwrite/)
|
|
@@ -34,25 +34,24 @@ Find mismatches. Keep missing evidence visible. Know what to check next.
|
|
|
34
34
|
|
|
35
35
|
</div>
|
|
36
36
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
Underwrite examines supplied artifacts and can retain selected bytes. It does
|
|
48
|
-
not rerun an agent, independently authenticate a producer, or assign a universal
|
|
49
|
-
quality score. No Docker, dashboard, or provider credentials are needed to
|
|
50
|
-
inspect the supported local files.
|
|
37
|
+
Evaluation tools run evaluations and export result files. Underwrite reads the
|
|
38
|
+
local files it supports and shows what those files establish, what remains
|
|
39
|
+
unknown, and what evidence to check next. For example, it checks whether
|
|
40
|
+
Promptfoo’s reported totals agree with its supplied result rows. You or your
|
|
41
|
+
coding agent use the JSON result to investigate and decide. You can also save
|
|
42
|
+
selected inputs with the result for later review.
|
|
43
|
+
|
|
44
|
+
Underwrite does not rerun evaluations, independently prove a producer's verdict,
|
|
45
|
+
or assign one quality score across tools. Checking supported local files needs
|
|
46
|
+
no Docker, dashboard, or provider credentials.
|
|
51
47
|
|
|
52
48
|
> [!NOTE]
|
|
53
|
-
> **This guide targets `underwrite==0.
|
|
54
|
-
>
|
|
55
|
-
>
|
|
49
|
+
> **This guide targets `underwrite==0.2.0`.** [Install the CLI](#installed-package--020)
|
|
50
|
+
> for automatic input handling, focused evidence views and retained trajectory review.
|
|
51
|
+
> A [source checkout](#development-checkout--current-main) is optional for repository
|
|
52
|
+
> examples and development. Use `capabilities --json` to confirm the installed routes.
|
|
53
|
+
> Upgrading from 0.1.1? Read the [migration guide](https://github.com/ntts9990/underwrite/blob/main/MIGRATION.md).
|
|
54
|
+
> The older `0.0.1` has a separate pin-manifest caution; see [Install](#install).
|
|
56
55
|
|
|
57
56
|
## Quickstart
|
|
58
57
|
|
|
@@ -61,18 +60,19 @@ synthetic download below uses `curl`.
|
|
|
61
60
|
File-based CLI workflows are tested with CPython 3.12 on Linux and macOS. They
|
|
62
61
|
require Unix/POSIX file APIs, including for ordinary artifact reads.
|
|
63
62
|
|
|
63
|
+
### Installed package · 0.2.0
|
|
64
|
+
|
|
64
65
|
**1. Install and check a file.** The app-only install resolves a compatible
|
|
65
66
|
`underwrite-core` automatically. Download the public synthetic example; no
|
|
66
67
|
checkout or API key is needed:
|
|
67
68
|
|
|
68
69
|
```sh
|
|
69
|
-
uv tool install 'underwrite==0.
|
|
70
|
+
uv tool install 'underwrite==0.2.0'
|
|
70
71
|
underwrite capabilities --json
|
|
71
72
|
example_dir=$(mktemp -d)
|
|
72
73
|
curl -fsSLo "$example_dir/promptfoo.json" \
|
|
73
74
|
https://raw.githubusercontent.com/ntts9990/underwrite/cb913d39bce628b93dcd3b06fcc4ce36f9dc66af/fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json
|
|
74
|
-
underwrite audit-promptfoo "$example_dir/promptfoo.json"
|
|
75
|
-
--version 0.123.0 --json
|
|
75
|
+
underwrite audit-promptfoo "$example_dir/promptfoo.json" --json
|
|
76
76
|
```
|
|
77
77
|
|
|
78
78
|
The same `uv tool install` command updates an older uv-managed `underwrite`
|
|
@@ -86,7 +86,7 @@ from its actual output:
|
|
|
86
86
|
|
|
87
87
|
```json
|
|
88
88
|
{
|
|
89
|
-
"schema": "promptfoo_accounting.
|
|
89
|
+
"schema": "promptfoo_accounting.v2",
|
|
90
90
|
"declared_relation": "inconsistent",
|
|
91
91
|
"producer_stats": {"errors": 1, "failures": 1, "successes": 1, "total": 3},
|
|
92
92
|
"row_tallies": {"errors": 1, "failures": 0, "successes": 1, "total": 2},
|
|
@@ -98,6 +98,8 @@ from its actual output:
|
|
|
98
98
|
"PLANNED_DENOMINATOR_UNKNOWN",
|
|
99
99
|
"CAUSE_UNKNOWN",
|
|
100
100
|
"SOURCE_CLAIMS_UNVERIFIED",
|
|
101
|
+
"UNCONSUMED_FIELDS_AND_POPULATIONS_NOT_ANALYZED",
|
|
102
|
+
"SOURCE_HASH_AND_VERSION_LINKABLE",
|
|
101
103
|
"NO_APPROVAL_AUTHORITY"
|
|
102
104
|
]
|
|
103
105
|
}
|
|
@@ -110,21 +112,40 @@ explicit. Exit `0` means the report was produced, not that the counts agree or
|
|
|
110
112
|
the source's verdict was verified. To keep the input and analysis together,
|
|
111
113
|
[save them in a local record](#keep-results-you-can-return-to).
|
|
112
114
|
|
|
113
|
-
For your own completed Promptfoo export, replace the
|
|
114
|
-
supported
|
|
115
|
-
not an unchanged producer export.
|
|
115
|
+
For your own completed Promptfoo export, replace the path and use a
|
|
116
|
+
[supported input shape](#supported-formats). The downloaded file is a
|
|
117
|
+
constructed example, not an unchanged producer export.
|
|
116
118
|
|
|
117
|
-
|
|
118
|
-
<summary><strong>Try the native measurement → acceptance workflow</strong></summary>
|
|
119
|
+
### Development checkout · current main
|
|
119
120
|
|
|
120
|
-
|
|
121
|
-
|
|
121
|
+
Use a checkout to work on the source or run repository examples. Run its commands
|
|
122
|
+
through `uv run` inside that directory, keeping the installed tool separate. If you
|
|
123
|
+
already have a checkout, open its root and start at `uv sync` below:
|
|
122
124
|
|
|
123
125
|
```sh
|
|
124
126
|
git clone https://github.com/ntts9990/underwrite.git
|
|
125
127
|
cd underwrite
|
|
126
128
|
uv sync --frozen --all-groups
|
|
129
|
+
uv run --frozen --no-sync underwrite capabilities --json
|
|
130
|
+
uv run --frozen --no-sync underwrite audit-promptfoo \
|
|
131
|
+
fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json --json
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
This uses the same synthetic counts as above: 3 declared results versus 2 rows.
|
|
135
|
+
The automatic reader produces `promptfoo_accounting.v2` and preserves this
|
|
136
|
+
fixture's declared `0.123.0` version as an unverified source claim. Inputs without
|
|
137
|
+
a version declaration retain an unknown release. No version flag, producer
|
|
138
|
+
installation, Goppi, or provider call is needed for this example.
|
|
139
|
+
Use this `uv run --frozen --no-sync underwrite` prefix for all checkout commands.
|
|
140
|
+
See [included workflows](#included-in-020) for the other routes.
|
|
127
141
|
|
|
142
|
+
<details>
|
|
143
|
+
<summary><strong>Try the native measurement → acceptance workflow</strong></summary>
|
|
144
|
+
|
|
145
|
+
After preparing the checkout above, run this example from its root. Its policies
|
|
146
|
+
and fixtures live in the repository, outside the installed package:
|
|
147
|
+
|
|
148
|
+
```sh
|
|
128
149
|
mkdir -p output
|
|
129
150
|
uv run --frozen --no-sync underwrite ingest \
|
|
130
151
|
--format underwrite.evidence-bundle --version v1 \
|
|
@@ -154,53 +175,46 @@ See [measurement meaning](https://github.com/ntts9990/underwrite/blob/main/AGENT
|
|
|
154
175
|
|
|
155
176
|
| Question after an evaluation | What underwrite gives you |
|
|
156
177
|
| --- | --- |
|
|
157
|
-
| Does the summary match the supplied records? |
|
|
158
|
-
| What is still unknown? |
|
|
159
|
-
| Can this evidence support a measurement? |
|
|
160
|
-
| Can I revisit the conclusion? |
|
|
178
|
+
| Does the summary match the supplied records? | Compare Promptfoo declared counts with rows. Separately account for DeepEval scored rows and evaluator errors, or Langfuse score associations; their aggregate summaries are not assumed checked. |
|
|
179
|
+
| What is still unknown? | See when planned cases are unknown or source claims are unverified; unknown is not counted as zero. |
|
|
180
|
+
| Can this evidence support a measurement? | Use the separate native binary-event path with a declared event and policy; external scores are not a universal quality verdict. |
|
|
181
|
+
| Can I revisit the conclusion? | Save selected input bytes and the analysis in a private record, then verify and retrieve them later. |
|
|
161
182
|
|
|
162
183
|
## How it works
|
|
163
184
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
C --> D[Typed JSON: finding, identity, limits]
|
|
169
|
-
D --> E[Agent or person gathers next evidence]
|
|
170
|
-
C --> F[Optional private record]
|
|
171
|
-
F --> G[Verify and retrieve]
|
|
172
|
-
H[Native event evidence + policy] --> I[Measure]
|
|
173
|
-
I --> D
|
|
174
|
-
```
|
|
185
|
+
- **Evaluation tool:** runs the evaluation and exports result files.
|
|
186
|
+
- **Underwrite:** checks the supplied results, returns JSON with findings and
|
|
187
|
+
unanswered questions, and can save selected original files with the result.
|
|
188
|
+
- **You or your agent:** check the missing evidence and decide what to do next.
|
|
175
189
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
190
|
+
Each command checks only its selected input profile. Source verdicts remain
|
|
191
|
+
source claims; Underwrite does not apply changes or grant merge authority. Its
|
|
192
|
+
separate native binary-event workflow measures only a declared event under an
|
|
193
|
+
explicit policy, not arbitrary external scores.
|
|
180
194
|
|
|
181
195
|
## Agent use
|
|
182
196
|
|
|
183
|
-
Give your coding agent
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
underwrite capabilities --json
|
|
189
|
-
```
|
|
197
|
+
Give your coding agent a completed local export and your chosen entry path.
|
|
198
|
+
Have it inspect `capabilities --json` and the command's `--help` using that same
|
|
199
|
+
executable: `underwrite` for the published tool, or
|
|
200
|
+
`uv run --frozen --no-sync underwrite` from the checkout. Package version alone
|
|
201
|
+
does not identify the available workflows.
|
|
190
202
|
|
|
191
203
|
Python callers can use
|
|
192
|
-
`underwrite.instrument.ingest.accounting.audit_promptfoo_file(Path(...))`
|
|
193
|
-
the
|
|
194
|
-
|
|
204
|
+
`underwrite.instrument.ingest.accounting.audit_promptfoo_file(Path(...))` in
|
|
205
|
+
the chosen environment. Automatic intake selects a supported reader from the
|
|
206
|
+
input shape without guessing a producer release. Use explicit historical selectors
|
|
207
|
+
when reproducing an older contract, as described in the migration guide.
|
|
195
208
|
|
|
196
209
|
Example task for an agent with terminal access:
|
|
197
210
|
|
|
198
|
-
> Review `/absolute/path/to/completed-promptfoo-export.json`
|
|
199
|
-
>
|
|
200
|
-
>
|
|
201
|
-
>
|
|
202
|
-
>
|
|
203
|
-
>
|
|
211
|
+
> Review `/absolute/path/to/completed-promptfoo-export.json` using our selected
|
|
212
|
+
> Underwrite entry path. Inspect its capabilities and help, then use the
|
|
213
|
+
> supported accounting route. Use automatic intake when advertised; otherwise
|
|
214
|
+
> require evidence of a supported published profile rather than guessing a version.
|
|
215
|
+
> Report count differences, input identity, missing evidence, interpretation
|
|
216
|
+
> limits, and the next useful check. Do not rerun the producer or treat exit `0`
|
|
217
|
+
> as agreement or approval.
|
|
204
218
|
|
|
205
219
|
The optional [underwrite-review skill](https://github.com/ntts9990/underwrite/blob/main/skills/underwrite-review/SKILL.md) provides
|
|
206
220
|
reusable guidance for Codex and Claude Code. [Setup and usage](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#optional-artifact-review-skill)
|
|
@@ -210,67 +224,76 @@ The same CLI and importable Python APIs are available to your own automation.
|
|
|
210
224
|
|
|
211
225
|
## Supported formats
|
|
212
226
|
|
|
213
|
-
Support
|
|
214
|
-
|
|
227
|
+
Support covers selected local payloads, not every producer API, export, or
|
|
228
|
+
version. Use capabilities from your chosen executable and the linked workflows
|
|
229
|
+
for exact input requirements.
|
|
230
|
+
|
|
231
|
+
### Supported profiles · 0.2.0
|
|
215
232
|
|
|
216
233
|
| Your input | Supported profile | Start here |
|
|
217
234
|
| --- | --- | --- |
|
|
218
|
-
| **
|
|
219
|
-
| **Langfuse** |
|
|
220
|
-
| **DeepEval** |
|
|
221
|
-
| **Promptfoo** |
|
|
222
|
-
| **External common result** | `evidence_result.v1`
|
|
235
|
+
| **Goppi** | Completed baseline exchange manifest v2; P7 handoff; adopted paired trajectory | [`audit-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#audit-a-completed-goppi-baseline-as-common-evidence) · [`project-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#project-a-completed-goppi-baseline-into-event-observations) · [`review-handoff`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#review-a-completed-paired-handoff-in-one-call) · [`audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#compare-a-goppi-paired-trajectory) |
|
|
236
|
+
| **Langfuse** | Compatible blob arrays or official-CLI list captures; automatic pair selection | [`audit-langfuse`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-langfuse-score-kinds-and-associations) |
|
|
237
|
+
| **DeepEval** | Compatible TestRun; automatic accounting, optional trace view and meaning bindings | [`audit-deepeval`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors) |
|
|
238
|
+
| **Promptfoo** | SummaryV3 eval output; automatic intake, historical selectors retained | [`audit-promptfoo`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#audit-promptfoo-counts) |
|
|
239
|
+
| **External common result** | `evidence_result.v1` or `evidence_result.v2` | [`inspect-result`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#inspect-an-external-common-result) |
|
|
223
240
|
| **Native evidence** | `underwrite.evidence-bundle` / `v1` | `ingest` → `measure` → `accept` |
|
|
224
241
|
| **Inspect AI** | `inspect.eval-log` / `0.3.263` | `ingest` |
|
|
225
242
|
| **OpenInference** | `openinference.traces` / `0.1.38` | `ingest` |
|
|
226
243
|
| **OTLP JSON** | `otlp-json.traces` / `0.160.0` | `ingest` |
|
|
227
244
|
|
|
228
|
-
|
|
229
|
-
|
|
245
|
+
### Included in 0.2.0
|
|
246
|
+
|
|
247
|
+
- **Automatic intake:** compatible Promptfoo SummaryV3, DeepEval TestRun, and
|
|
248
|
+
Langfuse blob arrays or [official-CLI list captures](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#read-official-langfuse-cli-captures-offline)
|
|
249
|
+
without selecting a producer version/profile. Actual producer release stays
|
|
250
|
+
unknown when the payload does not establish it; explicit selectors remain for
|
|
251
|
+
reproducing older contracts.
|
|
252
|
+
- **Focused views and storage:** selected DeepEval trace structure, version-neutral
|
|
253
|
+
`deepeval_metric_bindings.v2` packs, and Langfuse row-index recording. Follow the
|
|
254
|
+
linked producer workflows above for their separate contracts and limits.
|
|
255
|
+
- **Common results:** `inspect-result` also accepts `evidence_result.v2`.
|
|
256
|
+
- **Goppi steps:** [`audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#compare-a-goppi-paired-trajectory)
|
|
257
|
+
compares recorded tool states and evaluation results from a paired+trajectory
|
|
258
|
+
export, including runs that stopped early. This optional route requires installed
|
|
259
|
+
Goppi for offline verification; it does not run providers, establish causality,
|
|
260
|
+
or measure retry effects. [`records audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/STORAGE.md#retain-a-goppi-paired-trajectory)
|
|
261
|
+
retains the same inputs and result; `records read-trajectory` reopens them
|
|
262
|
+
without Goppi or the original export.
|
|
230
263
|
|
|
231
|
-
```sh
|
|
232
|
-
underwrite audit-promptfoo /path/to/promptfoo-output.json --version 0.124.1 --json
|
|
233
|
-
underwrite audit-deepeval /path/to/deepeval-test-run.json --version 4.2.8 --json
|
|
234
|
-
underwrite audit-langfuse --observations /path/to/observations.json \
|
|
235
|
-
--scores /path/to/scores.json --json
|
|
236
|
-
underwrite audit-baseline --exchange /path/to/goppi-baseline-exchange --json
|
|
237
|
-
```
|
|
238
|
-
|
|
239
|
-
DeepEval accounting defaults to `4.1.1`; select `--version 4.2.8` for that
|
|
240
|
-
profile. For a reusable owner-asserted metric-name pack and common descriptive
|
|
241
|
-
JSON, see the [DeepEval binding workflow](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors).
|
|
242
|
-
Promptfoo accounting defaults to `0.123.0`; select `--version 0.124.1`
|
|
243
|
-
for that profile. Langfuse's version identifies the server export, not its SDK.
|
|
244
264
|
See [artifact requirements](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#get-the-right-producer-artifact) and
|
|
245
|
-
[producer checks](https://github.com/ntts9990/underwrite/blob/main/EVIDENCE_REVIEW.md)
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
265
|
+
[producer checks](https://github.com/ntts9990/underwrite/blob/main/EVIDENCE_REVIEW.md)
|
|
266
|
+
for fixture origins and tested scope. A controlled fixture does not prove
|
|
267
|
+
live-server interoperability or export completeness.
|
|
268
|
+
|
|
269
|
+
Moving from published 0.1.1? See the [migration guide](https://github.com/ntts9990/underwrite/blob/main/MIGRATION.md)
|
|
270
|
+
for changed default JSON contracts, historical selectors, and existing records.
|
|
271
|
+
|
|
272
|
+
Generic `ingest` / `project` normalizes supported exports; external observations
|
|
273
|
+
do not enter the current `measure` path. Goppi `audit-baseline` produces
|
|
274
|
+
descriptive common evidence, while `project-baseline` can produce native event
|
|
275
|
+
observations. `inspect-result` screens the installed common-result contract and
|
|
276
|
+
counts items without verifying source truth or admitting them to measurement
|
|
277
|
+
or acceptance. Its action view counts recorded next checks, not executed tools.
|
|
258
278
|
|
|
259
279
|
## Keep results you can return to
|
|
260
280
|
|
|
261
281
|
Use `records` when you want the analysis and its selected source bytes retained
|
|
262
|
-
together.
|
|
282
|
+
together. This example uses the **published package** with a completed local
|
|
283
|
+
Promptfoo export:
|
|
263
284
|
|
|
264
285
|
```sh
|
|
265
286
|
underwrite records audit-promptfoo /path/to/promptfoo-output.json \
|
|
266
|
-
--
|
|
287
|
+
--root ./review-store --id review-001 --json
|
|
267
288
|
|
|
268
289
|
underwrite records read-json \
|
|
269
290
|
--root ./review-store --id review-001 \
|
|
270
291
|
--name analysis/accounting.json --json
|
|
271
292
|
```
|
|
272
293
|
|
|
273
|
-
|
|
294
|
+
Automatic intake needs no producer version flag. In a checkout, prefix both
|
|
295
|
+
commands with `uv run --frozen --no-sync`. Use a fresh ID for each capture;
|
|
296
|
+
existing records are never overwritten.
|
|
274
297
|
`records verify` checks stored integrity. `records measure` captures a native
|
|
275
298
|
measurement, and `records replay-measure` recalculates it from verified saved
|
|
276
299
|
inputs using the current engine, without executing the original producer.
|
|
@@ -304,7 +327,7 @@ Install the versioned app; its dependency resolves the compatible core. Use
|
|
|
304
327
|
profiles:
|
|
305
328
|
|
|
306
329
|
```sh
|
|
307
|
-
uv tool install 'underwrite==0.
|
|
330
|
+
uv tool install 'underwrite==0.2.0'
|
|
308
331
|
underwrite --help
|
|
309
332
|
underwrite capabilities --json
|
|
310
333
|
```
|
|
@@ -317,12 +340,12 @@ For the importable Python API, install into a Python 3.12 environment instead:
|
|
|
317
340
|
|
|
318
341
|
```sh
|
|
319
342
|
uv venv --python 3.12 .venv
|
|
320
|
-
uv pip install --python .venv/bin/python 'underwrite==0.
|
|
343
|
+
uv pip install --python .venv/bin/python 'underwrite==0.2.0'
|
|
321
344
|
.venv/bin/python - <<'PY'
|
|
322
345
|
from pathlib import Path
|
|
323
346
|
from underwrite.instrument.ingest.accounting import audit_promptfoo_file
|
|
324
347
|
|
|
325
|
-
result = audit_promptfoo_file(Path("/path/to/promptfoo-output.json")
|
|
348
|
+
result = audit_promptfoo_file(Path("/path/to/promptfoo-output.json"))
|
|
326
349
|
print(result["declared_relation"])
|
|
327
350
|
PY
|
|
328
351
|
```
|
|
@@ -358,46 +381,8 @@ a manifest with an expected hash, `check` examines release evidence linkage,
|
|
|
358
381
|
vocabulary, and `drift check` / `pins refresh` work with supplied pin registries.
|
|
359
382
|
[JSON contracts](https://github.com/ntts9990/underwrite/tree/main/contracts/) define each command family's results and errors.
|
|
360
383
|
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
`boundary scan` reads at most 4 MiB per selected JSON or JSONL file and 32 MiB
|
|
364
|
-
of captured bytes across the command, including its allowlist. It admits up to
|
|
365
|
-
4096 selected files, 100000 visited directory entries, 100000 physical JSONL
|
|
366
|
-
lines and 1 MiB per raw line (including its delimiter), and 4096 leaks including
|
|
367
|
-
allowed rows. Its selected JSON or human output is capped at 1 MiB including
|
|
368
|
-
the final newline. Duplicate keys and nonfinite numbers are invalid; supplied
|
|
369
|
-
`--now` and policy expiry dates must be real ASCII `YYYY-MM-DD` dates. Admission
|
|
370
|
-
failures return a typed `UNCONFIGURED` result with exit 2. An absent `--now`
|
|
371
|
-
still yields `NOW_REQUIRED` only when the semantic check needs a clock.
|
|
372
|
-
|
|
373
|
-
`absence` retains `Path.glob` matching and refuses more than 4096 matching
|
|
374
|
-
regular files; it does not bound directory traversal within `Path.glob` or
|
|
375
|
-
suppressed internal filesystem errors. Boundary scans follow an explicitly
|
|
376
|
-
selected directory link and regular file links, but do not descend into a child
|
|
377
|
-
directory link. These selections do not establish path containment against a
|
|
378
|
-
hostile filesystem.
|
|
379
|
-
|
|
380
|
-
`drift check` admits up to 256 manifests and 4096 declared pin files while
|
|
381
|
-
counting at most 100000 visited registry entries. Each manifest is limited to
|
|
382
|
-
1 MiB and 64 JSON container levels; each selected source or pin copy is limited
|
|
383
|
-
to 4 MiB. The 32 MiB command capture budget includes exemption bytes, manifests,
|
|
384
|
-
and both copies. Read-only inspection follows selected directory and file links;
|
|
385
|
-
it does not create an atomic filesystem snapshot or a hostile-path containment
|
|
386
|
-
guarantee. Missing or nonregular selected copies are reported as DRIFT; malformed,
|
|
387
|
-
unreadable, or excessive registry input returns a typed `UNCONFIGURED` result
|
|
388
|
-
without fabricated population counts.
|
|
389
|
-
|
|
390
|
-
`pins refresh` selects one registry repository and copies committed Git blobs.
|
|
391
|
-
Refresh paths must be normalized relative `/`-separated components; empty, dot,
|
|
392
|
-
parent, backslash, Unicode control/format, oversized, and colliding destinations
|
|
393
|
-
are refused.
|
|
394
|
-
The selected registry and its existing destinations are checked before mutation.
|
|
395
|
-
Each pin is replaced with a new inode under its opened registry parent, and
|
|
396
|
-
`PINS.json` is replaced last. Admission or Git-source failures leave the registry
|
|
397
|
-
unchanged. A later filesystem failure reports `publication: may_be_partial`;
|
|
398
|
-
refresh is not an all-files transaction or a durability guarantee.
|
|
399
|
-
Refresh captures each committed Git blob within 4 MiB and the 32 MiB aggregate
|
|
400
|
-
budget, and checks the fully serialized 1 MiB manifest before its first write.
|
|
384
|
+
For scan and pin resource limits and filesystem-selection caveats, see
|
|
385
|
+
[Scan and pin limits](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#scan-and-pin-limits).
|
|
401
386
|
|
|
402
387
|
<a id="development-checkout-evidence-pilots"></a>
|
|
403
388
|
|
|
@@ -430,10 +415,13 @@ Before using OIDC publishing, create the GitHub environment `pypi`, restrict it
|
|
|
430
415
|
`ntts9990`, repository `underwrite`, workflow `publish.yml`, and environment `pypi`.
|
|
431
416
|
Once configured, run **Publish PyPI** manually from `main` in GitHub Actions.
|
|
432
417
|
|
|
433
|
-
The workflow validates both packages
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
418
|
+
The workflow validates both packages and defaults to publishing only the app
|
|
419
|
+
(`publish_core: false`), after testing it with the published compatible core. Select
|
|
420
|
+
`publish_core: true` only for an explicitly approved core release; core upload must
|
|
421
|
+
succeed before app upload. The unchanged core remains `0.1.1` for this app release.
|
|
422
|
+
Only the publish job receives OIDC permission; no stored PyPI token is required.
|
|
423
|
+
Retry partial uploads with identical artifacts. Existing registry files must match
|
|
424
|
+
exactly; never re-upload a rebuilt, different archive under an existing version.
|
|
437
425
|
|
|
438
426
|
See the official [uv publishing guide](https://docs.astral.sh/uv/guides/package/)
|
|
439
427
|
and [PyPI Trusted Publisher guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
|