underwrite 0.1.1__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (318) hide show
  1. {underwrite-0.1.1 → underwrite-0.2.0}/PKG-INFO +138 -150
  2. {underwrite-0.1.1 → underwrite-0.2.0}/README.md +137 -149
  3. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/capabilities.v1.schema.json +1 -1
  4. underwrite-0.2.0/contracts/common_result_inspection.v2.schema.json +336 -0
  5. underwrite-0.2.0/contracts/common_result_inspection_actions.v2.schema.json +425 -0
  6. underwrite-0.2.0/contracts/deepeval_accounting.v3.schema.json +1232 -0
  7. underwrite-0.2.0/contracts/deepeval_bound_record.v2.schema.json +2587 -0
  8. underwrite-0.2.0/contracts/deepeval_metric_bindings.v2.schema.json +513 -0
  9. underwrite-0.2.0/contracts/deepeval_record.v3.schema.json +2294 -0
  10. underwrite-0.2.0/contracts/deepeval_trace_index.v1.schema.json +683 -0
  11. underwrite-0.2.0/contracts/deepeval_trace_index.v2.schema.json +703 -0
  12. underwrite-0.2.0/contracts/deepeval_trace_record.v1.schema.json +1749 -0
  13. underwrite-0.2.0/contracts/deepeval_trace_record.v2.schema.json +1765 -0
  14. underwrite-0.2.0/contracts/evidence_result.v2.schema.json +1063 -0
  15. underwrite-0.2.0/contracts/goppi_trajectory_review.v1.schema.json +1589 -0
  16. underwrite-0.2.0/contracts/goppi_trajectory_verifier_response.v1.schema.json +254 -0
  17. underwrite-0.2.0/contracts/langfuse_accounting.v2.schema.json +1849 -0
  18. underwrite-0.2.0/contracts/langfuse_cli_accounting.v1.schema.json +1231 -0
  19. underwrite-0.2.0/contracts/langfuse_cli_record.v1.schema.json +2439 -0
  20. underwrite-0.2.0/contracts/langfuse_cli_row_index.v1.schema.json +897 -0
  21. underwrite-0.2.0/contracts/langfuse_record.v2.schema.json +3057 -0
  22. underwrite-0.2.0/contracts/langfuse_row_index.v2.schema.json +1293 -0
  23. underwrite-0.2.0/contracts/langfuse_row_record.v1.schema.json +3786 -0
  24. underwrite-0.2.0/contracts/promptfoo_accounting.v2.schema.json +270 -0
  25. underwrite-0.2.0/contracts/promptfoo_record.v2.schema.json +1330 -0
  26. underwrite-0.2.0/contracts/promptfoo_row_index.v2.schema.json +369 -0
  27. underwrite-0.2.0/contracts/trajectory_record.v1.schema.json +2621 -0
  28. underwrite-0.2.0/contracts/trajectory_record_read.v1.schema.json +2596 -0
  29. underwrite-0.2.0/contracts/trajectory_verification_history.v1.schema.json +146 -0
  30. {underwrite-0.1.1 → underwrite-0.2.0}/pyproject.toml +33 -1
  31. {underwrite-0.1.1 → underwrite-0.2.0}/pyproject.toml.orig +33 -3
  32. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/capabilities.v1.schema.json +1 -1
  33. underwrite-0.2.0/src/underwrite/_contracts/common_result_inspection.v2.schema.json +336 -0
  34. underwrite-0.2.0/src/underwrite/_contracts/common_result_inspection_actions.v2.schema.json +425 -0
  35. underwrite-0.2.0/src/underwrite/_contracts/deepeval_accounting.v3.schema.json +1232 -0
  36. underwrite-0.2.0/src/underwrite/_contracts/deepeval_bound_record.v2.schema.json +2587 -0
  37. underwrite-0.2.0/src/underwrite/_contracts/deepeval_metric_bindings.v2.schema.json +513 -0
  38. underwrite-0.2.0/src/underwrite/_contracts/deepeval_record.v3.schema.json +2294 -0
  39. underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_index.v1.schema.json +683 -0
  40. underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_index.v2.schema.json +703 -0
  41. underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_record.v1.schema.json +1749 -0
  42. underwrite-0.2.0/src/underwrite/_contracts/deepeval_trace_record.v2.schema.json +1765 -0
  43. underwrite-0.2.0/src/underwrite/_contracts/evidence_result.v2.schema.json +1063 -0
  44. underwrite-0.2.0/src/underwrite/_contracts/goppi_trajectory_review.v1.schema.json +1589 -0
  45. underwrite-0.2.0/src/underwrite/_contracts/goppi_trajectory_verifier_response.v1.schema.json +254 -0
  46. underwrite-0.2.0/src/underwrite/_contracts/langfuse_accounting.v2.schema.json +1849 -0
  47. underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_accounting.v1.schema.json +1231 -0
  48. underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_record.v1.schema.json +2439 -0
  49. underwrite-0.2.0/src/underwrite/_contracts/langfuse_cli_row_index.v1.schema.json +897 -0
  50. underwrite-0.2.0/src/underwrite/_contracts/langfuse_record.v2.schema.json +3057 -0
  51. underwrite-0.2.0/src/underwrite/_contracts/langfuse_row_index.v2.schema.json +1293 -0
  52. underwrite-0.2.0/src/underwrite/_contracts/langfuse_row_record.v1.schema.json +3786 -0
  53. underwrite-0.2.0/src/underwrite/_contracts/promptfoo_accounting.v2.schema.json +270 -0
  54. underwrite-0.2.0/src/underwrite/_contracts/promptfoo_record.v2.schema.json +1330 -0
  55. underwrite-0.2.0/src/underwrite/_contracts/promptfoo_row_index.v2.schema.json +369 -0
  56. underwrite-0.2.0/src/underwrite/_contracts/trajectory_record.v1.schema.json +2621 -0
  57. underwrite-0.2.0/src/underwrite/_contracts/trajectory_record_read.v1.schema.json +2596 -0
  58. underwrite-0.2.0/src/underwrite/_contracts/trajectory_verification_history.v1.schema.json +146 -0
  59. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_deepeval.py +90 -17
  60. underwrite-0.2.0/src/underwrite/cli/audit_langfuse.py +171 -0
  61. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_promptfoo.py +19 -12
  62. underwrite-0.2.0/src/underwrite/cli/audit_trajectory.py +80 -0
  63. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/evidence.py +21 -3
  64. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/inspect_result.py +31 -11
  65. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/main.py +194 -23
  66. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_deepeval.py +212 -48
  67. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_deepeval_bound.py +66 -28
  68. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_langfuse.py +154 -31
  69. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_promptfoo.py +63 -27
  70. underwrite-0.2.0/src/underwrite/cli/record_trajectory.py +445 -0
  71. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/records.py +18 -0
  72. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/deepeval_binding.py +83 -12
  73. underwrite-0.2.0/src/underwrite/instrument/evidence/deepeval_compat.py +165 -0
  74. underwrite-0.2.0/src/underwrite/instrument/evidence/deepeval_trace.py +351 -0
  75. underwrite-0.2.0/src/underwrite/instrument/evidence/goppi_trajectory.py +693 -0
  76. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/langfuse_accounting.py +19 -1
  77. underwrite-0.2.0/src/underwrite/instrument/evidence/langfuse_cli.py +466 -0
  78. underwrite-0.2.0/src/underwrite/instrument/evidence/langfuse_compat.py +274 -0
  79. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/langfuse_row_index.py +14 -10
  80. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired_binary.py +0 -6
  81. underwrite-0.2.0/src/underwrite/instrument/evidence/promptfoo_compat.py +145 -0
  82. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/accounting.py +71 -18
  83. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/application.py +12 -0
  84. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/deepeval_accounting.py +14 -13
  85. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/deepeval_binding.py +37 -13
  86. underwrite-0.2.0/src/underwrite/instrument/ingest/deepeval_compat.py +84 -0
  87. underwrite-0.2.0/src/underwrite/instrument/ingest/deepeval_trace.py +96 -0
  88. underwrite-0.2.0/src/underwrite/instrument/ingest/goppi_trajectory.py +408 -0
  89. underwrite-0.2.0/src/underwrite/instrument/ingest/goppi_verifier.py +126 -0
  90. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/langfuse_accounting.py +86 -16
  91. underwrite-0.2.0/src/underwrite/instrument/ingest/langfuse_cli.py +131 -0
  92. underwrite-0.2.0/src/underwrite/instrument/ingest/langfuse_compat.py +121 -0
  93. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/local_capture.py +113 -1
  94. underwrite-0.1.1/src/underwrite/cli/audit_langfuse.py +0 -104
  95. {underwrite-0.1.1 → underwrite-0.2.0}/LICENSE +0 -0
  96. {underwrite-0.1.1 → underwrite-0.2.0}/NOTICE +0 -0
  97. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_decision.v1.schema.json +0 -0
  98. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_decision.v2.schema.json +0 -0
  99. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/acceptance_error.v1.schema.json +0 -0
  100. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_case.v1.schema.json +0 -0
  101. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_inspection.v1.schema.json +0 -0
  102. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/artifact_inspection_error.v1.schema.json +0 -0
  103. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/baseline_record.v1.schema.json +0 -0
  104. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/binary_run_audit.v1.schema.json +0 -0
  105. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/binary_run_inventory.v1.schema.json +0 -0
  106. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/boundary_allowlist.v1.schema.json +0 -0
  107. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/change_candidate.v1.schema.json +0 -0
  108. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/cli_result.v1.schema.json +0 -0
  109. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/common_result_inspection.v1.schema.json +0 -0
  110. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/common_result_inspection_actions.v1.schema.json +0 -0
  111. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_claims.v1.schema.json +0 -0
  112. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_claims.v2.schema.json +0 -0
  113. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_comparison.v1.schema.json +0 -0
  114. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/declared_comparison_case.v1.schema.json +0 -0
  115. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_accounting.v1.schema.json +0 -0
  116. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_accounting.v2.schema.json +0 -0
  117. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_bound_record.v1.schema.json +0 -0
  118. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_metric_bindings.v1.schema.json +0 -0
  119. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_record.v1.schema.json +0 -0
  120. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/deepeval_record.v2.schema.json +0 -0
  121. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/evidence_error.v1.schema.json +0 -0
  122. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/evidence_result.v1.schema.json +0 -0
  123. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/exemptions.v1.schema.json +0 -0
  124. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/gate_rules.v1.schema.json +0 -0
  125. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_accounting.v1.schema.json +0 -0
  126. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_record.v1.schema.json +0 -0
  127. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/langfuse_row_index.v1.schema.json +0 -0
  128. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_error.v1.schema.json +0 -0
  129. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_policy.v2.schema.json +0 -0
  130. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_policy.v3.schema.json +0 -0
  131. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_record.v1.schema.json +0 -0
  132. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_replay.v1.schema.json +0 -0
  133. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/measurement_semantics.v1.schema.json +0 -0
  134. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/metric_definition.v1.schema.json +0 -0
  135. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/observation.v1.schema.json +0 -0
  136. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_evidence.v1.schema.json +0 -0
  137. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_input.v1.schema.json +0 -0
  138. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_binary_policy.v1.schema.json +0 -0
  139. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_handoff_review.v1.schema.json +0 -0
  140. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_review.v1.schema.json +0 -0
  141. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/paired_review_context.v1.schema.json +0 -0
  142. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_accounting.v1.schema.json +0 -0
  143. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_record.v1.schema.json +0 -0
  144. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/promptfoo_row_index.v1.schema.json +0 -0
  145. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/read.v1.schema.json +0 -0
  146. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/read.v2.schema.json +0 -0
  147. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_catalog.v1.schema.json +0 -0
  148. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_document.v1.schema.json +0 -0
  149. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_error.v1.schema.json +0 -0
  150. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_spec.v1.schema.json +0 -0
  151. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_verification.v1.schema.json +0 -0
  152. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/record_view.v1.schema.json +0 -0
  153. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_case.v1.schema.json +0 -0
  154. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_inspection.v1.schema.json +0 -0
  155. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/release_inspection_error.v1.schema.json +0 -0
  156. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/result_record.v1.schema.json +0 -0
  157. {underwrite-0.1.1 → underwrite-0.2.0}/contracts/statistical_seed.v1.schema.json +0 -0
  158. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/__init__.py +0 -0
  159. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_decision.v1.schema.json +0 -0
  160. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_decision.v2.schema.json +0 -0
  161. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/acceptance_error.v1.schema.json +0 -0
  162. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_case.v1.schema.json +0 -0
  163. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_inspection.v1.schema.json +0 -0
  164. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/artifact_inspection_error.v1.schema.json +0 -0
  165. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/baseline_record.v1.schema.json +0 -0
  166. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/binary_run_audit.v1.schema.json +0 -0
  167. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/binary_run_inventory.v1.schema.json +0 -0
  168. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/boundary_allowlist.v1.schema.json +0 -0
  169. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/change_candidate.v1.schema.json +0 -0
  170. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/cli_result.v1.schema.json +0 -0
  171. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/common_result_inspection.v1.schema.json +0 -0
  172. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/common_result_inspection_actions.v1.schema.json +0 -0
  173. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_claims.v1.schema.json +0 -0
  174. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_claims.v2.schema.json +0 -0
  175. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_comparison.v1.schema.json +0 -0
  176. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/declared_comparison_case.v1.schema.json +0 -0
  177. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_accounting.v1.schema.json +0 -0
  178. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_accounting.v2.schema.json +0 -0
  179. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_bound_record.v1.schema.json +0 -0
  180. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_metric_bindings.v1.schema.json +0 -0
  181. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_record.v1.schema.json +0 -0
  182. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/deepeval_record.v2.schema.json +0 -0
  183. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/evidence_error.v1.schema.json +0 -0
  184. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/evidence_result.v1.schema.json +0 -0
  185. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/exemptions.v1.schema.json +0 -0
  186. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/gate_rules.v1.schema.json +0 -0
  187. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_accounting.v1.schema.json +0 -0
  188. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_record.v1.schema.json +0 -0
  189. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/langfuse_row_index.v1.schema.json +0 -0
  190. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_error.v1.schema.json +0 -0
  191. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_policy.v2.schema.json +0 -0
  192. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_policy.v3.schema.json +0 -0
  193. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_record.v1.schema.json +0 -0
  194. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_replay.v1.schema.json +0 -0
  195. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/measurement_semantics.v1.schema.json +0 -0
  196. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/metric_definition.v1.schema.json +0 -0
  197. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/observation.v1.schema.json +0 -0
  198. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_evidence.v1.schema.json +0 -0
  199. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_input.v1.schema.json +0 -0
  200. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_binary_policy.v1.schema.json +0 -0
  201. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_handoff_review.v1.schema.json +0 -0
  202. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_review.v1.schema.json +0 -0
  203. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/paired_review_context.v1.schema.json +0 -0
  204. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_accounting.v1.schema.json +0 -0
  205. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_record.v1.schema.json +0 -0
  206. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/promptfoo_row_index.v1.schema.json +0 -0
  207. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/read.v1.schema.json +0 -0
  208. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/read.v2.schema.json +0 -0
  209. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_catalog.v1.schema.json +0 -0
  210. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_document.v1.schema.json +0 -0
  211. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_error.v1.schema.json +0 -0
  212. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_spec.v1.schema.json +0 -0
  213. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_verification.v1.schema.json +0 -0
  214. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/record_view.v1.schema.json +0 -0
  215. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_case.v1.schema.json +0 -0
  216. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_inspection.v1.schema.json +0 -0
  217. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/release_inspection_error.v1.schema.json +0 -0
  218. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/result_record.v1.schema.json +0 -0
  219. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_contracts/statistical_seed.v1.schema.json +0 -0
  220. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_fileio.py +0 -0
  221. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/_json.py +0 -0
  222. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/__init__.py +0 -0
  223. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/claims.py +0 -0
  224. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/classify.py +0 -0
  225. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/decision.py +0 -0
  226. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/incident.py +0 -0
  227. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/acceptance/lifecycle.py +0 -0
  228. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/adjudication/__init__.py +0 -0
  229. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/__init__.py +0 -0
  230. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/accept.py +0 -0
  231. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_baseline.py +0 -0
  232. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_counts.py +0 -0
  233. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/audit_run.py +0 -0
  234. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/boundary.py +0 -0
  235. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/capabilities.py +0 -0
  236. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/commands.py +0 -0
  237. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/compare_declarations.py +0 -0
  238. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/gate_rules.py +0 -0
  239. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/handoff_limits.py +0 -0
  240. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/handoff_review.py +0 -0
  241. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/inspection.py +0 -0
  242. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/instrument.py +0 -0
  243. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/legacy_input.py +0 -0
  244. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/local_json.py +0 -0
  245. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/measure.py +0 -0
  246. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/measurement_flow.py +0 -0
  247. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/pair_binary.py +0 -0
  248. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/prepare_event_policy.py +0 -0
  249. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/project_baseline.py +0 -0
  250. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_baseline.py +0 -0
  251. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_capture.py +0 -0
  252. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/record_measure.py +0 -0
  253. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/release.py +0 -0
  254. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/render.py +0 -0
  255. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/replay_measure.py +0 -0
  256. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/cli/review_pair.py +0 -0
  257. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/__init__.py +0 -0
  258. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/_pin_storage.py +0 -0
  259. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/drift.py +0 -0
  260. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/fleet/pins.py +0 -0
  261. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/__init__.py +0 -0
  262. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/__init__.py +0 -0
  263. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/_resampling.py +0 -0
  264. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/aa.py +0 -0
  265. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/accounting.py +0 -0
  266. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/common_result.py +0 -0
  267. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/controlled.py +0 -0
  268. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/declarations.py +0 -0
  269. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/deepeval_accounting.py +0 -0
  270. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_baseline.py +0 -0
  271. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_identity.py +0 -0
  272. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/goppi_p7_handoff.py +0 -0
  273. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/judge.py +0 -0
  274. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired.py +0 -0
  275. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/paired_review.py +0 -0
  276. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/retrieval.py +0 -0
  277. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/run_audit.py +0 -0
  278. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/selfcheck.py +0 -0
  279. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/evidence/split.py +0 -0
  280. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/__init__.py +0 -0
  281. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/__init__.py +0 -0
  282. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/_shape.py +0 -0
  283. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/deepeval.py +0 -0
  284. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/evidence_bundle.py +0 -0
  285. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/inspect.py +0 -0
  286. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/langfuse.py +0 -0
  287. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/openinference.py +0 -0
  288. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/otlp_json.py +0 -0
  289. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/codecs/promptfoo.py +0 -0
  290. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/formats.py +0 -0
  291. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_baseline.py +0 -0
  292. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_p7_handoff.py +0 -0
  293. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/goppi_pilot.py +0 -0
  294. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/inspection.py +0 -0
  295. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/project.py +0 -0
  296. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release.py +0 -0
  297. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release_bindings.py +0 -0
  298. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/release_compose.py +0 -0
  299. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/schema.py +0 -0
  300. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/instrument/ingest/transport.py +0 -0
  301. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/__init__.py +0 -0
  302. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/availability.py +0 -0
  303. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/calibration.py +0 -0
  304. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/directional.py +0 -0
  305. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/eprocess.py +0 -0
  306. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/event_policy.py +0 -0
  307. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/event_read.py +0 -0
  308. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/jury.py +0 -0
  309. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/read.py +0 -0
  310. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/read_input.py +0 -0
  311. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement/stratify.py +0 -0
  312. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/__init__.py +0 -0
  313. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/read.py +0 -0
  314. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/measurement_contract/semantics.py +0 -0
  315. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/__init__.py +0 -0
  316. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/_record_errors.py +0 -0
  317. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/_record_format.py +0 -0
  318. {underwrite-0.1.1 → underwrite-0.2.0}/src/underwrite/service/records.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: underwrite
3
- Version: 0.1.1
3
+ Version: 0.2.0
4
4
  Summary: Local evidence normalization, measurement, and acceptance checks
5
5
  License-Expression: Apache-2.0
6
6
  License-File: LICENSE
@@ -21,7 +21,7 @@ Description-Content-Type: text/markdown
21
21
 
22
22
  Find mismatches. Keep missing evidence visible. Know what to check next.
23
23
 
24
- **Local CLI & Python library · Typed JSON · No server or model calls**
24
+ **Local CLI & Python library · JSON results · No server or model calls**
25
25
 
26
26
  [![CI](https://github.com/ntts9990/underwrite/actions/workflows/ci.yml/badge.svg)](https://github.com/ntts9990/underwrite/actions/workflows/ci.yml)
27
27
  [![PyPI](https://img.shields.io/pypi/v/underwrite)](https://pypi.org/project/underwrite/)
@@ -34,25 +34,24 @@ Find mismatches. Keep missing evidence visible. Know what to check next.
34
34
 
35
35
  </div>
36
36
 
37
- Already have evaluation exports from **goppi, Langfuse, DeepEval, or Promptfoo**?
38
- A score or summary alone does not show whether supplied rows match declared
39
- totals or whether the planned case inventory is complete. For supported local
40
- profiles, underwrite checks relationships it can verify and makes unknown
41
- coverage explicit. It returns typed CLI/Python results with findings, input
42
- identity, and interpretation limits. You or your agent can use them to request
43
- the next piece of evidence.
44
-
45
- Underwrite complements the tools that run evaluations: goppi owns execution,
46
- trace/verifier binding, and reruns; other producers own their exports and scores.
47
- Underwrite examines supplied artifacts and can retain selected bytes. It does
48
- not rerun an agent, independently authenticate a producer, or assign a universal
49
- quality score. No Docker, dashboard, or provider credentials are needed to
50
- inspect the supported local files.
37
+ Evaluation tools run evaluations and export result files. Underwrite reads the
38
+ local files it supports and shows what those files establish, what remains
39
+ unknown, and what evidence to check next. For example, it checks whether
40
+ Promptfoo’s reported totals agree with its supplied result rows. You or your
41
+ coding agent use the JSON result to investigate and decide. You can also save
42
+ selected inputs with the result for later review.
43
+
44
+ Underwrite does not rerun evaluations, independently prove a producer's verdict,
45
+ or assign one quality score across tools. Checking supported local files needs
46
+ no Docker, dashboard, or provider credentials.
51
47
 
52
48
  > [!NOTE]
53
- > **This guide targets `underwrite==0.1.1`.** Its CLI and Python workflows work
54
- > without a checkout. The older `0.0.1` artifacts lack these workflows and have
55
- > a separate `pins refresh` caution; see [Install](#install).
49
+ > **This guide targets `underwrite==0.2.0`.** [Install the CLI](#installed-package--020)
50
+ > for automatic input handling, focused evidence views and retained trajectory review.
51
+ > A [source checkout](#development-checkout--current-main) is optional for repository
52
+ > examples and development. Use `capabilities --json` to confirm the installed routes.
53
+ > Upgrading from 0.1.1? Read the [migration guide](https://github.com/ntts9990/underwrite/blob/main/MIGRATION.md).
54
+ > The older `0.0.1` has a separate pin-manifest caution; see [Install](#install).
56
55
 
57
56
  ## Quickstart
58
57
 
@@ -61,18 +60,19 @@ synthetic download below uses `curl`.
61
60
  File-based CLI workflows are tested with CPython 3.12 on Linux and macOS. They
62
61
  require Unix/POSIX file APIs, including for ordinary artifact reads.
63
62
 
63
+ ### Installed package · 0.2.0
64
+
64
65
  **1. Install and check a file.** The app-only install resolves a compatible
65
66
  `underwrite-core` automatically. Download the public synthetic example; no
66
67
  checkout or API key is needed:
67
68
 
68
69
  ```sh
69
- uv tool install 'underwrite==0.1.1'
70
+ uv tool install 'underwrite==0.2.0'
70
71
  underwrite capabilities --json
71
72
  example_dir=$(mktemp -d)
72
73
  curl -fsSLo "$example_dir/promptfoo.json" \
73
74
  https://raw.githubusercontent.com/ntts9990/underwrite/cb913d39bce628b93dcd3b06fcc4ce36f9dc66af/fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json
74
- underwrite audit-promptfoo "$example_dir/promptfoo.json" \
75
- --version 0.123.0 --json
75
+ underwrite audit-promptfoo "$example_dir/promptfoo.json" --json
76
76
  ```
77
77
 
78
78
  The same `uv tool install` command updates an older uv-managed `underwrite`
@@ -86,7 +86,7 @@ from its actual output:
86
86
 
87
87
  ```json
88
88
  {
89
- "schema": "promptfoo_accounting.v1",
89
+ "schema": "promptfoo_accounting.v2",
90
90
  "declared_relation": "inconsistent",
91
91
  "producer_stats": {"errors": 1, "failures": 1, "successes": 1, "total": 3},
92
92
  "row_tallies": {"errors": 1, "failures": 0, "successes": 1, "total": 2},
@@ -98,6 +98,8 @@ from its actual output:
98
98
  "PLANNED_DENOMINATOR_UNKNOWN",
99
99
  "CAUSE_UNKNOWN",
100
100
  "SOURCE_CLAIMS_UNVERIFIED",
101
+ "UNCONSUMED_FIELDS_AND_POPULATIONS_NOT_ANALYZED",
102
+ "SOURCE_HASH_AND_VERSION_LINKABLE",
101
103
  "NO_APPROVAL_AUTHORITY"
102
104
  ]
103
105
  }
@@ -110,21 +112,40 @@ explicit. Exit `0` means the report was produced, not that the counts agree or
110
112
  the source's verdict was verified. To keep the input and analysis together,
111
113
  [save them in a local record](#keep-results-you-can-return-to).
112
114
 
113
- For your own completed Promptfoo export, replace the example path and select its
114
- supported source version. The synthetic download above is a constructed example,
115
- not an unchanged producer export.
115
+ For your own completed Promptfoo export, replace the path and use a
116
+ [supported input shape](#supported-formats). The downloaded file is a
117
+ constructed example, not an unchanged producer export.
116
118
 
117
- <details>
118
- <summary><strong>Try the native measurement → acceptance workflow</strong></summary>
119
+ ### Development checkout · current main
119
120
 
120
- This example uses policies and fixtures in the GitHub checkout; they are not
121
- included in the installed package. To run it, prepare a checkout first:
121
+ Use a checkout to work on the source or run repository examples. Run its commands
122
+ through `uv run` inside that directory, keeping the installed tool separate. If you
123
+ already have a checkout, open its root and start at `uv sync` below:
122
124
 
123
125
  ```sh
124
126
  git clone https://github.com/ntts9990/underwrite.git
125
127
  cd underwrite
126
128
  uv sync --frozen --all-groups
129
+ uv run --frozen --no-sync underwrite capabilities --json
130
+ uv run --frozen --no-sync underwrite audit-promptfoo \
131
+ fixtures/examples/evidence/synthetic-promptfoo-count-mismatch.json --json
132
+ ```
133
+
134
+ This uses the same synthetic counts as above: 3 declared results versus 2 rows.
135
+ The automatic reader produces `promptfoo_accounting.v2` and preserves this
136
+ fixture's declared `0.123.0` version as an unverified source claim. Inputs without
137
+ a version declaration retain an unknown release. No version flag, producer
138
+ installation, Goppi, or provider call is needed for this example.
139
+ Use this `uv run --frozen --no-sync underwrite` prefix for all checkout commands.
140
+ See [included workflows](#included-in-020) for the other routes.
127
141
 
142
+ <details>
143
+ <summary><strong>Try the native measurement → acceptance workflow</strong></summary>
144
+
145
+ After preparing the checkout above, run this example from its root. Its policies
146
+ and fixtures live in the repository, outside the installed package:
147
+
148
+ ```sh
128
149
  mkdir -p output
129
150
  uv run --frozen --no-sync underwrite ingest \
130
151
  --format underwrite.evidence-bundle --version v1 \
@@ -154,53 +175,46 @@ See [measurement meaning](https://github.com/ntts9990/underwrite/blob/main/AGENT
154
175
 
155
176
  | Question after an evaluation | What underwrite gives you |
156
177
  | --- | --- |
157
- | Does the summary match the supplied records? | Profile-specific accounting: Promptfoo count deltas, DeepEval evaluator errors, or Langfuse score associations. |
158
- | What is still unknown? | Explicit missingness and limits such as an unknown planned denominator or unverified source claim, so a zero is not silently substituted. |
159
- | Can this evidence support a measurement? | A separate native binary-event path with declared semantics and policy; external scores do not become a universal quality verdict. |
160
- | Can I revisit the conclusion? | Typed CLI/Python JSON and optional private records tying selected input bytes to the analysis, with verification and retrieval. |
178
+ | Does the summary match the supplied records? | Compare Promptfoo declared counts with rows. Separately account for DeepEval scored rows and evaluator errors, or Langfuse score associations; their aggregate summaries are not assumed checked. |
179
+ | What is still unknown? | See when planned cases are unknown or source claims are unverified; unknown is not counted as zero. |
180
+ | Can this evidence support a measurement? | Use the separate native binary-event path with a declared event and policy; external scores are not a universal quality verdict. |
181
+ | Can I revisit the conclusion? | Save selected input bytes and the analysis in a private record, then verify and retrieve them later. |
161
182
 
162
183
  ## How it works
163
184
 
164
- ```mermaid
165
- flowchart LR
166
- A[Supported local eval file] --> B[Select exact profile]
167
- B --> C[Check supplied records and missingness]
168
- C --> D[Typed JSON: finding, identity, limits]
169
- D --> E[Agent or person gathers next evidence]
170
- C --> F[Optional private record]
171
- F --> G[Verify and retrieve]
172
- H[Native event evidence + policy] --> I[Measure]
173
- I --> D
174
- ```
185
+ - **Evaluation tool:** runs the evaluation and exports result files.
186
+ - **Underwrite:** checks the supplied results, returns JSON with findings and
187
+ unanswered questions, and can save selected original files with the result.
188
+ - **You or your agent:** check the missing evidence and decide what to do next.
175
189
 
176
- The command checks only what its selected profile supports. The calling agent
177
- interprets the JSON and can pursue the next evidence task; underwrite does not
178
- apply a proposed change or grant merge authority. Local records keep selected
179
- inputs and results for later review.
190
+ Each command checks only its selected input profile. Source verdicts remain
191
+ source claims; Underwrite does not apply changes or grant merge authority. Its
192
+ separate native binary-event workflow measures only a declared event under an
193
+ explicit policy, not arbitrary external scores.
180
194
 
181
195
  ## Agent use
182
196
 
183
- Give your coding agent the installed CLI and a completed local export. It can
184
- discover the available routes, execute them, and use the JSON in its response:
185
-
186
- ```sh
187
- underwrite --help
188
- underwrite capabilities --json
189
- ```
197
+ Give your coding agent a completed local export and your chosen entry path.
198
+ Have it inspect `capabilities --json` and the command's `--help` using that same
199
+ executable: `underwrite` for the published tool, or
200
+ `uv run --frozen --no-sync underwrite` from the checkout. Package version alone
201
+ does not identify the available workflows.
190
202
 
191
203
  Python callers can use
192
- `underwrite.instrument.ingest.accounting.audit_promptfoo_file(Path(...))` for
193
- the same result as a dictionary. Select an exact supported profile before
194
- interpreting any producer-specific field.
204
+ `underwrite.instrument.ingest.accounting.audit_promptfoo_file(Path(...))` in
205
+ the chosen environment. Automatic intake selects a supported reader from the
206
+ input shape without guessing a producer release. Use explicit historical selectors
207
+ when reproducing an older contract, as described in the migration guide.
195
208
 
196
209
  Example task for an agent with terminal access:
197
210
 
198
- > Review `/absolute/path/to/completed-promptfoo-export.json` with the installed
199
- > `underwrite` 0.1.1 CLI as a Promptfoo 0.123.0 export. Use `capabilities --json`
200
- > and `audit-promptfoo --version 0.123.0`.
201
- > Report `declared_relation`, `producer_stats`, `row_tallies`, `raw_hash`,
202
- > `limitations`, and the next useful action. Do not rerun the producer or treat
203
- > exit `0` as agreement or approval.
211
+ > Review `/absolute/path/to/completed-promptfoo-export.json` using our selected
212
+ > Underwrite entry path. Inspect its capabilities and help, then use the
213
+ > supported accounting route. Use automatic intake when advertised; otherwise
214
+ > require evidence of a supported published profile rather than guessing a version.
215
+ > Report count differences, input identity, missing evidence, interpretation
216
+ > limits, and the next useful check. Do not rerun the producer or treat exit `0`
217
+ > as agreement or approval.
204
218
 
205
219
  The optional [underwrite-review skill](https://github.com/ntts9990/underwrite/blob/main/skills/underwrite-review/SKILL.md) provides
206
220
  reusable guidance for Codex and Claude Code. [Setup and usage](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#optional-artifact-review-skill)
@@ -210,67 +224,76 @@ The same CLI and importable Python APIs are available to your own automation.
210
224
 
211
225
  ## Supported formats
212
226
 
213
- Support is for the **specific local payload profiles below**, not every API,
214
- export, or version offered by each producer. Select the version explicitly.
227
+ Support covers selected local payloads, not every producer API, export, or
228
+ version. Use capabilities from your chosen executable and the linked workflows
229
+ for exact input requirements.
230
+
231
+ ### Supported profiles · 0.2.0
215
232
 
216
233
  | Your input | Supported profile | Start here |
217
234
  | --- | --- | --- |
218
- | **goppi** | Completed baseline exchange manifest v2; separate paired-handoff route | [`audit-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#audit-a-completed-goppi-baseline-as-common-evidence) for descriptive common JSON · [`project-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#project-a-completed-goppi-baseline-into-event-observations) for native measurement · [`review-handoff`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#review-a-completed-paired-handoff-in-one-call) |
219
- | **Langfuse** | `4.35.0` server blob exports: observations-v2 and scores JSON arrays | [`audit-langfuse`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-langfuse-score-kinds-and-associations), with an optional content-minimized row index |
220
- | **DeepEval** | Ordinary TestRun JSON, `4.1.1` or `4.2.8` | [`audit-deepeval`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors) |
221
- | **Promptfoo** | Eval output JSON, `0.123.0` or `0.124.1` | [`audit-promptfoo`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#source-checkout-pilot-audit-promptfoo-counts) |
222
- | **External common result** | `evidence_result.v1` JSON from a producer | [`inspect-result`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#inspect-an-external-common-result) for contract screening and availability counts, with optional evidence-item `next_action` and declared metric-binding counts |
235
+ | **Goppi** | Completed baseline exchange manifest v2; P7 handoff; adopted paired trajectory | [`audit-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#audit-a-completed-goppi-baseline-as-common-evidence) · [`project-baseline`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#project-a-completed-goppi-baseline-into-event-observations) · [`review-handoff`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#review-a-completed-paired-handoff-in-one-call) · [`audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#compare-a-goppi-paired-trajectory) |
236
+ | **Langfuse** | Compatible blob arrays or official-CLI list captures; automatic pair selection | [`audit-langfuse`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-langfuse-score-kinds-and-associations) |
237
+ | **DeepEval** | Compatible TestRun; automatic accounting, optional trace view and meaning bindings | [`audit-deepeval`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors) |
238
+ | **Promptfoo** | SummaryV3 eval output; automatic intake, historical selectors retained | [`audit-promptfoo`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#audit-promptfoo-counts) |
239
+ | **External common result** | `evidence_result.v1` or `evidence_result.v2` | [`inspect-result`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#inspect-an-external-common-result) |
223
240
  | **Native evidence** | `underwrite.evidence-bundle` / `v1` | `ingest` → `measure` → `accept` |
224
241
  | **Inspect AI** | `inspect.eval-log` / `0.3.263` | `ingest` |
225
242
  | **OpenInference** | `openinference.traces` / `0.1.38` | `ingest` |
226
243
  | **OTLP JSON** | `otlp-json.traces` / `0.160.0` | `ingest` |
227
244
 
228
- For completed exports already on your machine, the installed CLI accepts them
229
- directly; select the exact producer profile you have:
245
+ ### Included in 0.2.0
246
+
247
+ - **Automatic intake:** compatible Promptfoo SummaryV3, DeepEval TestRun, and
248
+ Langfuse blob arrays or [official-CLI list captures](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#read-official-langfuse-cli-captures-offline)
249
+ without selecting a producer version/profile. Actual producer release stays
250
+ unknown when the payload does not establish it; explicit selectors remain for
251
+ reproducing older contracts.
252
+ - **Focused views and storage:** selected DeepEval trace structure, version-neutral
253
+ `deepeval_metric_bindings.v2` packs, and Langfuse row-index recording. Follow the
254
+ linked producer workflows above for their separate contracts and limits.
255
+ - **Common results:** `inspect-result` also accepts `evidence_result.v2`.
256
+ - **Goppi steps:** [`audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#compare-a-goppi-paired-trajectory)
257
+ compares recorded tool states and evaluation results from a paired+trajectory
258
+ export, including runs that stopped early. This optional route requires installed
259
+ Goppi for offline verification; it does not run providers, establish causality,
260
+ or measure retry effects. [`records audit-trajectory`](https://github.com/ntts9990/underwrite/blob/main/STORAGE.md#retain-a-goppi-paired-trajectory)
261
+ retains the same inputs and result; `records read-trajectory` reopens them
262
+ without Goppi or the original export.
230
263
 
231
- ```sh
232
- underwrite audit-promptfoo /path/to/promptfoo-output.json --version 0.124.1 --json
233
- underwrite audit-deepeval /path/to/deepeval-test-run.json --version 4.2.8 --json
234
- underwrite audit-langfuse --observations /path/to/observations.json \
235
- --scores /path/to/scores.json --json
236
- underwrite audit-baseline --exchange /path/to/goppi-baseline-exchange --json
237
- ```
238
-
239
- DeepEval accounting defaults to `4.1.1`; select `--version 4.2.8` for that
240
- profile. For a reusable owner-asserted metric-name pack and common descriptive
241
- JSON, see the [DeepEval binding workflow](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#account-for-deepeval-scores-and-evaluator-errors).
242
- Promptfoo accounting defaults to `0.123.0`; select `--version 0.124.1`
243
- for that profile. Langfuse's version identifies the server export, not its SDK.
244
264
  See [artifact requirements](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#get-the-right-producer-artifact) and
245
- [producer checks](https://github.com/ntts9990/underwrite/blob/main/EVIDENCE_REVIEW.md) for the tested scope and fixture origins.
246
-
247
- Generic `ingest` / `project` normalizes supported exports to an observation;
248
- external observations do not enter the current `measure` path. goppi uses its
249
- explicit exchange adapters: `audit-baseline` produces descriptive
250
- `evidence_result.v1` without a policy or directional verdict, while
251
- `project-baseline` can produce native event observations. `prepare-event-policy`
252
- binds a consumer configuration while leaving statistical assumptions unknown.
253
- For an already produced `evidence_result.v1` file, `inspect-result` checks the
254
- installed contract and item counts without verifying the producer's source or
255
- bundle, or admitting that result to measurement or acceptance.
256
- Its optional action view counts recorded next actions for evidence remediation,
257
- not executed tool calls or an agent trajectory.
265
+ [producer checks](https://github.com/ntts9990/underwrite/blob/main/EVIDENCE_REVIEW.md)
266
+ for fixture origins and tested scope. A controlled fixture does not prove
267
+ live-server interoperability or export completeness.
268
+
269
+ Moving from published 0.1.1? See the [migration guide](https://github.com/ntts9990/underwrite/blob/main/MIGRATION.md)
270
+ for changed default JSON contracts, historical selectors, and existing records.
271
+
272
+ Generic `ingest` / `project` normalizes supported exports; external observations
273
+ do not enter the current `measure` path. Goppi `audit-baseline` produces
274
+ descriptive common evidence, while `project-baseline` can produce native event
275
+ observations. `inspect-result` screens the installed common-result contract and
276
+ counts items without verifying source truth or admitting them to measurement
277
+ or acceptance. Its action view counts recorded next checks, not executed tools.
258
278
 
259
279
  ## Keep results you can return to
260
280
 
261
281
  Use `records` when you want the analysis and its selected source bytes retained
262
- together. For a completed local Promptfoo export:
282
+ together. This example uses the **published package** with a completed local
283
+ Promptfoo export:
263
284
 
264
285
  ```sh
265
286
  underwrite records audit-promptfoo /path/to/promptfoo-output.json \
266
- --version 0.123.0 --root ./review-store --id review-001 --json
287
+ --root ./review-store --id review-001 --json
267
288
 
268
289
  underwrite records read-json \
269
290
  --root ./review-store --id review-001 \
270
291
  --name analysis/accounting.json --json
271
292
  ```
272
293
 
273
- Use a fresh ID for each capture; existing records are never overwritten.
294
+ Automatic intake needs no producer version flag. In a checkout, prefix both
295
+ commands with `uv run --frozen --no-sync`. Use a fresh ID for each capture;
296
+ existing records are never overwritten.
274
297
  `records verify` checks stored integrity. `records measure` captures a native
275
298
  measurement, and `records replay-measure` recalculates it from verified saved
276
299
  inputs using the current engine, without executing the original producer.
@@ -304,7 +327,7 @@ Install the versioned app; its dependency resolves the compatible core. Use
304
327
  profiles:
305
328
 
306
329
  ```sh
307
- uv tool install 'underwrite==0.1.1'
330
+ uv tool install 'underwrite==0.2.0'
308
331
  underwrite --help
309
332
  underwrite capabilities --json
310
333
  ```
@@ -317,12 +340,12 @@ For the importable Python API, install into a Python 3.12 environment instead:
317
340
 
318
341
  ```sh
319
342
  uv venv --python 3.12 .venv
320
- uv pip install --python .venv/bin/python 'underwrite==0.1.1'
343
+ uv pip install --python .venv/bin/python 'underwrite==0.2.0'
321
344
  .venv/bin/python - <<'PY'
322
345
  from pathlib import Path
323
346
  from underwrite.instrument.ingest.accounting import audit_promptfoo_file
324
347
 
325
- result = audit_promptfoo_file(Path("/path/to/promptfoo-output.json"), version="0.123.0")
348
+ result = audit_promptfoo_file(Path("/path/to/promptfoo-output.json"))
326
349
  print(result["declared_relation"])
327
350
  PY
328
351
  ```
@@ -358,46 +381,8 @@ a manifest with an expected hash, `check` examines release evidence linkage,
358
381
  vocabulary, and `drift check` / `pins refresh` work with supplied pin registries.
359
382
  [JSON contracts](https://github.com/ntts9990/underwrite/tree/main/contracts/) define each command family's results and errors.
360
383
 
361
- Legacy `absence` and `drift check` exemption files and `boundary scan` allowlists
362
- are bounded to 1 MiB, 64 JSON container levels, and 1024 policy entries each.
363
- `boundary scan` reads at most 4 MiB per selected JSON or JSONL file and 32 MiB
364
- of captured bytes across the command, including its allowlist. It admits up to
365
- 4096 selected files, 100000 visited directory entries, 100000 physical JSONL
366
- lines and 1 MiB per raw line (including its delimiter), and 4096 leaks including
367
- allowed rows. Its selected JSON or human output is capped at 1 MiB including
368
- the final newline. Duplicate keys and nonfinite numbers are invalid; supplied
369
- `--now` and policy expiry dates must be real ASCII `YYYY-MM-DD` dates. Admission
370
- failures return a typed `UNCONFIGURED` result with exit 2. An absent `--now`
371
- still yields `NOW_REQUIRED` only when the semantic check needs a clock.
372
-
373
- `absence` retains `Path.glob` matching and refuses more than 4096 matching
374
- regular files; it does not bound directory traversal within `Path.glob` or
375
- suppressed internal filesystem errors. Boundary scans follow an explicitly
376
- selected directory link and regular file links, but do not descend into a child
377
- directory link. These selections do not establish path containment against a
378
- hostile filesystem.
379
-
380
- `drift check` admits up to 256 manifests and 4096 declared pin files while
381
- counting at most 100000 visited registry entries. Each manifest is limited to
382
- 1 MiB and 64 JSON container levels; each selected source or pin copy is limited
383
- to 4 MiB. The 32 MiB command capture budget includes exemption bytes, manifests,
384
- and both copies. Read-only inspection follows selected directory and file links;
385
- it does not create an atomic filesystem snapshot or a hostile-path containment
386
- guarantee. Missing or nonregular selected copies are reported as DRIFT; malformed,
387
- unreadable, or excessive registry input returns a typed `UNCONFIGURED` result
388
- without fabricated population counts.
389
-
390
- `pins refresh` selects one registry repository and copies committed Git blobs.
391
- Refresh paths must be normalized relative `/`-separated components; empty, dot,
392
- parent, backslash, Unicode control/format, oversized, and colliding destinations
393
- are refused.
394
- The selected registry and its existing destinations are checked before mutation.
395
- Each pin is replaced with a new inode under its opened registry parent, and
396
- `PINS.json` is replaced last. Admission or Git-source failures leave the registry
397
- unchanged. A later filesystem failure reports `publication: may_be_partial`;
398
- refresh is not an all-files transaction or a durability guarantee.
399
- Refresh captures each committed Git blob within 4 MiB and the 32 MiB aggregate
400
- budget, and checks the fully serialized 1 MiB manifest before its first write.
384
+ For scan and pin resource limits and filesystem-selection caveats, see
385
+ [Scan and pin limits](https://github.com/ntts9990/underwrite/blob/main/AGENT_USAGE.md#scan-and-pin-limits).
401
386
 
402
387
  <a id="development-checkout-evidence-pilots"></a>
403
388
 
@@ -430,10 +415,13 @@ Before using OIDC publishing, create the GitHub environment `pypi`, restrict it
430
415
  `ntts9990`, repository `underwrite`, workflow `publish.yml`, and environment `pypi`.
431
416
  Once configured, run **Publish PyPI** manually from `main` in GitHub Actions.
432
417
 
433
- The workflow validates both packages, then publishes core before the application;
434
- only the separate publish job receives OIDC permission. No stored PyPI API token
435
- is required. Retry a partial upload with the same artifacts: existing registry
436
- files must match exactly, and changed artifacts require a new version.
418
+ The workflow validates both packages and defaults to publishing only the app
419
+ (`publish_core: false`), after testing it with the published compatible core. Select
420
+ `publish_core: true` only for an explicitly approved core release; core upload must
421
+ succeed before app upload. The unchanged core remains `0.1.1` for this app release.
422
+ Only the publish job receives OIDC permission; no stored PyPI token is required.
423
+ Retry partial uploads with identical artifacts. Existing registry files must match
424
+ exactly; never re-upload a rebuilt, different archive under an existing version.
437
425
 
438
426
  See the official [uv publishing guide](https://docs.astral.sh/uv/guides/package/)
439
427
  and [PyPI Trusted Publisher guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).