agent-eval-flow 0.5.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (317) hide show
  1. agent_eval_flow-0.5.1/.gitattributes +6 -0
  2. agent_eval_flow-0.5.1/.github/ISSUE_TEMPLATE/bug_report.yml +65 -0
  3. agent_eval_flow-0.5.1/.github/ISSUE_TEMPLATE/feature_request.yml +58 -0
  4. agent_eval_flow-0.5.1/.github/pull_request_template.md +15 -0
  5. agent_eval_flow-0.5.1/.github/workflows/publish.yml +136 -0
  6. agent_eval_flow-0.5.1/.github/workflows/tests.yml +32 -0
  7. agent_eval_flow-0.5.1/CONTRIBUTING.md +30 -0
  8. agent_eval_flow-0.5.1/MANIFEST.in +8 -0
  9. agent_eval_flow-0.5.1/PKG-INFO +197 -0
  10. agent_eval_flow-0.5.1/README.md +169 -0
  11. agent_eval_flow-0.5.1/THIRD_PARTY_NOTICES.md +24 -0
  12. agent_eval_flow-0.5.1/docs/.nojekyll +0 -0
  13. agent_eval_flow-0.5.1/docs/AGENT_COMPATIBILITY_RED_TEAM.md +247 -0
  14. agent_eval_flow-0.5.1/docs/AGENT_EVAL_FLOW_REVIEW.md +618 -0
  15. agent_eval_flow-0.5.1/docs/AGENT_EVAL_FLOW_VISION.md +185 -0
  16. agent_eval_flow-0.5.1/docs/ARCHITECTURE_REUSE_REVIEW.md +259 -0
  17. agent_eval_flow-0.5.1/docs/DATA_CONTRACT.md +511 -0
  18. agent_eval_flow-0.5.1/docs/IMPLEMENTATION.md +182 -0
  19. agent_eval_flow-0.5.1/docs/INTEGRATION_DECISION.md +243 -0
  20. agent_eval_flow-0.5.1/docs/LIBRARY_OBJECT_MODEL.md +846 -0
  21. agent_eval_flow-0.5.1/docs/LLD_ARCHITECTURE.md +258 -0
  22. agent_eval_flow-0.5.1/docs/LLD_LEVEL_2.md +574 -0
  23. agent_eval_flow-0.5.1/docs/LLD_REUSE_RED_TEAM.md +317 -0
  24. agent_eval_flow-0.5.1/docs/MODULE_COMMUNICATION.md +229 -0
  25. agent_eval_flow-0.5.1/docs/PROJECT_PLAN.md +1306 -0
  26. agent_eval_flow-0.5.1/docs/RELEASE.md +60 -0
  27. agent_eval_flow-0.5.1/docs/UNIFIED_ASSESSMENT_FLOW.md +212 -0
  28. agent_eval_flow-0.5.1/docs/WHAT_AGENT_EVAL_FLOW_ADDS.md +104 -0
  29. agent_eval_flow-0.5.1/docs/WORKFLOW_EVALUATION_MODEL.md +1224 -0
  30. agent_eval_flow-0.5.1/docs/assets/agent-eval-flow-banner.png +0 -0
  31. agent_eval_flow-0.5.1/docs/assets/agent-eval-flow-banner.svg +79 -0
  32. agent_eval_flow-0.5.1/docs/contracts/agent_eval_flow.pyi +801 -0
  33. agent_eval_flow-0.5.1/docs/diagrams/01-pipeline.png +0 -0
  34. agent_eval_flow-0.5.1/docs/diagrams/01-pipeline.svg +97 -0
  35. agent_eval_flow-0.5.1/docs/diagrams/02-opensre.png +0 -0
  36. agent_eval_flow-0.5.1/docs/diagrams/02-opensre.svg +121 -0
  37. agent_eval_flow-0.5.1/docs/diagrams/03-openkritt.png +0 -0
  38. agent_eval_flow-0.5.1/docs/diagrams/03-openkritt.svg +117 -0
  39. agent_eval_flow-0.5.1/docs/diagrams/04-objectives.png +0 -0
  40. agent_eval_flow-0.5.1/docs/diagrams/04-objectives.svg +97 -0
  41. agent_eval_flow-0.5.1/docs/diagrams/AGENT_EVAL_FLOW_VISION.drawio +1 -0
  42. agent_eval_flow-0.5.1/docs/diagrams/Open-in-drawio.url +2 -0
  43. agent_eval_flow-0.5.1/docs/diagrams/build_module_map.py +148 -0
  44. agent_eval_flow-0.5.1/docs/diagrams/build_vision.py +317 -0
  45. agent_eval_flow-0.5.1/docs/diagrams/module-communication.drawio +330 -0
  46. agent_eval_flow-0.5.1/docs/diagrams/module-communication.png +0 -0
  47. agent_eval_flow-0.5.1/docs/diagrams/module-communication.svg +104 -0
  48. agent_eval_flow-0.5.1/docs/diagrams/vision.html +1 -0
  49. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre-walkthrough.html +31 -0
  50. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre-walkthrough.png +0 -0
  51. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre.html +45 -0
  52. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre.png +0 -0
  53. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt-walkthrough.html +31 -0
  54. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt-walkthrough.png +0 -0
  55. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt.html +45 -0
  56. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt.png +0 -0
  57. agent_eval_flow-0.5.1/docs/examples/agent-evaluation/README.md +107 -0
  58. agent_eval_flow-0.5.1/docs/implementation/local_openkritt.md +136 -0
  59. agent_eval_flow-0.5.1/docs/implementation/local_opensre.md +83 -0
  60. agent_eval_flow-0.5.1/docs/implementation/native_workflow_adapters.md +129 -0
  61. agent_eval_flow-0.5.1/docs/implementation/pypi_publishing.md +52 -0
  62. agent_eval_flow-0.5.1/docs/implementation/runtime_evidence.md +186 -0
  63. agent_eval_flow-0.5.1/docs/implementation/unified_assessment.md +160 -0
  64. agent_eval_flow-0.5.1/docs/index.html +121 -0
  65. agent_eval_flow-0.5.1/docs/lld/ASSESSMENT_CONTRACT.md +372 -0
  66. agent_eval_flow-0.5.1/docs/lld/README.md +330 -0
  67. agent_eval_flow-0.5.1/docs/lld/adapters/README.md +946 -0
  68. agent_eval_flow-0.5.1/docs/lld/evaluation/README.md +628 -0
  69. agent_eval_flow-0.5.1/docs/lld/execution/README.md +512 -0
  70. agent_eval_flow-0.5.1/docs/lld/objects/README.md +329 -0
  71. agent_eval_flow-0.5.1/docs/lld/pipeline/README.md +449 -0
  72. agent_eval_flow-0.5.1/docs/lld/reporting/README.md +199 -0
  73. agent_eval_flow-0.5.1/docs/lld/results/README.md +475 -0
  74. agent_eval_flow-0.5.1/docs/lld/storage/README.md +246 -0
  75. agent_eval_flow-0.5.1/docs/research/agent_redteam/browser_use_source_probe.json +82 -0
  76. agent_eval_flow-0.5.1/docs/research/agent_redteam/probe_browser_use_source.py +190 -0
  77. agent_eval_flow-0.5.1/docs/research/agent_redteam/probe_swe_agent_fixture.py +111 -0
  78. agent_eval_flow-0.5.1/docs/research/agent_redteam/swe_agent_fixture_probe.json +56 -0
  79. agent_eval_flow-0.5.1/docs/research/agentcompass_a2e_integration_evidence.md +157 -0
  80. agent_eval_flow-0.5.1/docs/research/agentcompass_a2e_lld_review.md +112 -0
  81. agent_eval_flow-0.5.1/docs/research/foundations_and_native_integration.md +76 -0
  82. agent_eval_flow-0.5.1/docs/research/harbor_skillevaluator_integration_evidence.md +62 -0
  83. agent_eval_flow-0.5.1/docs/research/inspect_architecture_review.md +117 -0
  84. agent_eval_flow-0.5.1/docs/research/nemo_integration_evidence.md +178 -0
  85. agent_eval_flow-0.5.1/docs/research/nemo_lld_review.md +166 -0
  86. agent_eval_flow-0.5.1/docs/research/promptfoo_mlflow_architecture_review.md +167 -0
  87. agent_eval_flow-0.5.1/docs/research/pydantic_evals_architecture_review.md +133 -0
  88. agent_eval_flow-0.5.1/docs/research/skill_harbor_lld_review.md +139 -0
  89. agent_eval_flow-0.5.1/examples/archive_review.py +143 -0
  90. agent_eval_flow-0.5.1/examples/assessment_review.py +88 -0
  91. agent_eval_flow-0.5.1/examples/integrations/README.openkritt-local.md +78 -0
  92. agent_eval_flow-0.5.1/examples/integrations/__init__.py +1 -0
  93. agent_eval_flow-0.5.1/examples/integrations/local_cli.py +250 -0
  94. agent_eval_flow-0.5.1/examples/integrations/openkritt_local.py +342 -0
  95. agent_eval_flow-0.5.1/examples/integrations/openkritt_observer.py +156 -0
  96. agent_eval_flow-0.5.1/examples/integrations/openkritt_setup.py +250 -0
  97. agent_eval_flow-0.5.1/examples/integrations/opensre_local.Dockerfile +17 -0
  98. agent_eval_flow-0.5.1/examples/integrations/opensre_local.py +215 -0
  99. agent_eval_flow-0.5.1/examples/integrations/opensre_local_session.py +292 -0
  100. agent_eval_flow-0.5.1/examples/integrations/opensre_local_worker.py +21 -0
  101. agent_eval_flow-0.5.1/examples/integrations/vertex.py +422 -0
  102. agent_eval_flow-0.5.1/examples/openkritt_attempts_report.py +183 -0
  103. agent_eval_flow-0.5.1/examples/openkritt_local_regrade.py +75 -0
  104. agent_eval_flow-0.5.1/examples/openkritt_local_review.py +443 -0
  105. agent_eval_flow-0.5.1/examples/opensre_attempts_report.py +62 -0
  106. agent_eval_flow-0.5.1/examples/opensre_local_review.py +373 -0
  107. agent_eval_flow-0.5.1/examples/profiles.local.example.json +31 -0
  108. agent_eval_flow-0.5.1/examples/workflows/README.md +74 -0
  109. agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v1.json +79 -0
  110. agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v1.provenance.json +18 -0
  111. agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v2.json +79 -0
  112. agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v2.provenance.json +22 -0
  113. agent_eval_flow-0.5.1/pyproject.toml +37 -0
  114. agent_eval_flow-0.5.1/pytest.ini +7 -0
  115. agent_eval_flow-0.5.1/requirements-test.txt +1 -0
  116. agent_eval_flow-0.5.1/scripts/smoke_wheel.py +27 -0
  117. agent_eval_flow-0.5.1/scripts/verify_acceptance_checkpoint.py +21 -0
  118. agent_eval_flow-0.5.1/scripts/verify_distribution.py +39 -0
  119. agent_eval_flow-0.5.1/setup.cfg +4 -0
  120. agent_eval_flow-0.5.1/src/agent_eval_flow/__init__.py +23 -0
  121. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/__init__.py +21 -0
  122. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/claude_code.py +134 -0
  123. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/cli.py +433 -0
  124. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/codex.py +97 -0
  125. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/common.py +81 -0
  126. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/configuration_files_assessment.py +157 -0
  127. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/harbor.py +305 -0
  128. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/harness_eval_assessment.py +325 -0
  129. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/nat.py +180 -0
  130. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/openkritt.py +709 -0
  131. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/opensre.py +509 -0
  132. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/process.py +308 -0
  133. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/skillevaluator.py +126 -0
  134. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/snapshot_binding_assessment.py +148 -0
  135. agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/worker.py +366 -0
  136. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/__init__.py +2 -0
  137. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/aggregation.py +116 -0
  138. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/assessment_projection.py +57 -0
  139. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/compiler.py +134 -0
  140. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/configuration.py +103 -0
  141. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/engine.py +333 -0
  142. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/primitives.py +26 -0
  143. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/runtime_checks.py +172 -0
  144. agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/scoring.py +85 -0
  145. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/__init__.py +4 -0
  146. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/capture.py +331 -0
  147. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/configuration_assessment.py +190 -0
  148. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/dispatch_assessment.py +324 -0
  149. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/importing.py +106 -0
  150. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/planning.py +72 -0
  151. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/preflight.py +99 -0
  152. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/runner.py +169 -0
  153. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/snapshot_assessment.py +14 -0
  154. agent_eval_flow-0.5.1/src/agent_eval_flow/execution/snapshot_binding_assessment.py +118 -0
  155. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/__init__.py +10 -0
  156. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/assessment.py +522 -0
  157. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/assessment_validation.py +651 -0
  158. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/base.py +79 -0
  159. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/candidate.py +33 -0
  160. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/dataset.py +121 -0
  161. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/errors.py +33 -0
  162. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/identity.py +118 -0
  163. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/records.py +871 -0
  164. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/runset.py +316 -0
  165. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/runtime_evidence.py +70 -0
  166. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/validation.py +141 -0
  167. agent_eval_flow-0.5.1/src/agent_eval_flow/objects/values.py +57 -0
  168. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/__init__.py +1 -0
  169. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/api.py +68 -0
  170. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment.py +284 -0
  171. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment_bindings.py +26 -0
  172. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment_preflight.py +105 -0
  173. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/bindings.py +22 -0
  174. agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/preflight.py +72 -0
  175. agent_eval_flow-0.5.1/src/agent_eval_flow/py.typed +0 -0
  176. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/__init__.py +1 -0
  177. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/assessment_html.py +53 -0
  178. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/html.py +150 -0
  179. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/assessment_report.html.j2 +67 -0
  180. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/report.css +1 -0
  181. agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/report.html.j2 +118 -0
  182. agent_eval_flow-0.5.1/src/agent_eval_flow/results/__init__.py +1 -0
  183. agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_comparison.py +74 -0
  184. agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_query.py +74 -0
  185. agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_selection.py +160 -0
  186. agent_eval_flow-0.5.1/src/agent_eval_flow/results/comparison.py +102 -0
  187. agent_eval_flow-0.5.1/src/agent_eval_flow/results/query.py +159 -0
  188. agent_eval_flow-0.5.1/src/agent_eval_flow/results/selection.py +193 -0
  189. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/__init__.py +1 -0
  190. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/artifacts.py +131 -0
  191. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/assessment_codec.py +107 -0
  192. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/assessment_manifests.py +38 -0
  193. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/codec.py +296 -0
  194. agent_eval_flow-0.5.1/src/agent_eval_flow/storage/manifests.py +111 -0
  195. agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/PKG-INFO +197 -0
  196. agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/SOURCES.txt +315 -0
  197. agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/dependency_links.txt +1 -0
  198. agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/requires.txt +11 -0
  199. agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/top_level.txt +1 -0
  200. agent_eval_flow-0.5.1/tests/ACCEPTANCE_MANIFEST.json +278 -0
  201. agent_eval_flow-0.5.1/tests/BASELINE.md +57 -0
  202. agent_eval_flow-0.5.1/tests/README.md +73 -0
  203. agent_eval_flow-0.5.1/tests/SHOWCASE_MANIFEST.json +352 -0
  204. agent_eval_flow-0.5.1/tests/__init__.py +1 -0
  205. agent_eval_flow-0.5.1/tests/adapters/test_cli_boundary.py +245 -0
  206. agent_eval_flow-0.5.1/tests/adapters/test_configuration_assessment.py +546 -0
  207. agent_eval_flow-0.5.1/tests/adapters/test_native_workflows.py +354 -0
  208. agent_eval_flow-0.5.1/tests/adapters/test_opensre_native_serialization.py +31 -0
  209. agent_eval_flow-0.5.1/tests/adapters/test_process_boundary.py +156 -0
  210. agent_eval_flow-0.5.1/tests/adapters/test_worker_boundary.py +179 -0
  211. agent_eval_flow-0.5.1/tests/conftest.py +44 -0
  212. agent_eval_flow-0.5.1/tests/contracts/__init__.py +1 -0
  213. agent_eval_flow-0.5.1/tests/contracts/test_assessment_consumers.py +258 -0
  214. agent_eval_flow-0.5.1/tests/contracts/test_assessment_objects.py +203 -0
  215. agent_eval_flow-0.5.1/tests/contracts/test_assessment_pipeline.py +319 -0
  216. agent_eval_flow-0.5.1/tests/contracts/test_assessment_receipts.py +238 -0
  217. agent_eval_flow-0.5.1/tests/contracts/test_capture.py +424 -0
  218. agent_eval_flow-0.5.1/tests/contracts/test_configuration.py +310 -0
  219. agent_eval_flow-0.5.1/tests/contracts/test_evaluation.py +444 -0
  220. agent_eval_flow-0.5.1/tests/contracts/test_native_jobs.py +297 -0
  221. agent_eval_flow-0.5.1/tests/contracts/test_pipeline_runtime.py +143 -0
  222. agent_eval_flow-0.5.1/tests/contracts/test_reducers.py +117 -0
  223. agent_eval_flow-0.5.1/tests/contracts/test_results_storage.py +257 -0
  224. agent_eval_flow-0.5.1/tests/e2e/BASELINE.md +83 -0
  225. agent_eval_flow-0.5.1/tests/e2e/PROFILES.md +280 -0
  226. agent_eval_flow-0.5.1/tests/e2e/README.md +125 -0
  227. agent_eval_flow-0.5.1/tests/e2e/SHOWCASE.md +90 -0
  228. agent_eval_flow-0.5.1/tests/e2e/__init__.py +1 -0
  229. agent_eval_flow-0.5.1/tests/e2e/cloud/test_openkritt_gcp.py +488 -0
  230. agent_eval_flow-0.5.1/tests/e2e/cloud/test_opensre_gcp.py +257 -0
  231. agent_eval_flow-0.5.1/tests/e2e/conftest.py +42 -0
  232. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/README.md +230 -0
  233. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/.gitignore +14 -0
  234. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/LICENSE.txt +28 -0
  235. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/README.rst +68 -0
  236. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/__init__.py +51 -0
  237. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/auth.py +116 -0
  238. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/blog.py +125 -0
  239. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/db.py +56 -0
  240. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/schema.sql +20 -0
  241. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/static/style.css +134 -0
  242. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/auth/login.html +15 -0
  243. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/auth/register.html +15 -0
  244. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/base.html +24 -0
  245. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/create.html +15 -0
  246. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/index.html +28 -0
  247. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/update.html +19 -0
  248. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/pyproject.toml +40 -0
  249. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/conftest.py +62 -0
  250. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/data.sql +8 -0
  251. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_auth.py +69 -0
  252. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_blog.py +83 -0
  253. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_db.py +29 -0
  254. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_factory.py +12 -0
  255. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/source_manifest.json +166 -0
  256. agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/workflow.json +79 -0
  257. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/README.md +160 -0
  258. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/SOURCES.json +110 -0
  259. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/context.json +18 -0
  260. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/incident_store.py +184 -0
  261. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/incidents.json +7 -0
  262. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/HDFS_2k.log +2000 -0
  263. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/LOGHUB_LICENSE +11 -0
  264. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/LICENSE +201 -0
  265. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/agent.py +183 -0
  266. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/events.py +306 -0
  267. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/harness.py +412 -0
  268. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/mixins.py +111 -0
  269. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/ports.py +317 -0
  270. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/react_loop.py +786 -0
  271. agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/turn_results.py +69 -0
  272. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/INSPECT.md +24 -0
  273. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/LICENSE +21 -0
  274. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/PROVENANCE.json +26 -0
  275. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/README.md +31 -0
  276. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/native-format.md +103 -0
  277. agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/pydicom__pydicom-1458.traj +257 -0
  278. agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/SKILL.md +9 -0
  279. agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/toy_agent.py +37 -0
  280. agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/toy_tool.py +16 -0
  281. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/MISSION.md +39 -0
  282. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/README.md +162 -0
  283. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/SOURCE_MANIFEST.json +47 -0
  284. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/cases.json +12 -0
  285. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/CHANGES.rst +213 -0
  286. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/LICENSE.txt +28 -0
  287. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/README.md +40 -0
  288. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/src/markupsafe/__init__.py +395 -0
  289. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/src/markupsafe/_native.py +8 -0
  290. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/tests/test_escape.py +68 -0
  291. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/investigation_tool.py +120 -0
  292. agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/report.schema.json +36 -0
  293. agent_eval_flow-0.5.1/tests/e2e/profiles.example.json +101 -0
  294. agent_eval_flow-0.5.1/tests/e2e/support.py +247 -0
  295. agent_eval_flow-0.5.1/tests/e2e/test_archived_trajectory.py +205 -0
  296. agent_eval_flow-0.5.1/tests/e2e/test_data_contract.py +263 -0
  297. agent_eval_flow-0.5.1/tests/e2e/test_live_toy.py +84 -0
  298. agent_eval_flow-0.5.1/tests/e2e/test_live_workflow.py +361 -0
  299. agent_eval_flow-0.5.1/tests/e2e/test_runtime_evidence.py +56 -0
  300. agent_eval_flow-0.5.1/tests/e2e/test_toy_pipeline.py +206 -0
  301. agent_eval_flow-0.5.1/tests/integrations/PROFILES.md +161 -0
  302. agent_eval_flow-0.5.1/tests/integrations/test_optional_libraries.py +256 -0
  303. agent_eval_flow-0.5.1/tests/unit/fixtures/openkritt_scan2_lineage.json +452 -0
  304. agent_eval_flow-0.5.1/tests/unit/test_cli_adapters.py +103 -0
  305. agent_eval_flow-0.5.1/tests/unit/test_openkritt_lineage.py +61 -0
  306. agent_eval_flow-0.5.1/tests/unit/test_openkritt_local_connection.py +283 -0
  307. agent_eval_flow-0.5.1/tests/unit/test_openkritt_local_review.py +105 -0
  308. agent_eval_flow-0.5.1/tests/unit/test_openkritt_setup.py +53 -0
  309. agent_eval_flow-0.5.1/tests/unit/test_openkritt_workflow_variant.py +139 -0
  310. agent_eval_flow-0.5.1/tests/unit/test_opensre_local_review.py +147 -0
  311. agent_eval_flow-0.5.1/tests/unit/test_opensre_local_runtime.py +53 -0
  312. agent_eval_flow-0.5.1/tests/unit/test_opensre_local_session.py +119 -0
  313. agent_eval_flow-0.5.1/tests/unit/test_optional_adapter_mappers.py +266 -0
  314. agent_eval_flow-0.5.1/tests/unit/test_runtime_evidence.py +149 -0
  315. agent_eval_flow-0.5.1/tests/unit/test_runtime_evidence_report.py +142 -0
  316. agent_eval_flow-0.5.1/tests/unit/test_showcase_recipe.py +96 -0
  317. agent_eval_flow-0.5.1/tests/unit/test_vertex_showcase.py +258 -0
@@ -0,0 +1,6 @@
1
+ # The acceptance checkpoint hashes exact original bytes, including upstream logs.
2
+ tests/** -text
3
+ pytest.ini -text
4
+ requirements-test.txt -text
5
+ # Workflow derivation receipts pin exact bytes; never normalize their line endings.
6
+ examples/workflows/*.json -text whitespace=cr-at-eol
@@ -0,0 +1,65 @@
1
+ {
2
+ "name": "Bug report",
3
+ "description": "Report unexpected behavior with a minimal reproduction.",
4
+ "body": [
5
+ {
6
+ "type": "markdown",
7
+ "attributes": {
8
+ "value": "Small, reproducible examples are most useful. Remove credentials, private task data, and sensitive prompts before sharing anything. Raw run logs are not required."
9
+ }
10
+ },
11
+ {
12
+ "type": "textarea",
13
+ "id": "environment",
14
+ "attributes": {
15
+ "label": "Versions and runtime",
16
+ "description": "Include the package version or commit, Python version, OS, and relevant runtime or importer. Use not applicable where appropriate.",
17
+ "placeholder": "Agent Eval Flow:\nPython / OS:\nRuntime, adapter or import format (and revision, if known):"
18
+ },
19
+ "validations": {
20
+ "required": true
21
+ }
22
+ },
23
+ {
24
+ "type": "textarea",
25
+ "id": "reproduction",
26
+ "attributes": {
27
+ "label": "Minimal reproduction",
28
+ "description": "Give the smallest code, configuration and steps that reproduce the problem. Use public or redacted sample data."
29
+ },
30
+ "validations": {
31
+ "required": true
32
+ }
33
+ },
34
+ {
35
+ "type": "textarea",
36
+ "id": "expected",
37
+ "attributes": {
38
+ "label": "Expected behavior",
39
+ "description": "What result or behavior did you expect?"
40
+ },
41
+ "validations": {
42
+ "required": true
43
+ }
44
+ },
45
+ {
46
+ "type": "textarea",
47
+ "id": "observed",
48
+ "attributes": {
49
+ "label": "Observed behavior",
50
+ "description": "What happened instead? Include relevant statuses or error messages."
51
+ },
52
+ "validations": {
53
+ "required": true
54
+ }
55
+ },
56
+ {
57
+ "type": "textarea",
58
+ "id": "evidence",
59
+ "attributes": {
60
+ "label": "Scrubbed evidence (optional)",
61
+ "description": "A short traceback, report excerpt or screenshot can help. Prefer the relevant excerpt over a complete capture, and remove sensitive content."
62
+ }
63
+ }
64
+ ]
65
+ }
@@ -0,0 +1,58 @@
1
+ {
2
+ "name": "Feature or use case",
3
+ "description": "Explain a workflow, missing capability, or useful improvement.",
4
+ "body": [
5
+ {
6
+ "type": "markdown",
7
+ "attributes": {
8
+ "value": "Describe the decision or workflow you want to support. A specific example helps, but private data and raw run logs are not needed."
9
+ }
10
+ },
11
+ {
12
+ "type": "textarea",
13
+ "id": "use_case",
14
+ "attributes": {
15
+ "label": "Problem or use case",
16
+ "description": "What are you evaluating, and what decision are you trying to make?"
17
+ },
18
+ "validations": {
19
+ "required": true
20
+ }
21
+ },
22
+ {
23
+ "type": "textarea",
24
+ "id": "desired_outcome",
25
+ "attributes": {
26
+ "label": "Desired outcome",
27
+ "description": "What would a useful result or supported workflow look like? An example input and output can help."
28
+ },
29
+ "validations": {
30
+ "required": true
31
+ }
32
+ },
33
+ {
34
+ "type": "textarea",
35
+ "id": "current_workflow",
36
+ "attributes": {
37
+ "label": "Current workflow (optional)",
38
+ "description": "Mention the runtime or imported format, relevant versions, and your current approach or workaround."
39
+ }
40
+ },
41
+ {
42
+ "type": "textarea",
43
+ "id": "proposal",
44
+ "attributes": {
45
+ "label": "Proposed approach or alternatives (optional)",
46
+ "description": "Share a possible solution, alternatives, or constraints. A proposed implementation is not required."
47
+ }
48
+ },
49
+ {
50
+ "type": "textarea",
51
+ "id": "examples",
52
+ "attributes": {
53
+ "label": "Public or redacted examples (optional)",
54
+ "description": "Link related issues or include a small example. Remove credentials, private prompts and task data."
55
+ }
56
+ }
57
+ ]
58
+ }
@@ -0,0 +1,15 @@
1
+ ## What changed
2
+
3
+ <!-- Describe the problem and resulting behavior. Link related issues if relevant. -->
4
+
5
+ ## Validation
6
+
7
+ <!-- List checks and their outcomes, including anything not run and why.
8
+ For a bug fix, include the regression check that demonstrates the behavior.
9
+ See CONTRIBUTING.md for the local checks. -->
10
+
11
+ ## Notes for review
12
+
13
+ <!-- Optional: explain changes to metric meaning, evidence/provenance,
14
+ missing values, or compatibility. Note remaining limitations.
15
+ Keep credentials, private task data and raw local captures out of the PR. -->
@@ -0,0 +1,136 @@
1
+ name: Publish to PyPI
2
+
3
+ on:
4
+ release:
5
+ types: [published]
6
+ # Retry a published release with: gh workflow run publish.yml --ref v0.5.1
7
+ # Dispatching a branch never enables either job.
8
+ workflow_dispatch:
9
+
10
+ permissions: {}
11
+
12
+ concurrency:
13
+ group: pypi-${{ github.ref }}
14
+ cancel-in-progress: false
15
+
16
+ jobs:
17
+ build:
18
+ if: github.repository == 'guybass/agent-eval-flow' && startsWith(github.ref, 'refs/tags/v')
19
+ runs-on: ubuntu-latest
20
+ timeout-minutes: 15
21
+ permissions:
22
+ contents: read
23
+ actions: read
24
+ outputs:
25
+ artifact-id: ${{ steps.distributions.outputs.artifact-id }}
26
+ steps:
27
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
28
+ with:
29
+ ref: ${{ github.sha }}
30
+ persist-credentials: false
31
+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
32
+ with:
33
+ python-version: '3.13'
34
+ - name: Verify published release, source version, and all six CI jobs
35
+ env:
36
+ GH_TOKEN: ${{ github.token }}
37
+ run: |
38
+ python - <<'PY'
39
+ import json
40
+ import os
41
+ from pathlib import Path
42
+ import re
43
+ import subprocess
44
+ import tomllib
45
+ from urllib.parse import urlencode
46
+ from urllib.request import Request, urlopen
47
+
48
+ def require(condition, message):
49
+ if not condition:
50
+ raise SystemExit(message)
51
+
52
+ repository = os.environ['GITHUB_REPOSITORY']
53
+ tag = os.environ['GITHUB_REF_NAME']
54
+ sha = os.environ['GITHUB_SHA']
55
+ require(repository == 'guybass/agent-eval-flow', 'Unexpected repository')
56
+ require(os.environ['GITHUB_REF'] == f'refs/tags/{tag}', 'A version tag is required')
57
+ require(re.fullmatch(r'v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)', tag),
58
+ 'Only final vMAJOR.MINOR.PATCH release tags are publishable')
59
+ require(os.environ['GITHUB_EVENT_NAME'] in {'release', 'workflow_dispatch'},
60
+ 'Unexpected publishing event')
61
+
62
+ def api(path):
63
+ request = Request(f'https://api.github.com/repos/{repository}/{path}', headers={
64
+ 'Authorization': f'Bearer {os.environ["GH_TOKEN"]}',
65
+ 'Accept': 'application/vnd.github+json',
66
+ 'X-GitHub-Api-Version': '2022-11-28',
67
+ })
68
+ with urlopen(request, timeout=30) as response:
69
+ return json.load(response)
70
+
71
+ release = api(f'releases/tags/{tag}')
72
+ require(release['tag_name'] == tag and not release['draft']
73
+ and not release['prerelease'] and release['published_at'],
74
+ 'The tag must have a published, non-prerelease GitHub release')
75
+ require(api(f'commits/{tag}')['sha'] == sha, 'Tag moved or event SHA differs')
76
+ require(subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() == sha,
77
+ 'Checkout differs from the verified release commit')
78
+ require(api(f'compare/{sha}...main')['status'] in {'ahead', 'identical'},
79
+ 'Release commit must belong to main history')
80
+ project = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8'))['project']
81
+ require(project['name'] == 'agent-eval-flow' and project['version'] == tag[1:],
82
+ 'Project name/version must match the release tag exactly')
83
+
84
+ query = urlencode({'head_sha': sha, 'event': 'push', 'branch': 'main', 'per_page': 100})
85
+ runs = api(f'actions/workflows/tests.yml/runs?{query}')['workflow_runs']
86
+ require(runs, 'No Tests and package run exists for this commit on main')
87
+ run = max(runs, key=lambda row: row['id'])
88
+ require(run['head_sha'] == sha and run['head_branch'] == 'main'
89
+ and run['event'] == 'push' and run['status'] == 'completed'
90
+ and run['conclusion'] == 'success',
91
+ 'Latest Tests and package run for this commit must finish successfully')
92
+ jobs = api(f'actions/runs/{run["id"]}/jobs?filter=latest&per_page=100')
93
+ expected = {f'offline ({system}, {version})'
94
+ for system in ('ubuntu-latest', 'windows-latest')
95
+ for version in ('3.11', '3.12', '3.13')}
96
+ require(jobs['total_count'] == len(expected) and len(jobs['jobs']) == len(expected)
97
+ and {job['name'] for job in jobs['jobs']} == expected
98
+ and all(job['status'] == 'completed' and job['conclusion'] == 'success'
99
+ for job in jobs['jobs']),
100
+ 'All six expected Python/OS matrix jobs must pass; skipped jobs do not qualify')
101
+ print(f'Verified {tag} at {sha}; CI: {run["html_url"]}')
102
+ PY
103
+ - name: Build and validate distributions
104
+ run: |
105
+ python -m pip install build twine
106
+ python -m build
107
+ python -m twine check --strict dist/*
108
+ python scripts/verify_distribution.py
109
+ - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
110
+ id: distributions
111
+ with:
112
+ name: distributions-${{ github.ref_name }}
113
+ path: dist/*
114
+ if-no-files-found: error
115
+ retention-days: 7
116
+
117
+ publish:
118
+ needs: build
119
+ if: github.repository == 'guybass/agent-eval-flow' && startsWith(github.ref, 'refs/tags/v')
120
+ runs-on: ubuntu-latest
121
+ timeout-minutes: 10
122
+ environment:
123
+ name: pypi
124
+ url: https://pypi.org/project/agent-eval-flow/
125
+ permissions:
126
+ id-token: write
127
+ steps:
128
+ - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
129
+ with:
130
+ artifact-ids: ${{ needs.build.outputs.artifact-id }}
131
+ path: dist/
132
+ digest-mismatch: error
133
+ - name: Publish verified distributions with Trusted Publishing
134
+ uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
135
+ with:
136
+ attestations: true
@@ -0,0 +1,32 @@
1
+ name: Tests and package
2
+
3
+ on:
4
+ push:
5
+ pull_request:
6
+
7
+ permissions:
8
+ contents: read
9
+
10
+ jobs:
11
+ offline:
12
+ strategy:
13
+ fail-fast: false
14
+ matrix:
15
+ os: [ubuntu-latest, windows-latest]
16
+ python: ['3.11', '3.12', '3.13']
17
+ runs-on: ${{ matrix.os }}
18
+ timeout-minutes: 15
19
+ steps:
20
+ - uses: actions/checkout@v7
21
+ with:
22
+ persist-credentials: false
23
+ - uses: actions/setup-python@v7
24
+ with:
25
+ python-version: ${{ matrix.python }}
26
+ cache: pip
27
+ - run: python -m pip install -e ".[test,cli]" build
28
+ - run: python scripts/verify_acceptance_checkpoint.py
29
+ - run: python -m pytest --junitxml=test-artifacts/pytest.xml
30
+ - run: python examples/archive_review.py --output demo-output
31
+ - run: python -m build
32
+ - run: python scripts/verify_distribution.py
@@ -0,0 +1,30 @@
1
+ # Contributing
2
+
3
+ Agent Eval Flow is a developer preview. Issues and small pull requests are welcome.
4
+ For a bug, include the package/Python versions, a minimal reproducible example,
5
+ the expected outcome and the observed result. Remove credentials and private
6
+ task data before sharing evidence.
7
+
8
+ ## Local checks
9
+
10
+ From a checkout, using Python 3.11 or later:
11
+
12
+ ```bash
13
+ python -m pip install -e ".[test,cli]" build
14
+ python scripts/verify_acceptance_checkpoint.py
15
+ python -m pytest
16
+ python examples/archive_review.py --output demo-output
17
+ python -m build
18
+ python scripts/verify_distribution.py
19
+ ```
20
+
21
+ Default tests do not start live models. Live profiles require explicit selection
22
+ and prepared runtimes; their setup is documented in [tests/e2e/SHOWCASE.md](tests/e2e/SHOWCASE.md).
23
+
24
+ Keep metric meaning, provenance and missingness explicit. Fixes should include
25
+ a regression check that demonstrates the affected behavior. Preserve the frozen
26
+ acceptance fixtures and their hashes; add new fixtures separately.
27
+
28
+ Local outputs belong in ignored `demo-output/` or `test-artifacts/` directories.
29
+ Include only deliberately selected, reviewed report artifacts in documentation.
30
+ Third-party fixtures retain their own licenses and provenance.
@@ -0,0 +1,8 @@
1
+ include README.md CONTRIBUTING.md THIRD_PARTY_NOTICES.md pyproject.toml pytest.ini requirements-test.txt .gitattributes
2
+ graft .github
3
+ graft docs
4
+ graft tests
5
+ graft examples
6
+ graft scripts
7
+ global-exclude *.pyc *.pyo
8
+ global-exclude __pycache__
@@ -0,0 +1,197 @@
1
+ Metadata-Version: 2.4
2
+ Name: agent-eval-flow
3
+ Version: 0.5.1
4
+ Summary: Evidence-preserving evaluation of complete agent systems
5
+ Project-URL: Homepage, https://guybass.github.io/agent-eval-flow/
6
+ Project-URL: Repository, https://github.com/guybass/agent-eval-flow
7
+ Project-URL: Issues, https://github.com/guybass/agent-eval-flow/issues
8
+ Project-URL: Documentation, https://github.com/guybass/agent-eval-flow/blob/main/README.md
9
+ Project-URL: Examples, https://guybass.github.io/agent-eval-flow/
10
+ Project-URL: Changelog, https://github.com/guybass/agent-eval-flow/releases
11
+ Keywords: agents,agent-evaluation,llm,evaluation,execution-evidence
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Requires-Python: >=3.11
18
+ Description-Content-Type: text/markdown
19
+ Requires-Dist: pydantic<3,>=2.13.5
20
+ Requires-Dist: anyio<5,>=4.15.1
21
+ Requires-Dist: jinja2<4,>=3.1.6
22
+ Requires-Dist: typing-extensions>=4.14
23
+ Provides-Extra: test
24
+ Requires-Dist: pytest<10,>=8; extra == "test"
25
+ Requires-Dist: jsonschema<5,>=4.26; extra == "test"
26
+ Provides-Extra: cli
27
+ Requires-Dist: jsonschema<5,>=4.26; extra == "cli"
28
+
29
+ [![Agent Eval Flow — capture evidence, apply checks, compare a change](https://raw.githubusercontent.com/guybass/agent-eval-flow/main/docs/assets/agent-eval-flow-banner.png)](https://guybass.github.io/agent-eval-flow/)
30
+
31
+ # Agent Eval Flow
32
+
33
+ [![Release](https://img.shields.io/github/v/release/guybass/agent-eval-flow?include_prereleases&label=release&color=245c50)](https://github.com/guybass/agent-eval-flow/releases)
34
+ [![Tests and package](https://github.com/guybass/agent-eval-flow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/guybass/agent-eval-flow/actions/workflows/tests.yml)
35
+ [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-49659c?logo=python&logoColor=white)](https://github.com/guybass/agent-eval-flow/blob/main/pyproject.toml)
36
+ [![Example reports](https://img.shields.io/badge/reports-OpenSRE%20%2B%20OpenKritt-786495)](https://guybass.github.io/agent-eval-flow/)
37
+
38
+ **Turn agent runs into evidence you can use to improve the system.**
39
+
40
+ Evaluate complete agent setups: models, instructions, skills, tools, loops,
41
+ memory and environment. Run through an existing runtime or import saved logs,
42
+ apply your checks, and compare changes in a human-readable report.
43
+
44
+ **[Browse the reports](https://guybass.github.io/agent-eval-flow/)** ·
45
+ **[Try the offline example](#try-it)** ·
46
+ **[Read the design](https://github.com/guybass/agent-eval-flow/blob/main/docs/lld/README.md)** ·
47
+ **[Report an issue](https://github.com/guybass/agent-eval-flow/issues)**
48
+
49
+ This is a developer preview. Missing evidence stays unknown; configuration
50
+ inspection and behavioral evaluation can run in parallel. The Python package
51
+ is `agent_eval_flow`.
52
+
53
+ ## Try it
54
+
55
+ ```bash
56
+ git clone https://github.com/guybass/agent-eval-flow.git
57
+ cd agent-eval-flow
58
+ python -m pip install -e ".[test,cli]"
59
+ python -m pytest
60
+ python examples/archive_review.py --output demo-output
61
+ ```
62
+
63
+ The example imports a **real, downloaded SWE-agent repair history** containing
64
+ 12 actions, failed edits, corrections and a submitted patch. It produces:
65
+
66
+ - `demo-output/report.html`: task results, diagnostics and linked native evidence;
67
+ - `demo-output/result/`: the typed saved evaluation;
68
+ - `demo-output/regraded/`: a changed evaluation over the same saved execution.
69
+
70
+ It makes no model calls and never executes commands from the archive. The
71
+ [example source](https://github.com/guybass/agent-eval-flow/blob/main/examples/archive_review.py) shows an application-owned importer
72
+ and custom metrics using the production library.
73
+
74
+ Requires Python 3.11 or later. The default tests and archive example need no
75
+ model credentials. Native agent runs require the integration-specific setup below.
76
+
77
+ ## Example reports
78
+
79
+ Two real agent workflows, controlled synthetic tasks, and specific changes
80
+ measured from retained runs. Click a report to open it in your browser.
81
+
82
+ | OpenSRE: recovery and stopping | OpenKritt: executed proof |
83
+ | --- | --- |
84
+ | [![OpenSRE before-and-after evaluation report](https://raw.githubusercontent.com/guybass/agent-eval-flow/main/docs/examples/agent-evaluation/01-opensre.png)](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre.html) | [![OpenKritt before-and-after evaluation report](https://raw.githubusercontent.com/guybass/agent-eval-flow/main/docs/examples/agent-evaluation/02-openkritt.png)](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt.html) |
85
+ | Safe recovery stayed **2/2**; post-report calls fell **1 → 0** after a terminal-tool binding fix. | The demo grounding contract passed **0/1 → 1/1** after requiring execution and clarifying the reporting contract. |
86
+ | [Read the report](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre.html) · [Task and fix walkthrough](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre-walkthrough.html) | [Read the report](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt.html) · [Task and fix walkthrough](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt-walkthrough.html) |
87
+
88
+ These are small development experiments with synthetic tasks, not general
89
+ reliability benchmarks. The gallery contains presentation reports; raw local
90
+ run captures and unpublished social drafts are excluded. The offline example
91
+ above is the reproducible starting point for trying the evaluation API.
92
+ See the [case details, run identifiers and limitations](https://github.com/guybass/agent-eval-flow/blob/main/docs/examples/agent-evaluation/README.md).
93
+
94
+ ## Configure once, then evaluate
95
+
96
+ ```python
97
+ from agent_eval_flow import EvaluationPipeline, EvaluationResult
98
+
99
+ pipeline = EvaluationPipeline(
100
+ study=study, # data, candidates, execution policy and suite
101
+ backends=backends, # existing agent runtimes
102
+ evaluators=evaluators, # your metric implementations
103
+ )
104
+ result = pipeline.eval()
105
+ result.save("results/experiment")
106
+ result.report("results/experiment.html")
107
+
108
+ saved = EvaluationResult.load("results/experiment")
109
+ explanation = saved.explain(saved.runs.runs[0].id)
110
+ comparison = saved.compare("baseline", "challenger", metrics=("success_rate",))
111
+ selection = saved.select(policy) # change cost/latency/quality priorities
112
+
113
+ # Regrade captured runs without binding or invoking an agent backend.
114
+ regraded = EvaluationPipeline(study=changed_study, evaluators=evaluators).eval(runs=saved.runs)
115
+ ```
116
+
117
+ In an async application, use `await pipeline.aeval()`. Construction and planning
118
+ do not start an agent. Changing the selection policy does not rerun agents or
119
+ metrics. Comparisons currently provide descriptive differences; they do not
120
+ invent confidence intervals.
121
+
122
+ ## The six objects
123
+
124
+ | Object | Responsibility |
125
+ | --- | --- |
126
+ | `Study` | Question, candidates, execution conditions and evaluation suite |
127
+ | `EvalDataset` | Keyed task tables, public inputs and private evaluator references |
128
+ | `Candidate` | Model, skill, tool, flow and harness configuration |
129
+ | `RunSet` | Every assignment, output, execution, event, resource observation and failure |
130
+ | `EvalSuite` | Versioned metrics, dependencies, score rules and summaries |
131
+ | `EvaluationResult` | Saved measurements, explanations, comparisons and selection |
132
+
133
+ Pydantic validates the shared records. AnyIO coordinates runtime calls. Jinja2
134
+ renders self-contained reports. Native agent frameworks keep their loops and
135
+ schedulers. Our code owns the common evidence and comparison contracts.
136
+
137
+ Missing usage remains unknown, failed work remains in the assignment inventory,
138
+ and shared grading activities are counted once. A captured JSON `null` is distinct
139
+ from absent output. Detail rows explain tasks; they do not inflate the sample size.
140
+
141
+ ## Runtime evidence checks
142
+
143
+ Use the same optional observation contract to check instructions, tools, model
144
+ selection, loops, memory and environment state. Versioned collectors retain
145
+ declared and observed values with phase, boundary, coverage and evidence.
146
+ Reusable checks run through the existing evaluation pipeline, including saved
147
+ run regrading; reports show the expected and observed values. Missing evidence
148
+ stays unknown. See the [runtime evidence guide](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/runtime_evidence.md).
149
+
150
+ ## Runtime integrations
151
+
152
+ Concrete adapter modules cover Codex, Claude Code, OpenSRE, OpenKritt, Harbor,
153
+ SkillEvaluator imports and NeMo Agent Toolkit batch grading. Their native
154
+ capture/lifecycle code is separate from the core. Prepared service connections,
155
+ version-specific configuration and deployment credentials remain runtime bindings.
156
+ See [implementation status and setup](https://github.com/guybass/agent-eval-flow/blob/main/docs/IMPLEMENTATION.md) for the supported
157
+ boundaries and the live checks still required.
158
+
159
+ OpenSRE and OpenKritt are separate studies: compare candidate versions **within**
160
+ each project. Their GCP showcase tests require prepared native runtimes and rich
161
+ capture observers. Default offline test success does not establish live compatibility.
162
+
163
+ A [live local OpenKritt + Codex example](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/local_openkritt.md)
164
+ has now completed against 22 real Flaskr files: eight workflow executions, four
165
+ post-processing executions, 20 tool results and two candidate findings. It saves
166
+ native evidence, verifies the result round trip, and regrades without new model
167
+ calls. Its score checks integration behavior, not security accuracy.
168
+
169
+ The separate [local OpenSRE + Codex example](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/local_opensre.md)
170
+ has completed an investigation of the downloaded HDFS sample: six ReAct
171
+ iterations, eight real tool calls and a cited incident report. It retains native
172
+ events and CLI receipts, then saves, reloads and regrades the captured result.
173
+
174
+ ## Verification and design
175
+
176
+ Release checks cover the offline suite, archived-run import and regrading,
177
+ package contents, and installed-wheel reporting. See the
178
+ [developer-preview release notes](https://github.com/guybass/agent-eval-flow/blob/main/docs/RELEASE.md) for the verified release scope.
179
+
180
+ CI runs on **Windows and Ubuntu with Python 3.11, 3.12 and 3.13**. Checks cover
181
+ native-format fixtures, evidence transport, missing outcomes, persistence,
182
+ regrading and package contents. All **90 frozen acceptance/fixture files** are
183
+ verified by hash. Live model tests are opt-in; offline success does not establish
184
+ compatibility with every deployed agent runtime. Historical checkpoints remain
185
+ in [implementation status](https://github.com/guybass/agent-eval-flow/blob/main/docs/IMPLEMENTATION.md).
186
+
187
+ - [Data structures and configuration](https://github.com/guybass/agent-eval-flow/blob/main/docs/DATA_CONTRACT.md)
188
+ - [Unified assessment flow: configuration and execution in parallel](https://github.com/guybass/agent-eval-flow/blob/main/docs/UNIFIED_ASSESSMENT_FLOW.md)
189
+ - [Module tree and communication graph](https://github.com/guybass/agent-eval-flow/blob/main/docs/lld/README.md)
190
+ - [Real-workflow E2E scenarios](https://github.com/guybass/agent-eval-flow/blob/main/tests/e2e/SHOWCASE.md)
191
+ - [Visual pipeline and score examples](https://github.com/guybass/agent-eval-flow/blob/main/docs/AGENT_EVAL_FLOW_VISION.md)
192
+ - [Library reuse decisions](https://github.com/guybass/agent-eval-flow/blob/main/docs/INTEGRATION_DECISION.md)
193
+ - [What this adds to existing work](https://github.com/guybass/agent-eval-flow/blob/main/docs/WHAT_AGENT_EVAL_FLOW_ADDS.md)
194
+
195
+ The older design and test-writing checkpoints are retained as historical records.
196
+
197
+ For changes and bug reports, see [CONTRIBUTING.md](https://github.com/guybass/agent-eval-flow/blob/main/CONTRIBUTING.md).