agent-eval-flow 0.5.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- agent_eval_flow-0.5.1/.gitattributes +6 -0
- agent_eval_flow-0.5.1/.github/ISSUE_TEMPLATE/bug_report.yml +65 -0
- agent_eval_flow-0.5.1/.github/ISSUE_TEMPLATE/feature_request.yml +58 -0
- agent_eval_flow-0.5.1/.github/pull_request_template.md +15 -0
- agent_eval_flow-0.5.1/.github/workflows/publish.yml +136 -0
- agent_eval_flow-0.5.1/.github/workflows/tests.yml +32 -0
- agent_eval_flow-0.5.1/CONTRIBUTING.md +30 -0
- agent_eval_flow-0.5.1/MANIFEST.in +8 -0
- agent_eval_flow-0.5.1/PKG-INFO +197 -0
- agent_eval_flow-0.5.1/README.md +169 -0
- agent_eval_flow-0.5.1/THIRD_PARTY_NOTICES.md +24 -0
- agent_eval_flow-0.5.1/docs/.nojekyll +0 -0
- agent_eval_flow-0.5.1/docs/AGENT_COMPATIBILITY_RED_TEAM.md +247 -0
- agent_eval_flow-0.5.1/docs/AGENT_EVAL_FLOW_REVIEW.md +618 -0
- agent_eval_flow-0.5.1/docs/AGENT_EVAL_FLOW_VISION.md +185 -0
- agent_eval_flow-0.5.1/docs/ARCHITECTURE_REUSE_REVIEW.md +259 -0
- agent_eval_flow-0.5.1/docs/DATA_CONTRACT.md +511 -0
- agent_eval_flow-0.5.1/docs/IMPLEMENTATION.md +182 -0
- agent_eval_flow-0.5.1/docs/INTEGRATION_DECISION.md +243 -0
- agent_eval_flow-0.5.1/docs/LIBRARY_OBJECT_MODEL.md +846 -0
- agent_eval_flow-0.5.1/docs/LLD_ARCHITECTURE.md +258 -0
- agent_eval_flow-0.5.1/docs/LLD_LEVEL_2.md +574 -0
- agent_eval_flow-0.5.1/docs/LLD_REUSE_RED_TEAM.md +317 -0
- agent_eval_flow-0.5.1/docs/MODULE_COMMUNICATION.md +229 -0
- agent_eval_flow-0.5.1/docs/PROJECT_PLAN.md +1306 -0
- agent_eval_flow-0.5.1/docs/RELEASE.md +60 -0
- agent_eval_flow-0.5.1/docs/UNIFIED_ASSESSMENT_FLOW.md +212 -0
- agent_eval_flow-0.5.1/docs/WHAT_AGENT_EVAL_FLOW_ADDS.md +104 -0
- agent_eval_flow-0.5.1/docs/WORKFLOW_EVALUATION_MODEL.md +1224 -0
- agent_eval_flow-0.5.1/docs/assets/agent-eval-flow-banner.png +0 -0
- agent_eval_flow-0.5.1/docs/assets/agent-eval-flow-banner.svg +79 -0
- agent_eval_flow-0.5.1/docs/contracts/agent_eval_flow.pyi +801 -0
- agent_eval_flow-0.5.1/docs/diagrams/01-pipeline.png +0 -0
- agent_eval_flow-0.5.1/docs/diagrams/01-pipeline.svg +97 -0
- agent_eval_flow-0.5.1/docs/diagrams/02-opensre.png +0 -0
- agent_eval_flow-0.5.1/docs/diagrams/02-opensre.svg +121 -0
- agent_eval_flow-0.5.1/docs/diagrams/03-openkritt.png +0 -0
- agent_eval_flow-0.5.1/docs/diagrams/03-openkritt.svg +117 -0
- agent_eval_flow-0.5.1/docs/diagrams/04-objectives.png +0 -0
- agent_eval_flow-0.5.1/docs/diagrams/04-objectives.svg +97 -0
- agent_eval_flow-0.5.1/docs/diagrams/AGENT_EVAL_FLOW_VISION.drawio +1 -0
- agent_eval_flow-0.5.1/docs/diagrams/Open-in-drawio.url +2 -0
- agent_eval_flow-0.5.1/docs/diagrams/build_module_map.py +148 -0
- agent_eval_flow-0.5.1/docs/diagrams/build_vision.py +317 -0
- agent_eval_flow-0.5.1/docs/diagrams/module-communication.drawio +330 -0
- agent_eval_flow-0.5.1/docs/diagrams/module-communication.png +0 -0
- agent_eval_flow-0.5.1/docs/diagrams/module-communication.svg +104 -0
- agent_eval_flow-0.5.1/docs/diagrams/vision.html +1 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre-walkthrough.html +31 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre-walkthrough.png +0 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre.html +45 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/01-opensre.png +0 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt-walkthrough.html +31 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt-walkthrough.png +0 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt.html +45 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/02-openkritt.png +0 -0
- agent_eval_flow-0.5.1/docs/examples/agent-evaluation/README.md +107 -0
- agent_eval_flow-0.5.1/docs/implementation/local_openkritt.md +136 -0
- agent_eval_flow-0.5.1/docs/implementation/local_opensre.md +83 -0
- agent_eval_flow-0.5.1/docs/implementation/native_workflow_adapters.md +129 -0
- agent_eval_flow-0.5.1/docs/implementation/pypi_publishing.md +52 -0
- agent_eval_flow-0.5.1/docs/implementation/runtime_evidence.md +186 -0
- agent_eval_flow-0.5.1/docs/implementation/unified_assessment.md +160 -0
- agent_eval_flow-0.5.1/docs/index.html +121 -0
- agent_eval_flow-0.5.1/docs/lld/ASSESSMENT_CONTRACT.md +372 -0
- agent_eval_flow-0.5.1/docs/lld/README.md +330 -0
- agent_eval_flow-0.5.1/docs/lld/adapters/README.md +946 -0
- agent_eval_flow-0.5.1/docs/lld/evaluation/README.md +628 -0
- agent_eval_flow-0.5.1/docs/lld/execution/README.md +512 -0
- agent_eval_flow-0.5.1/docs/lld/objects/README.md +329 -0
- agent_eval_flow-0.5.1/docs/lld/pipeline/README.md +449 -0
- agent_eval_flow-0.5.1/docs/lld/reporting/README.md +199 -0
- agent_eval_flow-0.5.1/docs/lld/results/README.md +475 -0
- agent_eval_flow-0.5.1/docs/lld/storage/README.md +246 -0
- agent_eval_flow-0.5.1/docs/research/agent_redteam/browser_use_source_probe.json +82 -0
- agent_eval_flow-0.5.1/docs/research/agent_redteam/probe_browser_use_source.py +190 -0
- agent_eval_flow-0.5.1/docs/research/agent_redteam/probe_swe_agent_fixture.py +111 -0
- agent_eval_flow-0.5.1/docs/research/agent_redteam/swe_agent_fixture_probe.json +56 -0
- agent_eval_flow-0.5.1/docs/research/agentcompass_a2e_integration_evidence.md +157 -0
- agent_eval_flow-0.5.1/docs/research/agentcompass_a2e_lld_review.md +112 -0
- agent_eval_flow-0.5.1/docs/research/foundations_and_native_integration.md +76 -0
- agent_eval_flow-0.5.1/docs/research/harbor_skillevaluator_integration_evidence.md +62 -0
- agent_eval_flow-0.5.1/docs/research/inspect_architecture_review.md +117 -0
- agent_eval_flow-0.5.1/docs/research/nemo_integration_evidence.md +178 -0
- agent_eval_flow-0.5.1/docs/research/nemo_lld_review.md +166 -0
- agent_eval_flow-0.5.1/docs/research/promptfoo_mlflow_architecture_review.md +167 -0
- agent_eval_flow-0.5.1/docs/research/pydantic_evals_architecture_review.md +133 -0
- agent_eval_flow-0.5.1/docs/research/skill_harbor_lld_review.md +139 -0
- agent_eval_flow-0.5.1/examples/archive_review.py +143 -0
- agent_eval_flow-0.5.1/examples/assessment_review.py +88 -0
- agent_eval_flow-0.5.1/examples/integrations/README.openkritt-local.md +78 -0
- agent_eval_flow-0.5.1/examples/integrations/__init__.py +1 -0
- agent_eval_flow-0.5.1/examples/integrations/local_cli.py +250 -0
- agent_eval_flow-0.5.1/examples/integrations/openkritt_local.py +342 -0
- agent_eval_flow-0.5.1/examples/integrations/openkritt_observer.py +156 -0
- agent_eval_flow-0.5.1/examples/integrations/openkritt_setup.py +250 -0
- agent_eval_flow-0.5.1/examples/integrations/opensre_local.Dockerfile +17 -0
- agent_eval_flow-0.5.1/examples/integrations/opensre_local.py +215 -0
- agent_eval_flow-0.5.1/examples/integrations/opensre_local_session.py +292 -0
- agent_eval_flow-0.5.1/examples/integrations/opensre_local_worker.py +21 -0
- agent_eval_flow-0.5.1/examples/integrations/vertex.py +422 -0
- agent_eval_flow-0.5.1/examples/openkritt_attempts_report.py +183 -0
- agent_eval_flow-0.5.1/examples/openkritt_local_regrade.py +75 -0
- agent_eval_flow-0.5.1/examples/openkritt_local_review.py +443 -0
- agent_eval_flow-0.5.1/examples/opensre_attempts_report.py +62 -0
- agent_eval_flow-0.5.1/examples/opensre_local_review.py +373 -0
- agent_eval_flow-0.5.1/examples/profiles.local.example.json +31 -0
- agent_eval_flow-0.5.1/examples/workflows/README.md +74 -0
- agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v1.json +79 -0
- agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v1.provenance.json +18 -0
- agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v2.json +79 -0
- agent_eval_flow-0.5.1/examples/workflows/openkritt_flaskr_local_v2.provenance.json +22 -0
- agent_eval_flow-0.5.1/pyproject.toml +37 -0
- agent_eval_flow-0.5.1/pytest.ini +7 -0
- agent_eval_flow-0.5.1/requirements-test.txt +1 -0
- agent_eval_flow-0.5.1/scripts/smoke_wheel.py +27 -0
- agent_eval_flow-0.5.1/scripts/verify_acceptance_checkpoint.py +21 -0
- agent_eval_flow-0.5.1/scripts/verify_distribution.py +39 -0
- agent_eval_flow-0.5.1/setup.cfg +4 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/__init__.py +23 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/__init__.py +21 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/claude_code.py +134 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/cli.py +433 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/codex.py +97 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/common.py +81 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/configuration_files_assessment.py +157 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/harbor.py +305 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/harness_eval_assessment.py +325 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/nat.py +180 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/openkritt.py +709 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/opensre.py +509 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/process.py +308 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/skillevaluator.py +126 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/snapshot_binding_assessment.py +148 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/adapters/worker.py +366 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/__init__.py +2 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/aggregation.py +116 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/assessment_projection.py +57 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/compiler.py +134 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/configuration.py +103 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/engine.py +333 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/primitives.py +26 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/runtime_checks.py +172 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/evaluation/scoring.py +85 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/__init__.py +4 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/capture.py +331 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/configuration_assessment.py +190 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/dispatch_assessment.py +324 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/importing.py +106 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/planning.py +72 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/preflight.py +99 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/runner.py +169 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/snapshot_assessment.py +14 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/execution/snapshot_binding_assessment.py +118 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/__init__.py +10 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/assessment.py +522 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/assessment_validation.py +651 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/base.py +79 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/candidate.py +33 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/dataset.py +121 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/errors.py +33 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/identity.py +118 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/records.py +871 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/runset.py +316 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/runtime_evidence.py +70 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/validation.py +141 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/objects/values.py +57 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/__init__.py +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/api.py +68 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment.py +284 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment_bindings.py +26 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/assessment_preflight.py +105 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/bindings.py +22 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/pipeline/preflight.py +72 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/py.typed +0 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/__init__.py +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/assessment_html.py +53 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/html.py +150 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/assessment_report.html.j2 +67 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/report.css +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/reporting/templates/report.html.j2 +118 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/__init__.py +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_comparison.py +74 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_query.py +74 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/assessment_selection.py +160 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/comparison.py +102 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/query.py +159 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/results/selection.py +193 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/__init__.py +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/artifacts.py +131 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/assessment_codec.py +107 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/assessment_manifests.py +38 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/codec.py +296 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow/storage/manifests.py +111 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/PKG-INFO +197 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/SOURCES.txt +315 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/dependency_links.txt +1 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/requires.txt +11 -0
- agent_eval_flow-0.5.1/src/agent_eval_flow.egg-info/top_level.txt +1 -0
- agent_eval_flow-0.5.1/tests/ACCEPTANCE_MANIFEST.json +278 -0
- agent_eval_flow-0.5.1/tests/BASELINE.md +57 -0
- agent_eval_flow-0.5.1/tests/README.md +73 -0
- agent_eval_flow-0.5.1/tests/SHOWCASE_MANIFEST.json +352 -0
- agent_eval_flow-0.5.1/tests/__init__.py +1 -0
- agent_eval_flow-0.5.1/tests/adapters/test_cli_boundary.py +245 -0
- agent_eval_flow-0.5.1/tests/adapters/test_configuration_assessment.py +546 -0
- agent_eval_flow-0.5.1/tests/adapters/test_native_workflows.py +354 -0
- agent_eval_flow-0.5.1/tests/adapters/test_opensre_native_serialization.py +31 -0
- agent_eval_flow-0.5.1/tests/adapters/test_process_boundary.py +156 -0
- agent_eval_flow-0.5.1/tests/adapters/test_worker_boundary.py +179 -0
- agent_eval_flow-0.5.1/tests/conftest.py +44 -0
- agent_eval_flow-0.5.1/tests/contracts/__init__.py +1 -0
- agent_eval_flow-0.5.1/tests/contracts/test_assessment_consumers.py +258 -0
- agent_eval_flow-0.5.1/tests/contracts/test_assessment_objects.py +203 -0
- agent_eval_flow-0.5.1/tests/contracts/test_assessment_pipeline.py +319 -0
- agent_eval_flow-0.5.1/tests/contracts/test_assessment_receipts.py +238 -0
- agent_eval_flow-0.5.1/tests/contracts/test_capture.py +424 -0
- agent_eval_flow-0.5.1/tests/contracts/test_configuration.py +310 -0
- agent_eval_flow-0.5.1/tests/contracts/test_evaluation.py +444 -0
- agent_eval_flow-0.5.1/tests/contracts/test_native_jobs.py +297 -0
- agent_eval_flow-0.5.1/tests/contracts/test_pipeline_runtime.py +143 -0
- agent_eval_flow-0.5.1/tests/contracts/test_reducers.py +117 -0
- agent_eval_flow-0.5.1/tests/contracts/test_results_storage.py +257 -0
- agent_eval_flow-0.5.1/tests/e2e/BASELINE.md +83 -0
- agent_eval_flow-0.5.1/tests/e2e/PROFILES.md +280 -0
- agent_eval_flow-0.5.1/tests/e2e/README.md +125 -0
- agent_eval_flow-0.5.1/tests/e2e/SHOWCASE.md +90 -0
- agent_eval_flow-0.5.1/tests/e2e/__init__.py +1 -0
- agent_eval_flow-0.5.1/tests/e2e/cloud/test_openkritt_gcp.py +488 -0
- agent_eval_flow-0.5.1/tests/e2e/cloud/test_opensre_gcp.py +257 -0
- agent_eval_flow-0.5.1/tests/e2e/conftest.py +42 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/README.md +230 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/.gitignore +14 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/LICENSE.txt +28 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/README.rst +68 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/__init__.py +51 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/auth.py +116 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/blog.py +125 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/db.py +56 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/schema.sql +20 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/static/style.css +134 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/auth/login.html +15 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/auth/register.html +15 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/base.html +24 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/create.html +15 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/index.html +28 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/flaskr/templates/blog/update.html +19 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/pyproject.toml +40 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/conftest.py +62 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/data.sql +8 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_auth.py +69 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_blog.py +83 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_db.py +29 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/review_target/tests/test_factory.py +12 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/source_manifest.json +166 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/openkritt/workflow.json +79 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/README.md +160 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/SOURCES.json +110 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/context.json +18 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/incident_store.py +184 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/incidents.json +7 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/HDFS_2k.log +2000 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/LOGHUB_LICENSE +11 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/LICENSE +201 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/agent.py +183 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/events.py +306 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/harness.py +412 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/mixins.py +111 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/ports.py +317 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/react_loop.py +786 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/opensre/upstream/opensre/turn_results.py +69 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/INSPECT.md +24 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/LICENSE +21 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/PROVENANCE.json +26 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/README.md +31 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/native-format.md +103 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/sweagent_archive/pydicom__pydicom-1458.traj +257 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/SKILL.md +9 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/toy_agent.py +37 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/toy/toy_tool.py +16 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/MISSION.md +39 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/README.md +162 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/SOURCE_MANIFEST.json +47 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/cases.json +12 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/CHANGES.rst +213 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/LICENSE.txt +28 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/README.md +40 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/src/markupsafe/__init__.py +395 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/src/markupsafe/_native.py +8 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/downloaded/tests/test_escape.py +68 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/investigation_tool.py +120 -0
- agent_eval_flow-0.5.1/tests/e2e/fixtures/workflow/report.schema.json +36 -0
- agent_eval_flow-0.5.1/tests/e2e/profiles.example.json +101 -0
- agent_eval_flow-0.5.1/tests/e2e/support.py +247 -0
- agent_eval_flow-0.5.1/tests/e2e/test_archived_trajectory.py +205 -0
- agent_eval_flow-0.5.1/tests/e2e/test_data_contract.py +263 -0
- agent_eval_flow-0.5.1/tests/e2e/test_live_toy.py +84 -0
- agent_eval_flow-0.5.1/tests/e2e/test_live_workflow.py +361 -0
- agent_eval_flow-0.5.1/tests/e2e/test_runtime_evidence.py +56 -0
- agent_eval_flow-0.5.1/tests/e2e/test_toy_pipeline.py +206 -0
- agent_eval_flow-0.5.1/tests/integrations/PROFILES.md +161 -0
- agent_eval_flow-0.5.1/tests/integrations/test_optional_libraries.py +256 -0
- agent_eval_flow-0.5.1/tests/unit/fixtures/openkritt_scan2_lineage.json +452 -0
- agent_eval_flow-0.5.1/tests/unit/test_cli_adapters.py +103 -0
- agent_eval_flow-0.5.1/tests/unit/test_openkritt_lineage.py +61 -0
- agent_eval_flow-0.5.1/tests/unit/test_openkritt_local_connection.py +283 -0
- agent_eval_flow-0.5.1/tests/unit/test_openkritt_local_review.py +105 -0
- agent_eval_flow-0.5.1/tests/unit/test_openkritt_setup.py +53 -0
- agent_eval_flow-0.5.1/tests/unit/test_openkritt_workflow_variant.py +139 -0
- agent_eval_flow-0.5.1/tests/unit/test_opensre_local_review.py +147 -0
- agent_eval_flow-0.5.1/tests/unit/test_opensre_local_runtime.py +53 -0
- agent_eval_flow-0.5.1/tests/unit/test_opensre_local_session.py +119 -0
- agent_eval_flow-0.5.1/tests/unit/test_optional_adapter_mappers.py +266 -0
- agent_eval_flow-0.5.1/tests/unit/test_runtime_evidence.py +149 -0
- agent_eval_flow-0.5.1/tests/unit/test_runtime_evidence_report.py +142 -0
- agent_eval_flow-0.5.1/tests/unit/test_showcase_recipe.py +96 -0
- agent_eval_flow-0.5.1/tests/unit/test_vertex_showcase.py +258 -0
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# The acceptance checkpoint hashes exact original bytes, including upstream logs.
|
|
2
|
+
tests/** -text
|
|
3
|
+
pytest.ini -text
|
|
4
|
+
requirements-test.txt -text
|
|
5
|
+
# Workflow derivation receipts pin exact bytes; never normalize their line endings.
|
|
6
|
+
examples/workflows/*.json -text whitespace=cr-at-eol
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "Bug report",
|
|
3
|
+
"description": "Report unexpected behavior with a minimal reproduction.",
|
|
4
|
+
"body": [
|
|
5
|
+
{
|
|
6
|
+
"type": "markdown",
|
|
7
|
+
"attributes": {
|
|
8
|
+
"value": "Small, reproducible examples are most useful. Remove credentials, private task data, and sensitive prompts before sharing anything. Raw run logs are not required."
|
|
9
|
+
}
|
|
10
|
+
},
|
|
11
|
+
{
|
|
12
|
+
"type": "textarea",
|
|
13
|
+
"id": "environment",
|
|
14
|
+
"attributes": {
|
|
15
|
+
"label": "Versions and runtime",
|
|
16
|
+
"description": "Include the package version or commit, Python version, OS, and relevant runtime or importer. Use not applicable where appropriate.",
|
|
17
|
+
"placeholder": "Agent Eval Flow:\nPython / OS:\nRuntime, adapter or import format (and revision, if known):"
|
|
18
|
+
},
|
|
19
|
+
"validations": {
|
|
20
|
+
"required": true
|
|
21
|
+
}
|
|
22
|
+
},
|
|
23
|
+
{
|
|
24
|
+
"type": "textarea",
|
|
25
|
+
"id": "reproduction",
|
|
26
|
+
"attributes": {
|
|
27
|
+
"label": "Minimal reproduction",
|
|
28
|
+
"description": "Give the smallest code, configuration and steps that reproduce the problem. Use public or redacted sample data."
|
|
29
|
+
},
|
|
30
|
+
"validations": {
|
|
31
|
+
"required": true
|
|
32
|
+
}
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"type": "textarea",
|
|
36
|
+
"id": "expected",
|
|
37
|
+
"attributes": {
|
|
38
|
+
"label": "Expected behavior",
|
|
39
|
+
"description": "What result or behavior did you expect?"
|
|
40
|
+
},
|
|
41
|
+
"validations": {
|
|
42
|
+
"required": true
|
|
43
|
+
}
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
"type": "textarea",
|
|
47
|
+
"id": "observed",
|
|
48
|
+
"attributes": {
|
|
49
|
+
"label": "Observed behavior",
|
|
50
|
+
"description": "What happened instead? Include relevant statuses or error messages."
|
|
51
|
+
},
|
|
52
|
+
"validations": {
|
|
53
|
+
"required": true
|
|
54
|
+
}
|
|
55
|
+
},
|
|
56
|
+
{
|
|
57
|
+
"type": "textarea",
|
|
58
|
+
"id": "evidence",
|
|
59
|
+
"attributes": {
|
|
60
|
+
"label": "Scrubbed evidence (optional)",
|
|
61
|
+
"description": "A short traceback, report excerpt or screenshot can help. Prefer the relevant excerpt over a complete capture, and remove sensitive content."
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
]
|
|
65
|
+
}
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "Feature or use case",
|
|
3
|
+
"description": "Explain a workflow, missing capability, or useful improvement.",
|
|
4
|
+
"body": [
|
|
5
|
+
{
|
|
6
|
+
"type": "markdown",
|
|
7
|
+
"attributes": {
|
|
8
|
+
"value": "Describe the decision or workflow you want to support. A specific example helps, but private data and raw run logs are not needed."
|
|
9
|
+
}
|
|
10
|
+
},
|
|
11
|
+
{
|
|
12
|
+
"type": "textarea",
|
|
13
|
+
"id": "use_case",
|
|
14
|
+
"attributes": {
|
|
15
|
+
"label": "Problem or use case",
|
|
16
|
+
"description": "What are you evaluating, and what decision are you trying to make?"
|
|
17
|
+
},
|
|
18
|
+
"validations": {
|
|
19
|
+
"required": true
|
|
20
|
+
}
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"type": "textarea",
|
|
24
|
+
"id": "desired_outcome",
|
|
25
|
+
"attributes": {
|
|
26
|
+
"label": "Desired outcome",
|
|
27
|
+
"description": "What would a useful result or supported workflow look like? An example input and output can help."
|
|
28
|
+
},
|
|
29
|
+
"validations": {
|
|
30
|
+
"required": true
|
|
31
|
+
}
|
|
32
|
+
},
|
|
33
|
+
{
|
|
34
|
+
"type": "textarea",
|
|
35
|
+
"id": "current_workflow",
|
|
36
|
+
"attributes": {
|
|
37
|
+
"label": "Current workflow (optional)",
|
|
38
|
+
"description": "Mention the runtime or imported format, relevant versions, and your current approach or workaround."
|
|
39
|
+
}
|
|
40
|
+
},
|
|
41
|
+
{
|
|
42
|
+
"type": "textarea",
|
|
43
|
+
"id": "proposal",
|
|
44
|
+
"attributes": {
|
|
45
|
+
"label": "Proposed approach or alternatives (optional)",
|
|
46
|
+
"description": "Share a possible solution, alternatives, or constraints. A proposed implementation is not required."
|
|
47
|
+
}
|
|
48
|
+
},
|
|
49
|
+
{
|
|
50
|
+
"type": "textarea",
|
|
51
|
+
"id": "examples",
|
|
52
|
+
"attributes": {
|
|
53
|
+
"label": "Public or redacted examples (optional)",
|
|
54
|
+
"description": "Link related issues or include a small example. Remove credentials, private prompts and task data."
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
]
|
|
58
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
## What changed
|
|
2
|
+
|
|
3
|
+
<!-- Describe the problem and resulting behavior. Link related issues if relevant. -->
|
|
4
|
+
|
|
5
|
+
## Validation
|
|
6
|
+
|
|
7
|
+
<!-- List checks and their outcomes, including anything not run and why.
|
|
8
|
+
For a bug fix, include the regression check that demonstrates the behavior.
|
|
9
|
+
See CONTRIBUTING.md for the local checks. -->
|
|
10
|
+
|
|
11
|
+
## Notes for review
|
|
12
|
+
|
|
13
|
+
<!-- Optional: explain changes to metric meaning, evidence/provenance,
|
|
14
|
+
missing values, or compatibility. Note remaining limitations.
|
|
15
|
+
Keep credentials, private task data and raw local captures out of the PR. -->
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
name: Publish to PyPI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
release:
|
|
5
|
+
types: [published]
|
|
6
|
+
# Retry a published release with: gh workflow run publish.yml --ref v0.5.1
|
|
7
|
+
# Dispatching a branch never enables either job.
|
|
8
|
+
workflow_dispatch:
|
|
9
|
+
|
|
10
|
+
permissions: {}
|
|
11
|
+
|
|
12
|
+
concurrency:
|
|
13
|
+
group: pypi-${{ github.ref }}
|
|
14
|
+
cancel-in-progress: false
|
|
15
|
+
|
|
16
|
+
jobs:
|
|
17
|
+
build:
|
|
18
|
+
if: github.repository == 'guybass/agent-eval-flow' && startsWith(github.ref, 'refs/tags/v')
|
|
19
|
+
runs-on: ubuntu-latest
|
|
20
|
+
timeout-minutes: 15
|
|
21
|
+
permissions:
|
|
22
|
+
contents: read
|
|
23
|
+
actions: read
|
|
24
|
+
outputs:
|
|
25
|
+
artifact-id: ${{ steps.distributions.outputs.artifact-id }}
|
|
26
|
+
steps:
|
|
27
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
28
|
+
with:
|
|
29
|
+
ref: ${{ github.sha }}
|
|
30
|
+
persist-credentials: false
|
|
31
|
+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
|
|
32
|
+
with:
|
|
33
|
+
python-version: '3.13'
|
|
34
|
+
- name: Verify published release, source version, and all six CI jobs
|
|
35
|
+
env:
|
|
36
|
+
GH_TOKEN: ${{ github.token }}
|
|
37
|
+
run: |
|
|
38
|
+
python - <<'PY'
|
|
39
|
+
import json
|
|
40
|
+
import os
|
|
41
|
+
from pathlib import Path
|
|
42
|
+
import re
|
|
43
|
+
import subprocess
|
|
44
|
+
import tomllib
|
|
45
|
+
from urllib.parse import urlencode
|
|
46
|
+
from urllib.request import Request, urlopen
|
|
47
|
+
|
|
48
|
+
def require(condition, message):
|
|
49
|
+
if not condition:
|
|
50
|
+
raise SystemExit(message)
|
|
51
|
+
|
|
52
|
+
repository = os.environ['GITHUB_REPOSITORY']
|
|
53
|
+
tag = os.environ['GITHUB_REF_NAME']
|
|
54
|
+
sha = os.environ['GITHUB_SHA']
|
|
55
|
+
require(repository == 'guybass/agent-eval-flow', 'Unexpected repository')
|
|
56
|
+
require(os.environ['GITHUB_REF'] == f'refs/tags/{tag}', 'A version tag is required')
|
|
57
|
+
require(re.fullmatch(r'v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)', tag),
|
|
58
|
+
'Only final vMAJOR.MINOR.PATCH release tags are publishable')
|
|
59
|
+
require(os.environ['GITHUB_EVENT_NAME'] in {'release', 'workflow_dispatch'},
|
|
60
|
+
'Unexpected publishing event')
|
|
61
|
+
|
|
62
|
+
def api(path):
|
|
63
|
+
request = Request(f'https://api.github.com/repos/{repository}/{path}', headers={
|
|
64
|
+
'Authorization': f'Bearer {os.environ["GH_TOKEN"]}',
|
|
65
|
+
'Accept': 'application/vnd.github+json',
|
|
66
|
+
'X-GitHub-Api-Version': '2022-11-28',
|
|
67
|
+
})
|
|
68
|
+
with urlopen(request, timeout=30) as response:
|
|
69
|
+
return json.load(response)
|
|
70
|
+
|
|
71
|
+
release = api(f'releases/tags/{tag}')
|
|
72
|
+
require(release['tag_name'] == tag and not release['draft']
|
|
73
|
+
and not release['prerelease'] and release['published_at'],
|
|
74
|
+
'The tag must have a published, non-prerelease GitHub release')
|
|
75
|
+
require(api(f'commits/{tag}')['sha'] == sha, 'Tag moved or event SHA differs')
|
|
76
|
+
require(subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() == sha,
|
|
77
|
+
'Checkout differs from the verified release commit')
|
|
78
|
+
require(api(f'compare/{sha}...main')['status'] in {'ahead', 'identical'},
|
|
79
|
+
'Release commit must belong to main history')
|
|
80
|
+
project = tomllib.loads(Path('pyproject.toml').read_text(encoding='utf-8'))['project']
|
|
81
|
+
require(project['name'] == 'agent-eval-flow' and project['version'] == tag[1:],
|
|
82
|
+
'Project name/version must match the release tag exactly')
|
|
83
|
+
|
|
84
|
+
query = urlencode({'head_sha': sha, 'event': 'push', 'branch': 'main', 'per_page': 100})
|
|
85
|
+
runs = api(f'actions/workflows/tests.yml/runs?{query}')['workflow_runs']
|
|
86
|
+
require(runs, 'No Tests and package run exists for this commit on main')
|
|
87
|
+
run = max(runs, key=lambda row: row['id'])
|
|
88
|
+
require(run['head_sha'] == sha and run['head_branch'] == 'main'
|
|
89
|
+
and run['event'] == 'push' and run['status'] == 'completed'
|
|
90
|
+
and run['conclusion'] == 'success',
|
|
91
|
+
'Latest Tests and package run for this commit must finish successfully')
|
|
92
|
+
jobs = api(f'actions/runs/{run["id"]}/jobs?filter=latest&per_page=100')
|
|
93
|
+
expected = {f'offline ({system}, {version})'
|
|
94
|
+
for system in ('ubuntu-latest', 'windows-latest')
|
|
95
|
+
for version in ('3.11', '3.12', '3.13')}
|
|
96
|
+
require(jobs['total_count'] == len(expected) and len(jobs['jobs']) == len(expected)
|
|
97
|
+
and {job['name'] for job in jobs['jobs']} == expected
|
|
98
|
+
and all(job['status'] == 'completed' and job['conclusion'] == 'success'
|
|
99
|
+
for job in jobs['jobs']),
|
|
100
|
+
'All six expected Python/OS matrix jobs must pass; skipped jobs do not qualify')
|
|
101
|
+
print(f'Verified {tag} at {sha}; CI: {run["html_url"]}')
|
|
102
|
+
PY
|
|
103
|
+
- name: Build and validate distributions
|
|
104
|
+
run: |
|
|
105
|
+
python -m pip install build twine
|
|
106
|
+
python -m build
|
|
107
|
+
python -m twine check --strict dist/*
|
|
108
|
+
python scripts/verify_distribution.py
|
|
109
|
+
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
110
|
+
id: distributions
|
|
111
|
+
with:
|
|
112
|
+
name: distributions-${{ github.ref_name }}
|
|
113
|
+
path: dist/*
|
|
114
|
+
if-no-files-found: error
|
|
115
|
+
retention-days: 7
|
|
116
|
+
|
|
117
|
+
publish:
|
|
118
|
+
needs: build
|
|
119
|
+
if: github.repository == 'guybass/agent-eval-flow' && startsWith(github.ref, 'refs/tags/v')
|
|
120
|
+
runs-on: ubuntu-latest
|
|
121
|
+
timeout-minutes: 10
|
|
122
|
+
environment:
|
|
123
|
+
name: pypi
|
|
124
|
+
url: https://pypi.org/project/agent-eval-flow/
|
|
125
|
+
permissions:
|
|
126
|
+
id-token: write
|
|
127
|
+
steps:
|
|
128
|
+
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
|
129
|
+
with:
|
|
130
|
+
artifact-ids: ${{ needs.build.outputs.artifact-id }}
|
|
131
|
+
path: dist/
|
|
132
|
+
digest-mismatch: error
|
|
133
|
+
- name: Publish verified distributions with Trusted Publishing
|
|
134
|
+
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
|
|
135
|
+
with:
|
|
136
|
+
attestations: true
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
name: Tests and package
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
pull_request:
|
|
6
|
+
|
|
7
|
+
permissions:
|
|
8
|
+
contents: read
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
offline:
|
|
12
|
+
strategy:
|
|
13
|
+
fail-fast: false
|
|
14
|
+
matrix:
|
|
15
|
+
os: [ubuntu-latest, windows-latest]
|
|
16
|
+
python: ['3.11', '3.12', '3.13']
|
|
17
|
+
runs-on: ${{ matrix.os }}
|
|
18
|
+
timeout-minutes: 15
|
|
19
|
+
steps:
|
|
20
|
+
- uses: actions/checkout@v7
|
|
21
|
+
with:
|
|
22
|
+
persist-credentials: false
|
|
23
|
+
- uses: actions/setup-python@v7
|
|
24
|
+
with:
|
|
25
|
+
python-version: ${{ matrix.python }}
|
|
26
|
+
cache: pip
|
|
27
|
+
- run: python -m pip install -e ".[test,cli]" build
|
|
28
|
+
- run: python scripts/verify_acceptance_checkpoint.py
|
|
29
|
+
- run: python -m pytest --junitxml=test-artifacts/pytest.xml
|
|
30
|
+
- run: python examples/archive_review.py --output demo-output
|
|
31
|
+
- run: python -m build
|
|
32
|
+
- run: python scripts/verify_distribution.py
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Agent Eval Flow is a developer preview. Issues and small pull requests are welcome.
|
|
4
|
+
For a bug, include the package/Python versions, a minimal reproducible example,
|
|
5
|
+
the expected outcome and the observed result. Remove credentials and private
|
|
6
|
+
task data before sharing evidence.
|
|
7
|
+
|
|
8
|
+
## Local checks
|
|
9
|
+
|
|
10
|
+
From a checkout, using Python 3.11 or later:
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
python -m pip install -e ".[test,cli]" build
|
|
14
|
+
python scripts/verify_acceptance_checkpoint.py
|
|
15
|
+
python -m pytest
|
|
16
|
+
python examples/archive_review.py --output demo-output
|
|
17
|
+
python -m build
|
|
18
|
+
python scripts/verify_distribution.py
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Default tests do not start live models. Live profiles require explicit selection
|
|
22
|
+
and prepared runtimes; their setup is documented in [tests/e2e/SHOWCASE.md](tests/e2e/SHOWCASE.md).
|
|
23
|
+
|
|
24
|
+
Keep metric meaning, provenance and missingness explicit. Fixes should include
|
|
25
|
+
a regression check that demonstrates the affected behavior. Preserve the frozen
|
|
26
|
+
acceptance fixtures and their hashes; add new fixtures separately.
|
|
27
|
+
|
|
28
|
+
Local outputs belong in ignored `demo-output/` or `test-artifacts/` directories.
|
|
29
|
+
Include only deliberately selected, reviewed report artifacts in documentation.
|
|
30
|
+
Third-party fixtures retain their own licenses and provenance.
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: agent-eval-flow
|
|
3
|
+
Version: 0.5.1
|
|
4
|
+
Summary: Evidence-preserving evaluation of complete agent systems
|
|
5
|
+
Project-URL: Homepage, https://guybass.github.io/agent-eval-flow/
|
|
6
|
+
Project-URL: Repository, https://github.com/guybass/agent-eval-flow
|
|
7
|
+
Project-URL: Issues, https://github.com/guybass/agent-eval-flow/issues
|
|
8
|
+
Project-URL: Documentation, https://github.com/guybass/agent-eval-flow/blob/main/README.md
|
|
9
|
+
Project-URL: Examples, https://guybass.github.io/agent-eval-flow/
|
|
10
|
+
Project-URL: Changelog, https://github.com/guybass/agent-eval-flow/releases
|
|
11
|
+
Keywords: agents,agent-evaluation,llm,evaluation,execution-evidence
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Requires-Python: >=3.11
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
Requires-Dist: pydantic<3,>=2.13.5
|
|
20
|
+
Requires-Dist: anyio<5,>=4.15.1
|
|
21
|
+
Requires-Dist: jinja2<4,>=3.1.6
|
|
22
|
+
Requires-Dist: typing-extensions>=4.14
|
|
23
|
+
Provides-Extra: test
|
|
24
|
+
Requires-Dist: pytest<10,>=8; extra == "test"
|
|
25
|
+
Requires-Dist: jsonschema<5,>=4.26; extra == "test"
|
|
26
|
+
Provides-Extra: cli
|
|
27
|
+
Requires-Dist: jsonschema<5,>=4.26; extra == "cli"
|
|
28
|
+
|
|
29
|
+
[](https://guybass.github.io/agent-eval-flow/)
|
|
30
|
+
|
|
31
|
+
# Agent Eval Flow
|
|
32
|
+
|
|
33
|
+
[](https://github.com/guybass/agent-eval-flow/releases)
|
|
34
|
+
[](https://github.com/guybass/agent-eval-flow/actions/workflows/tests.yml)
|
|
35
|
+
[](https://github.com/guybass/agent-eval-flow/blob/main/pyproject.toml)
|
|
36
|
+
[](https://guybass.github.io/agent-eval-flow/)
|
|
37
|
+
|
|
38
|
+
**Turn agent runs into evidence you can use to improve the system.**
|
|
39
|
+
|
|
40
|
+
Evaluate complete agent setups: models, instructions, skills, tools, loops,
|
|
41
|
+
memory and environment. Run through an existing runtime or import saved logs,
|
|
42
|
+
apply your checks, and compare changes in a human-readable report.
|
|
43
|
+
|
|
44
|
+
**[Browse the reports](https://guybass.github.io/agent-eval-flow/)** ·
|
|
45
|
+
**[Try the offline example](#try-it)** ·
|
|
46
|
+
**[Read the design](https://github.com/guybass/agent-eval-flow/blob/main/docs/lld/README.md)** ·
|
|
47
|
+
**[Report an issue](https://github.com/guybass/agent-eval-flow/issues)**
|
|
48
|
+
|
|
49
|
+
This is a developer preview. Missing evidence stays unknown; configuration
|
|
50
|
+
inspection and behavioral evaluation can run in parallel. The Python package
|
|
51
|
+
is `agent_eval_flow`.
|
|
52
|
+
|
|
53
|
+
## Try it
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
git clone https://github.com/guybass/agent-eval-flow.git
|
|
57
|
+
cd agent-eval-flow
|
|
58
|
+
python -m pip install -e ".[test,cli]"
|
|
59
|
+
python -m pytest
|
|
60
|
+
python examples/archive_review.py --output demo-output
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
The example imports a **real, downloaded SWE-agent repair history** containing
|
|
64
|
+
12 actions, failed edits, corrections and a submitted patch. It produces:
|
|
65
|
+
|
|
66
|
+
- `demo-output/report.html`: task results, diagnostics and linked native evidence;
|
|
67
|
+
- `demo-output/result/`: the typed saved evaluation;
|
|
68
|
+
- `demo-output/regraded/`: a changed evaluation over the same saved execution.
|
|
69
|
+
|
|
70
|
+
It makes no model calls and never executes commands from the archive. The
|
|
71
|
+
[example source](https://github.com/guybass/agent-eval-flow/blob/main/examples/archive_review.py) shows an application-owned importer
|
|
72
|
+
and custom metrics using the production library.
|
|
73
|
+
|
|
74
|
+
Requires Python 3.11 or later. The default tests and archive example need no
|
|
75
|
+
model credentials. Native agent runs require the integration-specific setup below.
|
|
76
|
+
|
|
77
|
+
## Example reports
|
|
78
|
+
|
|
79
|
+
Two real agent workflows, controlled synthetic tasks, and specific changes
|
|
80
|
+
measured from retained runs. Click a report to open it in your browser.
|
|
81
|
+
|
|
82
|
+
| OpenSRE: recovery and stopping | OpenKritt: executed proof |
|
|
83
|
+
| --- | --- |
|
|
84
|
+
| [](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre.html) | [](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt.html) |
|
|
85
|
+
| Safe recovery stayed **2/2**; post-report calls fell **1 → 0** after a terminal-tool binding fix. | The demo grounding contract passed **0/1 → 1/1** after requiring execution and clarifying the reporting contract. |
|
|
86
|
+
| [Read the report](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre.html) · [Task and fix walkthrough](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/01-opensre-walkthrough.html) | [Read the report](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt.html) · [Task and fix walkthrough](https://guybass.github.io/agent-eval-flow/examples/agent-evaluation/02-openkritt-walkthrough.html) |
|
|
87
|
+
|
|
88
|
+
These are small development experiments with synthetic tasks, not general
|
|
89
|
+
reliability benchmarks. The gallery contains presentation reports; raw local
|
|
90
|
+
run captures and unpublished social drafts are excluded. The offline example
|
|
91
|
+
above is the reproducible starting point for trying the evaluation API.
|
|
92
|
+
See the [case details, run identifiers and limitations](https://github.com/guybass/agent-eval-flow/blob/main/docs/examples/agent-evaluation/README.md).
|
|
93
|
+
|
|
94
|
+
## Configure once, then evaluate
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
from agent_eval_flow import EvaluationPipeline, EvaluationResult
|
|
98
|
+
|
|
99
|
+
pipeline = EvaluationPipeline(
|
|
100
|
+
study=study, # data, candidates, execution policy and suite
|
|
101
|
+
backends=backends, # existing agent runtimes
|
|
102
|
+
evaluators=evaluators, # your metric implementations
|
|
103
|
+
)
|
|
104
|
+
result = pipeline.eval()
|
|
105
|
+
result.save("results/experiment")
|
|
106
|
+
result.report("results/experiment.html")
|
|
107
|
+
|
|
108
|
+
saved = EvaluationResult.load("results/experiment")
|
|
109
|
+
explanation = saved.explain(saved.runs.runs[0].id)
|
|
110
|
+
comparison = saved.compare("baseline", "challenger", metrics=("success_rate",))
|
|
111
|
+
selection = saved.select(policy) # change cost/latency/quality priorities
|
|
112
|
+
|
|
113
|
+
# Regrade captured runs without binding or invoking an agent backend.
|
|
114
|
+
regraded = EvaluationPipeline(study=changed_study, evaluators=evaluators).eval(runs=saved.runs)
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
In an async application, use `await pipeline.aeval()`. Construction and planning
|
|
118
|
+
do not start an agent. Changing the selection policy does not rerun agents or
|
|
119
|
+
metrics. Comparisons currently provide descriptive differences; they do not
|
|
120
|
+
invent confidence intervals.
|
|
121
|
+
|
|
122
|
+
## The six objects
|
|
123
|
+
|
|
124
|
+
| Object | Responsibility |
|
|
125
|
+
| --- | --- |
|
|
126
|
+
| `Study` | Question, candidates, execution conditions and evaluation suite |
|
|
127
|
+
| `EvalDataset` | Keyed task tables, public inputs and private evaluator references |
|
|
128
|
+
| `Candidate` | Model, skill, tool, flow and harness configuration |
|
|
129
|
+
| `RunSet` | Every assignment, output, execution, event, resource observation and failure |
|
|
130
|
+
| `EvalSuite` | Versioned metrics, dependencies, score rules and summaries |
|
|
131
|
+
| `EvaluationResult` | Saved measurements, explanations, comparisons and selection |
|
|
132
|
+
|
|
133
|
+
Pydantic validates the shared records. AnyIO coordinates runtime calls. Jinja2
|
|
134
|
+
renders self-contained reports. Native agent frameworks keep their loops and
|
|
135
|
+
schedulers. Our code owns the common evidence and comparison contracts.
|
|
136
|
+
|
|
137
|
+
Missing usage remains unknown, failed work remains in the assignment inventory,
|
|
138
|
+
and shared grading activities are counted once. A captured JSON `null` is distinct
|
|
139
|
+
from absent output. Detail rows explain tasks; they do not inflate the sample size.
|
|
140
|
+
|
|
141
|
+
## Runtime evidence checks
|
|
142
|
+
|
|
143
|
+
Use the same optional observation contract to check instructions, tools, model
|
|
144
|
+
selection, loops, memory and environment state. Versioned collectors retain
|
|
145
|
+
declared and observed values with phase, boundary, coverage and evidence.
|
|
146
|
+
Reusable checks run through the existing evaluation pipeline, including saved
|
|
147
|
+
run regrading; reports show the expected and observed values. Missing evidence
|
|
148
|
+
stays unknown. See the [runtime evidence guide](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/runtime_evidence.md).
|
|
149
|
+
|
|
150
|
+
## Runtime integrations
|
|
151
|
+
|
|
152
|
+
Concrete adapter modules cover Codex, Claude Code, OpenSRE, OpenKritt, Harbor,
|
|
153
|
+
SkillEvaluator imports and NeMo Agent Toolkit batch grading. Their native
|
|
154
|
+
capture/lifecycle code is separate from the core. Prepared service connections,
|
|
155
|
+
version-specific configuration and deployment credentials remain runtime bindings.
|
|
156
|
+
See [implementation status and setup](https://github.com/guybass/agent-eval-flow/blob/main/docs/IMPLEMENTATION.md) for the supported
|
|
157
|
+
boundaries and the live checks still required.
|
|
158
|
+
|
|
159
|
+
OpenSRE and OpenKritt are separate studies: compare candidate versions **within**
|
|
160
|
+
each project. Their GCP showcase tests require prepared native runtimes and rich
|
|
161
|
+
capture observers. Default offline test success does not establish live compatibility.
|
|
162
|
+
|
|
163
|
+
A [live local OpenKritt + Codex example](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/local_openkritt.md)
|
|
164
|
+
has now completed against 22 real Flaskr files: eight workflow executions, four
|
|
165
|
+
post-processing executions, 20 tool results and two candidate findings. It saves
|
|
166
|
+
native evidence, verifies the result round trip, and regrades without new model
|
|
167
|
+
calls. Its score checks integration behavior, not security accuracy.
|
|
168
|
+
|
|
169
|
+
The separate [local OpenSRE + Codex example](https://github.com/guybass/agent-eval-flow/blob/main/docs/implementation/local_opensre.md)
|
|
170
|
+
has completed an investigation of the downloaded HDFS sample: six ReAct
|
|
171
|
+
iterations, eight real tool calls and a cited incident report. It retains native
|
|
172
|
+
events and CLI receipts, then saves, reloads and regrades the captured result.
|
|
173
|
+
|
|
174
|
+
## Verification and design
|
|
175
|
+
|
|
176
|
+
Release checks cover the offline suite, archived-run import and regrading,
|
|
177
|
+
package contents, and installed-wheel reporting. See the
|
|
178
|
+
[developer-preview release notes](https://github.com/guybass/agent-eval-flow/blob/main/docs/RELEASE.md) for the verified release scope.
|
|
179
|
+
|
|
180
|
+
CI runs on **Windows and Ubuntu with Python 3.11, 3.12 and 3.13**. Checks cover
|
|
181
|
+
native-format fixtures, evidence transport, missing outcomes, persistence,
|
|
182
|
+
regrading and package contents. All **90 frozen acceptance/fixture files** are
|
|
183
|
+
verified by hash. Live model tests are opt-in; offline success does not establish
|
|
184
|
+
compatibility with every deployed agent runtime. Historical checkpoints remain
|
|
185
|
+
in [implementation status](https://github.com/guybass/agent-eval-flow/blob/main/docs/IMPLEMENTATION.md).
|
|
186
|
+
|
|
187
|
+
- [Data structures and configuration](https://github.com/guybass/agent-eval-flow/blob/main/docs/DATA_CONTRACT.md)
|
|
188
|
+
- [Unified assessment flow: configuration and execution in parallel](https://github.com/guybass/agent-eval-flow/blob/main/docs/UNIFIED_ASSESSMENT_FLOW.md)
|
|
189
|
+
- [Module tree and communication graph](https://github.com/guybass/agent-eval-flow/blob/main/docs/lld/README.md)
|
|
190
|
+
- [Real-workflow E2E scenarios](https://github.com/guybass/agent-eval-flow/blob/main/tests/e2e/SHOWCASE.md)
|
|
191
|
+
- [Visual pipeline and score examples](https://github.com/guybass/agent-eval-flow/blob/main/docs/AGENT_EVAL_FLOW_VISION.md)
|
|
192
|
+
- [Library reuse decisions](https://github.com/guybass/agent-eval-flow/blob/main/docs/INTEGRATION_DECISION.md)
|
|
193
|
+
- [What this adds to existing work](https://github.com/guybass/agent-eval-flow/blob/main/docs/WHAT_AGENT_EVAL_FLOW_ADDS.md)
|
|
194
|
+
|
|
195
|
+
The older design and test-writing checkpoints are retained as historical records.
|
|
196
|
+
|
|
197
|
+
For changes and bug reports, see [CONTRIBUTING.md](https://github.com/guybass/agent-eval-flow/blob/main/CONTRIBUTING.md).
|