project-loop-harness 0.2.0__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {project_loop_harness-0.2.0/src/project_loop_harness.egg-info → project_loop_harness-0.2.2}/PKG-INFO +1 -1
- project_loop_harness-0.2.2/agent-tasks/0089-dogfood-fixture-propose.md +107 -0
- project_loop_harness-0.2.2/agent-tasks/0090-eval-baseline-record-compare.md +134 -0
- project_loop_harness-0.2.2/agent-tasks/0092-idempotent-status-transitions.md +78 -0
- project_loop_harness-0.2.2/agent-tasks/0093-evidence-add-command.md +95 -0
- project_loop_harness-0.2.2/agent-tasks/0094-job-completion-evidence.md +71 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/agent-adapter-contract.md +7 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/code-context.md +104 -11
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/data-model.md +36 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/distribution.md +1 -1
- project_loop_harness-0.2.2/docs/evidence-entry-paths-design.md +119 -0
- project_loop_harness-0.2.2/docs/feedback/2026-07-06-ax1-moc1-agent-feedback.md +55 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/golden-path.md +8 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/pyproject.toml +1 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/scripts/run_advisory_retrieval_eval.py +74 -23
- project_loop_harness-0.2.2/src/pcl/__init__.py +1 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/cli.py +117 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/__init__.py +3 -0
- project_loop_harness-0.2.2/src/pcl/code_context/eval.py +1043 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/store.py +63 -5
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/commands.py +16 -10
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/context.py +1 -40
- project_loop_harness-0.2.2/src/pcl/evidence.py +224 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/init_project.py +1 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/lifecycle.py +76 -12
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/stories.py +18 -10
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/tasks.py +11 -5
- project_loop_harness-0.2.2/src/pcl/token_estimation.py +44 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/validators.py +146 -3
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflows.py +1 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2/src/project_loop_harness.egg-info}/PKG-INFO +1 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/project_loop_harness.egg-info/SOURCES.txt +10 -0
- project_loop_harness-0.2.2/tests/fixtures/retrieval_v0.json +41 -0
- project_loop_harness-0.2.2/tests/test_advisory_retrieval_eval_script.py +146 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_code_index.py +685 -0
- project_loop_harness-0.2.2/tests/test_evidence_add.py +394 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_features.py +89 -4
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_lifecycle.py +268 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_stories.py +120 -1
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_tasks.py +118 -0
- project_loop_harness-0.2.0/src/pcl/__init__.py +0 -1
- project_loop_harness-0.2.0/src/pcl/code_context/eval.py +0 -234
- project_loop_harness-0.2.0/src/pcl/evidence.py +0 -25
- project_loop_harness-0.2.0/tests/fixtures/retrieval_v0.json +0 -19
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/LICENSE +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/MANIFEST.in +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/Makefile +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/README.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0001-hardening-cli.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0002-db-migrations.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0003-workflow-runner.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0004-dashboard-renderer.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0005-agent-integration.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0006-codex-plugin.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0007-mcp-server.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0008-loop-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0009-defect-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0010-reporting-evidence.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0011-validation-invariants.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0012-audit-log-integrity.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0013-validation-diagnostics.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0014-escalation-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0015-decision-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0016-escalation-decision-linkage.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0017-next-action-guided-loop.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0018-readme-golden-path.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0019-recovery-playbook.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0020-example-project-refresh.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0021-agent-adapter-contract.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0022-agent-output-validation.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0023-codex-exec-adapter-hardening.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0024-claude-manual-adapter-hardening.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0025-generic-shell-adapter.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0026-agent-job-evidence-ingestion.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0027-dashboard-data-contract.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0028-dashboard-evidence-navigation.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0029-dashboard-risk-and-blockers.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0030-distribution-readiness.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0031-workflow-proposal-mode.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0032-workflow-proposal-review.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0033-workflow-verifier.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0034-limited-execution-sandbox.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0035-automatic-workflow-executor.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0036-executor-dogfood-workflow.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0037-executor-retry-resume.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0038-story-test-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0039-workflow-yaml-rule-expressions.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0040-test-case-evidence-validation.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0041-feature-inspection-commands.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0042-report-coverage-context.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0043-feature-report.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0044-complete-csv-export.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0045-filtered-job-inspection.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0046-feature-status-lifecycle.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0047-feature-coverage-next-action.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0048-migration-status-command.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0049-render-json-artifact-paths.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0050-codex-plugin-package-inventory.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0051-mcp-render-artifact-paths.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0052-lifecycle-failure-job-cleanup.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0053-prompt-job-json-handoff.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0054-human-queue-linkage-cli.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0055-workflow-proposal-list-filter.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0056-sandbox-noop-execution-guard.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0057-executor-no-executable-step-guard.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0058-dogfood-usability-hardening.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0059-checkpoint-review-guidance.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0060-pypi-trusted-publishing.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0061-context-pack-v1.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0062-task-backlog-entity.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0063-structured-verification-rubric.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0064-task-loop-integration.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0065-dashboard-human-decisions.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0066-agent-registry-lease.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0067-context-pack-improvements.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0068-trust-hardening.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0069-explainable-code-context-v0.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0070-human-decision-cockpit.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0071-dogfood-impact-precision.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0072-sensitive-omission.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0073-code-context-module-split.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0074-search-snapshot-consistency.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0075-diff-source-modes.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0076-schema-version-integrity.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0077-index-output-budget-and-impact-noise.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0078-context-pack-code-context-bridge.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0079-receipt-human-summary.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0080-retrieval-eval-gate.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0081-diff-modes-completion.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0082-receipt-relevance-and-age.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0083-required-section-invariant.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0084-source-commands-honesty.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0085-distribution-source-completeness.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0086-command-surface-alignment.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0087-verification-suggestion-ids.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0088-verification-feedback.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/agent-tasks/0091-refresh-command-scope-fidelity.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/adoption-guide.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/adr/0001-hybrid-state.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/adr/0002-cli-is-the-runtime.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/adr/0003-static-workflows-before-dynamic-workflows.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/agent-output-template.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/agent-roles.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/architecture.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/context-pack.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/dashboard-data-contract.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/dashboard-design.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/implementation-plan.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/plh_v0_1_11_review_agenda.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/plh_v0_1_12_review_agenda.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/pypi-publishing.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/recovery-playbook.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/safety-permissions.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/v0.2.0-plan.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/verification-feedback-design.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/verification-feedback.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/verification-rubric.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/docs/workflow-contract.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/scripts/verify_sdist_contracts.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/setup.cfg +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/__main__.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/agents.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/checkpoints.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/diff.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/impact.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/receipts.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/scan.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/search.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/summary.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/symbols.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_context/test_hints.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/code_index.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/migrations/001_initial.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/migrations/002_tasks.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/migrations/003_agent_registry.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/migrations/004_code_index.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/migrations/005_verification_feedback.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db/schema.sql +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/db.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/decisions.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/dispatch.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/errors.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/escalations.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/events.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/exporters.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/guards.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/ids.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/links.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/locales.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/mcp_server.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/migrations.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/paths.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/receipt_show.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/registry.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/renderer.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/reports.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/resources.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/rubric.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/dashboard/dashboard.html +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/project/AGENTS.block.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/project/CLAUDE.block.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/project/gitignore.fragment +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/project/pcl.yaml +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/skills/project-control-loop/SKILL.md +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/workflows/defect_repair.yaml +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/workflows/executor_smoke.yaml +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/workflows/feature_coverage.yaml +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/templates/workflows/regression_loop.yaml +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/timeutil.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/update_check.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/verification_feedback.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/verifications.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_executor.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_proposal_validation.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_proposals.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_sandbox.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_verifier.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/pcl/workflow_yaml.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/project_loop_harness.egg-info/dependency_links.txt +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/project_loop_harness.egg-info/entry_points.txt +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/project_loop_harness.egg-info/requires.txt +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/src/project_loop_harness.egg-info/top_level.txt +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/fixtures/code_context_summary_v0.json +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/fixtures/context_receipt_v0.json +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/fixtures/context_receipt_v0_legacy_string_suggestions.json +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/fixtures/retrieval_adversarial_v0.json +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/fixtures/retrieval_real_history_v0.json +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_agent_adapter_contract.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_agent_output_validation.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_agents.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_audit_log_integrity.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_checkpoints.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_claude_manual_adapter.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_cli_init.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_code_context_summary.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_codex_exec_adapter.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_codex_plugin.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_context.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_dashboard.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_dashboard_data_contract.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_decisions.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_defects.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_dispatch.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_distribution.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_escalations.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_examples.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_generic_shell_adapter.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_golden_path.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_mcp_server.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_migrations.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_next_actions.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_pypi_publishing.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_receipt_show.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_recovery_playbook.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_registry.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_reports.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_rubric.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_update_check.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_validation.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_validation_diagnostics.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_verification_feedback.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_workflow_executor.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_workflow_proposals.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_workflow_sandbox.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_workflow_verifier.py +0 -0
- {project_loop_harness-0.2.0 → project_loop_harness-0.2.2}/tests/test_workflows.py +0 -0
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# Task 0089: Dogfood-to-Fixture Propose (v0.2.1)
|
|
2
|
+
|
|
3
|
+
Design source: `docs/verification-feedback-design.md` Part 4 plus the
|
|
4
|
+
Approval addendum item 4 (staging location). Where this spec and the
|
|
5
|
+
design doc disagree, the design doc wins. No migration, no new
|
|
6
|
+
runtime dependency.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
An eval suite whose fixtures do not grow from real usage is not
|
|
11
|
+
credible. Add `pcl eval fixture propose --from-receipt <evidence-id>`
|
|
12
|
+
so real context receipts become UNLABELED fixture candidates that a
|
|
13
|
+
human labels before adoption. PLH never fabricates ground-truth
|
|
14
|
+
labels.
|
|
15
|
+
|
|
16
|
+
## Design constraints (approved, do not relitigate)
|
|
17
|
+
|
|
18
|
+
- Staging area is repo-root `fixtures/proposed/` — a reviewable,
|
|
19
|
+
Git-tracked source asset, NOT `.project-loop/` state.
|
|
20
|
+
- Proposed candidates carry `labels_status: "unlabeled"` and EMPTY
|
|
21
|
+
`expected_files` / `expected_tests` / `critical_context` arrays.
|
|
22
|
+
Filling them is a human review step by definition.
|
|
23
|
+
- Adoption into `tests/fixtures/` is a manual `git mv`-style step
|
|
24
|
+
after human labeling. NO auto-adoption path exists, and none may
|
|
25
|
+
be added.
|
|
26
|
+
|
|
27
|
+
## Scope
|
|
28
|
+
|
|
29
|
+
### 1. `pcl eval fixture propose --from-receipt <evidence-id> [--json]`
|
|
30
|
+
|
|
31
|
+
- Loads the referenced evidence row; it must be `type =
|
|
32
|
+
'context_receipt'` with a readable `context-receipt/v0` artifact —
|
|
33
|
+
unknown evidence, wrong type, or unreadable artifact are distinct
|
|
34
|
+
typed errors (reuse the receipt-loading discipline from
|
|
35
|
+
`verification_feedback.py` where sensible).
|
|
36
|
+
- Emits `fixtures/proposed/<receipt-evidence-id>-retrieval.json`
|
|
37
|
+
(lowercased evidence id, deterministic name; refuse to overwrite an
|
|
38
|
+
existing file unless `--force` — overwriting a half-labeled
|
|
39
|
+
candidate silently would destroy human work).
|
|
40
|
+
- Candidate shape: `retrieval-fixture/v0` with ONE task:
|
|
41
|
+
- `id`: derived from the receipt evidence id (deterministic).
|
|
42
|
+
- `diff`: a minimal synthetic unified diff touching exactly the
|
|
43
|
+
receipt's `changed_files` paths (same mechanical style as
|
|
44
|
+
existing fixtures). Receipts do not store original diff text, so
|
|
45
|
+
the replay diff is synthesized; record that honestly via
|
|
46
|
+
`diff_synthesized_from_receipt: true` inside the task.
|
|
47
|
+
- `expected_files: []`, `expected_tests: []`,
|
|
48
|
+
`critical_context: []`, `labels_status: "unlabeled"`.
|
|
49
|
+
- `source_receipt` provenance block: `evidence_id`, `created_at`,
|
|
50
|
+
`diff_source`, `base_ref` (when present), and
|
|
51
|
+
`retrieved_candidate_paths` (the receipt's
|
|
52
|
+
`included_candidate_context` paths) — this is the raw material
|
|
53
|
+
the human labeler starts from, clearly provenance, not labels.
|
|
54
|
+
- Record one standard `eval_fixture_proposed` event via the shared
|
|
55
|
+
`append_event` path (SQLite events table + JSONL mirror, sequential
|
|
56
|
+
event id), recording the receipt evidence id and output path. The
|
|
57
|
+
event row is the ONLY permitted SQLite write; no other tables may
|
|
58
|
+
be touched.
|
|
59
|
+
**Clarified 2026-07-06 after first acceptance pass:** the original
|
|
60
|
+
wording "JSONL event ... No SQLite mutation" was over-constrained
|
|
61
|
+
and led to a JSONL-only event with a uuid id, which violates the
|
|
62
|
+
task-0012 audit integrity invariant (`validate --strict` errors on
|
|
63
|
+
JSONL event ids missing from the DB events table). Events always go
|
|
64
|
+
through `append_event` to both stores; do not invent side-channel
|
|
65
|
+
event writers.
|
|
66
|
+
|
|
67
|
+
### 2. Guard: eval refuses unlabeled fixtures
|
|
68
|
+
|
|
69
|
+
- `pcl eval retrieval` on a fixture whose tasks carry
|
|
70
|
+
`labels_status: "unlabeled"` (or empty expected+critical blocks
|
|
71
|
+
alongside that marker) fails with a typed error telling the
|
|
72
|
+
operator to label the candidate and move it into `tests/fixtures/`.
|
|
73
|
+
Labeled/legacy fixtures (no `labels_status` field) evaluate as
|
|
74
|
+
today.
|
|
75
|
+
|
|
76
|
+
### 3. Documentation
|
|
77
|
+
|
|
78
|
+
- `docs/code-context.md` (eval section): propose workflow, staging
|
|
79
|
+
directory, human labeling step, manual adoption, and the
|
|
80
|
+
no-fabricated-labels rule.
|
|
81
|
+
- Add `fixtures/proposed/.gitkeep` (or a one-line README.md in that
|
|
82
|
+
directory) so the staging area exists in Git.
|
|
83
|
+
|
|
84
|
+
## Acceptance Criteria
|
|
85
|
+
|
|
86
|
+
- Proposing from a real receipt fixture yields a candidate that is
|
|
87
|
+
byte-deterministic given the same receipt (no timestamps of its
|
|
88
|
+
own — reuse the receipt's `created_at`).
|
|
89
|
+
- Candidate visibly unlabeled: `labels_status: "unlabeled"`, all
|
|
90
|
+
three expected/critical arrays empty (contract test).
|
|
91
|
+
- `pcl eval retrieval` against the proposed candidate → typed error;
|
|
92
|
+
after a test labels it (filling expected blocks, removing/flipping
|
|
93
|
+
`labels_status`), evaluation runs and produces metrics.
|
|
94
|
+
- Unknown/wrong-type/unreadable receipt → distinct typed errors,
|
|
95
|
+
nothing written.
|
|
96
|
+
- Existing candidate without `--force` → typed error, file untouched.
|
|
97
|
+
- `ruff check .` passes; full `python3 -m pytest` passes; `pcl init`
|
|
98
|
+
smoke against a temp dir passes.
|
|
99
|
+
|
|
100
|
+
## Do Not
|
|
101
|
+
|
|
102
|
+
- Do not fill or guess expected/critical labels from receipt data.
|
|
103
|
+
- Do not add an auto-adoption or auto-labeling path.
|
|
104
|
+
- Do not write candidates anywhere under `.project-loop/`.
|
|
105
|
+
- Do not mutate SQLite; the only side effects are the candidate file
|
|
106
|
+
and one JSONL event.
|
|
107
|
+
- Do not use hit/miss/usefulness vocabulary.
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# Task 0090: Eval Baseline Record / Compare (v0.2.1)
|
|
2
|
+
|
|
3
|
+
Design source: `docs/verification-feedback-design.md` Part 3 plus the
|
|
4
|
+
Approval addendum. Where this spec and the design doc disagree, the
|
|
5
|
+
design doc wins. Depends on task 0089 (unlabeled-fixture guard) being
|
|
6
|
+
merged. No migration, no new runtime dependency.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
Make retrieval eval results durable, comparable evidence with full
|
|
11
|
+
provenance, so the semantic promotion gate can eventually be decided
|
|
12
|
+
by evidence instead of enthusiasm. Metric thresholds stay ADVISORY;
|
|
13
|
+
only broken measurement infrastructure blocks.
|
|
14
|
+
|
|
15
|
+
## Scope
|
|
16
|
+
|
|
17
|
+
### 1. Metrics completion (additive to `retrieval-eval/v0` output)
|
|
18
|
+
|
|
19
|
+
- Add `false_positive_rate` = (retrieved_total − true_positive_total)
|
|
20
|
+
/ retrieved_total (aggregate and per task; `null` on empty
|
|
21
|
+
denominator — same rule as everywhere else).
|
|
22
|
+
- Add `token_cost_estimate` = deterministic `charclass/v1` estimate
|
|
23
|
+
for the retrieved candidate files (per task and aggregate). Reuse
|
|
24
|
+
the existing estimator; it is an estimate label, never a billing
|
|
25
|
+
claim. Files missing from the index contribute 0 and are listed in
|
|
26
|
+
a `token_cost_unestimated_paths` array rather than silently
|
|
27
|
+
skipped.
|
|
28
|
+
**Clarified 2026-07-06 after first acceptance pass:** the estimate
|
|
29
|
+
basis is a per-file `token_estimate` INTEGER computed at index
|
|
30
|
+
build time (file text is already in memory during scan) and stored
|
|
31
|
+
in the index detail payload. The detail artifact must NEVER store
|
|
32
|
+
file content — the index stays hashes + symbol-lite metadata
|
|
33
|
+
(architecture boundary; 0077 budget discipline; 0072 sensitive
|
|
34
|
+
surface). Old details lacking `token_estimate` route those paths
|
|
35
|
+
into `token_cost_unestimated_paths`. The first implementation
|
|
36
|
+
embedded full `indexed_content` text into the detail; do not
|
|
37
|
+
repeat that.
|
|
38
|
+
- Existing precision / recall / missing_critical_context stay
|
|
39
|
+
unchanged.
|
|
40
|
+
|
|
41
|
+
### 2. Fixture kind coverage (5 kinds)
|
|
42
|
+
|
|
43
|
+
The approved kinds: code change, docs-only, config-only, rename/move,
|
|
44
|
+
secret omission. Code change, rename/move, and secret omission
|
|
45
|
+
already exist (`retrieval_v0.json`, `retrieval_adversarial_v0.json`).
|
|
46
|
+
Add small labeled fixture tasks for **docs-only** and **config-only**
|
|
47
|
+
changes in the same synthetic style. Keep the existing floor-test
|
|
48
|
+
discipline: assert metric floors per fixture in tests so regressions
|
|
49
|
+
fail loudly.
|
|
50
|
+
|
|
51
|
+
### 3. `pcl eval retrieval --record-baseline`
|
|
52
|
+
|
|
53
|
+
- Runs the eval, then stores the full evaluation payload as a normal
|
|
54
|
+
evidence artifact under `.project-loop/evidence/retrieval-eval/`
|
|
55
|
+
(evidence row + JSONL event, following the receipt-recording
|
|
56
|
+
pattern).
|
|
57
|
+
- Baseline artifact carries a `baseline_provenance` block with ALL
|
|
58
|
+
SIX approved fields, each REQUIRED:
|
|
59
|
+
1. `fixture_path` + `fixture_content_hash` (sha256 of fixture
|
|
60
|
+
bytes),
|
|
61
|
+
2. `git_head` of the target repo,
|
|
62
|
+
3. `index_run_id` + `index_detail_hash` (hash of the index detail
|
|
63
|
+
artifact),
|
|
64
|
+
4. `code_context_config_hash` (hash of the effective `code_index`
|
|
65
|
+
config subtree from `pcl.yaml`, canonicalized),
|
|
66
|
+
5. `pcl_version` (the real `pcl.__version__`),
|
|
67
|
+
6. `eval_contract_version` (`retrieval-eval/v0`).
|
|
68
|
+
- Missing provenance input (e.g., no index run exists, not a git
|
|
69
|
+
repo) → typed error, nothing recorded. Broken provenance is not a
|
|
70
|
+
baseline.
|
|
71
|
+
|
|
72
|
+
### 4. `pcl eval retrieval --compare-baseline`
|
|
73
|
+
|
|
74
|
+
- Finds the LATEST recorded baseline whose `fixture_content_hash`
|
|
75
|
+
matches the current fixture; reports per-metric deltas
|
|
76
|
+
(precision, recall, missing-critical-context count,
|
|
77
|
+
false_positive_rate, token_cost_estimate) plus both provenance
|
|
78
|
+
blocks side by side.
|
|
79
|
+
- No baseline with a matching fixture hash → typed
|
|
80
|
+
"not comparable" error naming the nearest baseline and why it does
|
|
81
|
+
not match (hash mismatch is stated explicitly, never silently
|
|
82
|
+
compared).
|
|
83
|
+
- Comparison output is a report, not a verdict: deltas and facts
|
|
84
|
+
only. No pass/fail field, no threshold evaluation.
|
|
85
|
+
|
|
86
|
+
### 5. CI advisory comparison
|
|
87
|
+
|
|
88
|
+
- Extend `scripts/run_advisory_retrieval_eval.py` (or add a sibling
|
|
89
|
+
script wired the same way) so CI runs the eval and, when a
|
|
90
|
+
committed baseline reference exists, prints the comparison as an
|
|
91
|
+
advisory artifact.
|
|
92
|
+
- Metric regressions NEVER fail CI.
|
|
93
|
+
- Eval INFRASTRUCTURE failures DO fail CI: eval command error,
|
|
94
|
+
fixture contract violation, unreadable fixture, provenance
|
|
95
|
+
computation failure. Broken measurement is not advisory.
|
|
96
|
+
|
|
97
|
+
### 6. Documentation
|
|
98
|
+
|
|
99
|
+
- `docs/code-context.md` eval section: baseline lifecycle, the six
|
|
100
|
+
provenance fields, the advisory-vs-blocking boundary, and the
|
|
101
|
+
semantic promotion gate linkage (evidence decides promotion).
|
|
102
|
+
|
|
103
|
+
## Acceptance Criteria
|
|
104
|
+
|
|
105
|
+
- Recording twice on an unchanged repo+fixture yields identical
|
|
106
|
+
metrics and identical provenance except evidence id/created_at.
|
|
107
|
+
- Compare on matching hash reports per-metric deltas; compare after
|
|
108
|
+
editing the fixture → typed not-comparable error that names the
|
|
109
|
+
hash mismatch.
|
|
110
|
+
- Missing index run → typed error from `--record-baseline`, no
|
|
111
|
+
evidence row, no event.
|
|
112
|
+
- `false_positive_rate` and `token_cost_estimate` appear per task
|
|
113
|
+
and aggregate, `null` on empty denominators;
|
|
114
|
+
`token_cost_unestimated_paths` lists unindexed files.
|
|
115
|
+
- Docs-only and config-only fixture tasks exist with floor tests.
|
|
116
|
+
- CI script: a deliberately corrupted fixture fails the script
|
|
117
|
+
(blocking path test); a metric delta does not (advisory path
|
|
118
|
+
test).
|
|
119
|
+
- `pcl validate --strict` untouched by baselines beyond standard
|
|
120
|
+
evidence integrity.
|
|
121
|
+
- `ruff check .` passes; full `python3 -m pytest` passes; `pcl init`
|
|
122
|
+
smoke against a temp dir passes.
|
|
123
|
+
|
|
124
|
+
## Do Not
|
|
125
|
+
|
|
126
|
+
- Do not implement metric thresholds, pass/fail verdicts, or release
|
|
127
|
+
gates on metric values.
|
|
128
|
+
- Do not compare across different fixture hashes "helpfully".
|
|
129
|
+
- Do not read generated dashboard HTML, mutate state from the eval
|
|
130
|
+
path beyond the evidence row + event, or add raw SQL.
|
|
131
|
+
- Do not use hit/miss/usefulness vocabulary; token cost is an
|
|
132
|
+
estimate, not a price.
|
|
133
|
+
- Do not touch embeddings / Tree-sitter / call graphs — the semantic
|
|
134
|
+
promotion gate is unchanged.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Task 0092: Idempotent Same-State Transitions (v0.2.2, F1)
|
|
2
|
+
|
|
3
|
+
Source: first external-project agent dogfood feedback
|
|
4
|
+
(`docs/feedback/2026-07-06-ax1-moc1-agent-feedback.md`, item F1).
|
|
5
|
+
`pcl feature status F-0001 --status passing` exits 2 when the feature
|
|
6
|
+
is already passing. For scripting agents a same-state request is a
|
|
7
|
+
satisfied postcondition, not an input error; exit 2 forces every
|
|
8
|
+
caller to pre-read state or wrap errors. No migration, no new
|
|
9
|
+
dependency.
|
|
10
|
+
|
|
11
|
+
## Behavior change (all four sites)
|
|
12
|
+
|
|
13
|
+
Same-state transition requests become successful no-ops:
|
|
14
|
+
|
|
15
|
+
- `pcl feature status <id> --status <current>` (`commands.py`
|
|
16
|
+
`set_feature_status`)
|
|
17
|
+
- `pcl goal complete` / goal lifecycle same-state (`lifecycle.py`,
|
|
18
|
+
the "Goal ... is already ..." site)
|
|
19
|
+
- `pcl test <verb>` when the test case is already in the requested
|
|
20
|
+
terminal state (`stories.py`, the "Test case ... is already ..."
|
|
21
|
+
site)
|
|
22
|
+
- `pcl task status <id> <current>` (`tasks.py`, the "Task ... is
|
|
23
|
+
already ..." site)
|
|
24
|
+
|
|
25
|
+
No-op semantics (identical across all four):
|
|
26
|
+
|
|
27
|
+
- Exit code 0; JSON gains `"changed": false` plus the current status.
|
|
28
|
+
- NOTHING is recorded: no status update, no inline evidence row, no
|
|
29
|
+
event append. A no-op that logs is not a no-op — repeated agent
|
|
30
|
+
retries must not inflate the audit trail.
|
|
31
|
+
- Human-readable output states plainly that the entity was already in
|
|
32
|
+
the requested state (e.g. `Feature F-0001 already passing; no
|
|
33
|
+
change recorded.`).
|
|
34
|
+
- Supplied `--evidence` / `--reason` text on a no-op is NOT recorded;
|
|
35
|
+
the response must make that explicit (e.g.
|
|
36
|
+
`"evidence_recorded": false`) so callers do not believe evidence
|
|
37
|
+
was attached. Attaching fresh evidence to an already-terminal state
|
|
38
|
+
(re-verification) is out of scope here and belongs to the evidence
|
|
39
|
+
entry-path design.
|
|
40
|
+
|
|
41
|
+
Changed transitions additionally gain `"changed": true` (additive
|
|
42
|
+
JSON field; existing fields untouched).
|
|
43
|
+
|
|
44
|
+
## What must NOT change
|
|
45
|
+
|
|
46
|
+
- Invalid TARGET states and disallowed transitions (e.g. rubric or
|
|
47
|
+
guard violations, unknown entities) keep their existing typed
|
|
48
|
+
errors and exit codes. Only the exact same-state case becomes a
|
|
49
|
+
no-op.
|
|
50
|
+
- Evidence-backed terminal-state validation invariants are untouched
|
|
51
|
+
(a no-op changes nothing, so nothing new to validate).
|
|
52
|
+
- No behavior change for transitions that DO change state, other
|
|
53
|
+
than the additive `changed` field.
|
|
54
|
+
|
|
55
|
+
## Acceptance Criteria
|
|
56
|
+
|
|
57
|
+
- For each of the four commands: same-state call → exit 0,
|
|
58
|
+
`changed: false`, row count of events/evidence unchanged (assert
|
|
59
|
+
counts before/after), status unchanged, and repeated invocation is
|
|
60
|
+
stable (call twice more, same result).
|
|
61
|
+
- Same-state call with `--evidence "..."` → `evidence_recorded:
|
|
62
|
+
false` in JSON and no evidence row.
|
|
63
|
+
- Changed-state calls still record exactly as before and now carry
|
|
64
|
+
`changed: true`.
|
|
65
|
+
- Unknown entity / invalid target status still typed errors (regression
|
|
66
|
+
tests kept).
|
|
67
|
+
- Docs: `docs/golden-path.md` or command docs note the idempotent
|
|
68
|
+
semantics in one short paragraph.
|
|
69
|
+
- `ruff check .` passes; full `python3 -m pytest` passes; `pcl init`
|
|
70
|
+
smoke against a temp dir passes.
|
|
71
|
+
|
|
72
|
+
## Do Not
|
|
73
|
+
|
|
74
|
+
- Do not make disallowed transitions succeed.
|
|
75
|
+
- Do not append events or evidence on no-ops.
|
|
76
|
+
- Do not add a `--force` re-record path (that is evidence-path design
|
|
77
|
+
territory).
|
|
78
|
+
- Do not touch schema or raw SQL outside the existing service layer.
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# Task 0093: `pcl evidence add` + Adhoc Bundles (v0.2.2, F2/F6)
|
|
2
|
+
|
|
3
|
+
Design source: `docs/evidence-entry-paths-design.md`
|
|
4
|
+
(**APPROVED 2026-07-07**, Parts 1-2 and the approval record). Where
|
|
5
|
+
this spec and the design doc disagree, the design doc wins. No
|
|
6
|
+
migration, no new runtime dependency; schema stays v5.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
Operators and agents produce reviewable artifacts outside the job
|
|
11
|
+
loop (test output, screenshots, visual-check reports) and PLH has no
|
|
12
|
+
honest way to accept them as evidence. Add the missing primitive.
|
|
13
|
+
|
|
14
|
+
## Scope
|
|
15
|
+
|
|
16
|
+
### 1. `pcl evidence add`
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
pcl evidence add --file work/reports/pytest-out.txt \
|
|
20
|
+
--summary "pytest run for suggestion E-0017/VS-01" \
|
|
21
|
+
--command "python3 -m pytest tests/test_context.py" --json
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
- `--file` is required and repeatable; every path must exist and be
|
|
25
|
+
readable at record time (missing/unreadable → typed error, nothing
|
|
26
|
+
recorded). `--summary` required. `--command` optional — it is the
|
|
27
|
+
CALLER'S claim of how the artifact was produced; PLH stores it
|
|
28
|
+
verbatim and never runs or verifies it (docs and help must say
|
|
29
|
+
this).
|
|
30
|
+
- One file → evidence `type = 'adhoc_artifact'` (approved). Two or
|
|
31
|
+
more files → `type = 'adhoc_bundle'` (approved).
|
|
32
|
+
- In both cases PLH writes a small manifest JSON under
|
|
33
|
+
`.project-loop/evidence/adhoc/` recording, per member: relative
|
|
34
|
+
path, size_bytes, sha256 — pinned AT RECORD TIME. Members are
|
|
35
|
+
referenced in place, NOT copied (approved; a `--copy` mode is a
|
|
36
|
+
possible later addition, do not add it now).
|
|
37
|
+
- The evidence row's `path` points at the manifest; `command` and
|
|
38
|
+
`summary` come from the flags. One evidence row per invocation, one
|
|
39
|
+
standard event via `append_event` (SQLite + JSONL both — never a
|
|
40
|
+
side-channel writer; see agent-tasks/0089 clarification).
|
|
41
|
+
- Manifest contract name: `adhoc-evidence/v0` with a
|
|
42
|
+
`contract_version` field. Deterministic member ordering (input
|
|
43
|
+
order preserved; duplicates by path → typed error).
|
|
44
|
+
- JSON output returns the evidence id, type, manifest path, and the
|
|
45
|
+
member list with hashes.
|
|
46
|
+
|
|
47
|
+
### 2. Strict validation integration
|
|
48
|
+
|
|
49
|
+
- `pcl validate --strict` verifies for adhoc evidence rows that the
|
|
50
|
+
manifest file exists and parses; missing/corrupt manifest is an
|
|
51
|
+
ERROR (state integrity).
|
|
52
|
+
- Member hash drift (file changed or deleted since recording) is a
|
|
53
|
+
WARNING, not an error (approved): drift is the working tree moving
|
|
54
|
+
on; the pinned hash remains the recorded claim. Warning text must
|
|
55
|
+
name the evidence id, member path, and which aspect drifted
|
|
56
|
+
(missing vs hash mismatch).
|
|
57
|
+
|
|
58
|
+
### 3. Documentation
|
|
59
|
+
|
|
60
|
+
- New section in `docs/data-model.md` or `docs/golden-path.md`:
|
|
61
|
+
the adhoc evidence flow, the claim/pointer epistemic boundary, and
|
|
62
|
+
the worked example ending in
|
|
63
|
+
`pcl verification feedback ... --evidence E-00xx`.
|
|
64
|
+
- `docs/evidence-entry-paths-design.md` stays the design record; do
|
|
65
|
+
not duplicate it wholesale.
|
|
66
|
+
|
|
67
|
+
## Acceptance Criteria
|
|
68
|
+
|
|
69
|
+
- Single file → `adhoc_artifact`; multiple files → `adhoc_bundle`;
|
|
70
|
+
manifest carries per-member path/size/sha256; JSON output matches.
|
|
71
|
+
- Determinism: same files, same flags → identical manifest content
|
|
72
|
+
except created_at/evidence id.
|
|
73
|
+
- Missing file, unreadable file, duplicate path → distinct typed
|
|
74
|
+
errors, no evidence row, no event, no manifest file left behind.
|
|
75
|
+
- `validate --strict`: green after add; deleting a member → WARNING
|
|
76
|
+
(still ok); editing a member → WARNING naming the hash mismatch;
|
|
77
|
+
deleting the MANIFEST → ERROR.
|
|
78
|
+
- End-to-end test: `evidence add` → `verification feedback
|
|
79
|
+
--status executed --evidence <new id>` succeeds (closes the M2
|
|
80
|
+
dogfood gap).
|
|
81
|
+
- Events mirrored in SQLite and JSONL; strict audit integrity stays
|
|
82
|
+
green (regression test).
|
|
83
|
+
- `ruff check .` passes; full `python3 -m pytest` passes; `pcl init`
|
|
84
|
+
smoke against a temp dir passes.
|
|
85
|
+
|
|
86
|
+
## Do Not
|
|
87
|
+
|
|
88
|
+
- Do not copy member files or add `--copy` yet.
|
|
89
|
+
- Do not execute `--command` or capture output (`--capture` is out of
|
|
90
|
+
scope by design).
|
|
91
|
+
- Do not add an M:N evidence link table or any migration.
|
|
92
|
+
- Do not use "verified"/"safe" language for adhoc evidence — it is a
|
|
93
|
+
recorded claim with pinned hashes.
|
|
94
|
+
- Do not touch `verification feedback` flags (`--output-file` is
|
|
95
|
+
formally superseded, not implemented).
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Task 0094: Job Completion Evidence Linkage (v0.2.2, F3)
|
|
2
|
+
|
|
3
|
+
Design source: `docs/evidence-entry-paths-design.md`
|
|
4
|
+
(**APPROVED 2026-07-07**, Part 3). Depends on 0093 being merged (its
|
|
5
|
+
adhoc evidence rows are natural inputs here, and both touch cli.py /
|
|
6
|
+
validators tests). No migration; schema stays v5.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
In the ax1-moc1 dogfood run, `pcl jobs list --json` showed empty
|
|
11
|
+
evidence for jobs whose `output_path` files existed, and there was no
|
|
12
|
+
way to attach evidence when completing a job. Close both gaps using
|
|
13
|
+
the EXISTING `latest_evidence_id` pointer on agent jobs.
|
|
14
|
+
|
|
15
|
+
## Scope
|
|
16
|
+
|
|
17
|
+
### 1. `pcl jobs complete <job-id> --evidence E-00xx`
|
|
18
|
+
|
|
19
|
+
- Optional repeatable? NO — single `--evidence` flag (the
|
|
20
|
+
one-pointer model matches `latest_evidence_id`; multiple artifacts
|
|
21
|
+
are one adhoc bundle from 0093).
|
|
22
|
+
- The referenced evidence row must exist → typed error otherwise,
|
|
23
|
+
nothing changed.
|
|
24
|
+
- Sets `latest_evidence_id` on the job as part of the existing
|
|
25
|
+
completion flow; completion event payload gains the evidence id.
|
|
26
|
+
All existing completion semantics (summary, status transition,
|
|
27
|
+
guards) unchanged.
|
|
28
|
+
|
|
29
|
+
### 2. Jobs-list evidence visibility audit (the ax1-moc1 symptom)
|
|
30
|
+
|
|
31
|
+
- Reproduce first: in the ax1-moc1 run, jobs completed through the
|
|
32
|
+
normal flow showed `evidence: []`/empty in `pcl jobs list --json`
|
|
33
|
+
even though `output_path` existed and ingest had run. Find out
|
|
34
|
+
why (linkage never set? read path not surfacing it? ingest path
|
|
35
|
+
only sets it on certain flows?) and fix the actual cause — do not
|
|
36
|
+
paper over it in the list renderer.
|
|
37
|
+
- After the fix: a job whose output was ingested surfaces its
|
|
38
|
+
evidence reference in `jobs read` / `jobs list --json` and
|
|
39
|
+
dashboard-data.
|
|
40
|
+
|
|
41
|
+
### 3. Documentation
|
|
42
|
+
|
|
43
|
+
- `docs/golden-path.md` / agent adapter docs: one short paragraph —
|
|
44
|
+
complete a job with `--evidence` when the artifact is already
|
|
45
|
+
recorded (0093 flow), otherwise ingest output as before.
|
|
46
|
+
|
|
47
|
+
## Acceptance Criteria
|
|
48
|
+
|
|
49
|
+
- `jobs complete --evidence E-00xx` sets `latest_evidence_id`,
|
|
50
|
+
appends the completion event with the evidence id, and the linkage
|
|
51
|
+
appears in `jobs read --json`, `jobs list --json`, and
|
|
52
|
+
dashboard-data.
|
|
53
|
+
- Unknown evidence id → typed error; job stays incomplete and
|
|
54
|
+
unchanged.
|
|
55
|
+
- Root-cause test for the visibility bug: a test that reproduces the
|
|
56
|
+
original empty-evidence symptom path and asserts it is fixed
|
|
57
|
+
(write the test against the actual discovered cause, and explain
|
|
58
|
+
the cause in the task report).
|
|
59
|
+
- Existing completion without `--evidence` behaves exactly as
|
|
60
|
+
before.
|
|
61
|
+
- `ruff check .` passes; full `python3 -m pytest` passes; `pcl init`
|
|
62
|
+
smoke against a temp dir passes; strict validate/audit integrity
|
|
63
|
+
regression stays green.
|
|
64
|
+
|
|
65
|
+
## Do Not
|
|
66
|
+
|
|
67
|
+
- Do not add repeatable `--evidence` or an M:N link table.
|
|
68
|
+
- Do not auto-create evidence from `output_path` without ingest —
|
|
69
|
+
linkage requires a real evidence row.
|
|
70
|
+
- Do not change ingest contracts (`agent-output/v1`) or job state
|
|
71
|
+
machine semantics.
|
|
@@ -78,6 +78,12 @@ The expected output path is deterministic:
|
|
|
78
78
|
|
|
79
79
|
The ingest path must be the project-scoped `.project-loop/evidence/agent-runs/<job_id>/output.md`. Files from another directory, even if they contain an `agent-runs/<job_id>` segment, are rejected. Ingest is also a guarded state transition: cancelled or failed jobs cannot be revived by late output, and inactive workflow runs cannot accept new agent output.
|
|
80
80
|
|
|
81
|
+
If a job artifact has already been recorded with `pcl evidence add`, complete
|
|
82
|
+
the job with `pcl jobs complete <job-id> --evidence E-00xx` to link that
|
|
83
|
+
existing evidence row. Otherwise, keep using `pcl ingest-agent-run` for raw
|
|
84
|
+
agent output so PLH validates `agent-output/v1`, creates the evidence row, and
|
|
85
|
+
marks the job passed in one guarded transition.
|
|
86
|
+
|
|
81
87
|
After ingest, `pcl jobs read J-0001 --json` and `pcl jobs list --json` expose the derived evidence linkage:
|
|
82
88
|
|
|
83
89
|
```json
|
|
@@ -88,7 +94,7 @@ After ingest, `pcl jobs read J-0001 --json` and `pcl jobs list --json` expose th
|
|
|
88
94
|
}
|
|
89
95
|
```
|
|
90
96
|
|
|
91
|
-
The linkage is derived from `agent_output_ingested` events and `evidence` rows, not from a mutable `agent_jobs.evidence_id` column. Repeated ingests can therefore preserve multiple evidence ids while the latest evidence remains easy to find from job surfaces and the dashboard.
|
|
97
|
+
The linkage is derived from `agent_output_ingested` events, evidence-bearing `agent_job_completed` events, and `evidence` rows, not from a mutable `agent_jobs.evidence_id` column. Repeated ingests can therefore preserve multiple evidence ids while the latest evidence remains easy to find from job surfaces and the dashboard.
|
|
92
98
|
|
|
93
99
|
## Minimal Output File
|
|
94
100
|
|
|
@@ -55,6 +55,8 @@ Evaluate retrieval behavior against labels:
|
|
|
55
55
|
|
|
56
56
|
```bash
|
|
57
57
|
pcl eval retrieval --fixture tests/fixtures/retrieval_v0.json --json
|
|
58
|
+
pcl eval retrieval --fixture tests/fixtures/retrieval_v0.json --record-baseline --json
|
|
59
|
+
pcl eval retrieval --fixture tests/fixtures/retrieval_v0.json --compare-baseline --json
|
|
58
60
|
```
|
|
59
61
|
|
|
60
62
|
## Index Contract
|
|
@@ -109,6 +111,7 @@ needs every file, ignored path, and hash-skip entry on stdout.
|
|
|
109
111
|
"mtime": 123456789,
|
|
110
112
|
"sha256": "abc123",
|
|
111
113
|
"line_count": 10,
|
|
114
|
+
"token_estimate": 42,
|
|
112
115
|
"symbol_summary": {
|
|
113
116
|
"contract_version": "symbol-summary/v0",
|
|
114
117
|
"symbols": [{"type": "function", "name": "pack_context_for_job", "line": 1}]
|
|
@@ -629,6 +632,29 @@ Task fields:
|
|
|
629
632
|
`assertion_note`, and `must_not_retrieve`, are metadata and are ignored by
|
|
630
633
|
the evaluator.
|
|
631
634
|
|
|
635
|
+
Dogfood receipts become fixture candidates through a staging workflow:
|
|
636
|
+
|
|
637
|
+
```bash
|
|
638
|
+
pcl eval fixture propose --from-receipt E-0001 --json
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
The command reads a real `context_receipt` evidence row and writes an
|
|
642
|
+
UNLABELED `retrieval-fixture/v0` candidate to repo-root `fixtures/proposed/`.
|
|
643
|
+
That directory is a Git-tracked source staging area, not `.project-loop/`
|
|
644
|
+
runtime state. The candidate has one task, a synthetic unified diff touching
|
|
645
|
+
the receipt's `changed_files` paths, `diff_synthesized_from_receipt: true`, and
|
|
646
|
+
source receipt provenance (`evidence_id`, `created_at`, `diff_source`,
|
|
647
|
+
optional `base_ref`, and retrieved candidate paths). The synthetic diff is only
|
|
648
|
+
a replay handle because receipts do not store original diff text.
|
|
649
|
+
|
|
650
|
+
PLH never fabricates ground-truth labels. Proposed tasks carry
|
|
651
|
+
`labels_status: "unlabeled"` and empty `expected_files`, `expected_tests`, and
|
|
652
|
+
`critical_context` arrays. `pcl eval retrieval` refuses these candidates with a
|
|
653
|
+
typed error. A human must inspect the source receipt, fill the expected and
|
|
654
|
+
critical arrays, remove or change the unlabeled marker, and manually adopt the
|
|
655
|
+
fixture into `tests/fixtures/` with a normal Git move. There is no auto-labeling
|
|
656
|
+
or auto-adoption path.
|
|
657
|
+
|
|
632
658
|
Fixture evolution is additive in v0:
|
|
633
659
|
|
|
634
660
|
- New optional fields may be added without a version bump.
|
|
@@ -640,6 +666,8 @@ Fixture evolution is additive in v0:
|
|
|
640
666
|
|
|
641
667
|
The checked-in fixture families have different purposes:
|
|
642
668
|
|
|
669
|
+
- `retrieval_v0.json`: small synthetic coverage for code change, docs-only,
|
|
670
|
+
and config-only tasks.
|
|
643
671
|
- `real-history`: derived from actual repository changes, currently following
|
|
644
672
|
the `tests/fixtures/retrieval_real_history_v0.json` pattern. These fixtures
|
|
645
673
|
measure ordinary retrieval quality against real change history.
|
|
@@ -649,23 +677,88 @@ The checked-in fixture families have different purposes:
|
|
|
649
677
|
omission, stale-index signaling, and an annotated renamed-file baseline miss.
|
|
650
678
|
|
|
651
679
|
`pcl eval retrieval --fixture <path> --json` returns `retrieval-eval/v0` with
|
|
652
|
-
precision, recall,
|
|
680
|
+
precision, recall, false-positive rate, token cost estimate,
|
|
681
|
+
`missing_critical_context`, and per-task retrieved paths.
|
|
682
|
+
|
|
683
|
+
Metric fields:
|
|
684
|
+
|
|
685
|
+
- `precision`: true positives divided by retrieved paths, preserving the v0
|
|
686
|
+
empty-denominator rule.
|
|
687
|
+
- `recall`: true positives divided by expected files/tests, preserving the v0
|
|
688
|
+
empty-denominator rule.
|
|
689
|
+
- `false_positive_rate`: `(retrieved - true_positives) / retrieved`; empty
|
|
690
|
+
retrieved denominators yield `null`.
|
|
691
|
+
- `token_cost_estimate`: deterministic `charclass/v1` estimate over retrieved
|
|
692
|
+
paths' per-file `token_estimate` integers from
|
|
693
|
+
`.project-loop/cache/code-index-detail.json`. The estimate is computed at
|
|
694
|
+
index build time from indexed text already in memory; the detail artifact does
|
|
695
|
+
not store raw file contents. It is an estimate of indexed text volume, not a
|
|
696
|
+
price or billing signal.
|
|
697
|
+
- `token_cost_unestimated_paths`: retrieved paths that are absent from the
|
|
698
|
+
detail artifact or present without a `token_estimate`, including details
|
|
699
|
+
built by older versions. They contribute `0` to the estimate and are listed
|
|
700
|
+
explicitly.
|
|
701
|
+
- `missing_critical_context`: labeled critical paths not retrieved.
|
|
702
|
+
|
|
653
703
|
Task output may also include additive diagnostic fields used by adversarial
|
|
654
704
|
fixtures: `retrieval_source`, `staleness_warnings`,
|
|
655
705
|
`staleness_affected_paths`, `sensitive_omitted_count`,
|
|
656
706
|
`retrieved_snapshot_consistency`, `excluded_changed_files`, and
|
|
657
707
|
`expected_misses`.
|
|
658
708
|
|
|
709
|
+
### Baseline Lifecycle
|
|
710
|
+
|
|
711
|
+
`pcl eval retrieval --fixture <path> --record-baseline --json` runs the eval
|
|
712
|
+
and stores the full payload as normal evidence under
|
|
713
|
+
`.project-loop/evidence/retrieval-eval/`. Recording writes an evidence row and
|
|
714
|
+
the standard mirrored event in SQLite and `.project-loop/events.jsonl`; it does
|
|
715
|
+
not use a JSONL-only side channel.
|
|
716
|
+
|
|
717
|
+
Every baseline artifact carries `baseline_provenance` with these required
|
|
718
|
+
fields:
|
|
719
|
+
|
|
720
|
+
- `fixture_path`
|
|
721
|
+
- `fixture_content_hash` (sha256 of fixture bytes)
|
|
722
|
+
- `git_head`
|
|
723
|
+
- `index_run_id`
|
|
724
|
+
- `index_detail_hash` (sha256 of the code index detail artifact)
|
|
725
|
+
- `code_context_config_hash` (canonical effective `code_index` subtree)
|
|
726
|
+
- `pcl_version`
|
|
727
|
+
- `eval_contract_version` (`retrieval-eval/v0`)
|
|
728
|
+
|
|
729
|
+
Missing provenance inputs, such as no index run, missing index detail artifact,
|
|
730
|
+
or a non-Git target, are typed errors. In those cases no baseline artifact,
|
|
731
|
+
evidence row, or event is recorded.
|
|
732
|
+
|
|
733
|
+
`pcl eval retrieval --fixture <path> --compare-baseline --json` compares the
|
|
734
|
+
current eval with the latest recorded baseline whose `fixture_content_hash`
|
|
735
|
+
matches the current fixture. It reports current metrics, baseline metrics,
|
|
736
|
+
metric deltas, current provenance, and baseline provenance. The reported deltas
|
|
737
|
+
cover precision, recall, missing-critical-context count, false-positive rate,
|
|
738
|
+
and token cost estimate.
|
|
739
|
+
|
|
740
|
+
Comparison never crosses fixture hashes. If only a different fixture hash is
|
|
741
|
+
available, the command returns a typed not-comparable error naming the nearest
|
|
742
|
+
baseline and the hash mismatch. The comparison output is facts and deltas only:
|
|
743
|
+
no threshold logic, no verdict field, and no release gate.
|
|
744
|
+
|
|
659
745
|
CI runs `python3 scripts/run_advisory_retrieval_eval.py` after pytest. The step
|
|
660
746
|
initializes and indexes the checked-out project, evaluates the checked-in
|
|
661
747
|
real-history fixtures, evaluates the checked-in adversarial fixture against a
|
|
662
|
-
prepared temp project, and prints a compact JSON summary.
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
748
|
+
prepared temp project, and prints a compact JSON summary. When a local
|
|
749
|
+
retrieval-eval baseline exists, the script also prints the same comparison
|
|
750
|
+
facts as advisory output.
|
|
751
|
+
|
|
752
|
+
Advisory-vs-blocking boundary:
|
|
753
|
+
|
|
754
|
+
- Metric deltas never fail CI in v0.2.1.
|
|
755
|
+
- Eval infrastructure integrity failures do fail CI: eval command errors,
|
|
756
|
+
fixture contract violations, unreadable fixtures, and provenance computation
|
|
757
|
+
failures.
|
|
758
|
+
- Broken measurement is not advisory.
|
|
759
|
+
|
|
760
|
+
Semantic promotion gate: richer retrieval approaches such as semantic
|
|
761
|
+
retrieval, Tree-sitter parsing, or call-graph retrieval remain out of scope for
|
|
762
|
+
v0.2.1. Future promotion is evidence-driven: baseline history and fixture
|
|
763
|
+
coverage decide whether richer retrieval is justified, not enthusiasm. This
|
|
764
|
+
document deliberately defines no metric thresholds or release gates.
|