@guilz-dev/sdlc-gh 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/CODEOWNERS +5 -0
- package/.github/ISSUE_TEMPLATE/bug_report.yml +68 -0
- package/.github/ISSUE_TEMPLATE/config.yml +1 -0
- package/.github/ISSUE_TEMPLATE/feature_request.yml +39 -0
- package/.github/ISSUE_TEMPLATE/support.yml +56 -0
- package/.github/ISSUE_TEMPLATE/task.yml +89 -0
- package/.github/agents/implementer.agent.md +17 -0
- package/.github/agents/reviewer.agent.md +18 -0
- package/.github/agents/triager.agent.md +13 -0
- package/.github/aw/actions-lock.json +9 -0
- package/.github/copilot-instructions.md +35 -0
- package/.github/hooks/hooks.json +12 -0
- package/.github/instructions/core.instructions.md +11 -0
- package/.github/instructions/profiles/go.instructions.md +10 -0
- package/.github/instructions/profiles/php.instructions.md +11 -0
- package/.github/instructions/profiles/python.instructions.md +11 -0
- package/.github/instructions/profiles/ruby.instructions.md +11 -0
- package/.github/instructions/profiles/typescript.instructions.md +11 -0
- package/.github/labels.yml +55 -0
- package/.github/pull_request_template.md +33 -0
- package/.github/ruleset.example.json +33 -0
- package/.github/ruleset.harness-eval.example.json +29 -0
- package/.github/skills/quality-loop/SKILL.md +23 -0
- package/.github/workflows/agent-retry-orchestrator.yml +161 -0
- package/.github/workflows/copilot-setup-steps.yml +64 -0
- package/.github/workflows/eval-ci.yml +169 -0
- package/.github/workflows/eval-drift.yml +75 -0
- package/.github/workflows/gh-aw-dogfood-ci.yml +73 -0
- package/.github/workflows/harness-ci.yml +244 -0
- package/.github/workflows/harness-sync.yml +28 -0
- package/.github/workflows/l1-readiness-check.yml +45 -0
- package/.github/workflows/labels-sync.yml +24 -0
- package/.github/workflows/nightly-harness-review.lock.yml +1643 -0
- package/.github/workflows/nightly-harness-review.md +87 -0
- package/.github/workflows/nightly-harness-review.yml +63 -0
- package/.github/workflows/npm-publish.yml +49 -0
- package/.github/workflows/pr-context-comment.yml +138 -0
- package/.github/workflows/product-ci-go.yml +33 -0
- package/.github/workflows/product-ci-php.yml +39 -0
- package/.github/workflows/product-ci-python.yml +34 -0
- package/.github/workflows/product-ci-ruby.yml +35 -0
- package/.github/workflows/product-ci-ts.yml +37 -0
- package/.github/workflows/task-issue-label-sync.yml +50 -0
- package/.github/workflows/weekly-redteam.lock.yml +1571 -0
- package/.github/workflows/weekly-redteam.md +76 -0
- package/.github/zizmor.yml +11 -0
- package/AGENTS.md +54 -0
- package/LICENSE +21 -0
- package/README.md +366 -0
- package/config/stacks.json +55 -0
- package/docs/adoption.md +126 -0
- package/docs/arch.md +535 -0
- package/docs/auth-boundaries.md +16 -0
- package/docs/coding-agent-l1.md +152 -0
- package/docs/exceptions/README.md +25 -0
- package/docs/exceptions/TEMPLATE.md +8 -0
- package/docs/failure-taxonomy.md +23 -0
- package/docs/gh-aw-dogfood.md +109 -0
- package/docs/kpi-baseline.md +9 -0
- package/docs/nightly-harness-review.md +94 -0
- package/docs/operations.md +108 -0
- package/docs/publishing.md +79 -0
- package/docs/revert-playbook.md +44 -0
- package/docs/shared-config.md +30 -0
- package/docs/telemetry-artifacts.md +78 -0
- package/docs/telemetry-schema.md +60 -0
- package/evals/.score-baseline.json +6 -0
- package/evals/e2e-bench/README.md +28 -0
- package/evals/e2e-bench/manifest.json +16 -0
- package/evals/e2e-bench/tasks/e2e-001.yml +10 -0
- package/evals/e2e-bench/tasks/e2e-002.yml +11 -0
- package/evals/e2e-bench/tasks/e2e-003.yml +10 -0
- package/evals/e2e-bench/tasks/e2e-004.yml +14 -0
- package/evals/e2e-bench/tasks/e2e-005.yml +11 -0
- package/evals/e2e-bench/tasks/e2e-006.yml +10 -0
- package/evals/e2e-bench/tasks/e2e-007.yml +10 -0
- package/evals/e2e-bench/tasks/e2e-008.yml +10 -0
- package/evals/e2e-bench/tasks/e2e-009.yml +10 -0
- package/evals/trajectories/rubric.md +12 -0
- package/evals/trajectories/test_harness_conventions.py +271 -0
- package/infra/README.md +49 -0
- package/infra/langfuse/docker-compose.yml +25 -0
- package/infra/otel/collector-config.yml +24 -0
- package/infra/samples/gh-aw-dogfood-report.json +44 -0
- package/infra/samples/harness-review-routing-plan.json +19 -0
- package/infra/samples/harness-review-summary.json +61 -0
- package/infra/samples/telemetry-artifact.json +29 -0
- package/infra/samples/telemetry-payload.json +19 -0
- package/package.json +85 -0
- package/prompts/triager-classify.prompt.yml +10 -0
- package/sample/go/add.go +5 -0
- package/sample/go/add_test.go +9 -0
- package/sample/go/go.mod +3 -0
- package/sample/php/composer.json +26 -0
- package/sample/php/composer.lock +1881 -0
- package/sample/php/phpunit.xml +8 -0
- package/sample/php/src/Add.php +13 -0
- package/sample/php/tests/AddTest.php +16 -0
- package/sample/python/requirements-dev.txt +2 -0
- package/sample/python/src/__init__.py +0 -0
- package/sample/python/src/greet.py +3 -0
- package/sample/python/tests/conftest.py +4 -0
- package/sample/python/tests/test_greet.py +5 -0
- package/sample/ruby/.rubocop.yml +10 -0
- package/sample/ruby/Gemfile +6 -0
- package/sample/ruby/Gemfile.lock +58 -0
- package/sample/ruby/lib/add.rb +9 -0
- package/sample/ruby/spec/add_spec.rb +11 -0
- package/sample/ts/biome.json +6 -0
- package/sample/ts/package-lock.json +1763 -0
- package/sample/ts/package.json +15 -0
- package/sample/ts/src/add.ts +3 -0
- package/sample/ts/tests/add.test.ts +8 -0
- package/sample/ts/tsconfig.json +12 -0
- package/scripts/aggregate-harness-review.mjs +48 -0
- package/scripts/bootstrap-harness.sh +411 -0
- package/scripts/check-diff-size.mjs +46 -0
- package/scripts/check-e2e-manifest.mjs +35 -0
- package/scripts/check-eval-score-drift.mjs +31 -0
- package/scripts/check-gh-aw-dogfood-scope.mjs +51 -0
- package/scripts/check-issue-spec.mjs +215 -0
- package/scripts/check-l1-readiness.mjs +82 -0
- package/scripts/check-open-pr-limit.mjs +34 -0
- package/scripts/doctor.mjs +177 -0
- package/scripts/emit-gh-aw-dogfood-report.mjs +112 -0
- package/scripts/emit-telemetry-artifact.mjs +99 -0
- package/scripts/fetch-telemetry-artifacts.mjs +176 -0
- package/scripts/harness-drift-report.mjs +99 -0
- package/scripts/lib/bootstrap-copy.mjs +123 -0
- package/scripts/lib/ccsd-contract.mjs +212 -0
- package/scripts/lib/diff-size.mjs +103 -0
- package/scripts/lib/doctor-local.mjs +179 -0
- package/scripts/lib/e2e-manifest.mjs +76 -0
- package/scripts/lib/gh-aw-dogfood.mjs +293 -0
- package/scripts/lib/github-config.mjs +94 -0
- package/scripts/lib/harness-ci-fragments.mjs +98 -0
- package/scripts/lib/harness-review-routing.mjs +244 -0
- package/scripts/lib/harness-review.mjs +388 -0
- package/scripts/lib/issue-form-label-sync.mjs +56 -0
- package/scripts/lib/l1-readiness.mjs +258 -0
- package/scripts/lib/merge-harness-package.mjs +36 -0
- package/scripts/lib/npm-package.mjs +129 -0
- package/scripts/lib/setup-wizard.mjs +224 -0
- package/scripts/lib/stacks.mjs +138 -0
- package/scripts/lib/telemetry-artifact.mjs +253 -0
- package/scripts/lib/template-root.mjs +39 -0
- package/scripts/merge-harness-package.mjs +14 -0
- package/scripts/route-harness-review.mjs +168 -0
- package/scripts/run-e2e-bench.mjs +216 -0
- package/scripts/sdlc-gh-cli.mjs +91 -0
- package/scripts/select-eval-jobs.mjs +41 -0
- package/scripts/setup-github.mjs +242 -0
- package/scripts/setup-github.sh +4 -0
- package/scripts/setup-wizard.mjs +426 -0
- package/scripts/test-bootstrap-guidance-scenarios.mjs +94 -0
- package/scripts/test-diff-size-scenarios.mjs +88 -0
- package/scripts/test-doctor-scenarios.mjs +70 -0
- package/scripts/test-e2e-manifest-scenarios.mjs +65 -0
- package/scripts/test-gh-aw-dogfood-scenarios.mjs +74 -0
- package/scripts/test-harness-review-routing-scenarios.mjs +130 -0
- package/scripts/test-harness-review-scenarios.mjs +92 -0
- package/scripts/test-hooks-scenarios.mjs +44 -0
- package/scripts/test-issue-form-label-sync-scenarios.mjs +48 -0
- package/scripts/test-issue-spec-scenarios.mjs +258 -0
- package/scripts/test-l1-readiness-scenarios.mjs +204 -0
- package/scripts/test-merge-harness-package-scenarios.mjs +53 -0
- package/scripts/test-npm-package-scenarios.mjs +31 -0
- package/scripts/test-sdlc-gh-cli-scenarios.mjs +54 -0
- package/scripts/test-setup-github-scenarios.mjs +103 -0
- package/scripts/test-setup-wizard-scenarios.mjs +114 -0
- package/scripts/test-telemetry-artifact-scenarios.mjs +69 -0
- package/scripts/trim-harness-ci.mjs +18 -0
- package/scripts/validate-gh-aw-compile.mjs +64 -0
- package/scripts/validate-harness.mjs +199 -0
- package/scripts/validate-telemetry.mjs +21 -0
- package/scripts/verify-bootstrap-stacks.sh +192 -0
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Optional shared config repository
|
|
2
|
+
|
|
3
|
+
For organizations with multiple product repos, use a **shared config repo** to distribute common agents and skills.
|
|
4
|
+
|
|
5
|
+
## Layout
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
org-harness-shared/
|
|
9
|
+
├── .github/
|
|
10
|
+
│ └── agents/
|
|
11
|
+
│ └── skills/
|
|
12
|
+
└── README.md
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Distribution options
|
|
16
|
+
|
|
17
|
+
1. **Manual copy** — periodic sync of `agents/` and `skills/` into product repos
|
|
18
|
+
2. **Subtree** — `git subtree pull` from shared repo into `.github/`
|
|
19
|
+
3. **harness-sync.yml** — extend with `TARGET_REPO` matrix for each product repo
|
|
20
|
+
|
|
21
|
+
## Conflict resolution
|
|
22
|
+
|
|
23
|
+
Product repo local overrides win for repo-specific instructions. Shared repo provides defaults only.
|
|
24
|
+
|
|
25
|
+
## When to use
|
|
26
|
+
|
|
27
|
+
- 3+ product repositories on the same harness template version
|
|
28
|
+
- Identical triager/implementer/reviewer definitions across teams
|
|
29
|
+
|
|
30
|
+
See [adoption.md](adoption.md) for bootstrap and sync procedures.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Telemetry artifacts
|
|
2
|
+
|
|
3
|
+
Machine-readable JSON records emitted by inner-loop workflows for nightly outer-loop aggregation. Span-level OTel export remains optional; these artifacts are the **canonical offline source** when Langfuse wiring is absent.
|
|
4
|
+
|
|
5
|
+
Parent schema fields: [telemetry-schema.md](telemetry-schema.md).
|
|
6
|
+
|
|
7
|
+
## Envelope shape
|
|
8
|
+
|
|
9
|
+
Each file is a single JSON object:
|
|
10
|
+
|
|
11
|
+
| Field | Required | Description |
|
|
12
|
+
|-------|----------|-------------|
|
|
13
|
+
| `schema_version` | yes | Currently `"1"` |
|
|
14
|
+
| `emitted_at` | yes | ISO-8601 timestamp |
|
|
15
|
+
| `source` | yes | Emitting workflow id (see table below) |
|
|
16
|
+
| `workflow` | best-effort | GitHub Actions workflow name |
|
|
17
|
+
| `workflow_run_id` | best-effort | `github.run_id` for correlation |
|
|
18
|
+
| `run_attempt` | best-effort | `github.run_attempt` |
|
|
19
|
+
| `event_name` | best-effort | `github.event_name` |
|
|
20
|
+
| `placeholders` | yes | Payload fields still using sentinel defaults |
|
|
21
|
+
| `payload` | yes | Telemetry fields per [telemetry-schema.md](telemetry-schema.md) |
|
|
22
|
+
|
|
23
|
+
Sample: [infra/samples/telemetry-artifact.json](../infra/samples/telemetry-artifact.json)
|
|
24
|
+
|
|
25
|
+
## Emitting workflows
|
|
26
|
+
|
|
27
|
+
| `source` | Workflow | When |
|
|
28
|
+
|----------|----------|------|
|
|
29
|
+
| `harness-ci` | `.github/workflows/harness-ci.yml` | Every PR after harness jobs complete |
|
|
30
|
+
| `eval-ci` | `.github/workflows/eval-ci.yml` | Pull request eval runs only (scheduled runs skip telemetry) |
|
|
31
|
+
| `agent-retry-orchestrator` | `.github/workflows/agent-retry-orchestrator.yml` | Failed check suite on a linked PR |
|
|
32
|
+
| `pr-context` | `.github/workflows/pr-context-comment.yml` | PR opened / synchronized |
|
|
33
|
+
|
|
34
|
+
Implementation: `node scripts/emit-telemetry-artifact.mjs` (see `scripts/lib/telemetry-artifact.mjs`).
|
|
35
|
+
|
|
36
|
+
## Storage and naming
|
|
37
|
+
|
|
38
|
+
**Runner path:** `telemetry-artifacts/` (repo root during the job).
|
|
39
|
+
|
|
40
|
+
**Filename:** `{source}-pr{number}-run{workflow_run_id}.json` (or `no-pr` when not PR-scoped).
|
|
41
|
+
|
|
42
|
+
**GitHub Actions artifact:** each workflow uploads the directory as `harness-telemetry-{run_id}`, `eval-telemetry-{run_id}`, `retry-telemetry-{run_id}`, or `pr-context-telemetry-{run_id}`.
|
|
43
|
+
|
|
44
|
+
Artifacts are retained per repository retention settings (default 90 days). Nightly aggregation should list workflow runs for the emitters above and download matching artifacts — no PR comment parsing required.
|
|
45
|
+
|
|
46
|
+
## Required vs best-effort payload fields
|
|
47
|
+
|
|
48
|
+
| Field | Inner-loop CI | Notes |
|
|
49
|
+
|-------|---------------|-------|
|
|
50
|
+
| `repo`, `pr_number`, `task_id` | required | `task_id` from linked Issue (`fixes #N`) or `pr-{number}` |
|
|
51
|
+
| `task_class`, `autonomy_level`, `retry_count` | required | From PR labels (`task:*`, `autonomy:*`, `retry:N`) |
|
|
52
|
+
| `changed_files`, `diff_loc` | required on PR workflows | From `git diff` when `BASE_SHA` is set |
|
|
53
|
+
| `wall_failure_type` | required | Empty when green; mapped from failed job names on harness-ci |
|
|
54
|
+
| `final_outcome` | required | `in_progress` until merge/close events wire later |
|
|
55
|
+
| `agent_type`, `execution_mode`, `model` | best-effort | Sentinel `n/a` / `ci` until agent runtime export |
|
|
56
|
+
| `tool_calls`, `cost`, `elapsed_time` | best-effort | Sentinel `-1` until Langfuse / OTel |
|
|
57
|
+
| `review_outcome` | best-effort | `pending` until review webhooks exist |
|
|
58
|
+
|
|
59
|
+
Fields listed in `placeholders` use documented sentinels and are safe for aggregation dashboards to filter.
|
|
60
|
+
|
|
61
|
+
## Validation
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
node scripts/validate-telemetry.mjs "$(cat infra/samples/telemetry-artifact.json)"
|
|
65
|
+
node scripts/emit-telemetry-artifact.mjs # in CI with TELEMETRY_SOURCE set
|
|
66
|
+
node scripts/test-telemetry-artifact-scenarios.mjs
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Set `HARNESS_STRICT_TELEMETRY=1` to fail when `placeholders` is non-empty (intended for post-wiring CI).
|
|
70
|
+
|
|
71
|
+
## Nightly consumption (outline)
|
|
72
|
+
|
|
73
|
+
1. Query Actions API for workflow runs of `Harness CI`, `Eval CI`, `Agent retry orchestrator`, and `PR context comment` in the last 24h.
|
|
74
|
+
2. Download `*-telemetry-*` artifacts from each run.
|
|
75
|
+
3. Parse JSON envelopes; dedupe by `workflow_run_id` + `source` + `payload.pr_number`.
|
|
76
|
+
4. Join rows on `repo`, `task_id`, `pr_number` for KPI rollups ([kpi-baseline.md](kpi-baseline.md)).
|
|
77
|
+
|
|
78
|
+
Classification and harness revision routing are out of scope for the emitters; see [failure-taxonomy.md](failure-taxonomy.md).
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Telemetry Schema
|
|
2
|
+
|
|
3
|
+
Minimum structured fields for agent harness observability (arch.md §5.4). Export via OpenTelemetry to Langfuse or any OTel-compatible backend.
|
|
4
|
+
|
|
5
|
+
## Required fields
|
|
6
|
+
|
|
7
|
+
| Field | Type | Description |
|
|
8
|
+
|-------|------|-------------|
|
|
9
|
+
| `task_id` | string | Unique task identifier (Issue number or UUID) |
|
|
10
|
+
| `pr_number` | integer | Pull request number, if applicable |
|
|
11
|
+
| `repo` | string | `owner/name` |
|
|
12
|
+
| `agent_type` | string | e.g. `implementer`, `triager`, `reviewer` |
|
|
13
|
+
| `execution_mode` | string | `cli`, `ide`, `coding_agent`, `gh_aw`, `sdk` |
|
|
14
|
+
| `model` | string | Model identifier used |
|
|
15
|
+
| `task_class` | string | `docs`, `test-fix`, `refactor`, etc. |
|
|
16
|
+
| `autonomy_level` | string | `L0`–`L3` |
|
|
17
|
+
| `tool_calls` | integer | Count of tool invocations |
|
|
18
|
+
| `retry_count` | integer | Inner-loop retry attempts |
|
|
19
|
+
| `wall_failure_type` | string | `test`, `lint`, `type`, `security`, `safe-output`, `diff-size`, or empty |
|
|
20
|
+
| `cost` | number | AI credits or token cost |
|
|
21
|
+
| `elapsed_time` | number | Seconds |
|
|
22
|
+
| `changed_files` | integer | Files in diff |
|
|
23
|
+
| `diff_loc` | integer | Lines changed (add + delete) |
|
|
24
|
+
| `final_outcome` | string | `merged`, `closed`, `escalated`, `in_progress` |
|
|
25
|
+
| `review_outcome` | string | `approved`, `changes_requested`, `pending` |
|
|
26
|
+
|
|
27
|
+
## KPI mapping
|
|
28
|
+
|
|
29
|
+
| KPI | Fields |
|
|
30
|
+
|-----|--------|
|
|
31
|
+
| PR rejection rate | `review_outcome`, `final_outcome` |
|
|
32
|
+
| First-pass wall rate | `wall_failure_type`, `retry_count` |
|
|
33
|
+
| Cost per task | `cost`, `task_id` |
|
|
34
|
+
| Autonomy distribution | `autonomy_level`, `task_class` |
|
|
35
|
+
| Adoption rate | `review_outcome`, `task_class` |
|
|
36
|
+
|
|
37
|
+
## Inner-loop artifacts
|
|
38
|
+
|
|
39
|
+
Workflows emit JSON artifacts (envelope + `payload`) for nightly aggregation without gh-aw. Storage, naming, emitters, and required vs best-effort fields: [telemetry-artifacts.md](telemetry-artifacts.md).
|
|
40
|
+
|
|
41
|
+
## Validation
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
node scripts/validate-telemetry.mjs "$(cat infra/samples/telemetry-payload.json)"
|
|
45
|
+
node scripts/validate-telemetry.mjs "$(cat infra/samples/telemetry-artifact.json)"
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Collector or `scripts/validate-telemetry.mjs` should reject spans missing required fields when `HARNESS_STRICT_TELEMETRY=1`. With strict mode, non-empty `placeholders` on artifacts also fails validation.
|
|
49
|
+
|
|
50
|
+
## PR context comment placeholders
|
|
51
|
+
|
|
52
|
+
When observability is not fully wired:
|
|
53
|
+
|
|
54
|
+
| Field | Behavior |
|
|
55
|
+
|-------|----------|
|
|
56
|
+
| Trace link | If `LANGFUSE_HOST` is unset, PR comment shows `_configure LANGFUSE_HOST; then search by repo=…, pr_number=…_` |
|
|
57
|
+
| AI credits | Informational only — `_set max-ai-credits in org settings_` until org policy is configured |
|
|
58
|
+
| Threat detection | `n/a` — gh-aw threat detection not active in this template |
|
|
59
|
+
|
|
60
|
+
Workflow display logic in `pr-context-comment.yml` is unchanged; this section documents the spec only.
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# E2E task bench
|
|
2
|
+
|
|
3
|
+
Executable acceptance checks for representative tasks. Each task definition carries
|
|
4
|
+
machine-checkable verifiers (`verification_commands`, `verification_contains`,
|
|
5
|
+
`verification_not_contains`) so the bench measures more than manifest/file presence.
|
|
6
|
+
|
|
7
|
+
This is still lighter than a full break-and-fix agent runner: it validates that task
|
|
8
|
+
fixtures are reproducible and acceptance checks are real. See `manifest.json`.
|
|
9
|
+
|
|
10
|
+
Run weekly via `eval-ci.yml` schedule. Current manifest: **9 tasks** (target 20–100 in a future break-and-fix runner).
|
|
11
|
+
|
|
12
|
+
## Runner boundary (current vs planned)
|
|
13
|
+
|
|
14
|
+
| Concern | Current (`run-e2e-bench.mjs`) | Planned break-and-fix runner |
|
|
15
|
+
|---------|-------------------------------|------------------------------|
|
|
16
|
+
| **Task input** | Static YAML fixture in `tasks/*.yml` | Issue + CC-SD contract + repo snapshot |
|
|
17
|
+
| **Expected artifact** | File content / command exit code | Agent-produced PR diff |
|
|
18
|
+
| **Verifier contract** | `verification_*` fields in task YAML | Same fields + agent execution harness |
|
|
19
|
+
| **Result summary** | Per-task ok/fail; class/stack counts; executed/skipped/failed totals | Above + pass@1, retry count, wall failure class |
|
|
20
|
+
|
|
21
|
+
Validation before run: `scripts/check-e2e-manifest.mjs` (duplicate id, orphan files, unsupported class, `min_tasks`, `last_rotated`).
|
|
22
|
+
|
|
23
|
+
Local:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npm run check-e2e # manifest only
|
|
27
|
+
npm run run-e2e # manifest + executable checks
|
|
28
|
+
```
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
{
|
|
2
|
+
"version": 1,
|
|
3
|
+
"min_tasks": 7,
|
|
4
|
+
"last_rotated": "2026-07-04T00:00:00Z",
|
|
5
|
+
"tasks": [
|
|
6
|
+
{ "id": "e2e-001", "class": "docs", "description": "Validate README heading acceptance", "stack": "any" },
|
|
7
|
+
{ "id": "e2e-002", "class": "test-fix", "description": "Validate sample/ts unit-test acceptance", "stack": "ts" },
|
|
8
|
+
{ "id": "e2e-003", "class": "test-fix", "description": "Validate sample/python unit-test acceptance", "stack": "python" },
|
|
9
|
+
{ "id": "e2e-004", "class": "refactor", "description": "Validate Go API rename without behavior change", "stack": "go" },
|
|
10
|
+
{ "id": "e2e-005", "class": "docs", "description": "Validate docstring presence on public API", "stack": "python" },
|
|
11
|
+
{ "id": "e2e-006", "class": "test-fix", "description": "Validate sample/ruby unit-test acceptance", "stack": "ruby" },
|
|
12
|
+
{ "id": "e2e-007", "class": "test-fix", "description": "Validate sample/php unit-test acceptance", "stack": "php" },
|
|
13
|
+
{ "id": "e2e-008", "class": "test-fix", "description": "Validate CC-SD contract module defines v1 enforced task classes", "stack": "any" },
|
|
14
|
+
{ "id": "e2e-009", "class": "test-fix", "description": "Validate diff-size autonomy limits match operations policy", "stack": "any" }
|
|
15
|
+
]
|
|
16
|
+
}
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-001
|
|
2
|
+
class: docs
|
|
3
|
+
stack: any
|
|
4
|
+
description: Validate README heading acceptance
|
|
5
|
+
acceptance:
|
|
6
|
+
- Heading spelling corrected
|
|
7
|
+
- No code files changed
|
|
8
|
+
verification_contains:
|
|
9
|
+
- README.md::# SDLC-GH
|
|
10
|
+
- README.md::deterministic guardrails for AI coding agents
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
id: e2e-002
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: ts
|
|
4
|
+
description: Validate sample/ts unit-test acceptance
|
|
5
|
+
acceptance:
|
|
6
|
+
- npm test passes in sample/ts
|
|
7
|
+
verification_commands:
|
|
8
|
+
- node>=22@sample/ts::npm test
|
|
9
|
+
verification_contains:
|
|
10
|
+
- sample/ts/src/add.ts::return a + b;
|
|
11
|
+
- sample/ts/tests/add.test.ts::expect(add(2, 3)).toBe(5);
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-003
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: python
|
|
4
|
+
description: Validate sample/python unit-test acceptance
|
|
5
|
+
acceptance:
|
|
6
|
+
- pytest passes in sample/python
|
|
7
|
+
verification_commands:
|
|
8
|
+
- sample/python::python3 -c "from src.greet import greet; assert greet('world') == 'Hello, world!'"
|
|
9
|
+
verification_contains:
|
|
10
|
+
- sample/python/tests/test_greet.py::assert greet("world") == "Hello, world!"
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
id: e2e-004
|
|
2
|
+
class: refactor
|
|
3
|
+
stack: go
|
|
4
|
+
description: Validate Go API rename without behavior change
|
|
5
|
+
acceptance:
|
|
6
|
+
- go test ./... passes
|
|
7
|
+
- Public API uses Sum
|
|
8
|
+
verification_commands:
|
|
9
|
+
- sample/go::env GOCACHE=/private/tmp/sdlc-gh-go-cache go test ./...
|
|
10
|
+
verification_contains:
|
|
11
|
+
- sample/go/add.go::func Sum(a, b int) int {
|
|
12
|
+
- sample/go/add_test.go::if got := Sum(2, 3); got != 5 {
|
|
13
|
+
verification_not_contains:
|
|
14
|
+
- sample/go/add.go::func Add(
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
id: e2e-005
|
|
2
|
+
class: docs
|
|
3
|
+
stack: python
|
|
4
|
+
description: Validate docstring presence on greet() public function
|
|
5
|
+
acceptance:
|
|
6
|
+
- Docstring present
|
|
7
|
+
- Tests still pass
|
|
8
|
+
verification_commands:
|
|
9
|
+
- sample/python::python3 -c "from src.greet import greet; assert greet('world') == 'Hello, world!'"
|
|
10
|
+
verification_contains:
|
|
11
|
+
- sample/python/src/greet.py::"""Return a friendly greeting for the provided name."""
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-006
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: ruby
|
|
4
|
+
description: Validate sample/ruby unit-test acceptance
|
|
5
|
+
acceptance:
|
|
6
|
+
- rspec passes in sample/ruby
|
|
7
|
+
verification_commands:
|
|
8
|
+
- cmd:bundle@sample/ruby::bundle exec rspec
|
|
9
|
+
verification_contains:
|
|
10
|
+
- sample/ruby/spec/add_spec.rb::expect(Add.add(2, 3)).to eq(5)
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-007
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: php
|
|
4
|
+
description: Validate sample/php unit-test acceptance
|
|
5
|
+
acceptance:
|
|
6
|
+
- phpunit passes in sample/php
|
|
7
|
+
verification_commands:
|
|
8
|
+
- cmd:composer@sample/php::composer test
|
|
9
|
+
verification_contains:
|
|
10
|
+
- sample/php/tests/AddTest.php::$this->assertSame(5, Add::add(2, 3));
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-008
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: any
|
|
4
|
+
description: Validate CC-SD contract module defines v1 enforced task classes
|
|
5
|
+
acceptance:
|
|
6
|
+
- CCSD_ENFORCED_TASK_CLASSES exports docs and test-fix only
|
|
7
|
+
verification_contains:
|
|
8
|
+
- scripts/lib/ccsd-contract.mjs::export const CCSD_ENFORCED_TASK_CLASSES
|
|
9
|
+
- scripts/lib/ccsd-contract.mjs::"docs"
|
|
10
|
+
- scripts/lib/ccsd-contract.mjs::"test-fix"
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
id: e2e-009
|
|
2
|
+
class: test-fix
|
|
3
|
+
stack: any
|
|
4
|
+
description: Validate diff-size autonomy limits match operations policy
|
|
5
|
+
acceptance:
|
|
6
|
+
- L2 and L3 limits are codified in diff-size module
|
|
7
|
+
verification_contains:
|
|
8
|
+
- scripts/lib/diff-size.mjs::L2: { loc: 120, files: 4 }
|
|
9
|
+
- scripts/lib/diff-size.mjs::L3: { loc: 60, files: 2 }
|
|
10
|
+
- scripts/lib/diff-size.mjs::L1: { loc: 300, files: 8 }
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
# Rubric for trajectory evaluation (G2)
|
|
2
|
+
|
|
3
|
+
## Good PR
|
|
4
|
+
|
|
5
|
+
- Meets all acceptance criteria
|
|
6
|
+
- Within autonomy size limits
|
|
7
|
+
- Tests adequately constrain behavior
|
|
8
|
+
- Clear rollback path
|
|
9
|
+
|
|
10
|
+
## Scoring
|
|
11
|
+
|
|
12
|
+
Use G-Eval or LLM-as-judge with `EVAL_JUDGE_API_KEY` in CI.
|
|
@@ -0,0 +1,271 @@
|
|
|
1
|
+
"""Harness convention compliance and regression checks."""
|
|
2
|
+
|
|
3
|
+
from pathlib import Path
|
|
4
|
+
import re
|
|
5
|
+
|
|
6
|
+
|
|
7
|
+
ROOT = Path(".")
|
|
8
|
+
|
|
9
|
+
|
|
10
|
+
def read(path: str) -> str:
|
|
11
|
+
return (ROOT / path).read_text()
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
def parse_frontmatter(path: str) -> tuple[dict[str, str], str]:
|
|
15
|
+
text = read(path)
|
|
16
|
+
match = re.match(r"^---\n(.*?)\n---\n(.*)$", text, re.S)
|
|
17
|
+
assert match, f"{path} missing YAML frontmatter"
|
|
18
|
+
frontmatter = {}
|
|
19
|
+
for line in match.group(1).splitlines():
|
|
20
|
+
if ":" in line:
|
|
21
|
+
key, value = line.split(":", 1)
|
|
22
|
+
frontmatter[key.strip()] = value.strip().strip('"')
|
|
23
|
+
return frontmatter, match.group(2)
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
def test_agents_have_frontmatter_and_expected_tools():
|
|
27
|
+
expected = {
|
|
28
|
+
"implementer.agent.md": {"read", "edit", "search", "execute"},
|
|
29
|
+
"reviewer.agent.md": {"read", "search"},
|
|
30
|
+
"triager.agent.md": {"read"},
|
|
31
|
+
}
|
|
32
|
+
for filename, tools in expected.items():
|
|
33
|
+
fm, _ = parse_frontmatter(f".github/agents/{filename}")
|
|
34
|
+
assert fm["name"]
|
|
35
|
+
tool_values = set(re.findall(r'"([^"]+)"', fm["tools"]))
|
|
36
|
+
assert tool_values == tools
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def test_issue_template_requires_acceptance_criteria_and_no_fixed_labels():
|
|
40
|
+
text = read(".github/ISSUE_TEMPLATE/task.yml")
|
|
41
|
+
assert "id: acceptance_criteria" in text
|
|
42
|
+
assert "id: goal" in text
|
|
43
|
+
assert "id: rollback_hints" in text
|
|
44
|
+
assert "type: textarea" in text
|
|
45
|
+
assert re.search(r"id: acceptance_criteria.*?required: true", text, re.S)
|
|
46
|
+
assert "labels:" not in text
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
def test_pr_template_has_harness_context_and_rollback():
|
|
50
|
+
text = read(".github/pull_request_template.md")
|
|
51
|
+
assert "## Harness context" in text
|
|
52
|
+
assert "## Rollback" in text
|
|
53
|
+
assert "## Goal implemented" in text
|
|
54
|
+
assert "Trace link" in text
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
def test_change_size_limits_align_between_docs_and_gate():
|
|
58
|
+
operations = read("docs/operations.md")
|
|
59
|
+
gate = read("scripts/lib/diff-size.mjs")
|
|
60
|
+
agents = read("AGENTS.md")
|
|
61
|
+
copilot = read(".github/copilot-instructions.md")
|
|
62
|
+
|
|
63
|
+
expected = {"L1": ("300", "8"), "L2": ("120", "4"), "L3": ("60", "2")}
|
|
64
|
+
for level, (loc, files) in expected.items():
|
|
65
|
+
assert f"| {level} | {loc} | {files} |" in operations
|
|
66
|
+
assert f"{level}: {{ loc: {loc}, files: {files} }}" in gate
|
|
67
|
+
assert f"- {level}: max {loc} LOC, {files} files" in copilot
|
|
68
|
+
assert "| `docs` | L3 | 60 | 2 |" in agents
|
|
69
|
+
assert "| `test-fix` | L2 | 120 | 4 |" in agents
|
|
70
|
+
assert "| `feature-small` | L1 | 300 | 8 |" in agents
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def test_telemetry_required_fields_align_with_validator():
|
|
74
|
+
schema = read("docs/telemetry-schema.md")
|
|
75
|
+
lib = read("scripts/lib/telemetry-artifact.mjs")
|
|
76
|
+
required = re.findall(r"^\| `([^`]+)` \|", schema, re.M)
|
|
77
|
+
match = re.search(r"export const TELEMETRY_REQUIRED_FIELDS = \[([\s\S]*?)\];", lib)
|
|
78
|
+
assert match, "TELEMETRY_REQUIRED_FIELDS not found in telemetry-artifact.mjs"
|
|
79
|
+
validator_fields = re.findall(r'"([^"]+)"', match.group(1))
|
|
80
|
+
assert set(required) == set(validator_fields)
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
def test_retry_policy_matches_operations_doc():
|
|
84
|
+
operations = read("docs/operations.md")
|
|
85
|
+
orchestrator = read(".github/workflows/agent-retry-orchestrator.yml")
|
|
86
|
+
assert "Max retries `N` | 3" in operations
|
|
87
|
+
assert "const MAX_RETRIES = 3;" in orchestrator
|
|
88
|
+
assert "Same failure signature | Stop after 2 consecutive identical" in operations
|
|
89
|
+
assert "Same failure signature detected twice" in orchestrator
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def test_gh_aw_dogfood_label_and_doc():
|
|
93
|
+
labels = read(".github/labels.yml")
|
|
94
|
+
assert "task:gh-aw-dogfood" in labels
|
|
95
|
+
dogfood = read("docs/gh-aw-dogfood.md")
|
|
96
|
+
assert "task:gh-aw-dogfood" in dogfood
|
|
97
|
+
assert "nightly-harness-review.yml" in dogfood
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def test_harness_review_classes_align_with_failure_taxonomy():
|
|
101
|
+
taxonomy = read("docs/failure-taxonomy.md")
|
|
102
|
+
lib = read("scripts/lib/harness-review.mjs")
|
|
103
|
+
match = re.search(r"export const FAILURE_CLASSES = \[([\s\S]*?)\];", lib)
|
|
104
|
+
assert match, "FAILURE_CLASSES not found in harness-review.mjs"
|
|
105
|
+
classes = re.findall(r'"([^"]+)"', match.group(1))
|
|
106
|
+
for label in ("FF不足", "壁不足", "モデル限界"):
|
|
107
|
+
assert label in taxonomy
|
|
108
|
+
assert label in classes
|
|
109
|
+
assert "unclassified" in classes
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
def test_telemetry_fetch_workflows_align_with_emitters():
|
|
113
|
+
fetch = read("scripts/fetch-telemetry-artifacts.mjs")
|
|
114
|
+
docs = read("docs/telemetry-artifacts.md")
|
|
115
|
+
bootstrap = read("scripts/bootstrap-harness.sh")
|
|
116
|
+
workflows = re.findall(r'workflow: "([^"]+\.yml)"', fetch)
|
|
117
|
+
assert workflows, "TELEMETRY_WORKFLOWS missing in fetch-telemetry-artifacts.mjs"
|
|
118
|
+
for wf in workflows:
|
|
119
|
+
assert (ROOT / ".github/workflows" / wf).is_file(), f"missing emitter workflow {wf}"
|
|
120
|
+
assert wf in bootstrap, f"bootstrap-harness.sh does not copy {wf}"
|
|
121
|
+
assert "harness-telemetry-" in fetch
|
|
122
|
+
assert "eval-telemetry-" in fetch
|
|
123
|
+
assert "retry-telemetry-" in fetch
|
|
124
|
+
assert "pr-context-telemetry-" in fetch
|
|
125
|
+
for source in ("harness-ci", "eval-ci", "agent-retry-orchestrator", "pr-context"):
|
|
126
|
+
assert f"`{source}`" in docs
|
|
127
|
+
|
|
128
|
+
|
|
129
|
+
def test_nightly_harness_review_bootstrap_and_workflow():
|
|
130
|
+
bootstrap = read("scripts/bootstrap-harness.sh")
|
|
131
|
+
assert "nightly-harness-review.yml" in bootstrap
|
|
132
|
+
assert "fetch-telemetry-artifacts.mjs" in bootstrap
|
|
133
|
+
assert "aggregate-harness-review.mjs" in bootstrap
|
|
134
|
+
assert "route-harness-review.mjs" in bootstrap
|
|
135
|
+
assert "harness-review.mjs" in bootstrap
|
|
136
|
+
assert (ROOT / ".github/workflows/nightly-harness-review.yml").is_file()
|
|
137
|
+
nightly = read(".github/workflows/nightly-harness-review.yml")
|
|
138
|
+
assert "fetch-telemetry-artifacts.mjs" in nightly
|
|
139
|
+
assert "aggregate-harness-review.mjs" in nightly
|
|
140
|
+
assert "route-harness-review.mjs" in nightly
|
|
141
|
+
|
|
142
|
+
|
|
143
|
+
def test_outer_loop_routing_labels_defined():
|
|
144
|
+
labels = read(".github/labels.yml")
|
|
145
|
+
routing = read("scripts/lib/harness-review-routing.mjs")
|
|
146
|
+
assert "outer-loop:harness-revision" in labels
|
|
147
|
+
assert "outer-loop:wall-addition" in labels
|
|
148
|
+
assert "outer-loop:harness-revision" in routing
|
|
149
|
+
assert "outer-loop:wall-addition" in routing
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def test_template_codeowners_keeps_placeholder():
|
|
153
|
+
codeowners = read(".github/CODEOWNERS")
|
|
154
|
+
validate = read("scripts/validate-harness.mjs")
|
|
155
|
+
assert "@your-org/harness-engineers" in codeowners
|
|
156
|
+
assert "detectRepoProfile" in validate
|
|
157
|
+
assert "CODEOWNERS_PLACEHOLDER" in validate
|
|
158
|
+
|
|
159
|
+
|
|
160
|
+
def test_gh_aw_sources_include_required_sections():
|
|
161
|
+
nightly = read(".github/workflows/nightly-harness-review.md")
|
|
162
|
+
weekly = read(".github/workflows/weekly-redteam.md")
|
|
163
|
+
for section in (
|
|
164
|
+
"## Required inputs",
|
|
165
|
+
"## Forbidden operations",
|
|
166
|
+
"## Expected outputs",
|
|
167
|
+
"## Promotion criteria",
|
|
168
|
+
):
|
|
169
|
+
assert section in nightly
|
|
170
|
+
assert section in weekly
|
|
171
|
+
assert "GH_AW_SOURCE_REQUIRED_SECTIONS" in read("scripts/lib/gh-aw-dogfood.mjs")
|
|
172
|
+
|
|
173
|
+
|
|
174
|
+
def _parse_ccsd_exports() -> tuple[list[str], list[str], list[str]]:
|
|
175
|
+
"""Read canonical CC-SD field names from scripts/lib/ccsd-contract.mjs."""
|
|
176
|
+
contract = read("scripts/lib/ccsd-contract.mjs")
|
|
177
|
+
required = re.findall(
|
|
178
|
+
r'export const CCSD_REQUIRED_FIELDS = \[\s*([\s\S]*?)\s*\];',
|
|
179
|
+
contract,
|
|
180
|
+
)[0]
|
|
181
|
+
optional = re.findall(
|
|
182
|
+
r'export const CCSD_OPTIONAL_FIELDS = \[\s*([\s\S]*?)\s*\];',
|
|
183
|
+
contract,
|
|
184
|
+
)[0]
|
|
185
|
+
pr_fields = re.findall(
|
|
186
|
+
r'export const CCSD_PR_SUMMARY_FIELDS = \[\s*([\s\S]*?)\s*\];',
|
|
187
|
+
contract,
|
|
188
|
+
)[0]
|
|
189
|
+
|
|
190
|
+
def names(block: str) -> list[str]:
|
|
191
|
+
return re.findall(r'"([^"]+)"', block)
|
|
192
|
+
|
|
193
|
+
return names(required), names(optional), names(pr_fields)
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
def test_task_template_contains_canonical_ccsd_fields():
|
|
197
|
+
required, optional, _ = _parse_ccsd_exports()
|
|
198
|
+
text = read(".github/ISSUE_TEMPLATE/task.yml")
|
|
199
|
+
|
|
200
|
+
for field in required:
|
|
201
|
+
assert f"label: {field}" in text, f"task.yml missing required field {field}"
|
|
202
|
+
|
|
203
|
+
for field in optional:
|
|
204
|
+
assert f"label: {field}" in text, f"task.yml missing optional field {field}"
|
|
205
|
+
|
|
206
|
+
assert "labels:" not in text
|
|
207
|
+
|
|
208
|
+
|
|
209
|
+
def test_task_template_placeholders_are_detected_by_validator():
|
|
210
|
+
contract = read("scripts/lib/ccsd-contract.mjs")
|
|
211
|
+
template = read(".github/ISSUE_TEMPLATE/task.yml")
|
|
212
|
+
snippets = re.findall(r'"([^"]+)"', contract.split("CCSD_PLACEHOLDER_SNIPPETS", 1)[1].split("];", 1)[0])
|
|
213
|
+
for snippet in snippets:
|
|
214
|
+
assert snippet in template or snippet in contract
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
def test_agents_and_quality_loop_reference_canonical_ccsd_fields():
|
|
218
|
+
required, _, _ = _parse_ccsd_exports()
|
|
219
|
+
paths = [
|
|
220
|
+
".github/agents/triager.agent.md",
|
|
221
|
+
".github/agents/implementer.agent.md",
|
|
222
|
+
".github/agents/reviewer.agent.md",
|
|
223
|
+
".github/skills/quality-loop/SKILL.md",
|
|
224
|
+
"AGENTS.md",
|
|
225
|
+
".github/copilot-instructions.md",
|
|
226
|
+
]
|
|
227
|
+
for path in paths:
|
|
228
|
+
text = read(path)
|
|
229
|
+
for field in required:
|
|
230
|
+
assert field in text, f"{path} missing canonical field {field}"
|
|
231
|
+
|
|
232
|
+
|
|
233
|
+
def test_pr_template_contains_ccsd_summary_fields():
|
|
234
|
+
_, _, pr_fields = _parse_ccsd_exports()
|
|
235
|
+
text = read(".github/pull_request_template.md")
|
|
236
|
+
for field in pr_fields:
|
|
237
|
+
assert f"## {field}" in text, f"PR template missing section {field}"
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
def test_coding_agent_l1_requires_ccsd_for_l1_docs_test_fix():
|
|
241
|
+
text = read("docs/coding-agent-l1.md")
|
|
242
|
+
assert "CC-SD" in text
|
|
243
|
+
assert "`task:docs`" in text
|
|
244
|
+
assert "`task:test-fix`" in text
|
|
245
|
+
assert "`autonomy:L1`" in text
|
|
246
|
+
assert "issue-spec-check" in text
|
|
247
|
+
|
|
248
|
+
|
|
249
|
+
def test_arch_documents_ccsd_contract():
|
|
250
|
+
text = read("docs/arch.md")
|
|
251
|
+
assert "CC-SD" in text
|
|
252
|
+
assert "ccsd-contract.mjs" in text
|
|
253
|
+
assert "issue-spec-check" in text
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
def test_adoption_describes_ccsd_as_l1_only_v1():
|
|
257
|
+
text = read("docs/adoption.md")
|
|
258
|
+
assert "CC-SD" in text
|
|
259
|
+
assert "v1" in text
|
|
260
|
+
assert "`task:docs`" in text
|
|
261
|
+
assert "`task:test-fix`" in text
|
|
262
|
+
assert "feature-small" in text
|
|
263
|
+
|
|
264
|
+
|
|
265
|
+
def test_validation_script_field_list_matches_template():
|
|
266
|
+
required, optional, _ = _parse_ccsd_exports()
|
|
267
|
+
template = read(".github/ISSUE_TEMPLATE/task.yml")
|
|
268
|
+
for field in required + optional:
|
|
269
|
+
assert f"label: {field}" in template
|
|
270
|
+
assert "CCSD_REQUIRED_FIELDS" in read("scripts/lib/ccsd-contract.mjs")
|
|
271
|
+
assert "check-issue-spec.mjs" in read(".github/workflows/harness-ci.yml")
|
package/infra/README.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Observability infrastructure
|
|
2
|
+
|
|
3
|
+
Scaffold only — no production wiring is included in the template. See [docs/telemetry-schema.md](../docs/telemetry-schema.md) for required span fields.
|
|
4
|
+
|
|
5
|
+
## Langfuse (self-hosted)
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
cd infra/langfuse
|
|
9
|
+
docker compose up -d
|
|
10
|
+
# UI: http://localhost:3000
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Change `NEXTAUTH_SECRET` and `SALT` before production use.
|
|
14
|
+
|
|
15
|
+
## OpenTelemetry collector
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
docker run -p 4317:4317 -p 4318:4318 \
|
|
19
|
+
-v "$(pwd)/otel/collector-config.yml:/etc/otelcol/config.yaml" \
|
|
20
|
+
otel/opentelemetry-collector:latest
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## Connect harness telemetry
|
|
24
|
+
|
|
25
|
+
1. Export spans with required fields per [docs/telemetry-schema.md](../docs/telemetry-schema.md), or consume inner-loop JSON artifacts from [docs/telemetry-artifacts.md](../docs/telemetry-artifacts.md).
|
|
26
|
+
2. Point exporters at collector `:4317` (gRPC) or `:4318` (HTTP).
|
|
27
|
+
3. Uncomment Langfuse OTLP exporter in `otel/collector-config.yml` when ready.
|
|
28
|
+
4. Validate payloads:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
node scripts/validate-telemetry.mjs "$(cat infra/samples/telemetry-payload.json)"
|
|
32
|
+
node scripts/validate-telemetry.mjs "$(cat infra/samples/telemetry-artifact.json)"
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## Environment variables (CI / local)
|
|
36
|
+
|
|
37
|
+
| Variable | Purpose |
|
|
38
|
+
|----------|---------|
|
|
39
|
+
| `LANGFUSE_HOST` | Base URL for trace deep links in PR comments. When unset, PR context shows a configure placeholder (see telemetry-schema.md) |
|
|
40
|
+
| `LANGFUSE_PUBLIC_KEY` | Optional export auth |
|
|
41
|
+
| `LANGFUSE_SECRET_KEY` | Optional export auth |
|
|
42
|
+
|
|
43
|
+
## PR context comment (informational fields)
|
|
44
|
+
|
|
45
|
+
| Display | Spec |
|
|
46
|
+
|---------|------|
|
|
47
|
+
| Trace | Langfuse search hint when `LANGFUSE_HOST` set; otherwise placeholder text |
|
|
48
|
+
| AI credits | Informational — org `max-ai-credits` not exposed to workflow |
|
|
49
|
+
| Threat detection | `n/a` until gh-aw outer loop is promoted beyond stub |
|