@next-friday/create-app 4.0.0 → 5.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/templates/base/SETUP.md +8 -13
- package/templates/nestjs/.github/workflows/ci.yml +0 -44
- package/templates/nestjs/Dockerfile +14 -4
- package/templates/nextjs/.github/workflows/ci.yml +0 -44
- package/templates/nextjs/Dockerfile +14 -4
- package/templates/nodejs/.github/workflows/ci.yml +0 -44
- package/templates/nodejs/Dockerfile +14 -4
- package/templates/payload/.github/workflows/ci.yml +0 -44
- package/templates/payload/Dockerfile +14 -4
- package/templates/turborepo/.github/workflows/ci.yml +0 -44
- package/templates/turborepo/apps/app/Dockerfile +14 -4
- package/templates/turborepo/apps/backoffice/Dockerfile +14 -4
- package/templates/base/.github/workflows/agent-evals.yml +0 -128
- package/templates/base/.github/workflows/claude.yml +0 -39
- package/templates/base/evals/README.md +0 -15
- package/templates/base/evals/check.sh +0 -17
- package/templates/base/lessons/README.md +0 -3
package/package.json
CHANGED
package/templates/base/SETUP.md
CHANGED
|
@@ -7,10 +7,10 @@ This project follows Anthropic's [AI-native SDLC Playbook](https://claude.com/re
|
|
|
7
7
|
1. **Plan:** an idea, a ticket, or an incident becomes `intent/<change>/intent.md` (`write-intent` skill). The product owner merges or closes it.
|
|
8
8
|
2. **Design:** `spec.md` is written from the accepted intent with the policy skills loaded (`write-spec` skill). The product owner resolves concerns with the policy owners.
|
|
9
9
|
3. **Build:** the engineer starts in plan mode, approves the plan, and commits it as `plan.md`. A tech lead approves higher-risk plans.
|
|
10
|
-
4. **Test:** the session runs **Verifying your work** in `AGENTS.md
|
|
11
|
-
5. **Review:** Claude Code Review and Bugbot review by `REVIEW.md
|
|
12
|
-
6. **Deploy:**
|
|
13
|
-
7. **Maintain:**
|
|
10
|
+
4. **Test:** the session runs **Verifying your work** in `AGENTS.md`.
|
|
11
|
+
5. **Review:** Claude Code Review and Bugbot review by `REVIEW.md`. A code owner approves; the agent never does.
|
|
12
|
+
6. **Deploy:** `agent-production-gate.sh` gates production.
|
|
13
|
+
7. **Maintain:** larger work becomes a new `intent.md`.
|
|
14
14
|
|
|
15
15
|
## 1. First commit and push
|
|
16
16
|
|
|
@@ -55,30 +55,26 @@ Also replace the owner placeholder in `.github/zizmor.yml`.
|
|
|
55
55
|
|
|
56
56
|
- **Actions → General → Workflow permissions:** select **Read repository contents and packages permissions**. Every workflow requests the extra permissions it needs.
|
|
57
57
|
- **Advanced Security:** enable private vulnerability reporting, Dependabot alerts, Dependabot security updates, secret scanning, and push protection. `SECURITY.md` sends reporters to private vulnerability reporting.
|
|
58
|
-
- **Secrets and variables → Actions:**
|
|
59
|
-
- the variables `ANTHROPIC_FEDERATION_RULE_ID` and `ANTHROPIC_ORGANIZATION_ID`, from step 4. They are identifiers, not secrets: `.github/workflows/claude.yml`, `.github/workflows/agent-evals.yml`, and the **Triage failed build** job in `ci.yml` exchange the workflow's OIDC token for short-lived Claude API access, so the repository stores no Claude token.
|
|
60
|
-
- the variable `EVALS_MIN_PASS_RATE`, from 0 to 100, once the team has decided it (step 6).
|
|
61
58
|
- **Rules → Rulesets:** protect `main` with:
|
|
62
59
|
- squash merges only, a linear history, and no force pushes or deletion;
|
|
63
|
-
- the required checks `All Checks
|
|
60
|
+
- the required checks `All Checks` and `Validate pull request title`;
|
|
64
61
|
- a pull request before merging, with at least one approval, review from Code Owners, and stale approvals dismissed when new commits are pushed. Claude Code Review follows the severity, exclusion, and nit-cap rules in `REVIEW.md` only on a default branch protected this way ([customize Code Review](https://code.claude.com/docs/en/code-review-customize)).
|
|
65
62
|
|
|
66
63
|
Rulesets in a private repository need GitHub Pro, Team, or Enterprise; on GitHub Free, make the repository public or upgrade.
|
|
67
64
|
|
|
68
65
|
- Create the `needs triage` label, which the issue forms apply: `gh label create "needs triage"`.
|
|
69
66
|
|
|
70
|
-
**Check:** a test pull request shows the
|
|
67
|
+
**Check:** a test pull request shows the two required checks and asks for a code owner's review.
|
|
71
68
|
|
|
72
69
|
## 4. Claude
|
|
73
70
|
|
|
74
71
|
**Who:** an Owner of the Claude organization, with an owner of the GitHub organization.
|
|
75
72
|
|
|
76
|
-
- **Code Review:** in [**Admin settings → Claude Code**](https://claude.ai/admin-settings/claude-code#code-review), turn on **Allow code review**, install the Claude GitHub App, and add this repository ([set up Code Review](https://code.claude.com/docs/en/code-review-setup)). It needs a Team or Enterprise plan ([availability](https://code.claude.com/docs/en/code-review#availability)). Keep **Review runs** at **After every push**, the default, so every push to a pull request gets a review. Reviews run on Anthropic's infrastructure, so the repository has no review workflow.
|
|
77
|
-
- **Workload identity federation:** in the [Claude Console](https://platform.claude.com), under **Settings → Workload identity**, register the issuer `https://token.actions.githubusercontent.com`, create a service account in the workspace the workflows should use, and create a federation rule for it that matches this repository's subject prefix, `repo:OWNER/REPOSITORY:` ([setup](https://github.com/anthropics/claude-code-action/blob/main/docs/setup.md), [workload identity federation](https://platform.claude.com/docs/en/manage-claude/workload-identity-federation)). Put the rule's `fdrl_...` ID and the organization ID in the repository variables of step 3. Usage is billed to the Console organization.
|
|
73
|
+
- **Code Review:** in [**Admin settings → Claude Code**](https://claude.ai/admin-settings/claude-code#code-review), turn on **Allow code review**, install the Claude GitHub App, and add this repository ([set up Code Review](https://code.claude.com/docs/en/code-review-setup)). It needs a Team or Enterprise plan ([availability](https://code.claude.com/docs/en/code-review#availability)). Keep **Review runs** at **After every push**, the default, so every push to a pull request gets a review. Reviews run on Anthropic's infrastructure, so the repository has no review workflow.
|
|
78
74
|
- **Managed settings:** deploy the non-negotiable settings and hooks as [managed settings](https://code.claude.com/docs/en/settings), tailored from the playbook's worked example in `.claude/managed-settings.example.json`. Engineers cannot override them.
|
|
79
75
|
- **Monitoring:** turn on the [OpenTelemetry export](https://code.claude.com/docs/en/monitoring-usage) that the playbook's indicators read.
|
|
80
76
|
|
|
81
|
-
**Check:** the test pull request gets a **Claude Code Review** check run within a few minutes.
|
|
77
|
+
**Check:** the test pull request gets a **Claude Code Review** check run within a few minutes.
|
|
82
78
|
|
|
83
79
|
## 5. Cursor
|
|
84
80
|
|
|
@@ -98,7 +94,6 @@ Also replace the owner placeholder in `.github/zizmor.yml`.
|
|
|
98
94
|
- **Policies (each policy owner, with an engineer):** write the policy in `.agents/skills/<area>-policy/SKILL.md` from the owner's source of truth, then run `pnpm rules:sync`. Test that the skill triggers: ask for the relevant task in different ways and confirm the skill loads each time. The policy owner approves every later change as its code owner.
|
|
99
95
|
- **Production gate (engineering leadership, with change management and compliance):** list the approvals that must survive. A person then changes `.agents/hooks/agent-production-gate.sh` to match this project's production deploy command and approval; agent settings deny the file to agents.
|
|
100
96
|
- **Review (tech lead):** replace `src/gen/` in `.agents/review.md` with this project's generated paths, then run `pnpm rules:sync`, which renders it to `REVIEW.md` for Claude Code Review. Once a month, rate the review findings and tune `.agents/review.md`.
|
|
101
|
-
- **Evals (platform engineer):** collect 20 to 50 real tasks in `evals/` (`evals/README.md`), and set `EVALS_MIN_PASS_RATE` (step 3).
|
|
102
97
|
- **People without Git experience:** connect GitHub to Claude (claude.ai or Cowork), so Claude commits `intent.md` for them.
|
|
103
98
|
|
|
104
99
|
## 7. Later, as the guardrails mature
|
|
@@ -166,50 +166,6 @@ jobs:
|
|
|
166
166
|
environment: dev
|
|
167
167
|
commit_sha: ${{ github.sha }}
|
|
168
168
|
|
|
169
|
-
triage:
|
|
170
|
-
name: Triage failed build
|
|
171
|
-
needs:
|
|
172
|
-
- checks
|
|
173
|
-
- image
|
|
174
|
-
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
|
|
175
|
-
runs-on: ubuntu-latest
|
|
176
|
-
timeout-minutes: 15
|
|
177
|
-
permissions:
|
|
178
|
-
actions: read # Needed to read the logs of the failed jobs.
|
|
179
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
180
|
-
pull-requests: write # Needed to post the summary on the pull request.
|
|
181
|
-
steps:
|
|
182
|
-
- name: Collect the build log
|
|
183
|
-
env:
|
|
184
|
-
GH_TOKEN: ${{ github.token }}
|
|
185
|
-
REPOSITORY: ${{ github.repository }}
|
|
186
|
-
RUN_ID: ${{ github.run_id }}
|
|
187
|
-
run: |
|
|
188
|
-
mkdir -p out
|
|
189
|
-
for job in $(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select(.conclusion == "failure") | .id'); do
|
|
190
|
-
gh api "repos/$REPOSITORY/actions/jobs/$job/logs" >> out/build.log
|
|
191
|
-
done
|
|
192
|
-
- name: Triage failed build
|
|
193
|
-
id: triage
|
|
194
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
195
|
-
with:
|
|
196
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
197
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
198
|
-
prompt: >
|
|
199
|
-
Read the build log at out/build.log. Identify the most
|
|
200
|
-
likely cause, say whether the failure looks flaky or real, and write a
|
|
201
|
-
three-line summary for the PR thread.
|
|
202
|
-
claude_args: >-
|
|
203
|
-
--allowedTools "Read"
|
|
204
|
-
--json-schema '{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"]}'
|
|
205
|
-
- name: Post the summary
|
|
206
|
-
env:
|
|
207
|
-
GH_TOKEN: ${{ github.token }}
|
|
208
|
-
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
209
|
-
REPOSITORY: ${{ github.repository }}
|
|
210
|
-
SUMMARY: ${{ fromJSON(steps.triage.outputs.structured_output).summary }}
|
|
211
|
-
run: gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "$SUMMARY"
|
|
212
|
-
|
|
213
169
|
ci:
|
|
214
170
|
name: All Checks
|
|
215
171
|
if: always()
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Vercel Container Runtime, Fly.io, Cloud Run). The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS base
|
|
3
5
|
WORKDIR /app
|
|
4
6
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
5
|
-
RUN
|
|
7
|
+
RUN corepack enable
|
|
6
8
|
|
|
7
9
|
FROM base AS build
|
|
8
10
|
RUN pnpm install --frozen-lockfile --ignore-scripts
|
|
@@ -13,12 +15,20 @@ RUN pnpm build
|
|
|
13
15
|
FROM base AS prod-deps
|
|
14
16
|
RUN pnpm install --frozen-lockfile --ignore-scripts --prod
|
|
15
17
|
|
|
16
|
-
FROM
|
|
18
|
+
FROM \
|
|
19
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
17
20
|
ENV NODE_ENV=production
|
|
18
21
|
WORKDIR /app
|
|
19
22
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
20
23
|
RUN apk upgrade --no-cache \
|
|
21
|
-
&& rm -rf
|
|
24
|
+
&& rm -rf \
|
|
25
|
+
/usr/local/lib/node_modules \
|
|
26
|
+
/usr/local/bin/npm \
|
|
27
|
+
/usr/local/bin/npx \
|
|
28
|
+
/usr/local/bin/corepack \
|
|
29
|
+
/usr/local/bin/yarn \
|
|
30
|
+
/usr/local/bin/yarnpkg \
|
|
31
|
+
/opt/yarn-*
|
|
22
32
|
COPY package.json ./
|
|
23
33
|
COPY --from=prod-deps /app/node_modules ./node_modules
|
|
24
34
|
COPY --from=build /app/dist ./dist
|
|
@@ -172,50 +172,6 @@ jobs:
|
|
|
172
172
|
environment: dev
|
|
173
173
|
commit_sha: ${{ github.sha }}
|
|
174
174
|
|
|
175
|
-
triage:
|
|
176
|
-
name: Triage failed build
|
|
177
|
-
needs:
|
|
178
|
-
- checks
|
|
179
|
-
- image
|
|
180
|
-
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
|
|
181
|
-
runs-on: ubuntu-latest
|
|
182
|
-
timeout-minutes: 15
|
|
183
|
-
permissions:
|
|
184
|
-
actions: read # Needed to read the logs of the failed jobs.
|
|
185
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
186
|
-
pull-requests: write # Needed to post the summary on the pull request.
|
|
187
|
-
steps:
|
|
188
|
-
- name: Collect the build log
|
|
189
|
-
env:
|
|
190
|
-
GH_TOKEN: ${{ github.token }}
|
|
191
|
-
REPOSITORY: ${{ github.repository }}
|
|
192
|
-
RUN_ID: ${{ github.run_id }}
|
|
193
|
-
run: |
|
|
194
|
-
mkdir -p out
|
|
195
|
-
for job in $(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select(.conclusion == "failure") | .id'); do
|
|
196
|
-
gh api "repos/$REPOSITORY/actions/jobs/$job/logs" >> out/build.log
|
|
197
|
-
done
|
|
198
|
-
- name: Triage failed build
|
|
199
|
-
id: triage
|
|
200
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
201
|
-
with:
|
|
202
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
203
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
204
|
-
prompt: >
|
|
205
|
-
Read the build log at out/build.log. Identify the most
|
|
206
|
-
likely cause, say whether the failure looks flaky or real, and write a
|
|
207
|
-
three-line summary for the PR thread.
|
|
208
|
-
claude_args: >-
|
|
209
|
-
--allowedTools "Read"
|
|
210
|
-
--json-schema '{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"]}'
|
|
211
|
-
- name: Post the summary
|
|
212
|
-
env:
|
|
213
|
-
GH_TOKEN: ${{ github.token }}
|
|
214
|
-
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
215
|
-
REPOSITORY: ${{ github.repository }}
|
|
216
|
-
SUMMARY: ${{ fromJSON(steps.triage.outputs.structured_output).summary }}
|
|
217
|
-
run: gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "$SUMMARY"
|
|
218
|
-
|
|
219
175
|
ci:
|
|
220
176
|
name: All Checks
|
|
221
177
|
if: always()
|
|
@@ -1,20 +1,30 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Fly.io, Cloud Run). On Vercel, deploy without it. The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS build
|
|
3
5
|
ENV NEXT_TELEMETRY_DISABLED=1
|
|
4
6
|
WORKDIR /app
|
|
5
7
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
6
|
-
RUN
|
|
8
|
+
RUN corepack enable
|
|
7
9
|
RUN pnpm install --frozen-lockfile --ignore-scripts
|
|
8
10
|
COPY next.config.ts postcss.config.mjs tsconfig.json ./
|
|
9
11
|
COPY app ./app
|
|
10
12
|
RUN pnpm build
|
|
11
13
|
|
|
12
|
-
FROM
|
|
14
|
+
FROM \
|
|
15
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
13
16
|
ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 HOSTNAME=0.0.0.0
|
|
14
17
|
WORKDIR /app
|
|
15
18
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
16
19
|
RUN apk upgrade --no-cache \
|
|
17
|
-
&& rm -rf
|
|
20
|
+
&& rm -rf \
|
|
21
|
+
/usr/local/lib/node_modules \
|
|
22
|
+
/usr/local/bin/npm \
|
|
23
|
+
/usr/local/bin/npx \
|
|
24
|
+
/usr/local/bin/corepack \
|
|
25
|
+
/usr/local/bin/yarn \
|
|
26
|
+
/usr/local/bin/yarnpkg \
|
|
27
|
+
/opt/yarn-*
|
|
18
28
|
# Next.js writes .next/cache at runtime. Add `COPY --from=build --chown=node:node /app/public ./public` when the app has a public/ folder, and allow it in .dockerignore.
|
|
19
29
|
COPY --from=build --chown=node:node /app/.next/standalone ./
|
|
20
30
|
COPY --from=build --chown=node:node /app/.next/static ./.next/static
|
|
@@ -164,50 +164,6 @@ jobs:
|
|
|
164
164
|
environment: dev
|
|
165
165
|
commit_sha: ${{ github.sha }}
|
|
166
166
|
|
|
167
|
-
triage:
|
|
168
|
-
name: Triage failed build
|
|
169
|
-
needs:
|
|
170
|
-
- checks
|
|
171
|
-
- image
|
|
172
|
-
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
|
|
173
|
-
runs-on: ubuntu-latest
|
|
174
|
-
timeout-minutes: 15
|
|
175
|
-
permissions:
|
|
176
|
-
actions: read # Needed to read the logs of the failed jobs.
|
|
177
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
178
|
-
pull-requests: write # Needed to post the summary on the pull request.
|
|
179
|
-
steps:
|
|
180
|
-
- name: Collect the build log
|
|
181
|
-
env:
|
|
182
|
-
GH_TOKEN: ${{ github.token }}
|
|
183
|
-
REPOSITORY: ${{ github.repository }}
|
|
184
|
-
RUN_ID: ${{ github.run_id }}
|
|
185
|
-
run: |
|
|
186
|
-
mkdir -p out
|
|
187
|
-
for job in $(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select(.conclusion == "failure") | .id'); do
|
|
188
|
-
gh api "repos/$REPOSITORY/actions/jobs/$job/logs" >> out/build.log
|
|
189
|
-
done
|
|
190
|
-
- name: Triage failed build
|
|
191
|
-
id: triage
|
|
192
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
193
|
-
with:
|
|
194
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
195
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
196
|
-
prompt: >
|
|
197
|
-
Read the build log at out/build.log. Identify the most
|
|
198
|
-
likely cause, say whether the failure looks flaky or real, and write a
|
|
199
|
-
three-line summary for the PR thread.
|
|
200
|
-
claude_args: >-
|
|
201
|
-
--allowedTools "Read"
|
|
202
|
-
--json-schema '{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"]}'
|
|
203
|
-
- name: Post the summary
|
|
204
|
-
env:
|
|
205
|
-
GH_TOKEN: ${{ github.token }}
|
|
206
|
-
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
207
|
-
REPOSITORY: ${{ github.repository }}
|
|
208
|
-
SUMMARY: ${{ fromJSON(steps.triage.outputs.structured_output).summary }}
|
|
209
|
-
run: gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "$SUMMARY"
|
|
210
|
-
|
|
211
167
|
ci:
|
|
212
168
|
name: All Checks
|
|
213
169
|
if: always()
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Fly.io, Cloud Run). The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS base
|
|
3
5
|
WORKDIR /app
|
|
4
6
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
5
|
-
RUN
|
|
7
|
+
RUN corepack enable
|
|
6
8
|
|
|
7
9
|
FROM base AS build
|
|
8
10
|
RUN pnpm install --frozen-lockfile --ignore-scripts
|
|
@@ -13,12 +15,20 @@ RUN pnpm build
|
|
|
13
15
|
FROM base AS prod-deps
|
|
14
16
|
RUN pnpm install --frozen-lockfile --ignore-scripts --prod
|
|
15
17
|
|
|
16
|
-
FROM
|
|
18
|
+
FROM \
|
|
19
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
17
20
|
ENV NODE_ENV=production
|
|
18
21
|
WORKDIR /app
|
|
19
22
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
20
23
|
RUN apk upgrade --no-cache \
|
|
21
|
-
&& rm -rf
|
|
24
|
+
&& rm -rf \
|
|
25
|
+
/usr/local/lib/node_modules \
|
|
26
|
+
/usr/local/bin/npm \
|
|
27
|
+
/usr/local/bin/npx \
|
|
28
|
+
/usr/local/bin/corepack \
|
|
29
|
+
/usr/local/bin/yarn \
|
|
30
|
+
/usr/local/bin/yarnpkg \
|
|
31
|
+
/opt/yarn-*
|
|
22
32
|
COPY package.json ./
|
|
23
33
|
COPY --from=prod-deps /app/node_modules ./node_modules
|
|
24
34
|
COPY --from=build /app/dist ./dist
|
|
@@ -189,50 +189,6 @@ jobs:
|
|
|
189
189
|
environment: dev
|
|
190
190
|
commit_sha: ${{ github.sha }}
|
|
191
191
|
|
|
192
|
-
triage:
|
|
193
|
-
name: Triage failed build
|
|
194
|
-
needs:
|
|
195
|
-
- checks
|
|
196
|
-
- image
|
|
197
|
-
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
|
|
198
|
-
runs-on: ubuntu-latest
|
|
199
|
-
timeout-minutes: 15
|
|
200
|
-
permissions:
|
|
201
|
-
actions: read # Needed to read the logs of the failed jobs.
|
|
202
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
203
|
-
pull-requests: write # Needed to post the summary on the pull request.
|
|
204
|
-
steps:
|
|
205
|
-
- name: Collect the build log
|
|
206
|
-
env:
|
|
207
|
-
GH_TOKEN: ${{ github.token }}
|
|
208
|
-
REPOSITORY: ${{ github.repository }}
|
|
209
|
-
RUN_ID: ${{ github.run_id }}
|
|
210
|
-
run: |
|
|
211
|
-
mkdir -p out
|
|
212
|
-
for job in $(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select(.conclusion == "failure") | .id'); do
|
|
213
|
-
gh api "repos/$REPOSITORY/actions/jobs/$job/logs" >> out/build.log
|
|
214
|
-
done
|
|
215
|
-
- name: Triage failed build
|
|
216
|
-
id: triage
|
|
217
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
218
|
-
with:
|
|
219
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
220
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
221
|
-
prompt: >
|
|
222
|
-
Read the build log at out/build.log. Identify the most
|
|
223
|
-
likely cause, say whether the failure looks flaky or real, and write a
|
|
224
|
-
three-line summary for the PR thread.
|
|
225
|
-
claude_args: >-
|
|
226
|
-
--allowedTools "Read"
|
|
227
|
-
--json-schema '{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"]}'
|
|
228
|
-
- name: Post the summary
|
|
229
|
-
env:
|
|
230
|
-
GH_TOKEN: ${{ github.token }}
|
|
231
|
-
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
232
|
-
REPOSITORY: ${{ github.repository }}
|
|
233
|
-
SUMMARY: ${{ fromJSON(steps.triage.outputs.structured_output).summary }}
|
|
234
|
-
run: gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "$SUMMARY"
|
|
235
|
-
|
|
236
192
|
ci:
|
|
237
193
|
name: All Checks
|
|
238
194
|
if: always()
|
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Fly.io, Cloud Run). On Vercel, deploy without it. The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS build
|
|
3
5
|
ENV NEXT_TELEMETRY_DISABLED=1
|
|
4
6
|
WORKDIR /app
|
|
5
7
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
6
|
-
RUN
|
|
8
|
+
RUN corepack enable
|
|
7
9
|
RUN pnpm install --frozen-lockfile --ignore-scripts
|
|
8
10
|
COPY next.config.ts tsconfig.json ./
|
|
9
11
|
COPY src ./src
|
|
@@ -11,12 +13,20 @@ COPY src ./src
|
|
|
11
13
|
# the platform sets the real ones at runtime, and the build never connects to the database.
|
|
12
14
|
RUN PAYLOAD_SECRET=build-only DATABASE_URL=postgres://build-only@localhost/build-only APP_URL=http://localhost:3000 pnpm build
|
|
13
15
|
|
|
14
|
-
FROM
|
|
16
|
+
FROM \
|
|
17
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
15
18
|
ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 HOSTNAME=0.0.0.0
|
|
16
19
|
WORKDIR /app
|
|
17
20
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
18
21
|
RUN apk upgrade --no-cache \
|
|
19
|
-
&& rm -rf
|
|
22
|
+
&& rm -rf \
|
|
23
|
+
/usr/local/lib/node_modules \
|
|
24
|
+
/usr/local/bin/npm \
|
|
25
|
+
/usr/local/bin/npx \
|
|
26
|
+
/usr/local/bin/corepack \
|
|
27
|
+
/usr/local/bin/yarn \
|
|
28
|
+
/usr/local/bin/yarnpkg \
|
|
29
|
+
/opt/yarn-* \
|
|
20
30
|
&& install -d -o node -g node media
|
|
21
31
|
# Next.js writes .next/cache at runtime. Add `COPY --from=build --chown=node:node /app/public ./public` when the app has a public/ folder, and allow it in .dockerignore. Uploads go to the container filesystem, which is lost on redeploy; use a storage adapter or a mounted volume in production.
|
|
22
32
|
COPY --from=build --chown=node:node /app/.next/standalone ./
|
|
@@ -297,50 +297,6 @@ jobs:
|
|
|
297
297
|
environment: dev
|
|
298
298
|
commit_sha: ${{ github.sha }}
|
|
299
299
|
|
|
300
|
-
triage:
|
|
301
|
-
name: Triage failed build
|
|
302
|
-
needs:
|
|
303
|
-
- checks
|
|
304
|
-
- image
|
|
305
|
-
if: failure() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
|
|
306
|
-
runs-on: ubuntu-latest
|
|
307
|
-
timeout-minutes: 15
|
|
308
|
-
permissions:
|
|
309
|
-
actions: read # Needed to read the logs of the failed jobs.
|
|
310
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
311
|
-
pull-requests: write # Needed to post the summary on the pull request.
|
|
312
|
-
steps:
|
|
313
|
-
- name: Collect the build log
|
|
314
|
-
env:
|
|
315
|
-
GH_TOKEN: ${{ github.token }}
|
|
316
|
-
REPOSITORY: ${{ github.repository }}
|
|
317
|
-
RUN_ID: ${{ github.run_id }}
|
|
318
|
-
run: |
|
|
319
|
-
mkdir -p out
|
|
320
|
-
for job in $(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select(.conclusion == "failure") | .id'); do
|
|
321
|
-
gh api "repos/$REPOSITORY/actions/jobs/$job/logs" >> out/build.log
|
|
322
|
-
done
|
|
323
|
-
- name: Triage failed build
|
|
324
|
-
id: triage
|
|
325
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
326
|
-
with:
|
|
327
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
328
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
329
|
-
prompt: >
|
|
330
|
-
Read the build log at out/build.log. Identify the most
|
|
331
|
-
likely cause, say whether the failure looks flaky or real, and write a
|
|
332
|
-
three-line summary for the PR thread.
|
|
333
|
-
claude_args: >-
|
|
334
|
-
--allowedTools "Read"
|
|
335
|
-
--json-schema '{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"]}'
|
|
336
|
-
- name: Post the summary
|
|
337
|
-
env:
|
|
338
|
-
GH_TOKEN: ${{ github.token }}
|
|
339
|
-
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
340
|
-
REPOSITORY: ${{ github.repository }}
|
|
341
|
-
SUMMARY: ${{ fromJSON(steps.triage.outputs.structured_output).summary }}
|
|
342
|
-
run: gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "$SUMMARY"
|
|
343
|
-
|
|
344
300
|
ci:
|
|
345
301
|
name: All Checks
|
|
346
302
|
if: always()
|
|
@@ -1,22 +1,32 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Fly.io, Cloud Run). On Vercel, deploy without it. The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS build
|
|
3
5
|
ENV NEXT_TELEMETRY_DISABLED=1
|
|
4
6
|
WORKDIR /app
|
|
5
7
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
6
8
|
COPY apps/app/package.json apps/app/package.json
|
|
7
9
|
COPY apps/backoffice/package.json apps/backoffice/package.json
|
|
8
|
-
RUN
|
|
10
|
+
RUN corepack enable
|
|
9
11
|
RUN pnpm install --frozen-lockfile --ignore-scripts --filter ./apps/app...
|
|
10
12
|
COPY apps/app/next.config.ts apps/app/postcss.config.mjs apps/app/tsconfig.json apps/app/
|
|
11
13
|
COPY apps/app/app apps/app/app
|
|
12
14
|
RUN pnpm --filter ./apps/app build
|
|
13
15
|
|
|
14
|
-
FROM
|
|
16
|
+
FROM \
|
|
17
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
15
18
|
ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 HOSTNAME=0.0.0.0
|
|
16
19
|
WORKDIR /app
|
|
17
20
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
18
21
|
RUN apk upgrade --no-cache \
|
|
19
|
-
&& rm -rf
|
|
22
|
+
&& rm -rf \
|
|
23
|
+
/usr/local/lib/node_modules \
|
|
24
|
+
/usr/local/bin/npm \
|
|
25
|
+
/usr/local/bin/npx \
|
|
26
|
+
/usr/local/bin/corepack \
|
|
27
|
+
/usr/local/bin/yarn \
|
|
28
|
+
/usr/local/bin/yarnpkg \
|
|
29
|
+
/opt/yarn-*
|
|
20
30
|
# Next.js writes .next/cache at runtime. Add `COPY --from=build --chown=node:node /app/apps/app/public ./apps/app/public` when the app has a public/ folder.
|
|
21
31
|
COPY --from=build --chown=node:node /app/apps/app/.next/standalone ./
|
|
22
32
|
COPY --from=build --chown=node:node /app/apps/app/.next/static ./apps/app/.next/static
|
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
# Runs on any OCI platform (Railway, Fly.io, Cloud Run). On Vercel, deploy without it. The platform sets `PORT`.
|
|
2
|
-
FROM
|
|
2
|
+
FROM \
|
|
3
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53 \
|
|
4
|
+
AS build
|
|
3
5
|
ENV NEXT_TELEMETRY_DISABLED=1
|
|
4
6
|
WORKDIR /app
|
|
5
7
|
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
|
|
6
8
|
COPY apps/app/package.json apps/app/package.json
|
|
7
9
|
COPY apps/backoffice/package.json apps/backoffice/package.json
|
|
8
|
-
RUN
|
|
10
|
+
RUN corepack enable
|
|
9
11
|
RUN pnpm install --frozen-lockfile --ignore-scripts --filter ./apps/backoffice...
|
|
10
12
|
COPY apps/backoffice/next.config.ts apps/backoffice/tsconfig.json apps/backoffice/
|
|
11
13
|
COPY apps/backoffice/src apps/backoffice/src
|
|
@@ -13,12 +15,20 @@ COPY apps/backoffice/src apps/backoffice/src
|
|
|
13
15
|
# The platform sets the real ones at runtime, and the build never connects to the database.
|
|
14
16
|
RUN PAYLOAD_SECRET=build-only DATABASE_URL=postgres://build-only@localhost/build-only APP_URL=http://localhost:3001 pnpm --filter ./apps/backoffice build
|
|
15
17
|
|
|
16
|
-
FROM
|
|
18
|
+
FROM \
|
|
19
|
+
node@sha256:06d7adf3e0ef61a13f1d28682726366f97a15aa1888c934d3ecec86c33918b53
|
|
17
20
|
ENV NODE_ENV=production NEXT_TELEMETRY_DISABLED=1 HOSTNAME=0.0.0.0
|
|
18
21
|
WORKDIR /app
|
|
19
22
|
# Apply Alpine security fixes released after the base image, and drop npm, corepack, and yarn: the app never runs them, and their bundled dependencies carry their own CVEs.
|
|
20
23
|
RUN apk upgrade --no-cache \
|
|
21
|
-
&& rm -rf
|
|
24
|
+
&& rm -rf \
|
|
25
|
+
/usr/local/lib/node_modules \
|
|
26
|
+
/usr/local/bin/npm \
|
|
27
|
+
/usr/local/bin/npx \
|
|
28
|
+
/usr/local/bin/corepack \
|
|
29
|
+
/usr/local/bin/yarn \
|
|
30
|
+
/usr/local/bin/yarnpkg \
|
|
31
|
+
/opt/yarn-* \
|
|
22
32
|
&& install -d -o node -g node apps/backoffice/media
|
|
23
33
|
# Next.js writes .next/cache at runtime. Uploads go to the container filesystem, which is lost on redeploy; use a storage adapter or a mounted volume in production.
|
|
24
34
|
COPY --from=build --chown=node:node /app/apps/backoffice/.next/standalone ./
|
|
@@ -1,128 +0,0 @@
|
|
|
1
|
-
name: Agent evals
|
|
2
|
-
|
|
3
|
-
on:
|
|
4
|
-
pull_request:
|
|
5
|
-
types: [opened, synchronize, reopened, ready_for_review]
|
|
6
|
-
schedule:
|
|
7
|
-
- cron: "0 2 * * *"
|
|
8
|
-
|
|
9
|
-
permissions: {}
|
|
10
|
-
|
|
11
|
-
concurrency:
|
|
12
|
-
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
|
|
13
|
-
cancel-in-progress: true
|
|
14
|
-
|
|
15
|
-
jobs:
|
|
16
|
-
list:
|
|
17
|
-
name: List evals
|
|
18
|
-
runs-on: ubuntu-latest
|
|
19
|
-
timeout-minutes: 5
|
|
20
|
-
permissions:
|
|
21
|
-
contents: read
|
|
22
|
-
outputs:
|
|
23
|
-
evals: ${{ steps.list.outputs.evals }}
|
|
24
|
-
steps:
|
|
25
|
-
- name: Checkout repository
|
|
26
|
-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
27
|
-
with:
|
|
28
|
-
fetch-depth: 0
|
|
29
|
-
persist-credentials: false
|
|
30
|
-
- name: List the evals to run
|
|
31
|
-
id: list
|
|
32
|
-
env:
|
|
33
|
-
EVENT_NAME: ${{ github.event_name }}
|
|
34
|
-
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
|
35
|
-
run: |
|
|
36
|
-
if [[ "$EVENT_NAME" == "schedule" ]] || git diff --name-only "$BASE_SHA" HEAD | grep -qE '^(CLAUDE\.md|AGENTS\.md|REVIEW\.md|\.mcp\.json|\.agents/|\.claude/|\.cursor/|evals/)'; then
|
|
37
|
-
echo "evals=$(find evals -maxdepth 1 -name '*.json' | sort | jq -R -s -c 'split("\n")[:-1]')" >> "$GITHUB_OUTPUT"
|
|
38
|
-
else
|
|
39
|
-
echo 'evals=[]' >> "$GITHUB_OUTPUT"
|
|
40
|
-
fi
|
|
41
|
-
|
|
42
|
-
eval:
|
|
43
|
-
name: Eval
|
|
44
|
-
needs: list
|
|
45
|
-
if: needs.list.outputs.evals != '[]'
|
|
46
|
-
runs-on: ubuntu-latest
|
|
47
|
-
timeout-minutes: 30
|
|
48
|
-
permissions:
|
|
49
|
-
contents: read
|
|
50
|
-
id-token: write # Needed to authenticate to Anthropic with the workflow's OIDC token.
|
|
51
|
-
strategy:
|
|
52
|
-
fail-fast: false
|
|
53
|
-
matrix:
|
|
54
|
-
eval: ${{ fromJSON(needs.list.outputs.evals) }}
|
|
55
|
-
steps:
|
|
56
|
-
- name: Checkout repository
|
|
57
|
-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
58
|
-
with:
|
|
59
|
-
persist-credentials: false
|
|
60
|
-
- name: Set up Node.js and pnpm
|
|
61
|
-
uses: pnpm/setup@fbda4c85fc2e1e08721cd8763afea8f48d60f024 # v3.0.0
|
|
62
|
-
with:
|
|
63
|
-
cache: true
|
|
64
|
-
require-lockfile: true
|
|
65
|
-
- name: Read the eval prompt
|
|
66
|
-
id: read
|
|
67
|
-
env:
|
|
68
|
-
EVAL: ${{ matrix.eval }}
|
|
69
|
-
run: |
|
|
70
|
-
{
|
|
71
|
-
echo "prompt<<EVAL_PROMPT_END"
|
|
72
|
-
jq -r '.prompt' "$EVAL"
|
|
73
|
-
echo "EVAL_PROMPT_END"
|
|
74
|
-
} >> "$GITHUB_OUTPUT"
|
|
75
|
-
- name: Run the eval task
|
|
76
|
-
id: claude
|
|
77
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
78
|
-
with:
|
|
79
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
80
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
81
|
-
prompt: ${{ steps.read.outputs.prompt }}
|
|
82
|
-
claude_args: --permission-mode acceptEdits --allowedTools "Read,Edit,Bash(pnpm test)"
|
|
83
|
-
- name: Check the result
|
|
84
|
-
env:
|
|
85
|
-
EVAL: ${{ matrix.eval }}
|
|
86
|
-
CONCLUSION: ${{ steps.claude.outputs.conclusion }}
|
|
87
|
-
run: ./evals/check.sh "$EVAL" "$CONCLUSION"
|
|
88
|
-
|
|
89
|
-
evals:
|
|
90
|
-
name: Agent evals
|
|
91
|
-
needs:
|
|
92
|
-
- list
|
|
93
|
-
- eval
|
|
94
|
-
if: always()
|
|
95
|
-
runs-on: ubuntu-latest
|
|
96
|
-
timeout-minutes: 5
|
|
97
|
-
permissions:
|
|
98
|
-
actions: read # Needed to read the result of each eval job.
|
|
99
|
-
steps:
|
|
100
|
-
- name: Check the pass rate
|
|
101
|
-
env:
|
|
102
|
-
EVALS: ${{ needs.list.outputs.evals }}
|
|
103
|
-
LIST_RESULT: ${{ needs.list.result }}
|
|
104
|
-
EVALS_MIN_PASS_RATE: ${{ vars.EVALS_MIN_PASS_RATE }}
|
|
105
|
-
GH_TOKEN: ${{ github.token }}
|
|
106
|
-
REPOSITORY: ${{ github.repository }}
|
|
107
|
-
RUN_ID: ${{ github.run_id }}
|
|
108
|
-
run: |
|
|
109
|
-
if [[ "$LIST_RESULT" != "success" ]]; then
|
|
110
|
-
echo "::error::Listing the evals failed."
|
|
111
|
-
exit 1
|
|
112
|
-
fi
|
|
113
|
-
total=$(jq 'length' <<< "$EVALS")
|
|
114
|
-
if ((total == 0)); then
|
|
115
|
-
echo "No evals to run."
|
|
116
|
-
exit 0
|
|
117
|
-
fi
|
|
118
|
-
if [[ -z "$EVALS_MIN_PASS_RATE" ]]; then
|
|
119
|
-
echo "::error::Set the repository variable EVALS_MIN_PASS_RATE (0 to 100)."
|
|
120
|
-
exit 1
|
|
121
|
-
fi
|
|
122
|
-
passed=$(gh api "repos/$REPOSITORY/actions/runs/$RUN_ID/jobs" --paginate --jq '.jobs[] | select((.name | startswith("Eval (")) and .conclusion == "success") | .id' | wc -l)
|
|
123
|
-
rate=$((passed * 100 / total))
|
|
124
|
-
echo "Pass rate: $rate% ($passed of $total)"
|
|
125
|
-
if ((rate < EVALS_MIN_PASS_RATE)); then
|
|
126
|
-
echo "::error::The pass rate $rate% is below EVALS_MIN_PASS_RATE ($EVALS_MIN_PASS_RATE%)."
|
|
127
|
-
exit 1
|
|
128
|
-
fi
|
|
@@ -1,39 +0,0 @@
|
|
|
1
|
-
name: Claude
|
|
2
|
-
|
|
3
|
-
on:
|
|
4
|
-
pull_request_review_comment:
|
|
5
|
-
types: [created]
|
|
6
|
-
pull_request_review:
|
|
7
|
-
types: [submitted]
|
|
8
|
-
|
|
9
|
-
permissions: {}
|
|
10
|
-
|
|
11
|
-
concurrency:
|
|
12
|
-
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
|
13
|
-
cancel-in-progress: false
|
|
14
|
-
|
|
15
|
-
jobs:
|
|
16
|
-
claude:
|
|
17
|
-
name: Address review comment
|
|
18
|
-
if: |
|
|
19
|
-
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
|
20
|
-
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude'))
|
|
21
|
-
runs-on: ubuntu-latest
|
|
22
|
-
timeout-minutes: 60
|
|
23
|
-
permissions:
|
|
24
|
-
contents: write # Needed to push the fix.
|
|
25
|
-
pull-requests: write # Needed to reply on the pull request.
|
|
26
|
-
id-token: write # Needed by the action to authenticate.
|
|
27
|
-
actions: read # Needed to read CI results on pull requests.
|
|
28
|
-
steps:
|
|
29
|
-
- name: Checkout repository
|
|
30
|
-
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
31
|
-
with:
|
|
32
|
-
fetch-depth: 1
|
|
33
|
-
persist-credentials: false
|
|
34
|
-
- name: Address the comment
|
|
35
|
-
uses: anthropics/claude-code-action@61a9c274fa7110a66dfafc1bbbced03833eb25fe # v1.0.269
|
|
36
|
-
with:
|
|
37
|
-
anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }}
|
|
38
|
-
anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }}
|
|
39
|
-
claude_args: --permission-mode acceptEdits
|
|
@@ -1,15 +0,0 @@
|
|
|
1
|
-
# Agent evals
|
|
2
|
-
|
|
3
|
-
Each eval is one real task from recent work with its accepted outcome, written as `evals/<name>.json`: the prompt, plus the check that defines an acceptable result.
|
|
4
|
-
|
|
5
|
-
```json
|
|
6
|
-
{
|
|
7
|
-
"prompt": "<the task, as a person would ask it>",
|
|
8
|
-
"check": "<a shell command that exits 0 when the result is acceptable, for example: pnpm test>"
|
|
9
|
-
}
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
- Collect 20 to 50 real tasks from recent work.
|
|
13
|
-
- `.github/workflows/agent-evals.yml` runs the suite every night, and on every pull request that changes `CLAUDE.md`, `AGENTS.md`, `.mcp.json`, `.agents/`, `.claude/`, `.cursor/`, or `evals/`. `evals/check.sh` checks one eval's result.
|
|
14
|
-
- Set the repository variable `EVALS_MIN_PASS_RATE`, from 0 to 100. When the pass rate falls below it, the `Agent evals` check fails and the pull request cannot merge.
|
|
15
|
-
- Each production incident gets an eval, written by the team that owned the incident, and it stays in the suite.
|
|
@@ -1,17 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env bash
|
|
2
|
-
set -euo pipefail
|
|
3
|
-
|
|
4
|
-
eval_file=$1
|
|
5
|
-
conclusion=$2
|
|
6
|
-
|
|
7
|
-
if [[ $conclusion != success ]]; then
|
|
8
|
-
echo "FAIL $eval_file: Claude Code reported $conclusion"
|
|
9
|
-
exit 1
|
|
10
|
-
fi
|
|
11
|
-
|
|
12
|
-
if bash -c "$(jq -r '.check' "$eval_file")"; then
|
|
13
|
-
echo "PASS $eval_file"
|
|
14
|
-
else
|
|
15
|
-
echo "FAIL $eval_file"
|
|
16
|
-
exit 1
|
|
17
|
-
fi
|