apify-test-tools 0.10.1-beta.4 → 0.10.1-beta.5

package/.claude-plugin/marketplace.json

@@ -0,0 +1,12 @@
+{
+    "name": "apify-test-tools",
+    "owner": {
+        "name": "Apify"
+    },
+    "plugins": [
+        {
+            "name": "platform-tests-investigate-and-fix",
+            "source": "./platform-tests-investigate-and-fix"
+        }
+    ]
+}

package/.github/workflows/public_platform-tests-claude-investigate-and-fix.yaml

@@ -39,16 +39,17 @@ on:
                 type: string
                 default: sonnet
                 description: Claude model to use for investigation and fix
+            max_flakiness_runs:
+                required: false
+                type: number
+                default: 200
+                description: The most times Claude may re-run the failing Actor run to measure how often the failure happens, when that isn't obvious. Claude decides how many of them it needs.
             investigate_prompt:
                 required: false
                 type: string
                 default: |
                     Investigate the failing E2E platform test and open a GitHub issue documenting your findings. Start by reviewing the test logs, error messages, and recent workflow run history to determine the root cause.
 
-                    The failure will typically fall into one of these categories:
-                    - Flaky test: The test is non-deterministic or depends on timing/external state and fails intermittently. Check past workflow runs to see if this test has failed before.
-                    - Real bug: Genuine broken functionality — either a pre-existing issue or an environment/dependency change outside the codebase (e.g. a website changed its structure, an external API changed behavior).
-
                     Use available tools to check previous workflow runs, git history, and existing GitHub issues to gather context and avoid creating duplicate issues. For deeper context,
                     use tools to investigate the run on Apify platform, review its logs, datasets, key-value stores.
 
@@ -56,9 +57,12 @@ on:
                     - Debug snapshots: often stored as a field on dataset items (e.g. `_debugSnapshots`) or as records in a key-value store, keyed so they can be matched to the log line for the failing request.
                     - Any other debug artifacts the Actor produced (extra logs, error/state fields on output items) that narrow down the failing request.
 
+                    Before you create a new issue, check whether the `platform-tests-investigate-and-fix:measure-flakiness` skill applies, and follow it if it does.
+
                     When you create the issue, write it like an incident postmortem: a short summary and impact up top for anyone skimming, full technical detail below for whoever fixes it. Don't water down the engineering detail to make the top readable.
                     - Title: short and clean, describing the actual problem in plain terms — no dates, run numbers, or other noise. Prefix it with a conventional-commit-style type, e.g. "fix: blocking issue on detail endpoint".
                     - Body, section 1 ("## Summary"): one line stating what's broken plus its impact, in plain language a PM can understand at a glance — no jargon, internal field names, or stack traces (e.g. "Scraping product detail pages fails when a listing has no reviews, affecting N% of runs").
+                    - If you measured flakiness, a "## Flakiness" section right after the Summary, as the `measure-flakiness` skill describes.
                     - A `---` divider, then section 2 ("## Details"): the engineering bug report — root cause, relevant log lines, links to the failing run, and any other internals an engineer needs. If a single item/request failed among otherwise-good ones, show or link that specific one (e.g. the dataset item index/id, or its matching debug snapshot) rather than just describing the run as failed.
 
                     If you create a new issue, you MUST write its issue number (just the number, nothing else) to $GITHUB_WORKSPACE/issue-number.txt as the very last thing you do. If you skip creating one because a duplicate already exists, do not create the file.
@@ -67,9 +71,9 @@ on:
             ANTHROPIC_API_KEY:
                 required: true
                 description: Claude API key
-            TESTER_APIFY_TOKEN_READ_ONLY:
+            TESTER_APIFY_TOKEN:
                 required: true
-                description: Apify token used only for read-only investigation — reading the failing run, its log and its storages.
+                description: Apify token for investigating the failing run — reading it, its log and its storages — and re-running it to measure how often the failure happens.
             NPM_TOKEN:
                 required: false
                 description: Read-only token for installing private npm packages before Claude runs.
@@ -92,7 +96,8 @@ jobs:
               id: claude
               uses: anthropics/claude-code-action@8251c103ac8c1d761882c86aba1412c7f583c844 # v1.0.213
               env:
-                  TESTER_APIFY_TOKEN_READ_ONLY: ${{ secrets.TESTER_APIFY_TOKEN_READ_ONLY }}
+                  TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
+                  MAX_FLAKINESS_RUNS: ${{ inputs.max_flakiness_runs }}
               with:
                   prompt: |
                       The failing E2E platform test ran in workflow run ${{ github.run_id }} of this repo.
@@ -100,13 +105,15 @@ jobs:
                       ${{ inputs.investigate_prompt }}
                   github_token: ${{ secrets.GITHUB_TOKEN }}
                   anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+                  plugin_marketplaces: https://github.com/apify/apify-test-tools.git
+                  plugins: platform-tests-investigate-and-fix@apify-test-tools
                   # The `tools` query parameter below selects which tools the Apify MCP server
                   # exposes: reading an Actor run — its status, exit code, storage ids and log — and
                   # reading that run's dataset items and its key-value store keys and records, e.g.
                   # INPUT, OUTPUT and error snapshots. WARN: the server adds `abort-actor-run` on top
                   # even when not listed here, see https://github.com/apify/apify-mcp-server
                   claude_args: |
-                      --mcp-config '{"mcpServers":{"apify":{"type":"http","url":"https://mcp.apify.com/?tools=get-actor-run,get-actor-log,get-dataset-list,get-dataset,get-dataset-items,get-key-value-store-list,get-key-value-store,get-key-value-store-keys,get-key-value-store-record","headers":{"Authorization":"Bearer ${TESTER_APIFY_TOKEN_READ_ONLY}"}}}}'
+                      --mcp-config '{"mcpServers":{"apify":{"type":"http","url":"https://mcp.apify.com/?tools=get-actor-run,get-actor-run-log,get-dataset-list,get-dataset,get-dataset-items,get-key-value-store-list,get-key-value-store,get-key-value-store-keys,get-key-value-store-record","headers":{"Authorization":"Bearer ${TESTER_APIFY_TOKEN}"}}}}'
                       --model ${{ inputs.model }}
                       --dangerously-skip-permissions
 

package/CHANGELOG.md

@@ -9,6 +9,7 @@ All notable changes to this project will be documented in this file.
 
 - **relative-path:** Path safety and manipulation lib ([#154](https://github.com/apify/apify-test-tools/pull/154)) ([c151a1c](https://github.com/apify/apify-test-tools/commit/c151a1c2755fef98b335d48a3461f5042b90def9)) by [@JuanGalilea](https://github.com/JuanGalilea)
 - **logger:** Add logger and ban console statements ([#170](https://github.com/apify/apify-test-tools/pull/170)) ([b295d36](https://github.com/apify/apify-test-tools/commit/b295d363276c92fc6c30dd34d7a90acef5d380bf)) by [@JuanGalilea](https://github.com/JuanGalilea), closes [#156](https://github.com/apify/apify-test-tools/issues/156)
+- Give claude investigator a skill to test flakiness ([#184](https://github.com/apify/apify-test-tools/pull/184)) ([b490298](https://github.com/apify/apify-test-tools/commit/b4902987dc2ae55882463ff8223e1e9f25b8b497)) by [@foxt451](https://github.com/foxt451), closes [#162](https://github.com/apify/apify-test-tools/issues/162)
 
 ### 🐛 Bug Fixes
 

package/README.md

@@ -258,13 +258,12 @@ it at a branch only to test a prompt change.
 Callers pass `secrets: inherit`. The workflows do not turn every inherited secret into job-wide
 environment variables, so a secret is only visible to the step that needs it:
 
-| Secret                                                                    | Reaches                                                                                                                                                                                                                |
-| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `NPM_TOKEN`                                                               | dependency install steps only, as both `NPM_TOKEN` and `NODE_AUTH_TOKEN`. Use a read-only token: npm granular tokens can be read-only, classic automation tokens can publish.                                          |
-| the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | the `build`, `release`, and `delete-old-builds` steps only                                                                                                                                                             |
-| `TESTER_APIFY_TOKEN`                                                      | the vitest step only                                                                                                                                                                                                   |
-| `SLACK_TOKEN_TESTS_BOT` / `SLACK_TOKEN_RELEASES_BOT`                      | the reporting and release steps only                                                                                                                                                                                   |
-| `TESTER_APIFY_TOKEN_READ_ONLY`                                            | the Claude investigation step only, as the Apify MCP server's bearer token. Required by `public_platform-tests-claude-investigate-and-fix`. Use a read-only token: it reads the failing run, its log and its storages. |
+| Secret                                                                    | Reaches                                                                                                                                                                                                                      |
+| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `NPM_TOKEN`                                                               | dependency install steps only, as both `NPM_TOKEN` and `NODE_AUTH_TOKEN`. Use a read-only token: npm granular tokens can be read-only, classic automation tokens can publish.                                                |
+| the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | the `build`, `release`, and `delete-old-builds` steps only                                                                                                                                                                   |
+| `TESTER_APIFY_TOKEN`                                                      | the vitest step, and the Claude investigation step: the Apify MCP server's bearer token for reading the failing run, its log and its storages, and Claude's token for re-running it to measure how often the failure happens |
+| `SLACK_TOKEN_TESTS_BOT` / `SLACK_TOKEN_RELEASES_BOT`                      | the reporting and release steps only                                                                                                                                                                                         |
 
 The Actor tokens are the one set that cannot be listed in the workflow, because each Actor names its
 own token via `tokenEnvVar` in `apify-test-tools.config.json`. Those steps pass

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "apify-test-tools",
-  "version": "0.10.1-beta.4",
+  "version": "0.10.1-beta.5",
   "type": "module",
   "description": "TBD",
   "repository": {

package/platform-tests-investigate-and-fix/.claude-plugin/plugin.json

@@ -0,0 +1,4 @@
+{
+    "name": "platform-tests-investigate-and-fix",
+    "description": "Skills for the platform-tests Claude investigate-and-fix workflow"
+}

package/platform-tests-investigate-and-fix/skills/measure-flakiness/SKILL.md

@@ -0,0 +1,28 @@
+---
+name: measure-flakiness
+description: Re-runs the failing Actor run many times in parallel to measure how often the failure happens. Use only when you are about to create a new issue for a failing platform test and its failure rate is not already obvious.
+---
+
+# Measure flakiness
+
+Use this only when both are true:
+
+- You are about to create a new issue — no existing issue already covers this failure.
+- How often the failure happens isn't obvious. Blocking, too few scraped items or a missing field can happen in 1% of runs or in all of them, and one failed run doesn't tell which.
+
+## Steps
+
+The Apify API token is in the `TESTER_APIFY_TOKEN` environment variable. It is always set, so there's no need to check for it.
+
+1. From the failing run (the run the failed test started; its link is in the test log), get the Actor ID, build number, memory, timeout and input. For example, you can get the first four from `GET https://api.apify.com/v2/actor-runs/{runId}`.
+2. Decide how to tell whether a finished run hit the same failure. Check what the failed test assertion checked — e.g. the run's status, its dataset item count against the expected minimum, a field missing from items, a log line.
+3. Start runs of the same Actor at once, with the same build, memory, timeout and input. For example, you can start each with `POST https://api.apify.com/v2/acts/{actorId}/runs`. If a run can't start because the account's memory limit is reached, start it once earlier runs finish.
+
+    Decide on your own how many runs you need, up to !`echo "$MAX_FLAKINESS_RUNS"` in total. The point isn't a statistically significant rate, but a rough idea of whether the failure is frequent, infrequent or a once-in-a-lifetime event. For example:
+    - Blocking that looks persistent: if around 20 runs all get blocked the same way, that's enough.
+    - A rare flake that's hard to reproduce, e.g. one item out of hundreds missing a field: start a lot of runs — it may take a hundred runs, and after that another hundred, to see it even a few times.
+
+4. Wait until every run has finished, then apply the check from step 2 to each. If there were no or almost no reproductions and you can't tell yet how often the failure happens, start more runs and check them the same way, but up to !`echo "$MAX_FLAKINESS_RUNS"` in total.
+5. Add a short `## Flakiness` section to the issue, right after the Summary:
+    - The rate, e.g. "Fails in 28 of 47 runs (60%)". Leave out runs that failed for another reason, e.g. never started — only mention how many there were.
+    - Links to up to 2 runs that hit the failure.