apify-test-tools 0.10.2-beta.8 → 0.10.2-beta.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. package/.github/actions/checkout-restore-dependencies/action.yaml +13 -0
  2. package/.github/cli/apify-test-tools.mjs +1 -1
  3. package/.github/scripts/check-major-tag-refs.mjs +1 -1
  4. package/.github/scripts/mask-config-secrets.sh +17 -0
  5. package/.github/scripts/run-with-apify-tokens.mjs +36 -11
  6. package/.github/workflows/public_platform-tests-claude-investigate-and-fix.yaml +6 -2
  7. package/.github/workflows/public_platform-tests.yaml +23 -0
  8. package/.github/workflows/public_pr-build-test.yaml +30 -1
  9. package/.github/workflows/public_push-build-latest.yaml +20 -3
  10. package/CHANGELOG.md +1 -0
  11. package/CONTRIBUTING.md +99 -8
  12. package/README.md +218 -699
  13. package/bin/git.ts +1 -1
  14. package/dist/bin/git.js +1 -1
  15. package/dist/bin/git.js.map +1 -1
  16. package/dist/test/unit/github-scripts/mask-config-secrets.test.d.ts +2 -0
  17. package/dist/test/unit/github-scripts/mask-config-secrets.test.d.ts.map +1 -0
  18. package/dist/test/unit/github-scripts/mask-config-secrets.test.js +50 -0
  19. package/dist/test/unit/github-scripts/mask-config-secrets.test.js.map +1 -0
  20. package/dist/test/unit/github-scripts/run-with-apify-tokens.test.d.ts +2 -0
  21. package/dist/test/unit/github-scripts/run-with-apify-tokens.test.d.ts.map +1 -0
  22. package/dist/test/unit/github-scripts/run-with-apify-tokens.test.js +102 -0
  23. package/dist/test/unit/github-scripts/run-with-apify-tokens.test.js.map +1 -0
  24. package/dist/tsconfig.tsbuildinfo +1 -1
  25. package/docs/cli.md +358 -0
  26. package/docs/lib.md +271 -0
  27. package/docs/workflows.md +366 -0
  28. package/package.json +1 -1
  29. package/test/unit/github-scripts/mask-config-secrets.test.ts +61 -0
  30. package/test/unit/github-scripts/run-with-apify-tokens.test.ts +115 -0
package/README.md CHANGED
@@ -1,8 +1,71 @@
1
1
  # Apify Test Tools
2
2
 
3
- [Contributing link](CONTRIBUTING.md)
3
+ Build, test and release Apify Actors from a Git repository or local source code. The package has three parts:
4
4
 
5
- ## Getting Started
5
+ 1. **A vitest library** for writing platform tests that run your Actors on Apify and check the results:
6
+
7
+ ```ts
8
+ testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
9
+ const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
10
+ await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
11
+ });
12
+ ```
13
+
14
+ 2. **A CLI** that finds the Actors a change affects, builds them, and reports the results. It runs anywhere, so you
15
+ can build your own pipeline on it in any CI:
16
+
17
+ ```bash
18
+ npx apify-test-tools get-affected-actors --base origin/master \
19
+ | jq '{ actors: .functionallyChangedActors }' \
20
+ | npx apify-test-tools build --version-number 0.99
21
+ ```
22
+
23
+ 3. **Ready-made GitHub Actions workflows** built on the CLI and the library, so a GitHub repo needs only a few lines
24
+ per workflow:
25
+
26
+ ```yaml
27
+ jobs:
28
+ buildDevelAndTest:
29
+ uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
30
+ secrets:
31
+ TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
32
+ # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
33
+ CONFIG_SECRETS: |
34
+ { "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}" }
35
+ ```
36
+
37
+ Most repos only need the library and the workflows: the workflows call the CLI for you. Use the CLI directly only to
38
+ build your own pipeline, e.g. on a CI platform other than GitHub, or to build and test locally.
39
+
40
+ ## Only what changed is retested
41
+
42
+ Builds and platform tests cost time and money, so the tool skips everything a change can't affect:
43
+
44
+ - **Only affected Actors.** A changed file only counts for the Actors whose build context it falls in, minus their
45
+ `.dockerignore`. In a monorepo, a change to one Actor doesn't build or test the others.
46
+ - **Cosmetic changes skip tests.** README, CHANGELOG and formatting-only schema changes don't trigger tests on a PR;
47
+ they're only rebuilt when released.
48
+ - **Only new changes on a PR.** Each push to a PR retests only the Actors that changed since its last green run, not
49
+ the whole PR again.
50
+ - **Only their tests.** Only the tests of the Actors that were built run.
51
+
52
+ See [Change detection](docs/cli.md#change-detection) for the exact rules.
53
+
54
+ This README covers what you need to get a repo running. The full reference lives in `docs/`:
55
+
56
+ - [Vitest library API](docs/lib.md) — `testActor`, `testStandbyActor`, `RunTestResult`, matchers, running tests locally
57
+ - [CLI API](docs/cli.md) — every command, change detection, releasing, building locally
58
+ - [Workflow API](docs/workflows.md) — every GitHub workflow, its inputs and secrets, versioning
59
+
60
+ [Contributing](CONTRIBUTING.md)
61
+
62
+ ## Getting started
63
+
64
+ Your Actors can be written in any programming language, e.g. Python: anything the platform builds from a Dockerfile
65
+ works. Only the tests and this tooling run on Node.js, so all a repo needs is a `package.json` at the root with
66
+ `apify-test-tools` and `vitest` in its `devDependencies`, and tests written in TypeScript or JavaScript. If you use the
67
+ [PR workflow](docs/workflows.md#public_pr-build-testyaml), also define the `build-check`, `lint` and `test` scripts
68
+ its static checks run, e.g. with your language's own linter and unit tests.
6
69
 
7
70
  ### 1. Install the package
8
71
 
@@ -10,118 +73,87 @@
10
73
  npm i -D apify-test-tools
11
74
  ```
12
75
 
13
- - Requires `vitest` version `3.2.0` or later (uses [annotate](https://vitest.dev/guide/test-context.html#annotate))
76
+ - Requires `vitest` version `3.2.4` or later (uses [annotate](https://vitest.dev/guide/test-context.html#annotate))
14
77
  - Make sure `target` and `module` in your `tsconfig.json`'s `compilerOptions` are set to `ES2022`
15
78
  - Every Actor must already have at least one build under its default build tag (usually `latest`), built
16
79
  from a Git repository version whose URL points to this repository. Build it once manually on the
17
- platform before CI takes over. See [Repository check](#repository-check).
80
+ platform before CI takes over. See [Repository check](docs/cli.md#repository-check).
18
81
 
19
82
  ### 2. Create the config file
20
83
 
21
- Every repo that uses `apify-test-tools` must have an `apify-test-tools.config.json` file at the root. It lists the actors in the repo, their folders, their full names on the platform, and which env var holds the token to use for each. A top-level `mode` field picks the file's layout; leave it out to use the default `"legacy"` layout.
84
+ Every repo must have an `apify-test-tools.config.json` file at the root that lists its Actors. See
85
+ [Config file](#config-file) below.
22
86
 
23
- See [Config file](#config-file) for the available modes and every field.
87
+ ### 3. Set up Actor folders
24
88
 
25
- ### 3. Set up actor folders
89
+ Each Actor in the config must have a `.actor/actor.json` file. The `dockerContextDir` field in `actor.json` defines
90
+ the build context boundary — this is what the tool uses to decide which files can affect the Actor's build (see
91
+ [Change detection](docs/cli.md#change-detection)).
26
92
 
27
- Each actor in the config must have a `.actor/actor.json` file. The `dockerContextDir` field in `actor.json` defines the build context boundary — this is what the tool uses to determine which files can affect the actor's build.
93
+ This example has two Actors that share the code at the repo root, and one standalone Actor with its own code:
28
94
 
29
95
  ```
30
96
  my-repo
31
97
  ├── apify-test-tools.config.json
98
+ ├── Dockerfile <- shared by web-scraper and email-sender
99
+ ├── package.json
100
+ ├── src/ <- shared by web-scraper and email-sender
32
101
  ├── actors
33
102
  │ ├── web-scraper
34
- │ │ ├── .actor
35
- │ │ │ └── actor.json
36
- │ │ └── src/
103
+ │ │ └── .actor
104
+ │ │ └── actor.json <- points to the repo root
37
105
  │ └── email-sender
106
+ │ └── .actor
107
+ │ └── actor.json <- points to the repo root
108
+ ├── standalone-actors
109
+ │ └── orchestrator
38
110
  │ ├── .actor
39
- │ │ └── actor.json
111
+ │ │ └── actor.json <- default context: its own folder
112
+ │ ├── Dockerfile
113
+ │ ├── package.json
40
114
  │ └── src/
41
115
  └── test
42
- ├── unit
43
116
  └── platform
44
- ├── core <- Core (hourly) tests
45
- │ └── core.test.ts
46
- ├── some.test.ts <- Daily tests can be anywhere inside platform/
47
- └── some-other.test.ts
117
+ ├── web-scraper.test.ts
118
+ ├── email-sender.test.ts
119
+ └── orchestrator
120
+ └── orchestrator.test.ts
48
121
  ```
49
122
 
50
- For a single-actor repo, set `"folder": "."` in the config and place `.actor/actor.json` at the repo root.
123
+ The Actors that share code point their `dockerfile` and `dockerContextDir` at the repo root. Both paths are relative
124
+ to the `.actor` folder:
51
125
 
52
- ### Change detection
53
-
54
- When a PR is opened or code is pushed, the tool determines which actors need to be built and tested based on the changed files. For each changed file, for each actor:
55
-
56
- 1. **Sibling exclusion** — files inside another actor's `folder` are excluded first. This prevents an actor with broad context from being triggered by changes that belong to a sibling actor.
57
- 2. **CHANGELOG classification** — a `CHANGELOG.md` file is always `cosmetic` (only triggers a release build, not tests), for every actor, regardless of context or folder. (See [issue #106](https://github.com/apify/apify-test-tools/issues/106).)
58
- 3. **Context matching** — the file must fall within one of the actor's context paths (`dockerContextDir` from `actor.json` by default, or `overrideActorContext` from config if set). Files outside every context path are skipped.
59
- 4. **Hardcoded ignore list, context-aware** — the file path is first "hoisted" relative to the context path it matched (e.g. a standalone actor's own `.eslintrc` is checked as just `.eslintrc`, not the full repo-root-relative path), then checked against repo-level dev file patterns (`.vscode/`, `.gitignore`, `.husky/`, `.eslintrc`, `eslint.config.mjs`, `.prettierrc`, `.editorconfig`). There's no hardcoded special-casing for legacy `code/`/`shared/` layouts — repos that need those directories treated as top-level must list them explicitly in `overrideActorContext`.
60
- 5. **`.dockerignore` filtering** — if a `.dockerignore` exists at the root of the actor's `dockerContextDir`, matching files are ignored. Patterns are resolved relative to `dockerContextDir`, matching Docker's own behavior. Test files are the exception: `get-affected-actors --test-files-glob <glob>` (relative to the repo root) skips this check for matching files, so a changed test triggers a build, and with it its tests, even when `.dockerignore` lists it. The PR workflow passes `test/platform/**`. Context matching still applies, so tests outside an actor's `dockerContextDir` (e.g. a root `test/` with standalone actors) need to be listed in its `overrideActorContext`; `get-affected-actors` logs a warning for changed test files outside every actor's context.
61
- 6. **README classification** — a `README.md` file is `cosmetic` (only triggers a release build, not tests) if it lives inside the actor's own `folder`; otherwise it's ignored entirely, since it isn't documentation for this actor.
62
- 7. **Cosmetic JSON classification** — `.json` files inside the actor's own `.actor/` directory with only cosmetic schema changes (whitespace, key ordering) only trigger a release build.
63
- 8. **Functional** — everything else triggers both build and tests.
64
-
65
- ### 4. Create test directories
66
-
67
- ```bash
68
- mkdir -p test/platform/core
126
+ ```json
127
+ {
128
+ "actorSpecification": 1,
129
+ "name": "web-scraper",
130
+ "version": "0.1",
131
+ "dockerfile": "../../../Dockerfile",
132
+ "dockerContextDir": "../../.."
133
+ }
69
134
  ```
70
135
 
71
- - Core (hourly) tests go in `test/platform/core`
72
- - Daily tests go anywhere in `test/platform`
136
+ A change to the shared `src/` then builds and tests both of them, while a change in `standalone-actors/orchestrator`
137
+ counts only for the orchestrator, even though it lies inside the others' context. The orchestrator leaves
138
+ `dockerContextDir` out, so its context is its own folder.
73
139
 
74
- ### 5. Set up GitHub workflows
140
+ For a single-Actor repo, set `"folder": "."` in the config and place `.actor/actor.json` at the repo root.
75
141
 
76
- See the [GitHub workflows](#github-worklows) section below.
142
+ ### 4. Write tests
77
143
 
78
- ## Config file
79
-
80
- `apify-test-tools.config.json` lives at the repo root and tells the tool which actors live in the repo, how to identify them, and which token to use.
81
-
82
- ### Actor fields
83
-
84
- Every actor in the config is described by these fields, whatever the mode:
144
+ Put platform tests in `test/platform`. How you organize them inside is up to you. See
145
+ [Writing tests](#writing-tests) below.
85
146
 
86
- | Field | Required | Description |
87
- | ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
88
- | `folder` | yes | Relative path from repo root to the actor's own project directory — the folder that directly contains `.actor/actor.json` (i.e. `<folder>/.actor/actor.json`), the actor's README/CHANGELOG, and its source. Use `"."` for a single-actor repo where `.actor/` is at the root. |
89
- | `actorFullName` | yes | Full actor identifier in `owner/name` format (e.g. `"apify/web-scraper"`). This is the source of truth for the actor name — the `name` field in `actor.json` is not used. |
90
- | `tokenEnvVar` | yes | Name of the environment variable holding the Apify API token for this actor. In [grouped mode](#grouped-mode) it's inherited from the group unless the actor sets its own. No fallback — if the env var is not set at build time, the build fails. |
91
- | `overrideActorContext` | no | Array of paths (relative to repo root) that define which files are relevant to this actor. When set, replaces the `dockerContextDir` from `actor.json` for change detection. Useful when an actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. `["", "code"]` or `["actors", "actors/foo"]` are rejected). The actor's own `folder` is always part of its context — if none of the listed entries reach it, it's added automatically. |
147
+ ### 5. Set up CI
92
148
 
93
- ### Modes
149
+ On GitHub, add the [GitHub workflows](#github-workflows) below. On another CI platform, call the
150
+ [CLI](docs/cli.md) from your own pipeline; the workflows are a good template for the steps.
94
151
 
95
- The file's layout is picked by the top-level `mode` field. Each mode is a different way of writing the same thing: every mode resolves to a list of actors with the fields above, and the same rules apply to all of them — no two actors may share a `folder`, and no two entries may point at the same `actorFullName`. If `mode` is left out, [`"legacy"`](#legacy-mode) is used, so existing config files keep working unchanged.
96
-
97
- ### Legacy mode
98
-
99
- The default. A flat `actors` array where every entry carries all of its settings. Good for repos with a few actors, or where every actor has different settings.
100
-
101
- ```json
102
- {
103
- "mode": "legacy",
104
- "actors": [
105
- {
106
- "folder": "actors/web-scraper",
107
- "actorFullName": "myteam/web-scraper",
108
- "tokenEnvVar": "APIFY_TOKEN_MYTEAM"
109
- },
110
- {
111
- "folder": "actors/email-sender",
112
- "actorFullName": "myteam/email-sender",
113
- "tokenEnvVar": "APIFY_TOKEN_MYTEAM",
114
- "overrideActorContext": ["actors/email-sender", "packages/shared"]
115
- }
116
- ]
117
- }
118
- ```
119
-
120
- `actors` must hold at least one entry. Every entry takes all the [actor fields](#actor-fields) directly, and `tokenEnvVar` is required on each one.
121
-
122
- ### Grouped mode
152
+ ## Config file
123
153
 
124
- Actors are organized into `groups`, and each group sets `tokenEnvVar` and `overrideActorContext` once for all of its actors. Good for repos where many actors share the same token or context.
154
+ `apify-test-tools.config.json` lives at the repo root and tells the tool which Actors live in the repo, where they
155
+ are, and which token builds them. Actors are organized into groups, and each group sets the token once for all of its
156
+ Actors:
125
157
 
126
158
  ```json
127
159
  {
@@ -129,72 +161,57 @@ Actors are organized into `groups`, and each group sets `tokenEnvVar` and `overr
129
161
  "groups": {
130
162
  "myteam": {
131
163
  "tokenEnvVar": "APIFY_TOKEN_MYTEAM",
132
- "overrideActorContext": ["packages/shared"],
133
164
  "actors": [
134
165
  { "folder": "actors/web-scraper", "actorFullName": "myteam/web-scraper" },
166
+ { "folder": "actors/email-sender", "actorFullName": "myteam/email-sender" },
135
167
  {
136
- "folder": "actors/email-sender",
137
- "actorFullName": "myteam/email-sender",
138
- "overrideActorContext": ["actors/email-sender", "packages/mailer"]
168
+ "folder": "standalone-actors/orchestrator",
169
+ "actorFullName": "myteam/orchestrator",
170
+ "overrideActorContext": ["standalone-actors/orchestrator", "test/platform/orchestrator"]
139
171
  }
140
172
  ]
141
- },
142
- "partner": {
143
- "tokenEnvVar": "APIFY_TOKEN_PARTNER",
144
- "actors": [{ "folder": "actors/partner-crawler", "actorFullName": "partner/crawler" }]
145
173
  }
146
174
  }
147
175
  }
148
176
  ```
149
177
 
150
- `groups` is an object with at least one group. The group key (`myteam`, `partner` above) is only a label for readability; it has no effect on the result, and it does not need to match any folder or username. Each group needs a `tokenEnvVar` and at least one actor, and can optionally set `overrideActorContext`.
178
+ The group key (`myteam` above) is only a label for readability; it doesn't need to match any folder or username. Each
179
+ group needs a `tokenEnvVar` and at least one Actor. Add another group for Actors that need a different token, e.g.
180
+ Actors owned by another account.
151
181
 
152
- An actor inside a group can set its own `tokenEnvVar` or `overrideActorContext`, and that value replaces the group's for that actor only. Values are replaced, not merged: in the example above, `myteam/email-sender` gets `["actors/email-sender", "packages/mailer"]`, not `packages/shared` as well.
182
+ The orchestrator sets `overrideActorContext` because its tests live outside its own folder: without it, a change to
183
+ only `test/platform/orchestrator` wouldn't build and test it. The other two already have the whole repo, tests
184
+ included, as their context.
153
185
 
154
- ## Github worklows
186
+ | Field | Where | Required | Description |
187
+ | ---------------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
188
+ | `folder` | Actor | yes | Relative path from repo root to the Actor's own project directory — the folder that directly contains `.actor/actor.json` (i.e. `<folder>/.actor/actor.json`), the Actor's README/CHANGELOG, and its source. Use `"."` for a single-Actor repo where `.actor/` is at the root. |
189
+ | `actorFullName` | Actor | yes | Full Actor identifier in `owner/name` format (e.g. `"apify/web-scraper"`). This is the source of truth for the Actor name — the `name` field in `actor.json` is not used. |
190
+ | `tokenEnvVar` | group or Actor | yes | Name of the environment variable holding the Apify API token that builds the group's Actors. No fallback — if the env var is not set at build time, the build fails. |
191
+ | `overrideActorContext` | group or Actor | no | Array of paths (relative to repo root) that define which files are relevant to the Actor. When set, replaces the `dockerContextDir` from `actor.json` for change detection. Useful when an Actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. `["", "code"]` or `["actors", "actors/foo"]` are rejected). The Actor's own `folder` is always part of its context — if none of the listed entries reach it, it's added automatically. |
155
192
 
156
- The reusable workflows live in this repo, alongside the package they call. They are the
157
- `public_`-prefixed files in `.github/workflows`; everything else there is this repo's own CI.
158
- Reference them at the `@workflows-v0` major tag, never at `@master` — see
159
- [Versioning and releases](#versioning-and-releases).
193
+ A value set on an Actor replaces the group's for that Actor only. Values are replaced, not merged: if the `myteam`
194
+ group also set `overrideActorContext`, the orchestrator would still get only the two paths it lists.
160
195
 
161
- There should be 4 GH workflow files in `.github/workflows`, plus an optional fifth for Claude reviews.
196
+ No two Actors may share a `folder`, and no two entries may point at the same `actorFullName`.
162
197
 
163
- ### `platform-tests-core.yaml`
198
+ ## GitHub workflows
164
199
 
165
- ```yaml
166
- name: Platform tests - Core
200
+ The workflows take every change straight to production, with tests in between:
167
201
 
168
- on:
169
- schedule:
170
- # Runs at the start of every hour
171
- - cron: '0 * * * *'
172
- workflow_dispatch:
202
+ - **Every push to a PR** builds the Actors the PR changes into a test version (`0.99`) that users never run, and runs
203
+ their platform tests against those builds. Only the Actors whose files changed are built and tested, and a later
204
+ push only re-tests what changed since the last green run.
205
+ - **Every merge to `master`** builds the changed Actors into their default version (`latest`), which is what users
206
+ run, and announces the release on Slack. There is no separate release step.
207
+ - **Scheduled runs** (optional) run the tests against the released builds to catch what breaks without a code change,
208
+ like a website changing its layout.
173
209
 
174
- jobs:
175
- platformTestsCore:
176
- uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
177
- with:
178
- subtest: core
179
- secrets: inherit
180
- ```
210
+ These are GitHub Actions workflows. On another CI platform, build the same flow with the [CLI](docs/cli.md).
181
211
 
182
- ### `platform-tests-daily.yaml`
183
-
184
- ```yaml
185
- name: Platform tests - Daily
186
-
187
- on:
188
- schedule:
189
- # Runs at 00:00 UTC every day
190
- - cron: '0 0 * * *'
191
- workflow_dispatch:
192
-
193
- jobs:
194
- platformTestsDaily:
195
- uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
196
- secrets: inherit
197
- ```
212
+ Add these files to `.github/workflows` in your repo. They call the reusable workflows in this repo, pinned at the
213
+ `@workflows-v0` major tag — never at `@master`. Every input, the optional Claude workflows, and how the tag moves are
214
+ in the [Workflow API](docs/workflows.md).
198
215
 
199
216
  ### `pr-build-devel-test.yaml`
200
217
 
@@ -208,7 +225,14 @@ on:
208
225
  jobs:
209
226
  buildDevelAndTest:
210
227
  uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
211
- secrets: inherit
228
+ secrets:
229
+ TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
230
+ # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
231
+ CONFIG_SECRETS: |
232
+ {
233
+ "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
234
+ "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
235
+ }
212
236
  ```
213
237
 
214
238
  ### `release-latest.yaml`
@@ -223,608 +247,103 @@ on:
223
247
  jobs:
224
248
  buildLatest:
225
249
  uses: apify/apify-test-tools/.github/workflows/public_push-build-latest.yaml@workflows-v0
226
- secrets: inherit
250
+ secrets:
251
+ SLACK_TOKEN_RELEASES_BOT: ${{ secrets.SLACK_TOKEN_RELEASES_BOT }}
252
+ # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
253
+ CONFIG_SECRETS: |
254
+ {
255
+ "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
256
+ "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
257
+ }
227
258
  ```
228
259
 
229
- This workflow runs one step per command: `get-affected-actors --since ${{ github.event.before }}`, then
230
- `build --version-number default` on every changed Actor, then `notify-release`. See [Releasing Actors](#releasing-actors) for how the
231
- released range is determined and what to pass if you call the CLI from somewhere else.
232
-
233
- ### `claude-review.yaml`
260
+ ### Scheduled tests (optional)
234
261
 
235
- Optional. Reviews a PR against the shared guidelines when you add the trigger label, and again on
236
- every push while that label is on.
262
+ How often to run which tests is up to you. For example, you can run every test once a day:
237
263
 
238
264
  ```yaml
239
- name: Claude review
265
+ name: Platform tests - Daily
240
266
 
241
267
  on:
242
- pull_request:
243
- types: [labeled, synchronize]
268
+ schedule:
269
+ # Runs at 00:00 UTC every day
270
+ - cron: '0 0 * * *'
271
+ workflow_dispatch:
244
272
 
245
273
  jobs:
246
- review:
247
- uses: apify/apify-test-tools/.github/workflows/public_review.yaml@workflows-v0
248
- secrets: inherit
274
+ platformTestsDaily:
275
+ uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
276
+ secrets:
277
+ TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
278
+ SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}
249
279
  ```
250
280
 
251
- The review instructions live in `.github/review-prompt.md` in this repo and are fetched at run time,
252
- because a reusable workflow doesn't get its own repo checked out. `prompt-ref` selects which ref to
253
- fetch them from and defaults to `workflows-v0`, so the instructions match the workflow you're calling — point
254
- it at a branch only to test a prompt change.
255
-
256
- ### Secrets
257
-
258
- Callers pass `secrets: inherit`. The workflows do not turn every inherited secret into job-wide
259
- environment variables, so a secret is only visible to the step that needs it:
260
-
261
- | Secret | Reaches |
262
- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
263
- | `NPM_TOKEN` | dependency install steps only, as both `NPM_TOKEN` and `NODE_AUTH_TOKEN`. Use a read-only token: npm granular tokens can be read-only, classic automation tokens can publish. |
264
- | the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | the build and `delete-old-builds` steps only |
265
- | `TESTER_APIFY_TOKEN` | the vitest step, and the Claude investigation step: the Apify MCP server's bearer token for reading the failing run, its log and its storages, and Claude's token for re-running it to measure how often the failure happens |
266
- | `SLACK_TOKEN_TESTS_BOT` / `SLACK_TOKEN_RELEASES_BOT` | the reporting and announcement steps only |
267
-
268
- The Actor tokens are the one set that cannot be listed in the workflow, because each Actor names its
269
- own token via `tokenEnvVar` in `apify-test-tools.config.json`. Those steps pass
270
- `${{ toJSON(secrets) }}` as `ALL_SECRETS` and run the command through
271
- `.github/scripts/run-with-apify-tokens.mjs`, which reads that same config file to decide which
272
- secrets to pass on:
281
+ and a few critical tests, kept in e.g. `test/platform/hourly`, every hour:
273
282
 
274
283
  ```yaml
275
- - name: Build
276
- shell: bash
277
- env:
278
- ALL_SECRETS: ${{ toJSON(secrets) }}
279
- run: |
280
- jq '{ actors: .functionallyChangedActors }' affected.json \
281
- | node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
282
- npx apify-test-tools build --version-number 0.99
283
- ```
284
-
285
- The wrapper passes only the tokens the config declares and drops `ALL_SECRETS`, so neither `npx` nor
286
- anything under `node_modules` sees the blob. Nothing is written to `$GITHUB_ENV`, so the tokens stay
287
- inside that one command rather than leaking into later steps. Reading the same file
288
- `apify-test-tools` reads means the two can't drift, and a secret that merely looks like an Actor
289
- token is not passed just because of its name.
290
-
291
- A token the config declares but the repo hasn't set is a warning, not a failure: a repo can carry an
292
- Actor whose token isn't configured and still build fine as long as that Actor never changes, and
293
- `apify-test-tools` raises a precise error naming the Actor at the point it actually needs the token.
294
-
295
- `scripts-path` comes from the setup action (give the step `id: setup`) and points at this repo's
296
- `.github/scripts/` directory inside the runner's action checkout, so workflows can run these helpers
297
- without checking this repo out again. The caller's workspace holds the caller's repo, not this one.
298
-
299
- Two tidier-looking alternatives don't work, so don't reach for them:
300
-
301
- - **Exporting to `$GITHUB_ENV`** would let the steps call `npx` directly with no wrapper, but
302
- `$GITHUB_ENV` applies to every later step in the job. In `pr-build-test` the vitest step runs after
303
- the build, so it would inherit Actor tokens it has no use for.
304
- - **Returning the tokens as a step output** would be scoped correctly, but the runner refuses to set
305
- an output whose value contains a registered secret. It logs `Skip output <name> since it may
306
- contain secret` and leaves the output empty, so anything reading it downstream gets nothing.
307
-
308
- The `unitTest` job runs static checks and needs `NPM_TOKEN` only. No job runs `npm ci` or
309
- `pnpm install` with Apify or Slack credentials in scope, so a postinstall script in the dependency
310
- tree cannot read them.
311
-
312
- ## Versioning and releases
313
-
314
- The workflows and the npm package live in one repo but ship on their own schedules:
315
-
316
- | What | Consumers get it through | Ships when |
317
- | -------------------------------------------- | ------------------------------------------ | ----------------------- |
318
- | the workflows and the CLI they call | the `@workflows-v0` tag in `uses:` | every master push |
319
- | the lib (`testActor`, `describe`, matchers…) | `apify-test-tools` in their `package.json` | a stable release on npm |
320
-
321
- The CLI is bundled with its dependencies into `.github/cli/apify-test-tools.mjs` (`npm run bundle`,
322
- run by the pre-commit hook and checked in CI), and the setup action runs it from there. Nothing is
323
- installed for it, so a tag is a complete statement: these workflows and this CLI, with the exact
324
- dependency versions from `package-lock.json` at that commit. The consumer's own `node_modules` is
325
- left alone, so their tests keep the lib version they locked.
326
-
327
- - **Workflow or CLI change** — merge it. `workflows-v0` moves once checks and tests pass, and it goes
328
- live. No release needed, even when a workflow starts using a new CLI feature: both are in the
329
- same commit.
330
- - **Lib change** — merge it, then cut a release when you want it out. Consumers pick it up when
331
- they update `apify-test-tools`.
332
-
333
- The CLI and the lib in one job can be different versions now, so the formats between them are a
334
- contract: the `ACTOR_BUILDS` JSON that `build` writes and the lib reads, and the vitest JSON report
335
- `report-tests` reads. Add fields freely; renaming or removing one needs a new major tag.
336
-
337
- `workflows-v0` moving on every master push means `@workflows-v0` is as live as `@master` was —
338
- there's no staging step. What the tag buys you is a `workflows-v1` for breaking workflow changes,
339
- so repos migrate one at a time instead of all at once, and a way to roll back by pointing the tag at
340
- an earlier commit (**Move major version tag** workflow). Bump `MAJOR_TAG` in
341
- `.github/workflows/_move_major_tag.yaml` to cut the next major; the old tag then freezes where it is
342
- and keeps working, CLI included, since the bundle is part of the frozen commit.
343
-
344
- The tag tracks the **workflows'** contract, not the npm package version. They move independently on
345
- purpose, so `@workflows-v0` is expected to stay `@workflows-v0` after the package reaches 1.0 —
346
- bump it when a workflow breaks its callers, not when the package does. Because `uses:` cannot take an expression, every ref
347
- into this repo repeats the tag literally; `check-major-tag-refs.mjs` fails the build if `MAJOR_TAG`
348
- and those refs disagree, which is the mistake that would otherwise ship silently during a bump.
349
-
350
- ### Testing workflow changes
351
-
352
- - Point [testing-repo-for-github-actions](https://github.com/apify-store/testing-repo-for-github-actions)
353
- at your branch (`uses: ...@your-branch`). It has real attached Actors and tests. The branch carries
354
- its own CLI bundle, so one branch tests the workflows and the CLI together.
355
- - To change the composite action itself, repoint the `uses:` refs inside the reusable workflows at
356
- your branch as well, and change them back before merging.
357
- - Make sure the shell code actually works on your laptop first.
358
- - After merging, watch the workflow on a real project before moving on.
359
-
360
- ## Writing tests
361
-
362
- ### Test structure
363
-
364
- `testActor` runs the actor and provides extended `expect` and `run` inside the callback.
365
-
366
- ```ts
367
- import { describe, testActor } from 'apify-test-tools';
368
-
369
- describe('test', () => {
370
- testActor(actorId, 'actor test 1', async ({ expect, run }) => {
371
- const runResult = await run({ input });
372
-
373
- // your checks
374
- });
375
-
376
- testActor(actorId, 'actor test 2', async ({ expect, run }) => {
377
- const runResult = await run({ input });
378
-
379
- // your checks
380
- });
381
- });
382
- ```
383
-
384
- ---
385
-
386
- ### Validating basic run attributes
387
-
388
- `toFinishWith` validates common run properties in a single call:
389
-
390
- ```ts
391
- await expect(runResult).toFinishWith({
392
- datasetItemCount: 100,
393
- });
394
- ```
395
-
396
- You can also specify a range:
397
-
398
- ```ts
399
- await expect(runResult).toFinishWith({
400
- datasetItemCount: { min: 80, max: 120 },
401
- });
402
- ```
403
-
404
- Here is full example of what you can validate with `toFinishWith`
405
-
406
- ```ts
407
- await expect(runResult).toFinishWith({
408
- // These are default
409
- status: 'SUCCEEDED',
410
- duration: {
411
- min: 600, // 0.6 sec
412
- max: 600_000, // 10 min
413
- },
414
- failedRequests: 0,
415
- requestsRetries: { max: 3 },
416
- forbiddenLogs: ['ReferenceError', 'TypeError'],
417
-
418
- // only datasetItemCount is required
419
- datasetItemCount: { min: 80, max: 120 },
420
-
421
- // optional
422
- chargedEventCounts: {
423
- 'actor-start': 1,
424
- 'place-scraped': 9,
425
- },
426
- });
427
- ```
428
-
429
- ---
430
-
431
- ### Custom validations
432
-
433
- ```ts
434
- expect(place.title, `London Eye's title`).toEqual('lastminute.com London Eye');
435
- ```
436
-
437
- ---
438
-
439
- ### Custom validation functions
440
-
441
- You can create your own functions wrapping a common validation logic in e.g. `test/platform/utils.ts` and import it in test files.
442
-
443
- ```ts
444
- import { ExpectStatic } from 'apify-test-tools'
445
-
446
- export const validateItem = (expect: ExpectStatic, item: any) {
447
- expect(item.title, 'Item title').toBeString();
448
- }
449
- ```
450
-
451
- ---
452
-
453
- ### Test options
454
-
455
- You can pass options as the fourth argument to `testActor`:
456
-
457
- ```ts
458
- testActor(
459
- actorId,
460
- 'slow actor test',
461
- async ({ expect, run }) => {
462
- const runResult = await run({ input });
463
- await expect(runResult).toFinishWith({ datasetItemCount: 100 });
464
- },
465
- {
466
- timeout: 2 * 60 * 60 * 1000, // 2 hours (default is 1 hour)
467
- retry: 3, // retry up to 3 times (default is 1)
468
- },
469
- );
470
- ```
471
-
472
- ---
473
-
474
- ### Using prefilled input
284
+ name: Platform tests - Hourly
475
285
 
476
- If the actor has a prefilled input on the platform, you can merge it with your test input:
286
+ on:
287
+ schedule:
288
+ # Runs at the start of every hour
289
+ - cron: '0 * * * *'
290
+ workflow_dispatch:
477
291
 
478
- ```ts
479
- testActor(actorId, 'with prefilled input', async ({ expect, run }) => {
480
- const runResult = await run({
481
- prefilledInput: true,
482
- input: { maxItems: 10 }, // merged on top of the prefilled input
483
- });
484
- await expect(runResult).toFinishWith({ datasetItemCount: 10 });
485
- });
292
+ jobs:
293
+ platformTestsHourly:
294
+ uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
295
+ with:
296
+ test-files-glob: hourly
297
+ secrets:
298
+ TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
299
+ SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}
486
300
  ```
487
301
 
488
- ---
489
-
490
- ### Testing an existing run
302
+ ### Secrets
491
303
 
492
- You can skip starting a new run and validate an existing one by passing `runId`:
304
+ Set these as repository or organization secrets and pass them in each job's `secrets:` block, as in the examples
305
+ above. The Actor tokens go packed in one JSON object, `CONFIG_SECRETS`, because their names differ per repo:
493
306
 
494
- ```ts
495
- testActor(actorId, 'validate existing run', async ({ expect, run }) => {
496
- const runResult = await run({ runId: 'some-run-id' });
497
- await expect(runResult).toFinishWith({ datasetItemCount: 100 });
498
- });
499
- ```
307
+ | Secret | Needed for |
308
+ | ------------------------------------------------------------------------- | ---------------------------------------------------------------- |
309
+ | the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | building Actors and deleting their old builds |
310
+ | `TESTER_APIFY_TOKEN` | running the platform tests |
311
+ | `SLACK_TOKEN_RELEASES_BOT` | announcing releases on Slack |
312
+ | `SLACK_TOKEN_TESTS_BOT` | reporting scheduled test results to Slack |
313
+ | `NPM_TOKEN` | optional, installing private npm packages. Use a read-only token |
500
314
 
501
- ---
315
+ Repos in the `apify` org can pass `secrets: inherit` instead. Each secret only reaches the step that needs it. See
316
+ [Secrets](docs/workflows.md#secrets) for both, and for passing `CONFIG_SECRETS` without listing the tokens.
502
317
 
503
- ### Accessing run data
318
+ ## Writing tests
504
319
 
505
- `RunTestResult` provides methods to access the run's data:
320
+ Platform tests use `describe` and `testActor` from `apify-test-tools` instead of the ones from `vitest`. `testActor`
321
+ registers a test that only runs when that Actor was built for it (or when you ask for all tests), and gives the
322
+ callback an extended `expect` and a `run` function that starts the Actor.
506
323
 
507
324
  ```ts
508
- testActor(actorId, 'check dataset items', async ({ expect, run }) => {
509
- const runResult = await run({ input });
510
-
511
- // Access dataset items
512
- const { items } = await runResult.getDataset();
513
- expect(items[0].title).toBeNonEmptyString();
514
-
515
- // Access run log
516
- const log = await runResult.getLog();
517
- expect(log).toContain('Crawl finished');
518
-
519
- // Access crawler statistics
520
- const stats = await runResult.getStatistics();
521
- expect(stats?.requestsFinished).toBeGreaterThan(0);
522
-
523
- // Access key-value store
524
- const kvs = runResult.getKeyValueStoreClient();
525
- const record = await kvs.getRecord('OUTPUT');
526
-
527
- // Access run info (refreshed from API)
528
- const runInfo = await runResult.getRunInfo();
529
- });
530
- ```
531
-
532
- ---
325
+ import { describe, testActor } from 'apify-test-tools';
533
326
 
534
- ### Testing standby actors
327
+ describe('web-scraper', () => {
328
+ testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
329
+ const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
535
330
 
536
- Use `testStandbyActor` for actors that support standby mode:
331
+ // Status SUCCEEDED, a sane duration, no failed requests, no errors in the log, and the item count
332
+ await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
537
333
 
538
- ```ts
539
- import { describe, testStandbyActor } from 'apify-test-tools';
540
-
541
- describe('standby tests', () => {
542
- testStandbyActor(actorId, 'standby request', async ({ expect, callStandby }) => {
543
- const { data, status } = await callStandby({
544
- input: { query: 'test' },
545
- path: '/search',
546
- headers: { 'Content-Type': 'application/json' },
547
- });
548
-
549
- expect(status).toBe(200);
550
- expect(data.results).toBeNonEmptyArray();
334
+ const { items } = await runResult.getDataset();
335
+ expect(items[0].title, 'Page title').toBeNonEmptyString();
551
336
  });
552
337
  });
553
338
  ```
554
339
 
555
- ---
556
-
557
- ### Custom matchers
558
-
559
- `testActor` extends `expect` with the following custom matchers:
560
-
561
- - `toBeArray()` / `toBeEmptyArray()` / `toBeNonEmptyArray()`
562
- - `toBeString()` / `toBeNonEmptyString()` / `toStartWith(prefix)`
563
- - `toBeNumber()` / `toBeWholeNumber()` / `toBeWithinRange(min, max)`
564
- - `toBeBoolean()` / `toBeTrue()` / `toBeFalse()`
565
- - `toBeObject()` / `toBeNonEmptyObject()`
566
- - `toFinishWith(options)` - validates run status, duration, dataset, logs, etc.
567
-
568
- ## CLI (`apify-test-tools` bin)
569
-
570
- The package includes a CLI binary used by CI workflows to build Actors, detect changes, and report test results. You can also run it locally.
571
-
572
- ### Dry runs (`--dry-run`)
573
-
574
- Every command that changes something (`build`, `build-from-local`, `notify-release`, `delete-old-builds`, `report-tests`) takes `--dry-run`. A dry run:
575
-
576
- - **Changes nothing.** No versions or builds are created, nothing is uploaded or deleted, and no Slack message is posted.
577
- - **Does every read and check the real run does**, so it fails wherever the real run would: change detection, the Apify token and Actor lookup, the [repository check](#repository-check), collecting and zipping local sources, and the Slack token (verified with Slack's read-only `auth.test`).
578
- - **Needs the same tokens as the real run.** Read-only tokens are enough, since nothing is written. A dry run can't tell whether a token is also allowed to write, because the Apify API has no dry-run mode of its own.
579
- - **Prints a plan, not results.** Logs start with `[DRY RUN] Would …`. `build` and `build-from-local` write `{ "dryRun": true, "builds": [...] }` to stdout instead of the build array, so the plan can't be passed to tests as `ACTOR_BUILDS` by mistake.
580
-
581
- ### Running locally
340
+ Platform tests are skipped unless `TESTER_APIFY_TOKEN` is set, so your regular unit test run stays unaffected.
582
341
 
583
- Running the testing library locally is useful when you only want to update the testing code in /test because you can iterate on it without pushing new code to the remote.
584
-
585
- If you don't need to change any source files and only iterate on /test code, you can skip steps 1-4. But if you want to test vs changed /src, you have to push that GitHub branch since it needs to build the Actors with that code.
586
-
587
- The main local flow is:
588
-
589
- 1. Switch to a dummy branch that you will push and can later delete
590
- 2. `npm i apify-test-tools@latest -D`
591
- 3. Push your code (changes you want to test)
592
- 4. Build Actors on Apify (with your new code)
593
- 5. Run tests against those builds. You can change tests and run on the same builds.
594
-
595
- Run every command from the root of the actor repository you want to work with.
596
-
597
- #### 4. Build affected Actors
598
-
599
- If you want to test vs existing src code, you can skip this and instead construct the output JSON manually from existing builds only for the Actors you need to test.
600
-
601
- Requires `APIFY_TOKEN_<USERNAME>` for all Apify users that own your Actors (e.g. `apify`, `compass`, `lukaskrivka` users). The username is derived from the actor name — uppercased with non-word chars replaced by `_` (e.g. Actor `john.doe/my-actor` needs `APIFY_TOKEN_JOHN_DOE`).
342
+ To run them locally against the Actors' current default builds:
602
343
 
603
344
  ```bash
604
- npx apify-test-tools get-affected-actors --base origin/master \
605
- | jq '{ actors: .functionallyChangedActors }' \
606
- | APIFY_TOKEN_JOHN_DOE=<token> npx apify-test-tools build --version-number 0.99 --dry-run
345
+ RUN_ALL_PLATFORM_TESTS=1 TESTER_APIFY_TOKEN=<token> npx vitest --run test/platform
607
346
  ```
608
347
 
609
- Run it with your dummy branch checked out: it builds the Actors that branch changed functionally, from the branch
610
- as it is on the remote, into the test version `0.99`. With `--dry-run`, it prints the plan (the version and Git URL
611
- each Actor would be built from). Remove `--dry-run` to actually trigger builds, and update `--base` to the branch
612
- you compare against. `build` outputs a JSON array of build objects to stdout:
613
-
614
- ```json
615
- [{ "buildId": "...", "actorRawId": "...", "buildNumber": "...", "actorFullName": "john.doe/my-actor" }]
616
- ```
617
-
618
- #### Build from local source (no push needed)
619
-
620
- If you don't want to push a dummy branch just to test a change and wait for all the tests to finish, `build-from-local` builds Actors directly from your local files (zipped into the Actor's `actor-<id>-source` key-value store and built as `TARBALL`, like `apify push`), skipping steps 1-4 above.
621
-
622
- ```bash
623
- APIFY_TOKEN_JOHN_DOE=<token> \
624
- npx apify-test-tools build-from-local --actors john.doe/my-actor
625
- ```
626
-
627
- Pass a hardcoded actor name via `--actors` to build only that Actor (comma-separate multiple names). Omit `--actors` to build all Actors in the repo, or add `--dry-run` to check the token, the Actor and the zipped sources without uploading or building. It outputs the same JSON build array as `build`, so you run tests against it the same way as in step 5 below:
628
-
629
- ```bash
630
- # Build from local source and capture output
631
- BUILDS=$(APIFY_TOKEN_JOHN_DOE=apify_api_xxx \
632
- npx apify-test-tools build-from-local --actors apify/my-actor)
633
- ```
634
-
635
- Since you already scoped the build to just the Actor(s) you care about, point vitest at a specific test file (or a `-t` name filter) instead of the whole `test/platform` directory — you get feedback on that one test without waiting for the full suite to run.
636
-
637
- #### 5. Run tests against the builds
638
-
639
- Pass the build output as `ACTOR_BUILDS` and provide `TESTER_APIFY_TOKEN`. The token can point to your own account (if you have enough memory) or you can use the testing account (xRGg9iAfJSymqartk). Platform test suites are skipped unless `TESTER_APIFY_TOKEN` is set, so regular unit test runs stay unaffected.
640
-
641
- If you want to run only certain tests, change the `test/platform` to be more specific.
642
-
643
- ```bash
644
- ACTOR_BUILDS='<JSON output from build command>' \
645
- TESTER_APIFY_TOKEN=<token> \
646
- npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform
647
- ```
648
-
649
- #### Full example
650
-
651
- Tests only run for Actors in `ACTOR_BUILDS`, so a custom workflow has to build (and pass) the Actors even when only tests changed. Pass `--test-files-glob` to `get-affected-actors` so test changes count even when `.dockerignore` lists them.
652
-
653
- ```bash
654
- # Build and capture output
655
- BUILDS=$(npx apify-test-tools get-affected-actors --base origin/master --test-files-glob 'test/platform/**' \
656
- | jq '{ actors: .functionallyChangedActors }' \
657
- | APIFY_TOKEN_JOHN_DOE=apify_api_xxx npx apify-test-tools build --version-number 0.99)
658
-
659
- # Run tests with the builds
660
- ACTOR_BUILDS="$BUILDS" \
661
- TESTER_APIFY_TOKEN=apify_api_yyy \
662
- npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform
663
- ```
664
-
665
- ### Finding affected Actors (`get-affected-actors`)
666
-
667
- `get-affected-actors` lists the Actors whose context changed in a commit range, using the rules in
668
- [Change detection](#change-detection). It only reads git and the config file, so it needs no tokens.
669
- The range always ends at the checked-out `HEAD`, so check out the branch you want to inspect first.
670
-
671
- ```bash
672
- # A PR: the branch's own changes, narrowed to the Actors that changed since its last validated commit
673
- npx apify-test-tools get-affected-actors --base origin/master --since <sha>
674
-
675
- # A release: everything after the last released commit
676
- npx apify-test-tools get-affected-actors --since <sha>
677
- ```
678
-
679
- | Flag | Default | Meaning |
680
- | ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
681
- | `--base` | | Only what `HEAD` changes compared to this branch counts, like `git diff <base>...HEAD` |
682
- | `--since` | | Only changes after this commit count (see below) |
683
- | `--test-files-glob` | | Test files (glob from the repo root) that count as changes even when `.dockerignore` lists them, see [Change detection](#change-detection) |
684
- | `--actors` | all | Only report these Actors |
685
- | `--ignore` | none | Never report these Actors |
686
-
687
- Pass `--base`, `--since`, or both:
688
-
689
- - **`--base` only** inspects what the branch changes, the same files GitHub's "Files changed" tab shows: the diff
690
- from where the branch forked from `--base`, or the last commit of `--base` it merged in. What it merged in from
691
- `--base` is already validated there, so it doesn't count.
692
- - **`--base` and `--since`** also inspects what changed after the branch's last validated commit, and reports only
693
- the Actors that changed in both. An Actor in only one of them is already validated, by this branch or on `--base`.
694
- When both changed an Actor, the changes can interact, so it is reported, but as functional only when both changes
695
- are functional. A validated functional change followed by README edits is cosmetic, and so is a functional
696
- change merged in from `--base` that meets only cosmetic changes of the branch. `--since` doesn't have to be on
697
- the branch, so a rebase only re-checks the Actors that `--base` changed too. If `--since` is not in the local
698
- history (a force-push), or it is `HEAD` (a rerun), it is ignored and the whole branch is checked.
699
- - **`--since` only** inspects everything after it up to `HEAD`. It must be an ancestor of `HEAD` and fails
700
- otherwise, see [Releasing Actors](#releasing-actors). If `HEAD` is `--since`, nothing is
701
- reported.
702
-
703
- `commits`, `changedFiles` and `changelog` describe the whole range, `--base...HEAD` or `--since..HEAD`: `--since`
704
- together with `--base` only narrows the Actor lists.
705
-
706
- The output is a JSON document:
707
-
708
- ```json
709
- {
710
- "version": 1,
711
- "head": "<sha of HEAD>",
712
- "commits": [{ "sha": "…", "author": "…", "date": "…", "message": "…" }],
713
- "changedFiles": ["src/main.ts", "CHANGELOG.md"],
714
- "changelog": "Lines added to CHANGELOG.md, or null",
715
- "functionallyChangedActors": [{ "actorFullName": "john.doe/my-actor" }],
716
- "cosmeticallyChangedActors": [{ "actorFullName": "john.doe/other-actor" }]
717
- }
718
- ```
719
-
720
- Each changed Actor is in exactly one of the lists. `functionallyChangedActors` have at least one functional change, so
721
- they need a new build and tests. `cosmeticallyChangedActors` only have cosmetic changes (see
722
- [Change detection](#change-detection)), so they need a new build only when released: the PR workflow builds
723
- `functionallyChangedActors`, the release workflow builds both.
724
-
725
- ### Building Actors (`build`)
726
-
727
- `build` builds exactly the Actors it is given on stdin, from the checked-out branch as it is on the remote, and
728
- prints the builds as the same JSON array `build-from-local` prints. A detached HEAD is an
729
- error, since there is no branch to build from. It doesn't look at git history: which Actors to build is
730
- up to you, usually picked from the output of [`get-affected-actors`](#finding-affected-actors-get-affected-actors).
731
-
732
- ```bash
733
- # A PR: the Actors a branch changed functionally, built into the test version
734
- npx apify-test-tools get-affected-actors --base origin/master \
735
- | jq '{ actors: .functionallyChangedActors }' \
736
- | npx apify-test-tools build --version-number 0.99
737
-
738
- # A release: every Actor changed since the last release, into the version users run
739
- npx apify-test-tools get-affected-actors --since <sha> \
740
- | jq '{ actors: (.functionallyChangedActors + .cosmeticallyChangedActors) }' \
741
- | npx apify-test-tools build --version-number default
742
-
743
- # One Actor, whatever changed
744
- echo '{ "actors": [{ "actorFullName": "john.doe/my-actor" }] }' | npx apify-test-tools build --version-number 0.99
745
- ```
746
-
747
- The input is `{ "actors": [{ "actorFullName": "<owner/name>" }] }`; other fields are ignored. An empty `actors` list builds nothing and prints `[]`. Empty or invalid input is an error,
748
- so a failed command earlier in the pipe can't turn into a successful run that built nothing. In GitHub
749
- Actions, give such a step `shell: bash` so the pipe also fails on its own (the default shell has no `pipefail`).
750
-
751
- | Flag | Default | Meaning |
752
- | ------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------------------- |
753
- | `--version-number` | required | Version to build, like `0.99`, or `default` for each Actor's default version (the one its default build tag points to) |
754
- | `--repo-url` | the `origin` remote | Repository to build from, see [Repository check](#repository-check) |
755
-
756
- `build` never sets or changes build tags. A build gets the tag its version is configured with on the platform,
757
- so building the default version moves its tag (usually `latest`) to the new build, and building a version
758
- without a tag leaves every tag where it is. Builds always use the Docker cache.
759
-
760
- ### Releasing Actors
761
-
762
- A release is three commands: find what changed since the last released commit, build those Actors into
763
- their default version (the one users run), and announce it.
764
-
765
- ```bash
766
- npx apify-test-tools get-affected-actors --since <sha> > affected.json
767
-
768
- jq '{ actors: (.functionallyChangedActors + .cosmeticallyChangedActors) }' affected.json \
769
- | npx apify-test-tools build --version-number default --dry-run
770
-
771
- npx apify-test-tools notify-release --release-slack-channel "#releases" --dry-run < affected.json
772
- ```
773
-
774
- `notify-release` reads the output of `get-affected-actors` on stdin. It posts the changelog additions to
775
- `--release-slack-channel` and the commits and changed files to `--report-slack-channel`, and posts nothing when
776
- there are no new commits. It names the release after the `origin` remote (or `--repo-url`) and the author of the
777
- newest commit, and needs `SLACK_TOKEN_RELEASES_BOT`.
778
-
779
- #### The last released commit (`--since`)
780
-
781
- The last commit that is **already released**. Everything after it, up to `HEAD`, is released. Git alone
782
- can't tell which commits a push brought, and guessing wrong silently leaves Actors on old code, so it has
783
- to be passed.
784
-
785
- The changed files come from diffing `--since` and `HEAD` directly, so this works for every merge strategy as
786
- long as `--since` is the branch tip from **before** the push:
787
-
788
- | Merge strategy | What lands on the branch | What to pass |
789
- | ------------------------------ | --------------------------------- | ---------------------------------------------------------------------- |
790
- | Squash and merge | 1 new commit | the push's "before" commit (`HEAD~1` also works) |
791
- | Merge commit | the PR's commits + 1 merge commit | the push's "before" commit (`HEAD^1`, the first parent, also works) |
792
- | Rebase and merge / direct push | N new commits | the push's "before" commit (`HEAD~1` would miss the first N-1 commits) |
793
-
794
- On GitHub, the push's "before" commit is `${{ github.event.before }}`, which the reusable workflow
795
- passes for you. On other CIs use the equivalent, e.g. `$CI_COMMIT_BEFORE_SHA` on GitLab. The checkout
796
- needs the full history (`fetch-depth: 0` in `actions/checkout`) so the commit is available locally.
797
-
798
- `get-affected-actors --since` without `--base` fails instead of guessing when:
799
-
800
- - the commit is all zeros (GitHub sends this for a push that created the branch)
801
- - the commit isn't in the local history (shallow checkout) or isn't an ancestor of `HEAD`
802
- (the branch was force-pushed)
803
-
804
- In those cases, rerun it with a `--since` you pick yourself: an ancestor of `HEAD` from before the changes
805
- you want to release (for a shallow checkout, fetch the full history instead). `--actors` only narrows which
806
- of the changed Actors are reported; it never forces a build, so on its own it doesn't get past these errors.
807
- If `HEAD` is `--since`, nothing is reported, so nothing is built or announced.
808
-
809
- A failed release isn't retried by the next push: that push's base commit is the failed one's head, so
810
- its changes are skipped. Re-run the failed workflow run, or release the Actors manually.
811
-
812
- #### Repository check
813
-
814
- `build` points each Actor's version at the `origin` remote. Because a build overwrites the
815
- version's Git URL, building from a fork or a mirror would repoint the Actor. So, before building, the
816
- remote is compared with the Git URL of the Actor's default version (the one its default build tag
817
- points to), and the command fails if they differ or if the default version isn't built from a Git
818
- repository. The URL form doesn't matter (`https://github.com/org/repo` equals
819
- `git@github.com:org/repo.git`).
820
-
821
- To build from a different repository on purpose, e.g. when moving an Actor, pass `--repo-url`. That
822
- skips the check.
823
-
824
- #### Dev mode
825
-
826
- For development on `apify-test-tools` itself, use `tsx` directly:
827
-
828
- ```bash
829
- cd local-clone && tsx ../bin/main.ts get-actor-configs
830
- ```
348
+ The [Vitest library API](docs/lib.md) covers everything `toFinishWith` checks, the other matchers, standby Actors,
349
+ reading run data, and running tests locally against builds of your own changes.