apify-test-tools 0.10.2-beta.8 → 0.10.2-beta.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/actions/checkout-restore-dependencies/action.yaml +13 -0
- package/.github/cli/apify-test-tools.mjs +1 -1
- package/.github/scripts/check-major-tag-refs.mjs +1 -1
- package/.github/scripts/mask-config-secrets.sh +17 -0
- package/.github/scripts/run-with-apify-tokens.mjs +36 -11
- package/.github/workflows/public_platform-tests-claude-investigate-and-fix.yaml +6 -2
- package/.github/workflows/public_platform-tests.yaml +23 -0
- package/.github/workflows/public_pr-build-test.yaml +30 -1
- package/.github/workflows/public_push-build-latest.yaml +20 -3
- package/CHANGELOG.md +1 -0
- package/CONTRIBUTING.md +99 -8
- package/README.md +218 -699
- package/bin/git.ts +1 -1
- package/dist/bin/git.js +1 -1
- package/dist/bin/git.js.map +1 -1
- package/dist/test/unit/github-scripts/mask-config-secrets.test.d.ts +2 -0
- package/dist/test/unit/github-scripts/mask-config-secrets.test.d.ts.map +1 -0
- package/dist/test/unit/github-scripts/mask-config-secrets.test.js +50 -0
- package/dist/test/unit/github-scripts/mask-config-secrets.test.js.map +1 -0
- package/dist/test/unit/github-scripts/run-with-apify-tokens.test.d.ts +2 -0
- package/dist/test/unit/github-scripts/run-with-apify-tokens.test.d.ts.map +1 -0
- package/dist/test/unit/github-scripts/run-with-apify-tokens.test.js +102 -0
- package/dist/test/unit/github-scripts/run-with-apify-tokens.test.js.map +1 -0
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/docs/cli.md +358 -0
- package/docs/lib.md +271 -0
- package/docs/workflows.md +366 -0
- package/package.json +1 -1
- package/test/unit/github-scripts/mask-config-secrets.test.ts +61 -0
- package/test/unit/github-scripts/run-with-apify-tokens.test.ts +115 -0
package/README.md
CHANGED
|
@@ -1,8 +1,71 @@
|
|
|
1
1
|
# Apify Test Tools
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Build, test and release Apify Actors from a Git repository or local source code. The package has three parts:
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
1. **A vitest library** for writing platform tests that run your Actors on Apify and check the results:
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
|
|
9
|
+
const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
|
|
10
|
+
await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
|
|
11
|
+
});
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
2. **A CLI** that finds the Actors a change affects, builds them, and reports the results. It runs anywhere, so you
|
|
15
|
+
can build your own pipeline on it in any CI:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
npx apify-test-tools get-affected-actors --base origin/master \
|
|
19
|
+
| jq '{ actors: .functionallyChangedActors }' \
|
|
20
|
+
| npx apify-test-tools build --version-number 0.99
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
3. **Ready-made GitHub Actions workflows** built on the CLI and the library, so a GitHub repo needs only a few lines
|
|
24
|
+
per workflow:
|
|
25
|
+
|
|
26
|
+
```yaml
|
|
27
|
+
jobs:
|
|
28
|
+
buildDevelAndTest:
|
|
29
|
+
uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
|
|
30
|
+
secrets:
|
|
31
|
+
TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
|
|
32
|
+
# One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
|
|
33
|
+
CONFIG_SECRETS: |
|
|
34
|
+
{ "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}" }
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Most repos only need the library and the workflows: the workflows call the CLI for you. Use the CLI directly only to
|
|
38
|
+
build your own pipeline, e.g. on a CI platform other than GitHub, or to build and test locally.
|
|
39
|
+
|
|
40
|
+
## Only what changed is retested
|
|
41
|
+
|
|
42
|
+
Builds and platform tests cost time and money, so the tool skips everything a change can't affect:
|
|
43
|
+
|
|
44
|
+
- **Only affected Actors.** A changed file only counts for the Actors whose build context it falls in, minus their
|
|
45
|
+
`.dockerignore`. In a monorepo, a change to one Actor doesn't build or test the others.
|
|
46
|
+
- **Cosmetic changes skip tests.** README, CHANGELOG and formatting-only schema changes don't trigger tests on a PR;
|
|
47
|
+
they're only rebuilt when released.
|
|
48
|
+
- **Only new changes on a PR.** Each push to a PR retests only the Actors that changed since its last green run, not
|
|
49
|
+
the whole PR again.
|
|
50
|
+
- **Only their tests.** Only the tests of the Actors that were built run.
|
|
51
|
+
|
|
52
|
+
See [Change detection](docs/cli.md#change-detection) for the exact rules.
|
|
53
|
+
|
|
54
|
+
This README covers what you need to get a repo running. The full reference lives in `docs/`:
|
|
55
|
+
|
|
56
|
+
- [Vitest library API](docs/lib.md) — `testActor`, `testStandbyActor`, `RunTestResult`, matchers, running tests locally
|
|
57
|
+
- [CLI API](docs/cli.md) — every command, change detection, releasing, building locally
|
|
58
|
+
- [Workflow API](docs/workflows.md) — every GitHub workflow, its inputs and secrets, versioning
|
|
59
|
+
|
|
60
|
+
[Contributing](CONTRIBUTING.md)
|
|
61
|
+
|
|
62
|
+
## Getting started
|
|
63
|
+
|
|
64
|
+
Your Actors can be written in any programming language, e.g. Python: anything the platform builds from a Dockerfile
|
|
65
|
+
works. Only the tests and this tooling run on Node.js, so all a repo needs is a `package.json` at the root with
|
|
66
|
+
`apify-test-tools` and `vitest` in its `devDependencies`, and tests written in TypeScript or JavaScript. If you use the
|
|
67
|
+
[PR workflow](docs/workflows.md#public_pr-build-testyaml), also define the `build-check`, `lint` and `test` scripts
|
|
68
|
+
its static checks run, e.g. with your language's own linter and unit tests.
|
|
6
69
|
|
|
7
70
|
### 1. Install the package
|
|
8
71
|
|
|
@@ -10,118 +73,87 @@
|
|
|
10
73
|
npm i -D apify-test-tools
|
|
11
74
|
```
|
|
12
75
|
|
|
13
|
-
- Requires `vitest` version `3.2.
|
|
76
|
+
- Requires `vitest` version `3.2.4` or later (uses [annotate](https://vitest.dev/guide/test-context.html#annotate))
|
|
14
77
|
- Make sure `target` and `module` in your `tsconfig.json`'s `compilerOptions` are set to `ES2022`
|
|
15
78
|
- Every Actor must already have at least one build under its default build tag (usually `latest`), built
|
|
16
79
|
from a Git repository version whose URL points to this repository. Build it once manually on the
|
|
17
|
-
platform before CI takes over. See [Repository check](#repository-check).
|
|
80
|
+
platform before CI takes over. See [Repository check](docs/cli.md#repository-check).
|
|
18
81
|
|
|
19
82
|
### 2. Create the config file
|
|
20
83
|
|
|
21
|
-
Every repo
|
|
84
|
+
Every repo must have an `apify-test-tools.config.json` file at the root that lists its Actors. See
|
|
85
|
+
[Config file](#config-file) below.
|
|
22
86
|
|
|
23
|
-
|
|
87
|
+
### 3. Set up Actor folders
|
|
24
88
|
|
|
25
|
-
|
|
89
|
+
Each Actor in the config must have a `.actor/actor.json` file. The `dockerContextDir` field in `actor.json` defines
|
|
90
|
+
the build context boundary — this is what the tool uses to decide which files can affect the Actor's build (see
|
|
91
|
+
[Change detection](docs/cli.md#change-detection)).
|
|
26
92
|
|
|
27
|
-
|
|
93
|
+
This example has two Actors that share the code at the repo root, and one standalone Actor with its own code:
|
|
28
94
|
|
|
29
95
|
```
|
|
30
96
|
my-repo
|
|
31
97
|
├── apify-test-tools.config.json
|
|
98
|
+
├── Dockerfile <- shared by web-scraper and email-sender
|
|
99
|
+
├── package.json
|
|
100
|
+
├── src/ <- shared by web-scraper and email-sender
|
|
32
101
|
├── actors
|
|
33
102
|
│ ├── web-scraper
|
|
34
|
-
│ │
|
|
35
|
-
│ │
|
|
36
|
-
│ │ └── src/
|
|
103
|
+
│ │ └── .actor
|
|
104
|
+
│ │ └── actor.json <- points to the repo root
|
|
37
105
|
│ └── email-sender
|
|
106
|
+
│ └── .actor
|
|
107
|
+
│ └── actor.json <- points to the repo root
|
|
108
|
+
├── standalone-actors
|
|
109
|
+
│ └── orchestrator
|
|
38
110
|
│ ├── .actor
|
|
39
|
-
│ │ └── actor.json
|
|
111
|
+
│ │ └── actor.json <- default context: its own folder
|
|
112
|
+
│ ├── Dockerfile
|
|
113
|
+
│ ├── package.json
|
|
40
114
|
│ └── src/
|
|
41
115
|
└── test
|
|
42
|
-
├── unit
|
|
43
116
|
└── platform
|
|
44
|
-
├──
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
117
|
+
├── web-scraper.test.ts
|
|
118
|
+
├── email-sender.test.ts
|
|
119
|
+
└── orchestrator
|
|
120
|
+
└── orchestrator.test.ts
|
|
48
121
|
```
|
|
49
122
|
|
|
50
|
-
|
|
123
|
+
The Actors that share code point their `dockerfile` and `dockerContextDir` at the repo root. Both paths are relative
|
|
124
|
+
to the `.actor` folder:
|
|
51
125
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
5. **`.dockerignore` filtering** — if a `.dockerignore` exists at the root of the actor's `dockerContextDir`, matching files are ignored. Patterns are resolved relative to `dockerContextDir`, matching Docker's own behavior. Test files are the exception: `get-affected-actors --test-files-glob <glob>` (relative to the repo root) skips this check for matching files, so a changed test triggers a build, and with it its tests, even when `.dockerignore` lists it. The PR workflow passes `test/platform/**`. Context matching still applies, so tests outside an actor's `dockerContextDir` (e.g. a root `test/` with standalone actors) need to be listed in its `overrideActorContext`; `get-affected-actors` logs a warning for changed test files outside every actor's context.
|
|
61
|
-
6. **README classification** — a `README.md` file is `cosmetic` (only triggers a release build, not tests) if it lives inside the actor's own `folder`; otherwise it's ignored entirely, since it isn't documentation for this actor.
|
|
62
|
-
7. **Cosmetic JSON classification** — `.json` files inside the actor's own `.actor/` directory with only cosmetic schema changes (whitespace, key ordering) only trigger a release build.
|
|
63
|
-
8. **Functional** — everything else triggers both build and tests.
|
|
64
|
-
|
|
65
|
-
### 4. Create test directories
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
mkdir -p test/platform/core
|
|
126
|
+
```json
|
|
127
|
+
{
|
|
128
|
+
"actorSpecification": 1,
|
|
129
|
+
"name": "web-scraper",
|
|
130
|
+
"version": "0.1",
|
|
131
|
+
"dockerfile": "../../../Dockerfile",
|
|
132
|
+
"dockerContextDir": "../../.."
|
|
133
|
+
}
|
|
69
134
|
```
|
|
70
135
|
|
|
71
|
-
|
|
72
|
-
|
|
136
|
+
A change to the shared `src/` then builds and tests both of them, while a change in `standalone-actors/orchestrator`
|
|
137
|
+
counts only for the orchestrator, even though it lies inside the others' context. The orchestrator leaves
|
|
138
|
+
`dockerContextDir` out, so its context is its own folder.
|
|
73
139
|
|
|
74
|
-
|
|
140
|
+
For a single-Actor repo, set `"folder": "."` in the config and place `.actor/actor.json` at the repo root.
|
|
75
141
|
|
|
76
|
-
|
|
142
|
+
### 4. Write tests
|
|
77
143
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
`apify-test-tools.config.json` lives at the repo root and tells the tool which actors live in the repo, how to identify them, and which token to use.
|
|
81
|
-
|
|
82
|
-
### Actor fields
|
|
83
|
-
|
|
84
|
-
Every actor in the config is described by these fields, whatever the mode:
|
|
144
|
+
Put platform tests in `test/platform`. How you organize them inside is up to you. See
|
|
145
|
+
[Writing tests](#writing-tests) below.
|
|
85
146
|
|
|
86
|
-
|
|
87
|
-
| ---------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
88
|
-
| `folder` | yes | Relative path from repo root to the actor's own project directory — the folder that directly contains `.actor/actor.json` (i.e. `<folder>/.actor/actor.json`), the actor's README/CHANGELOG, and its source. Use `"."` for a single-actor repo where `.actor/` is at the root. |
|
|
89
|
-
| `actorFullName` | yes | Full actor identifier in `owner/name` format (e.g. `"apify/web-scraper"`). This is the source of truth for the actor name — the `name` field in `actor.json` is not used. |
|
|
90
|
-
| `tokenEnvVar` | yes | Name of the environment variable holding the Apify API token for this actor. In [grouped mode](#grouped-mode) it's inherited from the group unless the actor sets its own. No fallback — if the env var is not set at build time, the build fails. |
|
|
91
|
-
| `overrideActorContext` | no | Array of paths (relative to repo root) that define which files are relevant to this actor. When set, replaces the `dockerContextDir` from `actor.json` for change detection. Useful when an actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. `["", "code"]` or `["actors", "actors/foo"]` are rejected). The actor's own `folder` is always part of its context — if none of the listed entries reach it, it's added automatically. |
|
|
147
|
+
### 5. Set up CI
|
|
92
148
|
|
|
93
|
-
|
|
149
|
+
On GitHub, add the [GitHub workflows](#github-workflows) below. On another CI platform, call the
|
|
150
|
+
[CLI](docs/cli.md) from your own pipeline; the workflows are a good template for the steps.
|
|
94
151
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
### Legacy mode
|
|
98
|
-
|
|
99
|
-
The default. A flat `actors` array where every entry carries all of its settings. Good for repos with a few actors, or where every actor has different settings.
|
|
100
|
-
|
|
101
|
-
```json
|
|
102
|
-
{
|
|
103
|
-
"mode": "legacy",
|
|
104
|
-
"actors": [
|
|
105
|
-
{
|
|
106
|
-
"folder": "actors/web-scraper",
|
|
107
|
-
"actorFullName": "myteam/web-scraper",
|
|
108
|
-
"tokenEnvVar": "APIFY_TOKEN_MYTEAM"
|
|
109
|
-
},
|
|
110
|
-
{
|
|
111
|
-
"folder": "actors/email-sender",
|
|
112
|
-
"actorFullName": "myteam/email-sender",
|
|
113
|
-
"tokenEnvVar": "APIFY_TOKEN_MYTEAM",
|
|
114
|
-
"overrideActorContext": ["actors/email-sender", "packages/shared"]
|
|
115
|
-
}
|
|
116
|
-
]
|
|
117
|
-
}
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
`actors` must hold at least one entry. Every entry takes all the [actor fields](#actor-fields) directly, and `tokenEnvVar` is required on each one.
|
|
121
|
-
|
|
122
|
-
### Grouped mode
|
|
152
|
+
## Config file
|
|
123
153
|
|
|
124
|
-
|
|
154
|
+
`apify-test-tools.config.json` lives at the repo root and tells the tool which Actors live in the repo, where they
|
|
155
|
+
are, and which token builds them. Actors are organized into groups, and each group sets the token once for all of its
|
|
156
|
+
Actors:
|
|
125
157
|
|
|
126
158
|
```json
|
|
127
159
|
{
|
|
@@ -129,72 +161,57 @@ Actors are organized into `groups`, and each group sets `tokenEnvVar` and `overr
|
|
|
129
161
|
"groups": {
|
|
130
162
|
"myteam": {
|
|
131
163
|
"tokenEnvVar": "APIFY_TOKEN_MYTEAM",
|
|
132
|
-
"overrideActorContext": ["packages/shared"],
|
|
133
164
|
"actors": [
|
|
134
165
|
{ "folder": "actors/web-scraper", "actorFullName": "myteam/web-scraper" },
|
|
166
|
+
{ "folder": "actors/email-sender", "actorFullName": "myteam/email-sender" },
|
|
135
167
|
{
|
|
136
|
-
"folder": "actors/
|
|
137
|
-
"actorFullName": "myteam/
|
|
138
|
-
"overrideActorContext": ["actors/
|
|
168
|
+
"folder": "standalone-actors/orchestrator",
|
|
169
|
+
"actorFullName": "myteam/orchestrator",
|
|
170
|
+
"overrideActorContext": ["standalone-actors/orchestrator", "test/platform/orchestrator"]
|
|
139
171
|
}
|
|
140
172
|
]
|
|
141
|
-
},
|
|
142
|
-
"partner": {
|
|
143
|
-
"tokenEnvVar": "APIFY_TOKEN_PARTNER",
|
|
144
|
-
"actors": [{ "folder": "actors/partner-crawler", "actorFullName": "partner/crawler" }]
|
|
145
173
|
}
|
|
146
174
|
}
|
|
147
175
|
}
|
|
148
176
|
```
|
|
149
177
|
|
|
150
|
-
|
|
178
|
+
The group key (`myteam` above) is only a label for readability; it doesn't need to match any folder or username. Each
|
|
179
|
+
group needs a `tokenEnvVar` and at least one Actor. Add another group for Actors that need a different token, e.g.
|
|
180
|
+
Actors owned by another account.
|
|
151
181
|
|
|
152
|
-
|
|
182
|
+
The orchestrator sets `overrideActorContext` because its tests live outside its own folder: without it, a change to
|
|
183
|
+
only `test/platform/orchestrator` wouldn't build and test it. The other two already have the whole repo, tests
|
|
184
|
+
included, as their context.
|
|
153
185
|
|
|
154
|
-
|
|
186
|
+
| Field | Where | Required | Description |
|
|
187
|
+
| ---------------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
188
|
+
| `folder` | Actor | yes | Relative path from repo root to the Actor's own project directory — the folder that directly contains `.actor/actor.json` (i.e. `<folder>/.actor/actor.json`), the Actor's README/CHANGELOG, and its source. Use `"."` for a single-Actor repo where `.actor/` is at the root. |
|
|
189
|
+
| `actorFullName` | Actor | yes | Full Actor identifier in `owner/name` format (e.g. `"apify/web-scraper"`). This is the source of truth for the Actor name — the `name` field in `actor.json` is not used. |
|
|
190
|
+
| `tokenEnvVar` | group or Actor | yes | Name of the environment variable holding the Apify API token that builds the group's Actors. No fallback — if the env var is not set at build time, the build fails. |
|
|
191
|
+
| `overrideActorContext` | group or Actor | no | Array of paths (relative to repo root) that define which files are relevant to the Actor. When set, replaces the `dockerContextDir` from `actor.json` for change detection. Useful when an Actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. `["", "code"]` or `["actors", "actors/foo"]` are rejected). The Actor's own `folder` is always part of its context — if none of the listed entries reach it, it's added automatically. |
|
|
155
192
|
|
|
156
|
-
|
|
157
|
-
`
|
|
158
|
-
Reference them at the `@workflows-v0` major tag, never at `@master` — see
|
|
159
|
-
[Versioning and releases](#versioning-and-releases).
|
|
193
|
+
A value set on an Actor replaces the group's for that Actor only. Values are replaced, not merged: if the `myteam`
|
|
194
|
+
group also set `overrideActorContext`, the orchestrator would still get only the two paths it lists.
|
|
160
195
|
|
|
161
|
-
|
|
196
|
+
No two Actors may share a `folder`, and no two entries may point at the same `actorFullName`.
|
|
162
197
|
|
|
163
|
-
|
|
198
|
+
## GitHub workflows
|
|
164
199
|
|
|
165
|
-
|
|
166
|
-
name: Platform tests - Core
|
|
200
|
+
The workflows take every change straight to production, with tests in between:
|
|
167
201
|
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
202
|
+
- **Every push to a PR** builds the Actors the PR changes into a test version (`0.99`) that users never run, and runs
|
|
203
|
+
their platform tests against those builds. Only the Actors whose files changed are built and tested, and a later
|
|
204
|
+
push only re-tests what changed since the last green run.
|
|
205
|
+
- **Every merge to `master`** builds the changed Actors into their default version (`latest`), which is what users
|
|
206
|
+
run, and announces the release on Slack. There is no separate release step.
|
|
207
|
+
- **Scheduled runs** (optional) run the tests against the released builds to catch what breaks without a code change,
|
|
208
|
+
like a website changing its layout.
|
|
173
209
|
|
|
174
|
-
|
|
175
|
-
platformTestsCore:
|
|
176
|
-
uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
|
|
177
|
-
with:
|
|
178
|
-
subtest: core
|
|
179
|
-
secrets: inherit
|
|
180
|
-
```
|
|
210
|
+
These are GitHub Actions workflows. On another CI platform, build the same flow with the [CLI](docs/cli.md).
|
|
181
211
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
name: Platform tests - Daily
|
|
186
|
-
|
|
187
|
-
on:
|
|
188
|
-
schedule:
|
|
189
|
-
# Runs at 00:00 UTC every day
|
|
190
|
-
- cron: '0 0 * * *'
|
|
191
|
-
workflow_dispatch:
|
|
192
|
-
|
|
193
|
-
jobs:
|
|
194
|
-
platformTestsDaily:
|
|
195
|
-
uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
|
|
196
|
-
secrets: inherit
|
|
197
|
-
```
|
|
212
|
+
Add these files to `.github/workflows` in your repo. They call the reusable workflows in this repo, pinned at the
|
|
213
|
+
`@workflows-v0` major tag — never at `@master`. Every input, the optional Claude workflows, and how the tag moves are
|
|
214
|
+
in the [Workflow API](docs/workflows.md).
|
|
198
215
|
|
|
199
216
|
### `pr-build-devel-test.yaml`
|
|
200
217
|
|
|
@@ -208,7 +225,14 @@ on:
|
|
|
208
225
|
jobs:
|
|
209
226
|
buildDevelAndTest:
|
|
210
227
|
uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
|
|
211
|
-
secrets:
|
|
228
|
+
secrets:
|
|
229
|
+
TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
|
|
230
|
+
# One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
|
|
231
|
+
CONFIG_SECRETS: |
|
|
232
|
+
{
|
|
233
|
+
"APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
|
|
234
|
+
"APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
|
|
235
|
+
}
|
|
212
236
|
```
|
|
213
237
|
|
|
214
238
|
### `release-latest.yaml`
|
|
@@ -223,608 +247,103 @@ on:
|
|
|
223
247
|
jobs:
|
|
224
248
|
buildLatest:
|
|
225
249
|
uses: apify/apify-test-tools/.github/workflows/public_push-build-latest.yaml@workflows-v0
|
|
226
|
-
secrets:
|
|
250
|
+
secrets:
|
|
251
|
+
SLACK_TOKEN_RELEASES_BOT: ${{ secrets.SLACK_TOKEN_RELEASES_BOT }}
|
|
252
|
+
# One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
|
|
253
|
+
CONFIG_SECRETS: |
|
|
254
|
+
{
|
|
255
|
+
"APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
|
|
256
|
+
"APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
|
|
257
|
+
}
|
|
227
258
|
```
|
|
228
259
|
|
|
229
|
-
|
|
230
|
-
`build --version-number default` on every changed Actor, then `notify-release`. See [Releasing Actors](#releasing-actors) for how the
|
|
231
|
-
released range is determined and what to pass if you call the CLI from somewhere else.
|
|
232
|
-
|
|
233
|
-
### `claude-review.yaml`
|
|
260
|
+
### Scheduled tests (optional)
|
|
234
261
|
|
|
235
|
-
|
|
236
|
-
every push while that label is on.
|
|
262
|
+
How often to run which tests is up to you. For example, you can run every test once a day:
|
|
237
263
|
|
|
238
264
|
```yaml
|
|
239
|
-
name:
|
|
265
|
+
name: Platform tests - Daily
|
|
240
266
|
|
|
241
267
|
on:
|
|
242
|
-
|
|
243
|
-
|
|
268
|
+
schedule:
|
|
269
|
+
# Runs at 00:00 UTC every day
|
|
270
|
+
- cron: '0 0 * * *'
|
|
271
|
+
workflow_dispatch:
|
|
244
272
|
|
|
245
273
|
jobs:
|
|
246
|
-
|
|
247
|
-
uses: apify/apify-test-tools/.github/workflows/
|
|
248
|
-
secrets:
|
|
274
|
+
platformTestsDaily:
|
|
275
|
+
uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
|
|
276
|
+
secrets:
|
|
277
|
+
TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
|
|
278
|
+
SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}
|
|
249
279
|
```
|
|
250
280
|
|
|
251
|
-
|
|
252
|
-
because a reusable workflow doesn't get its own repo checked out. `prompt-ref` selects which ref to
|
|
253
|
-
fetch them from and defaults to `workflows-v0`, so the instructions match the workflow you're calling — point
|
|
254
|
-
it at a branch only to test a prompt change.
|
|
255
|
-
|
|
256
|
-
### Secrets
|
|
257
|
-
|
|
258
|
-
Callers pass `secrets: inherit`. The workflows do not turn every inherited secret into job-wide
|
|
259
|
-
environment variables, so a secret is only visible to the step that needs it:
|
|
260
|
-
|
|
261
|
-
| Secret | Reaches |
|
|
262
|
-
| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
263
|
-
| `NPM_TOKEN` | dependency install steps only, as both `NPM_TOKEN` and `NODE_AUTH_TOKEN`. Use a read-only token: npm granular tokens can be read-only, classic automation tokens can publish. |
|
|
264
|
-
| the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | the build and `delete-old-builds` steps only |
|
|
265
|
-
| `TESTER_APIFY_TOKEN` | the vitest step, and the Claude investigation step: the Apify MCP server's bearer token for reading the failing run, its log and its storages, and Claude's token for re-running it to measure how often the failure happens |
|
|
266
|
-
| `SLACK_TOKEN_TESTS_BOT` / `SLACK_TOKEN_RELEASES_BOT` | the reporting and announcement steps only |
|
|
267
|
-
|
|
268
|
-
The Actor tokens are the one set that cannot be listed in the workflow, because each Actor names its
|
|
269
|
-
own token via `tokenEnvVar` in `apify-test-tools.config.json`. Those steps pass
|
|
270
|
-
`${{ toJSON(secrets) }}` as `ALL_SECRETS` and run the command through
|
|
271
|
-
`.github/scripts/run-with-apify-tokens.mjs`, which reads that same config file to decide which
|
|
272
|
-
secrets to pass on:
|
|
281
|
+
and a few critical tests, kept in e.g. `test/platform/hourly`, every hour:
|
|
273
282
|
|
|
274
283
|
```yaml
|
|
275
|
-
|
|
276
|
-
shell: bash
|
|
277
|
-
env:
|
|
278
|
-
ALL_SECRETS: ${{ toJSON(secrets) }}
|
|
279
|
-
run: |
|
|
280
|
-
jq '{ actors: .functionallyChangedActors }' affected.json \
|
|
281
|
-
| node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
|
|
282
|
-
npx apify-test-tools build --version-number 0.99
|
|
283
|
-
```
|
|
284
|
-
|
|
285
|
-
The wrapper passes only the tokens the config declares and drops `ALL_SECRETS`, so neither `npx` nor
|
|
286
|
-
anything under `node_modules` sees the blob. Nothing is written to `$GITHUB_ENV`, so the tokens stay
|
|
287
|
-
inside that one command rather than leaking into later steps. Reading the same file
|
|
288
|
-
`apify-test-tools` reads means the two can't drift, and a secret that merely looks like an Actor
|
|
289
|
-
token is not passed just because of its name.
|
|
290
|
-
|
|
291
|
-
A token the config declares but the repo hasn't set is a warning, not a failure: a repo can carry an
|
|
292
|
-
Actor whose token isn't configured and still build fine as long as that Actor never changes, and
|
|
293
|
-
`apify-test-tools` raises a precise error naming the Actor at the point it actually needs the token.
|
|
294
|
-
|
|
295
|
-
`scripts-path` comes from the setup action (give the step `id: setup`) and points at this repo's
|
|
296
|
-
`.github/scripts/` directory inside the runner's action checkout, so workflows can run these helpers
|
|
297
|
-
without checking this repo out again. The caller's workspace holds the caller's repo, not this one.
|
|
298
|
-
|
|
299
|
-
Two tidier-looking alternatives don't work, so don't reach for them:
|
|
300
|
-
|
|
301
|
-
- **Exporting to `$GITHUB_ENV`** would let the steps call `npx` directly with no wrapper, but
|
|
302
|
-
`$GITHUB_ENV` applies to every later step in the job. In `pr-build-test` the vitest step runs after
|
|
303
|
-
the build, so it would inherit Actor tokens it has no use for.
|
|
304
|
-
- **Returning the tokens as a step output** would be scoped correctly, but the runner refuses to set
|
|
305
|
-
an output whose value contains a registered secret. It logs `Skip output <name> since it may
|
|
306
|
-
contain secret` and leaves the output empty, so anything reading it downstream gets nothing.
|
|
307
|
-
|
|
308
|
-
The `unitTest` job runs static checks and needs `NPM_TOKEN` only. No job runs `npm ci` or
|
|
309
|
-
`pnpm install` with Apify or Slack credentials in scope, so a postinstall script in the dependency
|
|
310
|
-
tree cannot read them.
|
|
311
|
-
|
|
312
|
-
## Versioning and releases
|
|
313
|
-
|
|
314
|
-
The workflows and the npm package live in one repo but ship on their own schedules:
|
|
315
|
-
|
|
316
|
-
| What | Consumers get it through | Ships when |
|
|
317
|
-
| -------------------------------------------- | ------------------------------------------ | ----------------------- |
|
|
318
|
-
| the workflows and the CLI they call | the `@workflows-v0` tag in `uses:` | every master push |
|
|
319
|
-
| the lib (`testActor`, `describe`, matchers…) | `apify-test-tools` in their `package.json` | a stable release on npm |
|
|
320
|
-
|
|
321
|
-
The CLI is bundled with its dependencies into `.github/cli/apify-test-tools.mjs` (`npm run bundle`,
|
|
322
|
-
run by the pre-commit hook and checked in CI), and the setup action runs it from there. Nothing is
|
|
323
|
-
installed for it, so a tag is a complete statement: these workflows and this CLI, with the exact
|
|
324
|
-
dependency versions from `package-lock.json` at that commit. The consumer's own `node_modules` is
|
|
325
|
-
left alone, so their tests keep the lib version they locked.
|
|
326
|
-
|
|
327
|
-
- **Workflow or CLI change** — merge it. `workflows-v0` moves once checks and tests pass, and it goes
|
|
328
|
-
live. No release needed, even when a workflow starts using a new CLI feature: both are in the
|
|
329
|
-
same commit.
|
|
330
|
-
- **Lib change** — merge it, then cut a release when you want it out. Consumers pick it up when
|
|
331
|
-
they update `apify-test-tools`.
|
|
332
|
-
|
|
333
|
-
The CLI and the lib in one job can be different versions now, so the formats between them are a
|
|
334
|
-
contract: the `ACTOR_BUILDS` JSON that `build` writes and the lib reads, and the vitest JSON report
|
|
335
|
-
`report-tests` reads. Add fields freely; renaming or removing one needs a new major tag.
|
|
336
|
-
|
|
337
|
-
`workflows-v0` moving on every master push means `@workflows-v0` is as live as `@master` was —
|
|
338
|
-
there's no staging step. What the tag buys you is a `workflows-v1` for breaking workflow changes,
|
|
339
|
-
so repos migrate one at a time instead of all at once, and a way to roll back by pointing the tag at
|
|
340
|
-
an earlier commit (**Move major version tag** workflow). Bump `MAJOR_TAG` in
|
|
341
|
-
`.github/workflows/_move_major_tag.yaml` to cut the next major; the old tag then freezes where it is
|
|
342
|
-
and keeps working, CLI included, since the bundle is part of the frozen commit.
|
|
343
|
-
|
|
344
|
-
The tag tracks the **workflows'** contract, not the npm package version. They move independently on
|
|
345
|
-
purpose, so `@workflows-v0` is expected to stay `@workflows-v0` after the package reaches 1.0 —
|
|
346
|
-
bump it when a workflow breaks its callers, not when the package does. Because `uses:` cannot take an expression, every ref
|
|
347
|
-
into this repo repeats the tag literally; `check-major-tag-refs.mjs` fails the build if `MAJOR_TAG`
|
|
348
|
-
and those refs disagree, which is the mistake that would otherwise ship silently during a bump.
|
|
349
|
-
|
|
350
|
-
### Testing workflow changes
|
|
351
|
-
|
|
352
|
-
- Point [testing-repo-for-github-actions](https://github.com/apify-store/testing-repo-for-github-actions)
|
|
353
|
-
at your branch (`uses: ...@your-branch`). It has real attached Actors and tests. The branch carries
|
|
354
|
-
its own CLI bundle, so one branch tests the workflows and the CLI together.
|
|
355
|
-
- To change the composite action itself, repoint the `uses:` refs inside the reusable workflows at
|
|
356
|
-
your branch as well, and change them back before merging.
|
|
357
|
-
- Make sure the shell code actually works on your laptop first.
|
|
358
|
-
- After merging, watch the workflow on a real project before moving on.
|
|
359
|
-
|
|
360
|
-
## Writing tests
|
|
361
|
-
|
|
362
|
-
### Test structure
|
|
363
|
-
|
|
364
|
-
`testActor` runs the actor and provides extended `expect` and `run` inside the callback.
|
|
365
|
-
|
|
366
|
-
```ts
|
|
367
|
-
import { describe, testActor } from 'apify-test-tools';
|
|
368
|
-
|
|
369
|
-
describe('test', () => {
|
|
370
|
-
testActor(actorId, 'actor test 1', async ({ expect, run }) => {
|
|
371
|
-
const runResult = await run({ input });
|
|
372
|
-
|
|
373
|
-
// your checks
|
|
374
|
-
});
|
|
375
|
-
|
|
376
|
-
testActor(actorId, 'actor test 2', async ({ expect, run }) => {
|
|
377
|
-
const runResult = await run({ input });
|
|
378
|
-
|
|
379
|
-
// your checks
|
|
380
|
-
});
|
|
381
|
-
});
|
|
382
|
-
```
|
|
383
|
-
|
|
384
|
-
---
|
|
385
|
-
|
|
386
|
-
### Validating basic run attributes
|
|
387
|
-
|
|
388
|
-
`toFinishWith` validates common run properties in a single call:
|
|
389
|
-
|
|
390
|
-
```ts
|
|
391
|
-
await expect(runResult).toFinishWith({
|
|
392
|
-
datasetItemCount: 100,
|
|
393
|
-
});
|
|
394
|
-
```
|
|
395
|
-
|
|
396
|
-
You can also specify a range:
|
|
397
|
-
|
|
398
|
-
```ts
|
|
399
|
-
await expect(runResult).toFinishWith({
|
|
400
|
-
datasetItemCount: { min: 80, max: 120 },
|
|
401
|
-
});
|
|
402
|
-
```
|
|
403
|
-
|
|
404
|
-
Here is full example of what you can validate with `toFinishWith`
|
|
405
|
-
|
|
406
|
-
```ts
|
|
407
|
-
await expect(runResult).toFinishWith({
|
|
408
|
-
// These are default
|
|
409
|
-
status: 'SUCCEEDED',
|
|
410
|
-
duration: {
|
|
411
|
-
min: 600, // 0.6 sec
|
|
412
|
-
max: 600_000, // 10 min
|
|
413
|
-
},
|
|
414
|
-
failedRequests: 0,
|
|
415
|
-
requestsRetries: { max: 3 },
|
|
416
|
-
forbiddenLogs: ['ReferenceError', 'TypeError'],
|
|
417
|
-
|
|
418
|
-
// only datasetItemCount is required
|
|
419
|
-
datasetItemCount: { min: 80, max: 120 },
|
|
420
|
-
|
|
421
|
-
// optional
|
|
422
|
-
chargedEventCounts: {
|
|
423
|
-
'actor-start': 1,
|
|
424
|
-
'place-scraped': 9,
|
|
425
|
-
},
|
|
426
|
-
});
|
|
427
|
-
```
|
|
428
|
-
|
|
429
|
-
---
|
|
430
|
-
|
|
431
|
-
### Custom validations
|
|
432
|
-
|
|
433
|
-
```ts
|
|
434
|
-
expect(place.title, `London Eye's title`).toEqual('lastminute.com London Eye');
|
|
435
|
-
```
|
|
436
|
-
|
|
437
|
-
---
|
|
438
|
-
|
|
439
|
-
### Custom validation functions
|
|
440
|
-
|
|
441
|
-
You can create your own functions wrapping a common validation logic in e.g. `test/platform/utils.ts` and import it in test files.
|
|
442
|
-
|
|
443
|
-
```ts
|
|
444
|
-
import { ExpectStatic } from 'apify-test-tools'
|
|
445
|
-
|
|
446
|
-
export const validateItem = (expect: ExpectStatic, item: any) {
|
|
447
|
-
expect(item.title, 'Item title').toBeString();
|
|
448
|
-
}
|
|
449
|
-
```
|
|
450
|
-
|
|
451
|
-
---
|
|
452
|
-
|
|
453
|
-
### Test options
|
|
454
|
-
|
|
455
|
-
You can pass options as the fourth argument to `testActor`:
|
|
456
|
-
|
|
457
|
-
```ts
|
|
458
|
-
testActor(
|
|
459
|
-
actorId,
|
|
460
|
-
'slow actor test',
|
|
461
|
-
async ({ expect, run }) => {
|
|
462
|
-
const runResult = await run({ input });
|
|
463
|
-
await expect(runResult).toFinishWith({ datasetItemCount: 100 });
|
|
464
|
-
},
|
|
465
|
-
{
|
|
466
|
-
timeout: 2 * 60 * 60 * 1000, // 2 hours (default is 1 hour)
|
|
467
|
-
retry: 3, // retry up to 3 times (default is 1)
|
|
468
|
-
},
|
|
469
|
-
);
|
|
470
|
-
```
|
|
471
|
-
|
|
472
|
-
---
|
|
473
|
-
|
|
474
|
-
### Using prefilled input
|
|
284
|
+
name: Platform tests - Hourly
|
|
475
285
|
|
|
476
|
-
|
|
286
|
+
on:
|
|
287
|
+
schedule:
|
|
288
|
+
# Runs at the start of every hour
|
|
289
|
+
- cron: '0 * * * *'
|
|
290
|
+
workflow_dispatch:
|
|
477
291
|
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
}
|
|
292
|
+
jobs:
|
|
293
|
+
platformTestsHourly:
|
|
294
|
+
uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
|
|
295
|
+
with:
|
|
296
|
+
test-files-glob: hourly
|
|
297
|
+
secrets:
|
|
298
|
+
TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
|
|
299
|
+
SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}
|
|
486
300
|
```
|
|
487
301
|
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
### Testing an existing run
|
|
302
|
+
### Secrets
|
|
491
303
|
|
|
492
|
-
|
|
304
|
+
Set these as repository or organization secrets and pass them in each job's `secrets:` block, as in the examples
|
|
305
|
+
above. The Actor tokens go packed in one JSON object, `CONFIG_SECRETS`, because their names differ per repo:
|
|
493
306
|
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
307
|
+
| Secret | Needed for |
|
|
308
|
+
| ------------------------------------------------------------------------- | ---------------------------------------------------------------- |
|
|
309
|
+
| the Actor tokens named by `tokenEnvVar` in `apify-test-tools.config.json` | building Actors and deleting their old builds |
|
|
310
|
+
| `TESTER_APIFY_TOKEN` | running the platform tests |
|
|
311
|
+
| `SLACK_TOKEN_RELEASES_BOT` | announcing releases on Slack |
|
|
312
|
+
| `SLACK_TOKEN_TESTS_BOT` | reporting scheduled test results to Slack |
|
|
313
|
+
| `NPM_TOKEN` | optional, installing private npm packages. Use a read-only token |
|
|
500
314
|
|
|
501
|
-
|
|
315
|
+
Repos in the `apify` org can pass `secrets: inherit` instead. Each secret only reaches the step that needs it. See
|
|
316
|
+
[Secrets](docs/workflows.md#secrets) for both, and for passing `CONFIG_SECRETS` without listing the tokens.
|
|
502
317
|
|
|
503
|
-
|
|
318
|
+
## Writing tests
|
|
504
319
|
|
|
505
|
-
`
|
|
320
|
+
Platform tests use `describe` and `testActor` from `apify-test-tools` instead of the ones from `vitest`. `testActor`
|
|
321
|
+
registers a test that only runs when that Actor was built for it (or when you ask for all tests), and gives the
|
|
322
|
+
callback an extended `expect` and a `run` function that starts the Actor.
|
|
506
323
|
|
|
507
324
|
```ts
|
|
508
|
-
|
|
509
|
-
const runResult = await run({ input });
|
|
510
|
-
|
|
511
|
-
// Access dataset items
|
|
512
|
-
const { items } = await runResult.getDataset();
|
|
513
|
-
expect(items[0].title).toBeNonEmptyString();
|
|
514
|
-
|
|
515
|
-
// Access run log
|
|
516
|
-
const log = await runResult.getLog();
|
|
517
|
-
expect(log).toContain('Crawl finished');
|
|
518
|
-
|
|
519
|
-
// Access crawler statistics
|
|
520
|
-
const stats = await runResult.getStatistics();
|
|
521
|
-
expect(stats?.requestsFinished).toBeGreaterThan(0);
|
|
522
|
-
|
|
523
|
-
// Access key-value store
|
|
524
|
-
const kvs = runResult.getKeyValueStoreClient();
|
|
525
|
-
const record = await kvs.getRecord('OUTPUT');
|
|
526
|
-
|
|
527
|
-
// Access run info (refreshed from API)
|
|
528
|
-
const runInfo = await runResult.getRunInfo();
|
|
529
|
-
});
|
|
530
|
-
```
|
|
531
|
-
|
|
532
|
-
---
|
|
325
|
+
import { describe, testActor } from 'apify-test-tools';
|
|
533
326
|
|
|
534
|
-
|
|
327
|
+
describe('web-scraper', () => {
|
|
328
|
+
testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
|
|
329
|
+
const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
|
|
535
330
|
|
|
536
|
-
|
|
331
|
+
// Status SUCCEEDED, a sane duration, no failed requests, no errors in the log, and the item count
|
|
332
|
+
await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
|
|
537
333
|
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
describe('standby tests', () => {
|
|
542
|
-
testStandbyActor(actorId, 'standby request', async ({ expect, callStandby }) => {
|
|
543
|
-
const { data, status } = await callStandby({
|
|
544
|
-
input: { query: 'test' },
|
|
545
|
-
path: '/search',
|
|
546
|
-
headers: { 'Content-Type': 'application/json' },
|
|
547
|
-
});
|
|
548
|
-
|
|
549
|
-
expect(status).toBe(200);
|
|
550
|
-
expect(data.results).toBeNonEmptyArray();
|
|
334
|
+
const { items } = await runResult.getDataset();
|
|
335
|
+
expect(items[0].title, 'Page title').toBeNonEmptyString();
|
|
551
336
|
});
|
|
552
337
|
});
|
|
553
338
|
```
|
|
554
339
|
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
### Custom matchers
|
|
558
|
-
|
|
559
|
-
`testActor` extends `expect` with the following custom matchers:
|
|
560
|
-
|
|
561
|
-
- `toBeArray()` / `toBeEmptyArray()` / `toBeNonEmptyArray()`
|
|
562
|
-
- `toBeString()` / `toBeNonEmptyString()` / `toStartWith(prefix)`
|
|
563
|
-
- `toBeNumber()` / `toBeWholeNumber()` / `toBeWithinRange(min, max)`
|
|
564
|
-
- `toBeBoolean()` / `toBeTrue()` / `toBeFalse()`
|
|
565
|
-
- `toBeObject()` / `toBeNonEmptyObject()`
|
|
566
|
-
- `toFinishWith(options)` - validates run status, duration, dataset, logs, etc.
|
|
567
|
-
|
|
568
|
-
## CLI (`apify-test-tools` bin)
|
|
569
|
-
|
|
570
|
-
The package includes a CLI binary used by CI workflows to build Actors, detect changes, and report test results. You can also run it locally.
|
|
571
|
-
|
|
572
|
-
### Dry runs (`--dry-run`)
|
|
573
|
-
|
|
574
|
-
Every command that changes something (`build`, `build-from-local`, `notify-release`, `delete-old-builds`, `report-tests`) takes `--dry-run`. A dry run:
|
|
575
|
-
|
|
576
|
-
- **Changes nothing.** No versions or builds are created, nothing is uploaded or deleted, and no Slack message is posted.
|
|
577
|
-
- **Does every read and check the real run does**, so it fails wherever the real run would: change detection, the Apify token and Actor lookup, the [repository check](#repository-check), collecting and zipping local sources, and the Slack token (verified with Slack's read-only `auth.test`).
|
|
578
|
-
- **Needs the same tokens as the real run.** Read-only tokens are enough, since nothing is written. A dry run can't tell whether a token is also allowed to write, because the Apify API has no dry-run mode of its own.
|
|
579
|
-
- **Prints a plan, not results.** Logs start with `[DRY RUN] Would …`. `build` and `build-from-local` write `{ "dryRun": true, "builds": [...] }` to stdout instead of the build array, so the plan can't be passed to tests as `ACTOR_BUILDS` by mistake.
|
|
580
|
-
|
|
581
|
-
### Running locally
|
|
340
|
+
Platform tests are skipped unless `TESTER_APIFY_TOKEN` is set, so your regular unit test run stays unaffected.
|
|
582
341
|
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
If you don't need to change any source files and only iterate on /test code, you can skip steps 1-4. But if you want to test vs changed /src, you have to push that GitHub branch since it needs to build the Actors with that code.
|
|
586
|
-
|
|
587
|
-
The main local flow is:
|
|
588
|
-
|
|
589
|
-
1. Switch to a dummy branch that you will push and can later delete
|
|
590
|
-
2. `npm i apify-test-tools@latest -D`
|
|
591
|
-
3. Push your code (changes you want to test)
|
|
592
|
-
4. Build Actors on Apify (with your new code)
|
|
593
|
-
5. Run tests against those builds. You can change tests and run on the same builds.
|
|
594
|
-
|
|
595
|
-
Run every command from the root of the actor repository you want to work with.
|
|
596
|
-
|
|
597
|
-
#### 4. Build affected Actors
|
|
598
|
-
|
|
599
|
-
If you want to test vs existing src code, you can skip this and instead construct the output JSON manually from existing builds only for the Actors you need to test.
|
|
600
|
-
|
|
601
|
-
Requires `APIFY_TOKEN_<USERNAME>` for all Apify users that own your Actors (e.g. `apify`, `compass`, `lukaskrivka` users). The username is derived from the actor name — uppercased with non-word chars replaced by `_` (e.g. Actor `john.doe/my-actor` needs `APIFY_TOKEN_JOHN_DOE`).
|
|
342
|
+
To run them locally against the Actors' current default builds:
|
|
602
343
|
|
|
603
344
|
```bash
|
|
604
|
-
npx
|
|
605
|
-
| jq '{ actors: .functionallyChangedActors }' \
|
|
606
|
-
| APIFY_TOKEN_JOHN_DOE=<token> npx apify-test-tools build --version-number 0.99 --dry-run
|
|
345
|
+
RUN_ALL_PLATFORM_TESTS=1 TESTER_APIFY_TOKEN=<token> npx vitest --run test/platform
|
|
607
346
|
```
|
|
608
347
|
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
each Actor would be built from). Remove `--dry-run` to actually trigger builds, and update `--base` to the branch
|
|
612
|
-
you compare against. `build` outputs a JSON array of build objects to stdout:
|
|
613
|
-
|
|
614
|
-
```json
|
|
615
|
-
[{ "buildId": "...", "actorRawId": "...", "buildNumber": "...", "actorFullName": "john.doe/my-actor" }]
|
|
616
|
-
```
|
|
617
|
-
|
|
618
|
-
#### Build from local source (no push needed)
|
|
619
|
-
|
|
620
|
-
If you don't want to push a dummy branch just to test a change and wait for all the tests to finish, `build-from-local` builds Actors directly from your local files (zipped into the Actor's `actor-<id>-source` key-value store and built as `TARBALL`, like `apify push`), skipping steps 1-4 above.
|
|
621
|
-
|
|
622
|
-
```bash
|
|
623
|
-
APIFY_TOKEN_JOHN_DOE=<token> \
|
|
624
|
-
npx apify-test-tools build-from-local --actors john.doe/my-actor
|
|
625
|
-
```
|
|
626
|
-
|
|
627
|
-
Pass a hardcoded actor name via `--actors` to build only that Actor (comma-separate multiple names). Omit `--actors` to build all Actors in the repo, or add `--dry-run` to check the token, the Actor and the zipped sources without uploading or building. It outputs the same JSON build array as `build`, so you run tests against it the same way as in step 5 below:
|
|
628
|
-
|
|
629
|
-
```bash
|
|
630
|
-
# Build from local source and capture output
|
|
631
|
-
BUILDS=$(APIFY_TOKEN_JOHN_DOE=apify_api_xxx \
|
|
632
|
-
npx apify-test-tools build-from-local --actors apify/my-actor)
|
|
633
|
-
```
|
|
634
|
-
|
|
635
|
-
Since you already scoped the build to just the Actor(s) you care about, point vitest at a specific test file (or a `-t` name filter) instead of the whole `test/platform` directory — you get feedback on that one test without waiting for the full suite to run.
|
|
636
|
-
|
|
637
|
-
#### 5. Run tests against the builds
|
|
638
|
-
|
|
639
|
-
Pass the build output as `ACTOR_BUILDS` and provide `TESTER_APIFY_TOKEN`. The token can point to your own account (if you have enough memory) or you can use the testing account (xRGg9iAfJSymqartk). Platform test suites are skipped unless `TESTER_APIFY_TOKEN` is set, so regular unit test runs stay unaffected.
|
|
640
|
-
|
|
641
|
-
If you want to run only certain tests, change the `test/platform` to be more specific.
|
|
642
|
-
|
|
643
|
-
```bash
|
|
644
|
-
ACTOR_BUILDS='<JSON output from build command>' \
|
|
645
|
-
TESTER_APIFY_TOKEN=<token> \
|
|
646
|
-
npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform
|
|
647
|
-
```
|
|
648
|
-
|
|
649
|
-
#### Full example
|
|
650
|
-
|
|
651
|
-
Tests only run for Actors in `ACTOR_BUILDS`, so a custom workflow has to build (and pass) the Actors even when only tests changed. Pass `--test-files-glob` to `get-affected-actors` so test changes count even when `.dockerignore` lists them.
|
|
652
|
-
|
|
653
|
-
```bash
|
|
654
|
-
# Build and capture output
|
|
655
|
-
BUILDS=$(npx apify-test-tools get-affected-actors --base origin/master --test-files-glob 'test/platform/**' \
|
|
656
|
-
| jq '{ actors: .functionallyChangedActors }' \
|
|
657
|
-
| APIFY_TOKEN_JOHN_DOE=apify_api_xxx npx apify-test-tools build --version-number 0.99)
|
|
658
|
-
|
|
659
|
-
# Run tests with the builds
|
|
660
|
-
ACTOR_BUILDS="$BUILDS" \
|
|
661
|
-
TESTER_APIFY_TOKEN=apify_api_yyy \
|
|
662
|
-
npx vitest --run --maxConcurrency 20 --fileParallelism=true --maxWorkers 100 test/platform
|
|
663
|
-
```
|
|
664
|
-
|
|
665
|
-
### Finding affected Actors (`get-affected-actors`)
|
|
666
|
-
|
|
667
|
-
`get-affected-actors` lists the Actors whose context changed in a commit range, using the rules in
|
|
668
|
-
[Change detection](#change-detection). It only reads git and the config file, so it needs no tokens.
|
|
669
|
-
The range always ends at the checked-out `HEAD`, so check out the branch you want to inspect first.
|
|
670
|
-
|
|
671
|
-
```bash
|
|
672
|
-
# A PR: the branch's own changes, narrowed to the Actors that changed since its last validated commit
|
|
673
|
-
npx apify-test-tools get-affected-actors --base origin/master --since <sha>
|
|
674
|
-
|
|
675
|
-
# A release: everything after the last released commit
|
|
676
|
-
npx apify-test-tools get-affected-actors --since <sha>
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
| Flag | Default | Meaning |
|
|
680
|
-
| ------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
681
|
-
| `--base` | | Only what `HEAD` changes compared to this branch counts, like `git diff <base>...HEAD` |
|
|
682
|
-
| `--since` | | Only changes after this commit count (see below) |
|
|
683
|
-
| `--test-files-glob` | | Test files (glob from the repo root) that count as changes even when `.dockerignore` lists them, see [Change detection](#change-detection) |
|
|
684
|
-
| `--actors` | all | Only report these Actors |
|
|
685
|
-
| `--ignore` | none | Never report these Actors |
|
|
686
|
-
|
|
687
|
-
Pass `--base`, `--since`, or both:
|
|
688
|
-
|
|
689
|
-
- **`--base` only** inspects what the branch changes, the same files GitHub's "Files changed" tab shows: the diff
|
|
690
|
-
from where the branch forked from `--base`, or the last commit of `--base` it merged in. What it merged in from
|
|
691
|
-
`--base` is already validated there, so it doesn't count.
|
|
692
|
-
- **`--base` and `--since`** also inspects what changed after the branch's last validated commit, and reports only
|
|
693
|
-
the Actors that changed in both. An Actor in only one of them is already validated, by this branch or on `--base`.
|
|
694
|
-
When both changed an Actor, the changes can interact, so it is reported, but as functional only when both changes
|
|
695
|
-
are functional. A validated functional change followed by README edits is cosmetic, and so is a functional
|
|
696
|
-
change merged in from `--base` that meets only cosmetic changes of the branch. `--since` doesn't have to be on
|
|
697
|
-
the branch, so a rebase only re-checks the Actors that `--base` changed too. If `--since` is not in the local
|
|
698
|
-
history (a force-push), or it is `HEAD` (a rerun), it is ignored and the whole branch is checked.
|
|
699
|
-
- **`--since` only** inspects everything after it up to `HEAD`. It must be an ancestor of `HEAD` and fails
|
|
700
|
-
otherwise, see [Releasing Actors](#releasing-actors). If `HEAD` is `--since`, nothing is
|
|
701
|
-
reported.
|
|
702
|
-
|
|
703
|
-
`commits`, `changedFiles` and `changelog` describe the whole range, `--base...HEAD` or `--since..HEAD`: `--since`
|
|
704
|
-
together with `--base` only narrows the Actor lists.
|
|
705
|
-
|
|
706
|
-
The output is a JSON document:
|
|
707
|
-
|
|
708
|
-
```json
|
|
709
|
-
{
|
|
710
|
-
"version": 1,
|
|
711
|
-
"head": "<sha of HEAD>",
|
|
712
|
-
"commits": [{ "sha": "…", "author": "…", "date": "…", "message": "…" }],
|
|
713
|
-
"changedFiles": ["src/main.ts", "CHANGELOG.md"],
|
|
714
|
-
"changelog": "Lines added to CHANGELOG.md, or null",
|
|
715
|
-
"functionallyChangedActors": [{ "actorFullName": "john.doe/my-actor" }],
|
|
716
|
-
"cosmeticallyChangedActors": [{ "actorFullName": "john.doe/other-actor" }]
|
|
717
|
-
}
|
|
718
|
-
```
|
|
719
|
-
|
|
720
|
-
Each changed Actor is in exactly one of the lists. `functionallyChangedActors` have at least one functional change, so
|
|
721
|
-
they need a new build and tests. `cosmeticallyChangedActors` only have cosmetic changes (see
|
|
722
|
-
[Change detection](#change-detection)), so they need a new build only when released: the PR workflow builds
|
|
723
|
-
`functionallyChangedActors`, the release workflow builds both.
|
|
724
|
-
|
|
725
|
-
### Building Actors (`build`)
|
|
726
|
-
|
|
727
|
-
`build` builds exactly the Actors it is given on stdin, from the checked-out branch as it is on the remote, and
|
|
728
|
-
prints the builds as the same JSON array `build-from-local` prints. A detached HEAD is an
|
|
729
|
-
error, since there is no branch to build from. It doesn't look at git history: which Actors to build is
|
|
730
|
-
up to you, usually picked from the output of [`get-affected-actors`](#finding-affected-actors-get-affected-actors).
|
|
731
|
-
|
|
732
|
-
```bash
|
|
733
|
-
# A PR: the Actors a branch changed functionally, built into the test version
|
|
734
|
-
npx apify-test-tools get-affected-actors --base origin/master \
|
|
735
|
-
| jq '{ actors: .functionallyChangedActors }' \
|
|
736
|
-
| npx apify-test-tools build --version-number 0.99
|
|
737
|
-
|
|
738
|
-
# A release: every Actor changed since the last release, into the version users run
|
|
739
|
-
npx apify-test-tools get-affected-actors --since <sha> \
|
|
740
|
-
| jq '{ actors: (.functionallyChangedActors + .cosmeticallyChangedActors) }' \
|
|
741
|
-
| npx apify-test-tools build --version-number default
|
|
742
|
-
|
|
743
|
-
# One Actor, whatever changed
|
|
744
|
-
echo '{ "actors": [{ "actorFullName": "john.doe/my-actor" }] }' | npx apify-test-tools build --version-number 0.99
|
|
745
|
-
```
|
|
746
|
-
|
|
747
|
-
The input is `{ "actors": [{ "actorFullName": "<owner/name>" }] }`; other fields are ignored. An empty `actors` list builds nothing and prints `[]`. Empty or invalid input is an error,
|
|
748
|
-
so a failed command earlier in the pipe can't turn into a successful run that built nothing. In GitHub
|
|
749
|
-
Actions, give such a step `shell: bash` so the pipe also fails on its own (the default shell has no `pipefail`).
|
|
750
|
-
|
|
751
|
-
| Flag | Default | Meaning |
|
|
752
|
-
| ------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
|
753
|
-
| `--version-number` | required | Version to build, like `0.99`, or `default` for each Actor's default version (the one its default build tag points to) |
|
|
754
|
-
| `--repo-url` | the `origin` remote | Repository to build from, see [Repository check](#repository-check) |
|
|
755
|
-
|
|
756
|
-
`build` never sets or changes build tags. A build gets the tag its version is configured with on the platform,
|
|
757
|
-
so building the default version moves its tag (usually `latest`) to the new build, and building a version
|
|
758
|
-
without a tag leaves every tag where it is. Builds always use the Docker cache.
|
|
759
|
-
|
|
760
|
-
### Releasing Actors
|
|
761
|
-
|
|
762
|
-
A release is three commands: find what changed since the last released commit, build those Actors into
|
|
763
|
-
their default version (the one users run), and announce it.
|
|
764
|
-
|
|
765
|
-
```bash
|
|
766
|
-
npx apify-test-tools get-affected-actors --since <sha> > affected.json
|
|
767
|
-
|
|
768
|
-
jq '{ actors: (.functionallyChangedActors + .cosmeticallyChangedActors) }' affected.json \
|
|
769
|
-
| npx apify-test-tools build --version-number default --dry-run
|
|
770
|
-
|
|
771
|
-
npx apify-test-tools notify-release --release-slack-channel "#releases" --dry-run < affected.json
|
|
772
|
-
```
|
|
773
|
-
|
|
774
|
-
`notify-release` reads the output of `get-affected-actors` on stdin. It posts the changelog additions to
|
|
775
|
-
`--release-slack-channel` and the commits and changed files to `--report-slack-channel`, and posts nothing when
|
|
776
|
-
there are no new commits. It names the release after the `origin` remote (or `--repo-url`) and the author of the
|
|
777
|
-
newest commit, and needs `SLACK_TOKEN_RELEASES_BOT`.
|
|
778
|
-
|
|
779
|
-
#### The last released commit (`--since`)
|
|
780
|
-
|
|
781
|
-
The last commit that is **already released**. Everything after it, up to `HEAD`, is released. Git alone
|
|
782
|
-
can't tell which commits a push brought, and guessing wrong silently leaves Actors on old code, so it has
|
|
783
|
-
to be passed.
|
|
784
|
-
|
|
785
|
-
The changed files come from diffing `--since` and `HEAD` directly, so this works for every merge strategy as
|
|
786
|
-
long as `--since` is the branch tip from **before** the push:
|
|
787
|
-
|
|
788
|
-
| Merge strategy | What lands on the branch | What to pass |
|
|
789
|
-
| ------------------------------ | --------------------------------- | ---------------------------------------------------------------------- |
|
|
790
|
-
| Squash and merge | 1 new commit | the push's "before" commit (`HEAD~1` also works) |
|
|
791
|
-
| Merge commit | the PR's commits + 1 merge commit | the push's "before" commit (`HEAD^1`, the first parent, also works) |
|
|
792
|
-
| Rebase and merge / direct push | N new commits | the push's "before" commit (`HEAD~1` would miss the first N-1 commits) |
|
|
793
|
-
|
|
794
|
-
On GitHub, the push's "before" commit is `${{ github.event.before }}`, which the reusable workflow
|
|
795
|
-
passes for you. On other CIs use the equivalent, e.g. `$CI_COMMIT_BEFORE_SHA` on GitLab. The checkout
|
|
796
|
-
needs the full history (`fetch-depth: 0` in `actions/checkout`) so the commit is available locally.
|
|
797
|
-
|
|
798
|
-
`get-affected-actors --since` without `--base` fails instead of guessing when:
|
|
799
|
-
|
|
800
|
-
- the commit is all zeros (GitHub sends this for a push that created the branch)
|
|
801
|
-
- the commit isn't in the local history (shallow checkout) or isn't an ancestor of `HEAD`
|
|
802
|
-
(the branch was force-pushed)
|
|
803
|
-
|
|
804
|
-
In those cases, rerun it with a `--since` you pick yourself: an ancestor of `HEAD` from before the changes
|
|
805
|
-
you want to release (for a shallow checkout, fetch the full history instead). `--actors` only narrows which
|
|
806
|
-
of the changed Actors are reported; it never forces a build, so on its own it doesn't get past these errors.
|
|
807
|
-
If `HEAD` is `--since`, nothing is reported, so nothing is built or announced.
|
|
808
|
-
|
|
809
|
-
A failed release isn't retried by the next push: that push's base commit is the failed one's head, so
|
|
810
|
-
its changes are skipped. Re-run the failed workflow run, or release the Actors manually.
|
|
811
|
-
|
|
812
|
-
#### Repository check
|
|
813
|
-
|
|
814
|
-
`build` points each Actor's version at the `origin` remote. Because a build overwrites the
|
|
815
|
-
version's Git URL, building from a fork or a mirror would repoint the Actor. So, before building, the
|
|
816
|
-
remote is compared with the Git URL of the Actor's default version (the one its default build tag
|
|
817
|
-
points to), and the command fails if they differ or if the default version isn't built from a Git
|
|
818
|
-
repository. The URL form doesn't matter (`https://github.com/org/repo` equals
|
|
819
|
-
`git@github.com:org/repo.git`).
|
|
820
|
-
|
|
821
|
-
To build from a different repository on purpose, e.g. when moving an Actor, pass `--repo-url`. That
|
|
822
|
-
skips the check.
|
|
823
|
-
|
|
824
|
-
#### Dev mode
|
|
825
|
-
|
|
826
|
-
For development on `apify-test-tools` itself, use `tsx` directly:
|
|
827
|
-
|
|
828
|
-
```bash
|
|
829
|
-
cd local-clone && tsx ../bin/main.ts get-actor-configs
|
|
830
|
-
```
|
|
348
|
+
The [Vitest library API](docs/lib.md) covers everything `toFinishWith` checks, the other matchers, standby Actors,
|
|
349
|
+
reading run data, and running tests locally against builds of your own changes.
|