apify-test-tools 0.10.2-beta.1 → 0.10.2-beta.10

package/.github/workflows/public_push-build-latest.yaml

@@ -8,6 +8,22 @@ on:
                 type: string
                 default: ''
                 required: false
+        # Every secret is optional. The ones this workflow reads are declared by name. The secrets that
+        # apify-test-tools.config.json names (e.g. each Actor's `tokenEnvVar`) differ per repo, so they
+        # come packed in CONFIG_SECRETS. See the Secrets section of docs/workflows.md.
+        secrets:
+            NPM_TOKEN:
+                required: false
+                description: Read-only token for installing private npm packages.
+            SLACK_TOKEN_RELEASES_BOT:
+                required: false
+                description: Slack token for announcing releases.
+            CONFIG_SECRETS:
+                required: false
+                description: >-
+                    JSON object of the secrets apify-test-tools.config.json names, secret name -> value.
+                    Today that is each Actor's `tokenEnvVar`. Passing `toJSON(secrets)` works too, only
+                    the names the config declares are used.
 
 jobs:
     pushBuildLatest:
@@ -24,23 +40,40 @@ jobs:
               uses: apify/apify-test-tools/.github/actions/checkout-restore-dependencies@workflows-v0
               with:
                   npm-token: ${{ secrets.NPM_TOKEN }}
+                  config-secrets: ${{ secrets.CONFIG_SECRETS }}
+
+            # --since is the last commit that is already released: the push's "before" commit.
+            # Everything after it up to HEAD gets released. See the "Releasing Actors" section of docs/cli.md.
+            # Needs no tokens, it only reads git and apify-test-tools.config.json.
+            - name: Find changes since the last release
+              run: |
+                  apify-test-tools get-affected-actors --since ${{ github.event.before }} > affected.json
 
-            # --base-commit is the last commit that is already released: the push's "before" commit.
-            # Everything after it up to HEAD gets released. See the "Releasing Actors" section of the README.
+            # A release ships cosmetic changes too (README, input schema descriptions), so both lists are built,
+            # into each Actor's default version: the one its default build tag points to, which users run.
             # run-with-apify-tokens.mjs reads the token names from in apify-test-tools.config.json
             # and hands the build those secrets and nothing else, so the tokens stay in this one command instead of the whole job.
             # The library never sees secrets it doesn't need
-            - name: Release Actors and notify
+            - name: Build the default versions
+              # bash rather than the default shell, for pipefail
+              shell: bash
+              env:
+                  WORKFLOW_SECRETS: ${{ toJSON(secrets) }}
+              run: |
+                  jq '{ actors: (.functionallyChangedActors + .cosmeticallyChangedActors) }' affected.json \
+                    | node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
+                        apify-test-tools build --version-number default
+
+            # Skips the announcement when the push brought no new commits
+            - name: Announce the release on Slack
               env:
-                  ALL_SECRETS: ${{ toJSON(secrets) }}
                   SLACK_TOKEN_RELEASES_BOT: ${{ secrets.SLACK_TOKEN_RELEASES_BOT }}
               run: |
-                  node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
-                    npx apify-test-tools release --base-commit ${{ github.event.before }} --release-slack-channel "#delivery-public-actors" --report-slack-channel "${{ inputs.report-slack-channel || format('#notif-{0}', github.event.repository.name) }}" --use-docker-cache
+                  apify-test-tools notify-release --release-slack-channel "#delivery-public-actors" --report-slack-channel "${{ inputs.report-slack-channel || format('#notif-{0}', github.event.repository.name) }}" < affected.json
 
             - name: Delete old builds
               env:
-                  ALL_SECRETS: ${{ toJSON(secrets) }}
+                  WORKFLOW_SECRETS: ${{ toJSON(secrets) }}
               run: |
                   node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
-                    npx apify-test-tools delete-old-builds
+                    apify-test-tools delete-old-builds

package/.husky/pre-commit

@@ -1,6 +1,11 @@
 echo "pre-commit: typechecking, linting, testing, checking code formatting, checking for unused exports & validating schemas"
 
 
+# The workflows run the CLI from this bundle, so it ships with every commit that changes bin/ or its
+# dependencies. Rebuilding takes well under a second.
+npm run -s bundle
+git add .github/cli/apify-test-tools.mjs
+
 npm test
 npm run format:check
 # We typecheck & lint last when we know our code works

package/.prettierignore

@@ -8,3 +8,6 @@ CHANGELOG.md
 
 # Fixtures must stay byte-exact — several are deliberately malformed or empty
 test/fixtures
+
+# Generated by `npm run bundle`
+.github/cli/

package/CHANGELOG.md

@@ -5,10 +5,26 @@ All notable changes to this project will be documented in this file.
 <!-- git-cliff-unreleased-start -->
 ## 0.10.2 - **not yet released**
 
+### 🚀 Features
+
+- **ci:** Ship the workflows&#x27; CLI as a bundle in the action instead of installing it ([#200](https://github.com/apify/apify-test-tools/pull/200)) ([a0ef975](https://github.com/apify/apify-test-tools/commit/a0ef9756123a30a8210590b080a2a289c6a9c17c)) by [@metalwarrior665](https://github.com/metalwarrior665)
+- [**breaking**] Revamp get-affected-actors to be a core logic command ([#190](https://github.com/apify/apify-test-tools/pull/190)) ([ea5f244](https://github.com/apify/apify-test-tools/commit/ea5f244cfd42bf3fa9ffd38cd2ad499dde2e83f8)) by [@metalwarrior665](https://github.com/metalwarrior665)
+- [**breaking**] Add logic-less build command and use command chaining in workflows ([#193](https://github.com/apify/apify-test-tools/pull/193)) ([e5b0b86](https://github.com/apify/apify-test-tools/commit/e5b0b86ee06fca51d4be0ddc81b0a4a469d31603)) by [@metalwarrior665](https://github.com/metalwarrior665)
+- **get-affected-actors:** Three-dot diff, narrowed per Actor with --since ([#204](https://github.com/apify/apify-test-tools/pull/204)) ([fa9a0e7](https://github.com/apify/apify-test-tools/commit/fa9a0e7518764dd4722ec9ecbf67b2a26a5a062e)) by [@metalwarrior665](https://github.com/metalwarrior665)
+- Add effort input to Claude investigate-and-fix workflow ([#221](https://github.com/apify/apify-test-tools/pull/221)) ([c2ac078](https://github.com/apify/apify-test-tools/commit/c2ac078aec9ef02337af0e5c06db1b947b6c969b)) by [@foxt451](https://github.com/foxt451)
+- **ci:** Support pnpm ([#214](https://github.com/apify/apify-test-tools/pull/214)) ([ee7d734](https://github.com/apify/apify-test-tools/commit/ee7d7340564c5b8657a8a1f0cf920d67b40ae013)) by [@ruocco-l](https://github.com/ruocco-l)
+- Support flexible secret passing methods in workflows ([#225](https://github.com/apify/apify-test-tools/pull/225)) ([22f7e80](https://github.com/apify/apify-test-tools/commit/22f7e807ca3949656c15b79f848a192d71b15946)) by [@metalwarrior665](https://github.com/metalwarrior665), closes [#205](https://github.com/apify/apify-test-tools/issues/205)
+- [**breaking**] Require env vars of test versions in sync with the default version ([#217](https://github.com/apify/apify-test-tools/pull/217)) ([a2481dd](https://github.com/apify/apify-test-tools/commit/a2481ddf09004308cb54e49b783c8b21a25dcbe4)) by [@metalwarrior665](https://github.com/metalwarrior665)
+
 ### 🐛 Bug Fixes
 
 - **toBeObject:** Reject null ([#191](https://github.com/apify/apify-test-tools/pull/191)) ([6bdae49](https://github.com/apify/apify-test-tools/commit/6bdae4905756e0ae8dd6146942e6fc04dbead48e)) by [@JuanGalilea](https://github.com/JuanGalilea), closes [#88](https://github.com/apify/apify-test-tools/issues/88)
 - Build and test on test-only changes even when .dockerignore lists the tests ([#177](https://github.com/apify/apify-test-tools/pull/177)) ([5cb1b5a](https://github.com/apify/apify-test-tools/commit/5cb1b5ab5f6f43ec279f9624b402e73d55a5974a)) by [@artogahr](https://github.com/artogahr), closes [#145](https://github.com/apify/apify-test-tools/issues/145)
+- **git:** List both paths of moved files when detecting changes ([#215](https://github.com/apify/apify-test-tools/pull/215)) ([44ba565](https://github.com/apify/apify-test-tools/commit/44ba565242c1ac3daea281a648acb28712bb4363)) by [@metalwarrior665](https://github.com/metalwarrior665)
+
+### 🚜 Refactor
+
+- [**breaking**] Remove build-for-test and release composite commands ([#203](https://github.com/apify/apify-test-tools/pull/203)) ([85efc54](https://github.com/apify/apify-test-tools/commit/85efc54697e98208c827c4f450906e00eebcd9ca)) by [@metalwarrior665](https://github.com/metalwarrior665)
 
 
 <!-- git-cliff-unreleased-end -->

package/CONTRIBUTING.md

@@ -56,6 +56,8 @@ npx tsx /path/to/apify-test-tools/bin/main.ts --help
 npx tsx /path/to/apify-test-tools/bin/main.ts get-commits --target-branch master --source-branch feat/testing-feature-branch
 ```
 
+Or clone a test repo into `local-clone/` inside this repo and run `npm run start:dev -- <command>`.
+
 #### Working on the library
 
 You need to istall the local version of `apify-test-tools` in your cloned `testing-repo-for-github-actions`:
@@ -85,28 +87,14 @@ The tag is not semver, so `git-cliff` would abort on it. It is skipped only beca
 The action's "Nothing to release" guard resolves to this tag and no longer fires, so an empty stable
 release fails at `npm publish` instead of erroring cleanly.
 
-Consumers pin `@workflows-v0`, not `@master`. See
-[Versioning and releases](./README.md#versioning-and-releases) in the README for how the tag and the
-npm release relate — the short version:
-
-- changing only a workflow needs no npm release
-- changing only the package needs no workflow change
-- a workflow that calls a **new** CLI feature must set `.github/workflows-package-version` to the
-  version that will contain it, in the same PR. The `workflows-v0` tag is then held until that version is on
-  npm, so merging can't ship a workflow that calls a CLI that doesn't exist yet.
-
-`.github/workflows-package-version` holds one exact version, and the stable release writes it. The
-workflows install exactly it, which is what makes a frozen major tag stay frozen — it keeps the
-library it was tested with instead of following `latest` forever.
-
-The `Package version bump needed` check enforces the point above: a PR touching both a `public_`
-workflow (or the composite action) and `bin/`, `lib/` or `index.ts` has to move the pin. When the
-two changes are unrelated and the workflow doesn't need the new code, label the PR
-`no-version-bump-needed`.
+Consumers pin `@workflows-v0`, not `@master`. See [Versioning and releases](#versioning-and-releases) below.
 
-It only looks at a single PR, so it won't catch a workflow that starts using a CLI feature merged in
-an earlier, still-unreleased PR. That needs someone to land a CLI change and sit on it unreleased;
-catching it would mean flagging every workflow edit made while any package change is unreleased.
+- `npm test` covers the bundle without network (`test/bundle/`). `npm run test:e2e` runs it against
+  dummy Actors in an isolated sandbox account (`test/e2e/`) and skips itself unless
+  `E2E_TEST_APIFY_TOKEN_DBYPODOIHUZABNQTR` is set. CI passes that secret only to the `E2E tests` job.
+  The git `build` and release pipeline tests clone
+  [testing-repo-for-github-actions](https://github.com/apify-store/testing-repo-for-github-actions) and
+  use its `e2e/base` and `e2e/feature` branches as fixtures, so never rebase or delete those.
 
 `npm run lint` and `actionlint` (via the `Code checks` workflow) both gate master, so run them before
 pushing workflow changes.
@@ -115,3 +103,99 @@ pushing workflow changes.
 reusable workflow runs with the caller's repo checked out and never gets its own. Its `prompt-ref`
 input defaults to `workflows-v0` so the instructions come from the same release as the workflow — leaving it
 at `master` would run released workflows against unreleased instructions.
+
+## Versioning and releases
+
+The workflows and the npm package live in one repo but ship on their own schedules:
+
+| What                                         | Consumers get it through                   | Ships when              |
+| -------------------------------------------- | ------------------------------------------ | ----------------------- |
+| the workflows and the CLI they call          | the `@workflows-v0` tag in `uses:`         | every master push       |
+| the lib (`testActor`, `describe`, matchers…) | `apify-test-tools` in their `package.json` | a stable release on npm |
+
+The CLI is bundled with its dependencies into `.github/cli/apify-test-tools.mjs` (`npm run bundle`, run by the
+pre-commit hook and checked in CI), and the setup action runs it from there. Nothing is installed for it, so a tag is
+a complete statement: these workflows and this CLI, with the exact dependency versions from `package-lock.json` at
+that commit. The consumer's own `node_modules` is left alone, so their tests keep the lib version they locked.
+
+- **Workflow or CLI change** — merge it. `workflows-v0` moves once checks and tests pass, and it goes live. No
+  release needed, even when a workflow starts using a new CLI feature: both are in the same commit.
+- **Lib change** — merge it, then cut a release when you want it out. Consumers pick it up when they update
+  `apify-test-tools`.
+
+The CLI and the lib in one job can be different versions now, so the formats between them are a contract: the
+`ACTOR_BUILDS` JSON that `build` writes and the lib reads, and the vitest JSON report `report-tests` reads. Add
+fields freely; renaming or removing one needs a new major tag.
+
+`workflows-v0` moving on every master push means `@workflows-v0` is as live as `@master` was — there's no staging
+step. What the tag buys you is a `workflows-v1` for breaking workflow changes, so repos migrate one at a time instead
+of all at once, and a way to roll back by pointing the tag at an earlier commit (**Move major version tag**
+workflow). Bump `MAJOR_TAG` in `.github/workflows/_move_major_tag.yaml` to cut the next major; the old tag then
+freezes where it is and keeps working, CLI included, since the bundle is part of the frozen commit.
+
+The tag tracks the **workflows'** contract, not the npm package version. They move independently on purpose, so
+`@workflows-v0` is expected to stay `@workflows-v0` after the package reaches 1.0 — bump it when a workflow breaks
+its callers, not when the package does. Because `uses:` cannot take an expression, every ref into this repo repeats
+the tag literally; `check-major-tag-refs.mjs` fails the build if `MAJOR_TAG` and those refs disagree, which is the
+mistake that would otherwise ship silently during a bump.
+
+## Testing workflow changes
+
+- Point [testing-repo-for-github-actions](https://github.com/apify-store/testing-repo-for-github-actions) at your
+  branch (`uses: ...@your-branch`). It has real attached Actors and tests. The branch carries its own CLI bundle, so
+  one branch tests the workflows and the CLI together.
+- To change the composite action itself, repoint the `uses:` refs inside the reusable workflows at your branch as
+  well, and change them back before merging.
+- Make sure the shell code actually works on your laptop first.
+- After merging, watch the workflow on a real project before moving on.
+
+## How the workflows scope secrets
+
+The public side of this is the [Secrets](./docs/workflows.md#secrets) table.
+
+The Actor tokens are the one set that cannot be listed in the workflow, because each Actor names its own token via
+`tokenEnvVar` in `apify-test-tools.config.json`, and a reusable workflow only accepts secrets it declares by name.
+Callers pack them into the `CONFIG_SECRETS` secret. Those steps pass `${{ toJSON(secrets) }}` as `WORKFLOW_SECRETS`
+and run the command through `.github/scripts/run-with-apify-tokens.mjs`, which reads that same config file to decide
+which secrets to pass on. It looks each one up in `CONFIG_SECRETS` first, then among the secrets `secrets: inherit`
+brought:
+
+```yaml
+- name: Build
+  shell: bash
+  env:
+      WORKFLOW_SECRETS: ${{ toJSON(secrets) }}
+  run: |
+      jq '{ actors: .functionallyChangedActors }' affected.json \
+        | node "${{ steps.setup.outputs.scripts-path }}/run-with-apify-tokens.mjs" \
+          apify-test-tools build --version-number 0.99
+```
+
+The wrapper passes only the tokens the config declares and drops `WORKFLOW_SECRETS`, so neither the CLI nor anything
+under `node_modules` sees the blob. Nothing is written to `$GITHUB_ENV`, so the tokens stay inside that one command
+rather than leaking into later steps. Reading the same file `apify-test-tools` reads means the two can't drift, and a
+secret that merely looks like an Actor token is not passed just because of its name.
+
+A token the config declares but the repo hasn't set is a warning, not a failure: a repo can carry an Actor whose
+token isn't configured and still build fine as long as that Actor never changes, and `apify-test-tools` raises a
+precise error naming the Actor at the point it actually needs the token.
+
+The runner masks a secret by its exact value, so it only knows `CONFIG_SECRETS` as a whole blob. The setup action
+runs `.github/scripts/mask-config-secrets.sh` first, which registers each value inside it with `::add-mask::`, so a
+token taken out of the blob is masked too.
+
+`scripts-path` comes from the setup action (give the step `id: setup`) and points at this repo's `.github/scripts/`
+directory inside the runner's action checkout, so workflows can run these helpers without checking this repo out
+again. The caller's workspace holds the caller's repo, not this one.
+
+Two tidier-looking alternatives don't work, so don't reach for them:
+
+- **Exporting to `$GITHUB_ENV`** would let the steps call the CLI directly with no wrapper, but `$GITHUB_ENV`
+  applies to every later step in the job. In `pr-build-test` the vitest step runs after the build, so it would
+  inherit Actor tokens it has no use for.
+- **Returning the tokens as a step output** would be scoped correctly, but the runner refuses to set an output whose
+  value contains a registered secret. It logs `Skip output <name> since it may contain secret` and leaves the output
+  empty, so anything reading it downstream gets nothing.
+
+The `unitTest` job runs static checks and needs `NPM_TOKEN` only. No job runs `npm ci` or `pnpm install` with Apify
+or Slack credentials in scope, so a postinstall script in the dependency tree cannot read them.