@gfdelarue/testkit 0.6.0 → 0.6.2

package/docs/bug-records.md

@@ -93,31 +93,37 @@ Testkit redacts the secrets it recognises in the title, in every section, and in
 
 - The value of a `KEY=value` assignment whose key names a secret.
 - The password in a URL.
-- The value of an `Authorization`, `Proxy-Authorization`, `Cookie`, or `Set-Cookie` header.
+- The value of an `Authorization`, `Proxy-Authorization`, `Cookie`, `Set-Cookie`, `Api-Key`, or `X-Api-Key` header. When a double-quoted string holds the header, the value ends at the quote that closes the string, at the string's own escape depth, so a value that ends in an escaped backslash ends there too.
 - A JSON string value whose key names a secret, including JSON that is itself inside a JSON string, as in `{"message":"{\"token\":\"…\"}"}`. Testkit reads the key's escapes, so `"\u0074oken"` names a secret too.
-- The value of a single-quoted `'password': '…'` member, in any letter case. It ends at the next apostrophe that no backslash precedes, or with the line.
+- Every string in a JSON array or object whose key names a secret, as in `{"tokens":["…"]}`, unless the key names a secret only through `AUTH`, `COOKIE`, or `SESSION`. So `session` and `auth` settings stay. In an object with a `name` string and a `value` string, only the `value` and members whose own key names a secret count. Numbers and booleans stay. In a `cookies` array, the `value` string of each object directly in it counts, and the rules above still apply inside it.
+- The value of a single-quoted member whose key names a secret, as in `{'token': '…'}`. It ends at the next apostrophe that no backslash precedes, or with the line.
+- The value after a key whose last word is `PASSWORD` or `PASSWD` and a colon, outside JSON, as in `{ password: '…' }` or a YAML `password: …` line. A quoted value counts anywhere and ends as a quoted assigned value does. An unquoted value counts at the start of a line, after spaces and an optional `- `, when it is the line's one word. A value that is one word of letters, perhaps followed by `.`, `,`, `;`, `:`, `!`, `?`, or `)`, stays, so `password: 'Required'`, `password: required!`, and `rejects a password: too short` stay as they are.
+- The token after `Bearer` and a space or a tab, wherever it appears, when it is at least eight characters long and holds a digit. The token is made of letters, digits, `-`, `.`, `_`, `~`, `+`, and `/`, and may end in `=` signs. `Bearer authentication` stays as it is.
+- The value in a `"name": "…", "value": "…"` pair whose name names a secret, as traces store headers, as in `{"name":"Authorization","value":"…"}`.
 
-A key names a secret when it contains `PASSWORD`, `TOKEN`, `SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY` in any letter case. Still keep credentials, tokens, and passwords out of the record.
+A key names a secret when it contains `PASSWORD`, `TOKEN`, `SECRET`, `PRIVATE_KEY`, `AUTH`, `COOKIE`, `SESSION`, or `API_KEY` in any letter case, or when its last word is `PASS`, `PASSWD`, `PAT`, `DSN`, `CREDENTIAL`, or `CREDENTIALS`. `AUTH` doesn't count in `AUTHOR` or `AUTHORITY`, so `GIT_AUTHOR_NAME` and `author` name no secret, though `AUTHORIZATION` and `AUTHORISE` still do. `SESSION` doesn't count when an optional `S`, at most one other character, and then `COUNT`, `DURATION`, `EXPIR`, `LENGTH`, `LIMIT`, `MAX`, `TIMEOUT`, or `TTL` follow it, as in `sessionCount` and `SESSION_TIMEOUT`. A key whose last two words are `API`, `PRIVATE`, `SIGNING`, or `ENCRYPTION` and then `KEY`, as in `apiKey`, `x-api-key`, and `privateKey`, names a secret too, and so does an upper-case key whose last word is `KEY`, such as `STRIPE_KEY` or `SIGNING_KEY`, unless the word before it is `PUBLIC` or `PUBLISHABLE`. Words break at any character other than a letter or a digit, and before an upper-case letter that follows a lower-case letter or a digit, so `GITHUB_PAT` and `dbPass` name secrets and `PATH`, `BYPASS`, and `cacheKey` don't. Still keep credentials, tokens, and passwords out of the record.
 
 An assigned value ends at the first space, with four exceptions.
 
 - A value that starts with `"` or `'` ends at the next such quote on its line. When the line has no such quote, the value ends with the line.
 - A quoted value inside a JSON string (`\"`), or inside JSON that is itself inside a JSON string (`\\\"`), ends at the next quote escaped to the same depth. A quote escaped more deeply belongs to the value. The value ends sooner at a quote escaped less deeply, such as the quote that closes the JSON string, at an escaped line break at its own depth or less, or with the line. A value quoted with an escaped apostrophe, `\'`, works the same way. JSON doesn't escape apostrophes, so each JSON layer only doubles the backslashes (`\\'`). A plain apostrophe inside the value belongs to it.
 - A value that starts with `Basic`, `Bearer`, `Digest`, `Negotiate`, or `Token` and then a space or a tab ends with the line.
-- Under a key that ends in `_URL`, Testkit keeps a value that starts with a scheme, such as `https://`, and redacts the URL's password only.
+- Under a key that ends in `_URL`, Testkit keeps a value that starts with a scheme, such as `https://`, or with a single `/`, such as `/oauth/authorize`. A value that starts with `//` is redacted whole. It redacts what the other rules find in it, such as the URL's password or a `token=` query parameter.
 
-When `testkit run` stores a result, it redacts JSON one string value at a time. That covers text that is a JSON object, array, or string, and a line whose text from its first `{` or `[` is JSON. Each string value gets the rules above, and so does JSON inside a string. A value that runs to a space or to the end of its line therefore ends with its string, and the record's other values stay as they are. Other text gets the rules as written, and so does JSON in two cases:
+When `testkit run` stores a result, it redacts JSON one string value at a time. That covers text that is a JSON object, array, or string, and a line whose text from its first `{` or `[` is JSON. Each string value gets the rules above, and so does JSON inside a string. A value that runs to a space or to the end of its line therefore ends with its string, and the record's other values stay as they are. In an object with `name` and `value` members, in either order, the `value` string is redacted when the name names a secret, and always when the object is in a `cookies` array. Other text gets the rules as written, and so does JSON in two cases:
 
 - a rule reading the whole line would redact across the line's first `{` or `[`;
 - a secret Testkit resolved from the run's environment, whole or cut short, crosses the JSON's edge or spans more than one string.
 
 A string that holds such a secret in escaped form is redacted as decoded text when the escaped text decodes to the secret, or, for a secret cut short, to the start of one, and redacted whole otherwise. Where these rules and a resolved secret overlap, Testkit redacts every character that either would redact.
 
+Testkit applies every rule to the same text and redacts each character that any rule finds. A value that runs to a space therefore doesn't hide the secret after it, so `PASSWORD=…"Cookie: sid=…` loses both.
+
 Testkit writes `<redacted>` in place of the value. A quoted value keeps its quotes, as in `PASSWORD="<redacted>"`, so redacting the text again changes nothing. A `<redacted>` that the text already holds counts as part of the value, so `TOKEN=<redacted>example-secret` becomes `TOKEN=<redacted>`. A secret resolved from the run's environment can itself contain part of `<redacted>`, or a character Testkit reserves. Such a secret can match text that redaction creates, so where it takes part, `testkit run` redacts the whole line of a log, or the whole JSON string, when it stores the result. When such a secret spans lines, or takes part only across lines, `testkit run` redacts every line between the nearest JSON lines instead, or the whole text when it holds none.
 
-A URL password is the text between `://user:` and the last `@` before the next `/` or space. The user can be empty, as in `redis://:password@cache`. Testkit does not recognise a password that holds a `/`, or a token that stands alone as the user, as in `https://token@host`.
+A URL password is the text after `://user:`, up to the first `@` whose host runs to a `/`, `?`, `#`, space, or the end of the text with no other `@`. The user can be empty, as in `redis://:password@cache`. The password can hold `/` and `@`, as in `postgres://app:pa/ss@db`, A password that starts with digits and then a `/`, `?`, or `#`, and also ends with one of them before the `@`, reads as a port and a path, so `https://registry.example.com:8443/@scope/pkg` stays and `postgres://app:1234/secret@db` loses `1234/secret`. The user can't hold `[` or `]`, so an IPv6 host such as `http://[::1]:5173/@vite/client` stays. A directory name keeps one slash of a URL, so Testkit also reads `postgres:/app:password@db`. In an `http` or `https` URL, a user that stands alone, as in `https://token@host`, is redacted when it is at least 20 characters of letters, digits, `-`, `.`, `_`, and `~`, and holds a digit, so `https://ada@host` and `ssh://deploy2024@host` stay.
 
-Testkit does not rewrite a path or an area's name. `testkit bug record` rejects a record whose `test.file` or `fault.path` holds text that Testkit redacts. `testkit bug file` stops when the area's name or the reproduce command holds such text.
+Testkit does not rewrite a path or an area's name. `testkit bug record` rejects a record whose `test.file`, `fault.path`, or reproduce command holds text that Testkit redacts. The reproduce command holds the product directory's path, the test's path from it, and the `--vitest-config` and `--vitest-project` selectors. `testkit bug file` stops when the area's name or the reproduce command holds such text.
 
 Testkit reads the facts back from the body before it files the issue. It files nothing when they do not read back as written. That happens when `test.title`, `test.file`, `fault.path`, or the area's name holds one of the two marker comments.
 
@@ -234,7 +240,7 @@ A named failure's title comes from the engine.
 | Vitest     | The names of the `describe` blocks around the test and the test's own name, joined by single spaces.  |
 | Playwright | The names of the `describe` blocks around the spec and the spec's own title, joined by single spaces. |
 
-A Playwright title holds neither the file path nor the project name. A spec that fails in two projects is one named failure.
+A Playwright title holds neither the file path nor the project name. A spec that fails in two projects is one named failure. When a Playwright config sets `failOnFlakyTests`, a spec that passes only on a retry fails the run, and it is a named failure.
 
 When a file reports named failures, `testkit bug file` rejects a bug record whose `test.title` is not one of their titles. Its message lists the titles a bug record can use.
 
@@ -248,8 +254,8 @@ A file also reports one unnamed failure beside its named failures when the engin
 
 - A Vitest file runs past its timeout, writes more output than Testkit keeps, or ends on a signal or with an exit code other than 1.
 - A Playwright run is interrupted or runs past its timeout.
-- A Playwright process ends on a signal or with an exit code other than 1. A reporter in the Playwright config that ends the process does that.
-- A Playwright config sets `failOnFlakyTests` and a spec passes only on a retry. A spec that passes on a retry is no named failure, so no issue can cover it.
+- A Playwright process ends on a signal, with an exit code other than 1, or before every reporter in the Playwright config has finished. A reporter that ends the process from one of its hooks does that, whatever its exit code. A reporter that ends the process with exit code 1 after its hooks have returned, such as from a timer, still reads as the failed specs.
+- Playwright does not run a spec of the selected file that was expected to run. A failed spec stops the later specs of a serial group, a failed `beforeAll` hook stops the specs after the one it failed, and a run that reaches `maxFailures` stops the rest. A spec that the file skips with `test.skip` or `test.fixme` is not such a spec.
 - A Playwright spec fails in a file other than the selected one, such as a spec in a project that the selected file's project depends on or a spec in a teardown project. That spec is no named failure of the selected file, so no issue can cover it. A spec's file is the file that Playwright loaded. A compiled file whose source map names the selected file is still another file.
 - Testkit cannot keep a file that a Playwright spec attached, or cannot store the evidence of the Playwright run.
 

package/docs/engines.md

@@ -55,6 +55,45 @@ Back restores a page from the back/forward cache, drop that flag in the file:
 test.use({ launchOptions: { ignoreDefaultArgs: ["--disable-back-forward-cache"] } });
 ```
 
+Testkit starts managed Chromium with its sandbox on, which confines a
+compromised renderer. Playwright on its own starts Chromium with
+`--no-sandbox`, where a renderer runs with the user's access to the checkout and
+their files. A project's `launchOptions` and a file's `test.use` keep the
+config's `chromiumSandbox` unless they set it.
+
+Some hosts can't start the sandbox, such as root in a container, or a Linux host
+that doesn't allow unprivileged user namespaces. There Chromium fails to launch,
+and Testkit reports an infrastructure failure that names the opt-out. Turn the
+sandbox off in `playwright.config.ts`:
+
+```ts
+export default defineTestkitConfig({
+  use: { launchOptions: { chromiumSandbox: false } },
+});
+```
+
+`playwright.config.ts` sees only the environment that its group's services
+declare, so to turn the sandbox off on one host only, pass the host's variable
+through a service's `env` in `testkit.config.ts` and read it in
+`playwright.config.ts`:
+
+```ts
+// testkit.config.ts, in a service the UI group uses
+env: { CHROMIUM_SANDBOX: process.env.CHROMIUM_SANDBOX ?? "on" },
+
+// playwright.config.ts
+use: { launchOptions: { chromiumSandbox: process.env.CHROMIUM_SANDBOX !== "off" } },
+```
+
+`app.node(...)` and `app.next(...)` take the same value as
+`env: { values: { CHROMIUM_SANDBOX: … } }`.
+
+The config's value also applies to browsers that `globalSetup` and
+`globalTeardown` launch.
+
+A file can turn it off for itself with
+`test.use({ launchOptions: { chromiumSandbox: false } })`.
+
 ## Typechecking
 
 `testkit typecheck` groups the discovered test files by engine and by their
@@ -83,7 +122,14 @@ over conflicting consumer packages. A helper shared by several engines must
 typecheck in each engine's program. Testkit's generated configs don't set
 `baseUrl`, so they need no deprecation suppression and you never edit them.
 Aliases inherited from a config with `baseUrl` keep their target directory and
-fallback order.
+fallback order. The generated configs also clear the inherited output paths,
+`outDir`, `outFile`, `declarationDir`, `sourceRoot`, `mapRoot`, and, for test
+programs, `rootDir`. TypeScript holds every file to an explicit `rootDir`, and
+when an output path is set without one, TypeScript 6 uses the tsconfig's own
+directory. With both cleared, a test file outside the tsconfig's `rootDir`, or
+outside TypeScript 6's default, still type-checks. When the tsconfig sets `incremental`
+or `composite`, Testkit writes its build info beside the generated config in
+`.testkit/_typecheck`, never over the repository's own.
 
 A plain file's program also includes the TypeScript setup files of its Vitest
 project, because Vitest loads them before the file and a setup file can declare
@@ -123,8 +169,10 @@ support files that join the Playwright program. Use it for global setup and
 teardown, reporters, fixtures, and dependency and teardown projects, including
 every variant of a multi-project config. An empty companion can supply options
 alone. Without a companion, Testkit checks the implicit roots, what they import,
-and the existing Next ambient and fixture defaults. Project references don't
-run.
+and the existing Next ambient and fixture defaults. A companion replaces those
+Next defaults: list `next-env.d.ts`, `tests/playwright-fixtures`, or generated
+types such as `.next/types` in it when the suites need them. Project references
+don't run.
 
 TypeScript follows imported helpers, but it can't find a file that only a
 runtime string or function names. Include those files in the companion.
@@ -183,7 +231,7 @@ environment:
 | Engine        | Reserved prefixes                                | Other reserved keys                                                                                                                                                                                                                                                                                                                                                                 | Consumer environment limit                                                                 |
 | ------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
 | `node-vitest` | none                                             | `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`, `UPDATE_SNAPSHOT`, `VITEST_POOL_ID`, `VITEST_WORKER_ID`, `TESTKIT_ACTIVE`, `TESTKIT_DATA_ARTIFACT_DIR`, `TESTKIT_INVOCATION_ID`, `TESTKIT_LEASE_DIR`, `TESTKIT_LEASE_ID`, `TESTKIT_MODE`, `TESTKIT_NODE_VITEST_INPUT_FILE`, `TESTKIT_NODE_VITEST_INPUT_SHA256`, `TESTKIT_RUNTIME_ID`, `TESTKIT_TEST_FILE`, `TESTKIT_TEST_ID` | 512 entries and 4 MiB of UTF-8 key/value data; 8 MiB for all of the test's inputs together |
-| `playwright`  | `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, `PW_TEST_` | `BASE_URL`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`                                                                                                                                                                                                                                                                                                                               | 512 entries and 2 MiB of UTF-8 key/value data; 4 MiB for all of the test's inputs together |
+| `playwright`  | `TESTKIT_`, `PLAYWRIGHT_`, `PWTEST_`, `PW_TEST_` | `BASE_URL`, `DBUS_SESSION_BUS_ADDRESS`, `FC_FONTATIONS`, `FORCE_COLOR`, `NODE_OPTIONS`, `NO_COLOR`                                                                                                                                                                                                                                                                                  | 512 entries and 2 MiB of UTF-8 key/value data; 4 MiB for all of the test's inputs together |
 
 Prefixes and keys match in any letter case. The processes Testkit starts have
 separate environment limits, which consumer values don't count toward: 24,576