eslint-plugin-test-governance 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -1
- package/README.md +64 -34
- package/dist/rules/no-conflicting-states.d.ts +1 -1
- package/dist/rules/no-conflicting-states.js +7 -7
- package/dist/utils/analyze.js +34 -14
- package/dist/utils/constants.d.ts +4 -1
- package/dist/utils/constants.js +3 -1
- package/dist/utils/frameworks.d.ts +7 -0
- package/dist/utils/frameworks.js +9 -3
- package/dist/utils/options.d.ts +9 -5
- package/dist/utils/options.js +4 -2
- package/dist/utils/schema.js +10 -4
- package/docs/rules/marker-matches-state.md +3 -3
- package/docs/rules/no-conflicting-states.md +9 -8
- package/docs/rules/require-ticket.md +8 -7
- package/docs/upgrading.md +50 -0
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,46 @@ All notable changes to this project are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.6.0] - 2026-10-14
|
|
10
|
+
|
|
11
|
+
This release renames the `unstable` state to `quarantined`.
|
|
12
|
+
[Upgrading to 0.6](https://github.com/Huldoser/eslint-plugin-test-governance/blob/main/docs/upgrading.md#upgrading-to-06)
|
|
13
|
+
shows what to change.
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
- `states` can rename the tag of the `new` and `quarantined` states, for teams that call them
|
|
18
|
+
something else: `states: { quarantined: { tag: '@flaky', marker: 'FLAKY' } }` makes `@flaky` tests
|
|
19
|
+
need a `// FLAKY:` marker. The rules check the new tag as they did the default one, and
|
|
20
|
+
`@quarantined` no longer counts.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
|
|
24
|
+
- **Breaking:** the `unstable` state is renamed to `quarantined`. `lifecycleTags: true` now turns on
|
|
25
|
+
`@quarantined`, with the `// QUARANTINED:` marker, instead of `@unstable` and `// UNSTABLE:`, so rename
|
|
26
|
+
those in your tests, or keep them with
|
|
27
|
+
`states: { quarantined: { tag: '@unstable', marker: 'UNSTABLE' } }`. A config that sets
|
|
28
|
+
`states.unstable` fails with a message that names `states.quarantined`. A config that only sets
|
|
29
|
+
`lifecycleTags: true` still loads, but no longer checks `@unstable` tests.
|
|
30
|
+
|
|
31
|
+
## [0.5.1] - 2026-10-13
|
|
32
|
+
|
|
33
|
+
### Fixed
|
|
34
|
+
|
|
35
|
+
- In Jest and Vitest, a test or describe whose title is a variable and whose body is defined
|
|
36
|
+
elsewhere, as in `test.skip(title, placeOrder)`, is checked. Before, it was read as Playwright's
|
|
37
|
+
runtime `test.skip(condition, description)` and left alone, though Jest and Vitest have no such call.
|
|
38
|
+
In Playwright, a title kept in a `const` in the same file now counts as a title too.
|
|
39
|
+
- Vitest's `test.skip('title')` and `describe.skip('title')` with no body need a `// SKIP:` marker, since
|
|
40
|
+
Vitest reports them as skipped. Before, only `.todo` was checked without a body.
|
|
41
|
+
- Vitest's `context.skip('note')` always skips, so it needs a `// SKIP:` marker. Before, the note was
|
|
42
|
+
read as a condition and the skip as conditional.
|
|
43
|
+
- In Playwright, `describe` taken out of `test`, as in `const { describe } = test`, works like
|
|
44
|
+
`test.describe`: `describe.skip(...)` needs a marker, and tags in its title apply to the tests inside.
|
|
45
|
+
Before, it wasn't recognized.
|
|
46
|
+
- A function declared inside another function that happens to be called `test` is no longer taken for
|
|
47
|
+
the framework's `test`, as a local variable already wasn't.
|
|
48
|
+
|
|
9
49
|
## [0.5.0] - 2026-10-08
|
|
10
50
|
|
|
11
51
|
This release changes how the plugin is configured, ahead of 1.0.
|
|
@@ -245,7 +285,9 @@ First release.
|
|
|
245
285
|
- Support for ESLint 9 and 10 flat config, JavaScript and TypeScript test files, `.extend()`
|
|
246
286
|
fixtures, describe blocks, runtime skips and tags in titles or the `tag` option.
|
|
247
287
|
|
|
248
|
-
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.
|
|
288
|
+
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.6.0...HEAD
|
|
289
|
+
[0.6.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.5.1...v0.6.0
|
|
290
|
+
[0.5.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.5.0...v0.5.1
|
|
249
291
|
[0.5.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.4.0...v0.5.0
|
|
250
292
|
[0.4.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.3.1...v0.4.0
|
|
251
293
|
[0.3.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.3.0...v0.3.1
|
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
7
|
ESLint rules that keep a Playwright, Jest or Vitest suite honest. Every skipped, fixme, todo, `@new`
|
|
8
|
-
or `@
|
|
8
|
+
or `@quarantined` test must point at the ticket that tracks it:
|
|
9
9
|
|
|
10
10
|
<!-- example: valid -->
|
|
11
11
|
|
|
@@ -20,8 +20,8 @@ Without the comment, `test-governance/require-ticket` fails the lint. When the t
|
|
|
20
20
|
skip removed, `no-orphaned-marker` flags the leftover comment. The markers are plain
|
|
21
21
|
comments, so `grep -rn "SKIP:" tests/` lists every disabled test and its ticket.
|
|
22
22
|
|
|
23
|
-
> **Status:** early release (0.x). Options may still change before 1.0. Coming from 0.
|
|
24
|
-
> [Upgrading to 0.
|
|
23
|
+
> **Status:** early release (0.x). Options may still change before 1.0. Coming from 0.5?
|
|
24
|
+
> [Upgrading to 0.6](docs/upgrading.md#upgrading-to-06) shows what to change.
|
|
25
25
|
|
|
26
26
|
## Install
|
|
27
27
|
|
|
@@ -90,7 +90,7 @@ export default [
|
|
|
90
90
|
framework: 'playwright',
|
|
91
91
|
// Only accept Jira keys from these projects (or Jira URLs).
|
|
92
92
|
ticket: { preset: 'jira', projects: ['TRADE', 'RISK'] },
|
|
93
|
-
// Turn on the @new and @
|
|
93
|
+
// Turn on the @new and @quarantined states.
|
|
94
94
|
lifecycleTags: true,
|
|
95
95
|
// Any tag your team uses, with the same rules as the built-ins.
|
|
96
96
|
customStates: {
|
|
@@ -129,19 +129,20 @@ export default [
|
|
|
129
129
|
`configure()` for the same files keeps that config's options and changes only the framework. All three
|
|
130
130
|
turn on the same rules. What puts a test in a state in each framework:
|
|
131
131
|
|
|
132
|
-
| State (marker) | Playwright | Jest | Vitest
|
|
133
|
-
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
|
134
|
-
| `skip` (`// SKIP:`) | `test.skip(...)`, `test.describe.skip`, `test.skip()` in a body, `testInfo.skip()`, `test.step.skip`, `step.skip()` | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xtest`, `xdescribe` | `test.skip`, `it.skip`, `describe.skip`, `suite.skip`, `skipIf(true)`, `runIf(false)`, `{ skip: true }`, `context.skip()` or `skip()` in a body or in `beforeEach` |
|
|
135
|
-
| `todo` (`// TODO:`) | | `test.todo`, `it.todo` | `test.todo`, `describe.todo`, `{ todo: true }`
|
|
136
|
-
| `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | |
|
|
137
|
-
| `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }`
|
|
138
|
-
| `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | |
|
|
139
|
-
| conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()` | | `skipIf(condition)`, `runIf(condition)`, `{ skip: condition }`, `context.skip(condition)`, `if (...) context.skip()`
|
|
140
|
-
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new`
|
|
132
|
+
| State (marker) | Playwright | Jest | Vitest |
|
|
133
|
+
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
134
|
+
| `skip` (`// SKIP:`) | `test.skip(...)`, `test.describe.skip`, `test.skip()` in a body, `testInfo.skip()`, `test.step.skip`, `step.skip()` | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xtest`, `xdescribe` | `test.skip`, `it.skip`, `describe.skip`, `suite.skip`, `skipIf(true)`, `runIf(false)`, `{ skip: true }`, `context.skip()`, `context.skip('note')` or `skip()` in a body or in `beforeEach` |
|
|
135
|
+
| `todo` (`// TODO:`) | | `test.todo`, `it.todo` | `test.todo`, `describe.todo`, `{ todo: true }` |
|
|
136
|
+
| `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | | |
|
|
137
|
+
| `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }` |
|
|
138
|
+
| `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | | |
|
|
139
|
+
| conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()` | | `skipIf(condition)`, `runIf(condition)`, `{ skip: condition }`, `context.skip(condition)`, `if (...) context.skip()` |
|
|
140
|
+
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` |
|
|
141
141
|
|
|
142
142
|
Table-driven tests count too: `test.skip.each(table)(...)`, `describe.skip.each` and Vitest's `.for`.
|
|
143
143
|
A literal condition decides on its own: `skipIf(true)` and `runIf(false)` always skip, and `skipIf(false)`,
|
|
144
144
|
`runIf(true)` and `{ skip: false }` never do.
|
|
145
|
+
In Vitest, `test.skip('title')` and `describe.skip('title')` with no body are skipped too.
|
|
145
146
|
Vitest writes tag names without the `@`, as in `{ tags: ['flaky'] }`, so a name in `tags` counts as the
|
|
146
147
|
tag with an `@` in front: `'flaky'` is `@flaky`, and a custom state with `when: '@flaky'` matches it.
|
|
147
148
|
|
|
@@ -174,7 +175,7 @@ The defaults follow how most teams already work, so each framework config is use
|
|
|
174
175
|
| Setting | Default | Why |
|
|
175
176
|
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
176
177
|
| States that need a ticket | `skip`, `fixme`; in Jest and Vitest `skip`, `todo` | Every team skips tests. Chromium DevTools, GitLab and Slack all require a bug link on disabled tests. |
|
|
177
|
-
| `@new`, `@
|
|
178
|
+
| `@new`, `@quarantined` | off; `lifecycleTags: true` turns both on | A promotion workflow is not universal. Teams that say `@flaky` instead [rename the tag](#tag-names). |
|
|
178
179
|
| Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true` changes this. |
|
|
179
180
|
| `test.fail`, `test.slow`, Jest's `test.failing`, Vitest's `test.fails` | off | They don't remove coverage. Turn them on with `states: { fail: true, slow: true }`. |
|
|
180
181
|
| Marker format | `// SKIP: TRADE-123` or `// SKIP: TRADE-123, TRADE-124` | Uppercase keyword and colon, like `TODO:`. Easy to grep. |
|
|
@@ -193,20 +194,47 @@ yourself. `framework` is required, and every other option has a default. The rul
|
|
|
193
194
|
of their own: they all read these shared settings, so they always agree on ticket formats, states
|
|
194
195
|
and markers. To turn a rule off, set it to `'off'` as usual.
|
|
195
196
|
|
|
196
|
-
| Option | Type
|
|
197
|
-
| ----------------------------- |
|
|
198
|
-
| `framework` | `'playwright' \| 'jest' \| 'vitest'`
|
|
199
|
-
| `ticket` | `TicketSpec \| TicketSpec[]`
|
|
200
|
-
| `placeholders` | `string[]`
|
|
201
|
-
| `lifecycleTags` | `boolean`
|
|
202
|
-
| `states` | `{ [state]: boolean \| { enabled?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `new`, `
|
|
203
|
-
| `customStates` | `{ [name]: { when, marker, ticket? } }`
|
|
204
|
-
| `requireTicketForConditional` | `boolean`
|
|
205
|
-
| `allowBlankLine` | `boolean`
|
|
206
|
-
| `allowNotes` | `boolean`
|
|
207
|
-
| `comments` | `false \| { keywords?: string[] }`
|
|
208
|
-
| `testFunctions` | `string[]`
|
|
209
|
-
| `reportUnreadableTags` | `boolean`
|
|
197
|
+
| Option | Type | Default | Description |
|
|
198
|
+
| ----------------------------- | -------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
199
|
+
| `framework` | `'playwright' \| 'jest' \| 'vitest'` | required | The test framework of the files the config applies to. `configs.playwright`, `configs.jest` and `configs.vitest` set it, and `configure()` needs it. See [Frameworks](#frameworks). |
|
|
200
|
+
| `ticket` | `TicketSpec \| TicketSpec[]` | `{ preset: 'any' }` | Accepted ticket formats. An array accepts a ticket that matches any entry. See [Ticket presets](#ticket-presets). |
|
|
201
|
+
| `placeholders` | `string[]` | see above | Tickets rejected as placeholders. `*` matches any characters. Case-insensitive; a leading `#` is ignored. Your list replaces the default one, so repeat the defaults you want to keep. Tickets numbered 0 are always rejected. |
|
|
202
|
+
| `lifecycleTags` | `boolean` | `false` | Turns on the `new` (`@new`) and `quarantined` (`@quarantined`) states. |
|
|
203
|
+
| `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `new`, `quarantined`) on or off, renames their tag or marker, or gives them their own ticket format. `fail` and `slow` are off by default; a setting for `new` or `quarantined` here wins over `lifecycleTags`. Only `new` and `quarantined` have a tag to rename; see [Tag names](#tag-names). `fixme` and `slow` exist only in Playwright and `todo` only in Jest and Vitest, so turning one on elsewhere has no effect. |
|
|
204
|
+
| `customStates` | `{ [name]: { when, marker, ticket? } }` | `{}` | Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. `when` is a tag and `marker` an uppercase keyword; neither can be shared with another state. |
|
|
205
|
+
| `requireTicketForConditional` | `boolean` | `false` | Require a ticket for conditional skips too: `test.skip(condition, ...)`, `if (...) test.skip()`, a skip in a `catch` block or after an early `return` or `throw`. |
|
|
206
|
+
| `allowBlankLine` | `boolean` | `false` | Let blank lines separate a marker from its test. |
|
|
207
|
+
| `allowNotes` | `boolean` | `false` | Allow free text after the tickets, e.g. `// SKIP: TRADE-123 flaky on CI`. Applies to markers and to `FIXME` / `TODO` comments. |
|
|
208
|
+
| `comments` | `false \| { keywords?: string[] }` | `{ keywords: ['FIXME', 'TODO'] }` | Comment keywords that must start with a ticket anywhere in a test file, e.g. `// TODO: TRADE-123`. Your keywords replace the default ones. `false` allows free-form comments. |
|
|
209
|
+
| `testFunctions` | `string[]` | `['test']` | Names treated as the framework's `test`. Jest and Vitest default to `['test', 'it']`. Your list replaces the default one, so include those names if you still need them. See below. |
|
|
210
|
+
| `reportUnreadableTags` | `boolean` | `false` | When a tag state is on, report titles that aren't static text, and tags or details that can't be read in the file, such as ones imported from another module, since their tags can't be checked. |
|
|
211
|
+
|
|
212
|
+
### Tag names
|
|
213
|
+
|
|
214
|
+
Teams name the stages of a test differently, so `states` can rename the tag of `new` and
|
|
215
|
+
`quarantined`. The marker keeps its name unless you rename it too:
|
|
216
|
+
|
|
217
|
+
```js
|
|
218
|
+
testGovernance.configure({
|
|
219
|
+
framework: 'playwright',
|
|
220
|
+
lifecycleTags: true,
|
|
221
|
+
states: { quarantined: { tag: '@flaky', marker: 'FLAKY' } },
|
|
222
|
+
});
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
<!-- example: valid settings={"lifecycleTags":true,"states":{"quarantined":{"tag":"@flaky","marker":"FLAKY"}}} -->
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
// FLAKY: TRADE-311
|
|
229
|
+
test('streams level 2 quotes @flaky', async ({ page }) => {
|
|
230
|
+
// ...
|
|
231
|
+
});
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
The new tag replaces the default one, so `@quarantined` no longer puts a test in the state, and the
|
|
235
|
+
rules check `@flaky` as they checked `@quarantined`: `@new` together with `@flaky` is reported, and so
|
|
236
|
+
is a near-miss such as `@flakey`. For a tag that isn't one of these stages, such as `@needs-data`, add
|
|
237
|
+
a `customStates` entry.
|
|
210
238
|
|
|
211
239
|
### Ticket presets
|
|
212
240
|
|
|
@@ -234,7 +262,7 @@ testGovernance.configure({
|
|
|
234
262
|
framework: 'playwright',
|
|
235
263
|
ticket: { preset: 'jira', projects: ['TRADE'] },
|
|
236
264
|
lifecycleTags: true,
|
|
237
|
-
states: {
|
|
265
|
+
states: { quarantined: { ticket: { preset: 'github' } } },
|
|
238
266
|
});
|
|
239
267
|
```
|
|
240
268
|
|
|
@@ -266,16 +294,17 @@ In Jest and Vitest files:
|
|
|
266
294
|
A name imported or required from another test runner (Playwright, `vitest`, `@jest/globals`,
|
|
267
295
|
`node:test`, `bun:test`, `mocha`, `ava`, `tap`, `uvu`), or a variable set from one such as
|
|
268
296
|
`const test = require('ava')`, is never treated as the framework's `test`, and neither is a local
|
|
269
|
-
variable declared inside a function that happens to be called `test`.
|
|
297
|
+
variable or function declared inside a function that happens to be called `test`.
|
|
270
298
|
|
|
271
299
|
### Syntax the rules understand
|
|
272
300
|
|
|
273
301
|
The [Frameworks](#frameworks) table lists what each framework's tests can do. In Playwright:
|
|
274
302
|
|
|
275
303
|
- `test.skip('title', fn)`, `test.fixme(...)`, `test.fail(...)`, with or without a details object,
|
|
276
|
-
and with a body defined elsewhere, as in `test.skip('places a limit order', placeLimitOrder)`.
|
|
277
|
-
|
|
278
|
-
|
|
304
|
+
and with a body defined elsewhere, as in `test.skip('places a limit order', placeLimitOrder)`. The
|
|
305
|
+
title can also be a `const` declared in the same file.
|
|
306
|
+
- `test.describe.skip(...)` and `test.describe.fixme(...)`, including `serial` and `parallel` variants,
|
|
307
|
+
and `describe.skip(...)` after `const { describe } = test`. The state covers every test inside.
|
|
279
308
|
- Runtime calls inside a test, a describe or a hook such as `test.beforeEach`: `test.skip()`,
|
|
280
309
|
`test.skip(true)`, `test.fixme()`, `test.fail()`, `test.slow()`, the same methods on `testInfo`,
|
|
281
310
|
and `test.info().skip()`.
|
|
@@ -289,7 +318,8 @@ The [Frameworks](#frameworks) table lists what each framework's tests can do. In
|
|
|
289
318
|
|
|
290
319
|
Not covered in this version: tests generated in loops with dynamic titles, and tags or details imported
|
|
291
320
|
from other modules (see `reportUnreadableTags`), fixtures imported from other modules under a name not in
|
|
292
|
-
`testFunctions`,
|
|
321
|
+
`testFunctions`, a Vitest test with only a title, which Vitest runs as a todo but which looks the same as a
|
|
322
|
+
helper called `test`, Playwright annotations as markers, and other runners such as Mocha, Bun and `node:test`.
|
|
293
323
|
|
|
294
324
|
## TypeScript
|
|
295
325
|
|
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
declare const _default: import("../utils/create-rule.ts").Rule<"
|
|
1
|
+
declare const _default: import("../utils/create-rule.ts").Rule<"newAndQuarantined" | "newSkipped" | "tagCase" | "tagTypo">;
|
|
2
2
|
export default _default;
|
|
@@ -13,7 +13,7 @@ export default createRule({
|
|
|
13
13
|
fixable: 'code',
|
|
14
14
|
schema: [],
|
|
15
15
|
messages: {
|
|
16
|
-
|
|
16
|
+
newAndQuarantined: '`{{new}}` and `{{quarantined}}` are separate stages; a test should be in only one of them.',
|
|
17
17
|
newSkipped: "`{{new}}` on a skipped test: a new test that doesn't run can't be promoted. Fix the test or drop the tag.",
|
|
18
18
|
tagCase: '`{{found}}` should be written `{{expected}}`.',
|
|
19
19
|
tagTypo: '`{{found}}` looks like a typo of `{{expected}}`.',
|
|
@@ -22,7 +22,7 @@ export default createRule({
|
|
|
22
22
|
create(context, analysis, options) {
|
|
23
23
|
const tagStates = options.states.filter(isTagState);
|
|
24
24
|
const newTag = options.states.find((s) => s.name === 'new')?.tag;
|
|
25
|
-
const
|
|
25
|
+
const quarantinedTag = options.states.find((s) => s.name === 'quarantined')?.tag;
|
|
26
26
|
for (const subject of analysis.subjects) {
|
|
27
27
|
for (const occurrence of subject.tags)
|
|
28
28
|
checkSpelling(occurrence);
|
|
@@ -31,13 +31,13 @@ export default createRule({
|
|
|
31
31
|
const own = (tag) => subject.tags.find((t) => t.tag === tag);
|
|
32
32
|
const has = (tag) => own(tag) !== undefined || subject.inheritedTags.has(tag);
|
|
33
33
|
const ownNew = own(newTag);
|
|
34
|
-
if (
|
|
35
|
-
const occurrence = ownNew ?? own(
|
|
34
|
+
if (quarantinedTag !== undefined && has(newTag) && has(quarantinedTag)) {
|
|
35
|
+
const occurrence = ownNew ?? own(quarantinedTag);
|
|
36
36
|
if (occurrence) {
|
|
37
37
|
context.report({
|
|
38
38
|
node: occurrence.node,
|
|
39
|
-
messageId: '
|
|
40
|
-
data: { new: newTag,
|
|
39
|
+
messageId: 'newAndQuarantined',
|
|
40
|
+
data: { new: newTag, quarantined: quarantinedTag },
|
|
41
41
|
});
|
|
42
42
|
}
|
|
43
43
|
}
|
|
@@ -67,7 +67,7 @@ export default createRule({
|
|
|
67
67
|
return;
|
|
68
68
|
}
|
|
69
69
|
// A typo changes, adds, drops or swaps a letter or two. A tag that is two or more letters
|
|
70
|
-
// shorter or longer, like `@
|
|
70
|
+
// shorter or longer, like `@unquarantined` next to `@quarantined`, is a different word.
|
|
71
71
|
const near = tagStates.find((s) => {
|
|
72
72
|
const expected = s.tag.toLowerCase();
|
|
73
73
|
return (Math.abs(lower.length - expected.length) < 2 &&
|
package/dist/utils/analyze.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { isOtherRunner, RUNTIME_MODIFIERS } from "./frameworks.js";
|
|
2
2
|
const TEST = { kind: 'test', modifiers: [] };
|
|
3
|
+
const DESCRIBE = { kind: 'describe', modifiers: [] };
|
|
3
4
|
/** Modifiers that keep a test from running. */
|
|
4
5
|
const SKIPPING_MODIFIERS = new Set(['skip', 'fixme', 'todo']);
|
|
5
6
|
/** What a step's `TestStepInfo` can do at runtime: `step.skip()`. */
|
|
@@ -173,6 +174,15 @@ function collectTestNames(program, framework, configured) {
|
|
|
173
174
|
}
|
|
174
175
|
}
|
|
175
176
|
else if (declarator.id.type === 'ObjectPattern') {
|
|
177
|
+
// Playwright's `const { describe } = test`, used as `describe.skip(...)`.
|
|
178
|
+
if (framework.playwright && resolve(init)) {
|
|
179
|
+
for (const prop of declarator.id.properties) {
|
|
180
|
+
if (prop.type === 'Property' && prop.value.type === 'Identifier' && propertyName(prop) === 'describe') {
|
|
181
|
+
roots.set(prop.value.name, DESCRIBE);
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
continue;
|
|
185
|
+
}
|
|
176
186
|
const source = requireSource(init);
|
|
177
187
|
if (source === undefined)
|
|
178
188
|
continue;
|
|
@@ -209,8 +219,8 @@ function requireSource(node) {
|
|
|
209
219
|
return first?.type === 'Literal' && typeof first.value === 'string' ? first.value : undefined;
|
|
210
220
|
}
|
|
211
221
|
/**
|
|
212
|
-
* Whether `id` refers to a variable declared inside a function or block, such as a local
|
|
213
|
-
* happens to be called `test`. Top-level variables, imports, parameters and globals still count.
|
|
222
|
+
* Whether `id` refers to a variable or function declared inside a function or block, such as a local
|
|
223
|
+
* helper that happens to be called `test`. Top-level variables, imports, parameters and globals still count.
|
|
214
224
|
*/
|
|
215
225
|
function isLocalVariable(sourceCode, id) {
|
|
216
226
|
for (let scope = sourceCode.getScope(id); scope; scope = scope.upper) {
|
|
@@ -221,7 +231,7 @@ function isLocalVariable(sourceCode, id) {
|
|
|
221
231
|
// wraps the module.
|
|
222
232
|
if (scope.block.type === 'Program')
|
|
223
233
|
return false;
|
|
224
|
-
return variable.defs.some((def) => def.type === 'Variable');
|
|
234
|
+
return variable.defs.some((def) => def.type === 'Variable' || def.type === 'FunctionName');
|
|
225
235
|
}
|
|
226
236
|
return false;
|
|
227
237
|
}
|
|
@@ -478,28 +488,38 @@ function collectTags(sourceCode, call, hasTitle, framework, details) {
|
|
|
478
488
|
return { tags, dynamic: title.unread.size > 0, unread, opaqueDetails: details.unread !== undefined };
|
|
479
489
|
}
|
|
480
490
|
/**
|
|
481
|
-
* Whether a call declares a test or describe block: a title and a body, as in `test('title', fn)
|
|
482
|
-
* `test('title', details, fn)
|
|
483
|
-
*
|
|
484
|
-
*
|
|
491
|
+
* Whether a call declares a test or describe block: a title and a body, as in `test('title', fn)`,
|
|
492
|
+
* `test('title', details, fn)` or `test.skip(title, payWithCard)` with the body defined elsewhere, a
|
|
493
|
+
* describe with only a body, or a `todo` with only a title, and in Vitest a `skip` with only a title.
|
|
494
|
+
* Only Playwright has a runtime `test.skip(isMobile, 'why')`; there a title, written out or held in a
|
|
495
|
+
* constant, or a body written in place tells the two apart.
|
|
485
496
|
*/
|
|
486
|
-
function isDeclaration(kind, args,
|
|
497
|
+
function isDeclaration(sourceCode, framework, kind, args, members) {
|
|
487
498
|
const first = args.at(0);
|
|
488
499
|
if (isFunction(first))
|
|
489
500
|
return kind === 'describe';
|
|
490
501
|
if (first === undefined)
|
|
491
502
|
return false;
|
|
492
503
|
if (args.length === 1)
|
|
493
|
-
return todo;
|
|
494
|
-
|
|
504
|
+
return members.includes('todo') || (framework.skipWithoutBody && members.includes('skip'));
|
|
505
|
+
if (!framework.playwright)
|
|
506
|
+
return true;
|
|
507
|
+
const title = unwrap(first);
|
|
508
|
+
const value = constantValue(sourceCode, title);
|
|
509
|
+
return isStaticText(title) || (value !== undefined && isStaticText(unwrap(value))) || args.slice(1).some(isFunction);
|
|
495
510
|
}
|
|
496
511
|
/** The value of a `true` or `false` literal; undefined for anything else. */
|
|
497
512
|
function booleanOf(node) {
|
|
498
513
|
return node?.type === 'Literal' && typeof node.value === 'boolean' ? node.value : undefined;
|
|
499
514
|
}
|
|
500
|
-
|
|
515
|
+
/**
|
|
516
|
+
* Whether a runtime call always applies: it has no condition, or `true`, or with `notes` only a note, as
|
|
517
|
+
* Vitest's `context.skip('Broker sandbox is down')`, and nothing around it makes it conditional.
|
|
518
|
+
*/
|
|
519
|
+
function isUnconditional(call, notes) {
|
|
501
520
|
const first = call.arguments.at(0);
|
|
502
|
-
|
|
521
|
+
const always = first === undefined || booleanOf(first) === true || (notes && isStaticText(first));
|
|
522
|
+
return always && !isGuarded(call);
|
|
503
523
|
}
|
|
504
524
|
const FUNCTIONS = new Set(['FunctionDeclaration', 'FunctionExpression', 'ArrowFunctionExpression']);
|
|
505
525
|
/** Whether a statement always leaves the function or block: `return`, `throw`, or a block containing one. */
|
|
@@ -736,7 +756,7 @@ function runAnalysis(sourceCode, options) {
|
|
|
736
756
|
}
|
|
737
757
|
const members = [...root.modifiers, ...path];
|
|
738
758
|
const allowed = kind === 'test' ? framework.testMembers : framework.describeMembers;
|
|
739
|
-
if (members.every((m) => allowed.has(m)) && isDeclaration(kind, args, members
|
|
759
|
+
if (members.every((m) => allowed.has(m)) && isDeclaration(sourceCode, framework, kind, args, members)) {
|
|
740
760
|
// A declaration always has a first argument: its title, or the body of a describe without one.
|
|
741
761
|
const details = detailsOf(sourceCode, call);
|
|
742
762
|
const tagInfo = collectTags(sourceCode, call, !isFunction(args[0]), framework, details);
|
|
@@ -778,7 +798,7 @@ function runAnalysis(sourceCode, options) {
|
|
|
778
798
|
const runtimeSubject = (call, modifier, step) => {
|
|
779
799
|
if (!states.some((s) => s.modifier === modifier))
|
|
780
800
|
return undefined;
|
|
781
|
-
const subject = makeSubject('runtime', call, new Map([[modifier, isUnconditional(call)]]), NO_TAGS, step ?? stack.findLast((s) => s.kind !== 'step'));
|
|
801
|
+
const subject = makeSubject('runtime', call, new Map([[modifier, isUnconditional(call, framework.skipNotes)]]), NO_TAGS, step ?? stack.findLast((s) => s.kind !== 'step'));
|
|
782
802
|
const parent = subject.parent;
|
|
783
803
|
if (parent) {
|
|
784
804
|
parent.runtimeStates.add(subject.states[0].state.name);
|
|
@@ -2,8 +2,11 @@
|
|
|
2
2
|
export declare const MARKER_RE: RegExp;
|
|
3
3
|
/** A tag as written in options: `@` followed by letters, digits, `-` and `_`. */
|
|
4
4
|
export declare const TAG_RE: RegExp;
|
|
5
|
-
export declare const BUILTIN_STATE_NAMES: readonly ["skip", "fixme", "fail", "slow", "todo", "new", "
|
|
5
|
+
export declare const BUILTIN_STATE_NAMES: readonly ["skip", "fixme", "fail", "slow", "todo", "new", "quarantined"];
|
|
6
6
|
export type BuiltinStateName = (typeof BUILTIN_STATE_NAMES)[number];
|
|
7
|
+
/** The built-in states a tag puts a test in, so the only ones whose tag can be renamed. */
|
|
8
|
+
export declare const TAG_STATE_NAMES: readonly ["new", "quarantined"];
|
|
9
|
+
export type TagStateName = (typeof TAG_STATE_NAMES)[number];
|
|
7
10
|
/** The test frameworks the rules understand. */
|
|
8
11
|
export declare const FRAMEWORK_NAMES: readonly ["playwright", "jest", "vitest"];
|
|
9
12
|
export type FrameworkName = (typeof FRAMEWORK_NAMES)[number];
|
package/dist/utils/constants.js
CHANGED
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
export const MARKER_RE = /^[A-Z][A-Z0-9_-]*$/;
|
|
3
3
|
/** A tag as written in options: `@` followed by letters, digits, `-` and `_`. */
|
|
4
4
|
export const TAG_RE = /^@[\w-]+$/;
|
|
5
|
-
export const BUILTIN_STATE_NAMES = ['skip', 'fixme', 'fail', 'slow', 'todo', 'new', '
|
|
5
|
+
export const BUILTIN_STATE_NAMES = ['skip', 'fixme', 'fail', 'slow', 'todo', 'new', 'quarantined'];
|
|
6
|
+
/** The built-in states a tag puts a test in, so the only ones whose tag can be renamed. */
|
|
7
|
+
export const TAG_STATE_NAMES = ['new', 'quarantined'];
|
|
6
8
|
/** The test frameworks the rules understand. */
|
|
7
9
|
export const FRAMEWORK_NAMES = ['playwright', 'jest', 'vitest'];
|
|
8
10
|
export const TICKET_PRESETS = [
|
|
@@ -55,6 +55,13 @@ export interface Framework {
|
|
|
55
55
|
param: number;
|
|
56
56
|
methods: ReadonlySet<string>;
|
|
57
57
|
};
|
|
58
|
+
/** Whether `skip` with only a title declares a skipped test or describe, as Vitest's `test.skip('title')`. */
|
|
59
|
+
skipWithoutBody: boolean;
|
|
60
|
+
/**
|
|
61
|
+
* Whether a runtime skip with only text always skips, the text being a note, as Vitest's
|
|
62
|
+
* `context.skip('why')`. In Playwright the first argument is always a condition.
|
|
63
|
+
*/
|
|
64
|
+
skipNotes: boolean;
|
|
58
65
|
/** Built-in states a test can be put in with this framework. */
|
|
59
66
|
states: ReadonlySet<BuiltinStateName>;
|
|
60
67
|
/** Playwright's own API on `test`: `test.describe`, `test.step`, `test.info()` and runtime `test.skip()`. */
|
package/dist/utils/frameworks.js
CHANGED
|
@@ -31,7 +31,9 @@ const playwright = {
|
|
|
31
31
|
tagKey: 'tag',
|
|
32
32
|
plainTags: false,
|
|
33
33
|
info: { param: 1, methods: RUNTIME_MODIFIERS },
|
|
34
|
-
|
|
34
|
+
skipWithoutBody: false,
|
|
35
|
+
skipNotes: false,
|
|
36
|
+
states: new Set(['skip', 'fixme', 'fail', 'slow', 'new', 'quarantined']),
|
|
35
37
|
playwright: true,
|
|
36
38
|
};
|
|
37
39
|
const JEST_FUNCTIONS = new Map([
|
|
@@ -60,7 +62,9 @@ const jest = {
|
|
|
60
62
|
]),
|
|
61
63
|
optionEffects: new Map(),
|
|
62
64
|
plainTags: false,
|
|
63
|
-
|
|
65
|
+
skipWithoutBody: false,
|
|
66
|
+
skipNotes: false,
|
|
67
|
+
states: new Set(['skip', 'todo', 'fail', 'new', 'quarantined']),
|
|
64
68
|
playwright: false,
|
|
65
69
|
};
|
|
66
70
|
const VITEST_FACTORIES = ['each()', 'for()', 'skipIf()', 'runIf()'];
|
|
@@ -98,7 +102,9 @@ const vitest = {
|
|
|
98
102
|
tagKey: 'tags',
|
|
99
103
|
plainTags: true,
|
|
100
104
|
info: { param: 0, methods: new Set(['skip']) },
|
|
101
|
-
|
|
105
|
+
skipWithoutBody: true,
|
|
106
|
+
skipNotes: true,
|
|
107
|
+
states: new Set(['skip', 'todo', 'fail', 'new', 'quarantined']),
|
|
102
108
|
playwright: false,
|
|
103
109
|
};
|
|
104
110
|
export const FRAMEWORKS = { playwright, jest, vitest };
|
package/dist/utils/options.d.ts
CHANGED
|
@@ -5,6 +5,8 @@ export declare const SETTINGS_KEY = "test-governance";
|
|
|
5
5
|
/** A state override in `states`. `false` turns a state off, `true` turns it on. */
|
|
6
6
|
export interface StateOverride {
|
|
7
7
|
enabled?: boolean;
|
|
8
|
+
/** The tag that puts a test in the state, e.g. `@flaky`. Only `new` and `quarantined` have a tag. */
|
|
9
|
+
tag?: string;
|
|
8
10
|
marker?: string;
|
|
9
11
|
ticket?: TicketSpec | TicketSpec[];
|
|
10
12
|
}
|
|
@@ -35,15 +37,17 @@ export interface GovernanceOptions {
|
|
|
35
37
|
* one (`TODO`, `TBD`, `XXX-*`, ...); tickets numbered 0 are always rejected.
|
|
36
38
|
*/
|
|
37
39
|
placeholders?: string[];
|
|
38
|
-
/** Turns on the `@new` and `@
|
|
40
|
+
/** Turns on the `@new` and `@quarantined` states. */
|
|
39
41
|
lifecycleTags?: boolean;
|
|
40
42
|
/**
|
|
41
|
-
* Turns built-in states on or off, renames their marker or gives them their own ticket format.
|
|
42
|
-
* `fixme` and `todo` are on by default, `fail` and `slow` off; a setting for `new` or `
|
|
43
|
-
* `lifecycleTags`.
|
|
43
|
+
* Turns built-in states on or off, renames their tag or marker, or gives them their own ticket format.
|
|
44
|
+
* `skip`, `fixme` and `todo` are on by default, `fail` and `slow` off; a setting for `new` or `quarantined`
|
|
45
|
+
* wins over `lifecycleTags`. Only `new` and `quarantined` have a tag, e.g.
|
|
46
|
+
* `{ quarantined: { tag: '@flaky', marker: 'FLAKY' } }`. `fixme` and `slow` exist only in Playwright and
|
|
47
|
+
* `todo` only in Jest and Vitest.
|
|
44
48
|
*/
|
|
45
49
|
states?: Partial<Record<BuiltinStateName, boolean | StateOverride>>;
|
|
46
|
-
/** Adds a state for a tag, e.g. `{
|
|
50
|
+
/** Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. */
|
|
47
51
|
customStates?: Record<string, CustomState>;
|
|
48
52
|
/** Require a ticket for conditional skips such as `test.skip(browserName === 'webkit')`. */
|
|
49
53
|
requireTicketForConditional?: boolean;
|
package/dist/utils/options.js
CHANGED
|
@@ -8,6 +8,8 @@ export const SETTINGS_KEY = 'test-governance';
|
|
|
8
8
|
export function isTagState(state) {
|
|
9
9
|
return state.tag !== undefined;
|
|
10
10
|
}
|
|
11
|
+
// A state in `TAG_STATE_NAMES` must have a tag and every other one a modifier, so the schema offers `tag`
|
|
12
|
+
// to the right states.
|
|
11
13
|
const BUILTIN_STATES = {
|
|
12
14
|
skip: { marker: 'SKIP', modifier: 'skip', enabled: true },
|
|
13
15
|
fixme: { marker: 'FIXME', modifier: 'fixme', enabled: true },
|
|
@@ -15,7 +17,7 @@ const BUILTIN_STATES = {
|
|
|
15
17
|
slow: { marker: 'SLOW', modifier: 'slow', enabled: false },
|
|
16
18
|
todo: { marker: 'TODO', modifier: 'todo', enabled: true },
|
|
17
19
|
new: { marker: 'NEW', tag: '@new', lifecycle: true, enabled: false },
|
|
18
|
-
|
|
20
|
+
quarantined: { marker: 'QUARANTINED', tag: '@quarantined', lifecycle: true, enabled: false },
|
|
19
21
|
};
|
|
20
22
|
export { ConfigError } from "./errors.js";
|
|
21
23
|
function toSpecs(spec) {
|
|
@@ -103,7 +105,7 @@ export function compileOptions(options) {
|
|
|
103
105
|
marker: override.marker ?? def.marker,
|
|
104
106
|
ticket: override.ticket === undefined ? defaultMatcher : compile(override.ticket),
|
|
105
107
|
};
|
|
106
|
-
states.push(def.modifier === undefined ? { ...base, tag: def.tag } : { ...base, modifier: def.modifier });
|
|
108
|
+
states.push(def.modifier === undefined ? { ...base, tag: override.tag ?? def.tag } : { ...base, modifier: def.modifier });
|
|
107
109
|
}
|
|
108
110
|
for (const [name, custom] of Object.entries(options.customStates ?? {})) {
|
|
109
111
|
states.push({
|
package/dist/utils/schema.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { BUILTIN_STATE_NAMES, FRAMEWORK_NAMES, MARKER_RE, TAG_RE, TICKET_PRESETS } from "./constants.js";
|
|
1
|
+
import { BUILTIN_STATE_NAMES, FRAMEWORK_NAMES, MARKER_RE, TAG_RE, TAG_STATE_NAMES, TICKET_PRESETS, } from "./constants.js";
|
|
2
2
|
const MARKER_HINT = 'an uppercase keyword such as NEEDS-DATA (letters, digits, "-" and "_")';
|
|
3
3
|
const TAG_HINT = 'a tag such as @needs-data';
|
|
4
4
|
/** Jira project and Linear team keys, as they appear in tickets: `WEB` in `WEB-123`. */
|
|
@@ -25,20 +25,22 @@ const ticketSpecSchema = {
|
|
|
25
25
|
const ticketSchema = {
|
|
26
26
|
anyOf: [ticketSpecSchema, { type: 'array', items: ticketSpecSchema, minItems: 1 }],
|
|
27
27
|
};
|
|
28
|
-
|
|
28
|
+
/** A setting in `states`. Only a state that a tag puts a test in, such as `new`, can rename its tag. */
|
|
29
|
+
const stateOverrideSchema = (hasTag) => ({
|
|
29
30
|
anyOf: [
|
|
30
31
|
{ type: 'boolean' },
|
|
31
32
|
{
|
|
32
33
|
type: 'object',
|
|
33
34
|
properties: {
|
|
34
35
|
enabled: { type: 'boolean' },
|
|
36
|
+
...(hasTag && { tag: { type: 'string', pattern: TAG_RE.source, patternHint: TAG_HINT } }),
|
|
35
37
|
marker: { type: 'string', pattern: MARKER_RE.source, patternHint: MARKER_HINT },
|
|
36
38
|
ticket: ticketSchema,
|
|
37
39
|
},
|
|
38
40
|
additionalProperties: false,
|
|
39
41
|
},
|
|
40
42
|
],
|
|
41
|
-
};
|
|
43
|
+
});
|
|
42
44
|
/** The shape of `settings['test-governance']`, the options of `configure()`. */
|
|
43
45
|
export const optionsSchema = {
|
|
44
46
|
type: 'object',
|
|
@@ -50,8 +52,12 @@ export const optionsSchema = {
|
|
|
50
52
|
lifecycleTags: { type: 'boolean' },
|
|
51
53
|
states: {
|
|
52
54
|
type: 'object',
|
|
53
|
-
properties: Object.fromEntries(BUILTIN_STATE_NAMES.map((name) => [
|
|
55
|
+
properties: Object.fromEntries(BUILTIN_STATE_NAMES.map((name) => [
|
|
56
|
+
name,
|
|
57
|
+
stateOverrideSchema(TAG_STATE_NAMES.includes(name)),
|
|
58
|
+
])),
|
|
54
59
|
additionalProperties: false,
|
|
60
|
+
renamed: { unstable: 'quarantined' },
|
|
55
61
|
},
|
|
56
62
|
customStates: {
|
|
57
63
|
type: 'object',
|
|
@@ -30,7 +30,7 @@ test.fixme('shows the unfilled quantity of a partly filled order', async () => {
|
|
|
30
30
|
|
|
31
31
|
```js
|
|
32
32
|
// SKIP: TRADE-500
|
|
33
|
-
test.skip('shows the unfilled quantity of a partly filled order @
|
|
33
|
+
test.skip('shows the unfilled quantity of a partly filled order @quarantined', async () => {});
|
|
34
34
|
```
|
|
35
35
|
|
|
36
36
|
Correct:
|
|
@@ -46,8 +46,8 @@ test.fixme('shows the unfilled quantity of a partly filled order', async () => {
|
|
|
46
46
|
|
|
47
47
|
```js
|
|
48
48
|
// SKIP: TRADE-500
|
|
49
|
-
//
|
|
50
|
-
test.skip('shows the unfilled quantity of a partly filled order @
|
|
49
|
+
// QUARANTINED: TRADE-500
|
|
50
|
+
test.skip('shows the unfilled quantity of a partly filled order @quarantined', async () => {});
|
|
51
51
|
```
|
|
52
52
|
|
|
53
53
|
## Options
|
|
@@ -8,13 +8,13 @@
|
|
|
8
8
|
|
|
9
9
|
<!-- end auto-generated rule header -->
|
|
10
10
|
|
|
11
|
-
Tags are matched exactly, so `@New` or `@
|
|
11
|
+
Tags are matched exactly, so `@New` or `@quarantiend` silently puts a test in no state at all. This
|
|
12
12
|
rule reports:
|
|
13
13
|
|
|
14
|
-
- `@new` together with `@
|
|
14
|
+
- `@new` together with `@quarantined` on the same test, including tags inherited from a `describe`;
|
|
15
15
|
- `@new` on a test that is skipped or fixme'd, since a test that doesn't run can't be promoted;
|
|
16
16
|
- a state tag written in the wrong case, such as `@New` (autofixed to `@new`);
|
|
17
|
-
- a near-miss spelling of a state tag, such as `@
|
|
17
|
+
- a near-miss spelling of a state tag, such as `@quarantiend`.
|
|
18
18
|
|
|
19
19
|
The `@new` checks run when the `new` state is on (`lifecycleTags: true`). Case and typo checks cover
|
|
20
20
|
every tag state that is on, including custom states. A tag kept in a `const` in the same file is
|
|
@@ -28,7 +28,7 @@ Incorrect:
|
|
|
28
28
|
<!-- example: invalid settings={"lifecycleTags":true} -->
|
|
29
29
|
|
|
30
30
|
```js
|
|
31
|
-
test('places a limit order @new @
|
|
31
|
+
test('places a limit order @new @quarantined', async () => {});
|
|
32
32
|
```
|
|
33
33
|
|
|
34
34
|
<!-- example: invalid settings={"lifecycleTags":true} -->
|
|
@@ -48,7 +48,7 @@ test('exports the trade history as CSV @New', async () => {});
|
|
|
48
48
|
<!-- example: invalid settings={"lifecycleTags":true} -->
|
|
49
49
|
|
|
50
50
|
```js
|
|
51
|
-
test('shows dividend payments in the account history @
|
|
51
|
+
test('shows dividend payments in the account history @quarantiend', async () => {});
|
|
52
52
|
```
|
|
53
53
|
|
|
54
54
|
Correct:
|
|
@@ -56,7 +56,7 @@ Correct:
|
|
|
56
56
|
<!-- example: valid settings={"lifecycleTags":true} -->
|
|
57
57
|
|
|
58
58
|
```js
|
|
59
|
-
test('places a limit order @
|
|
59
|
+
test('places a limit order @quarantined', async () => {});
|
|
60
60
|
```
|
|
61
61
|
|
|
62
62
|
## Options
|
|
@@ -64,7 +64,8 @@ test('places a limit order @unstable', async () => {});
|
|
|
64
64
|
This rule has no options of its own. It reads the shared options described in the
|
|
65
65
|
[README](../../README.md#options). These change what it reports:
|
|
66
66
|
|
|
67
|
-
- `lifecycleTags`, or `states` for `new` and `
|
|
68
|
-
is on, and `@new` together with `@
|
|
67
|
+
- `lifecycleTags`, or `states` for `new` and `quarantined`: the `@new` checks run when the `new` state
|
|
68
|
+
is on, and `@new` together with `@quarantined` is reported when both are on. A tag renamed in
|
|
69
|
+
`states`, such as `@flaky`, is checked in place of the default one.
|
|
69
70
|
- `customStates`: the tags of custom states get the same case and typo checks.
|
|
70
71
|
- `framework` and `testFunctions`: which functions count as the framework's `test`.
|
|
@@ -19,18 +19,19 @@ A test is in a state when it is:
|
|
|
19
19
|
| `fixme` | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | `// FIXME:` | yes |
|
|
20
20
|
| `todo` | Jest's and Vitest's `test.todo` | `// TODO:` | yes, in Jest and Vitest |
|
|
21
21
|
| `new` | the `@new` tag | `// NEW:` | with `lifecycleTags: true` |
|
|
22
|
-
| `
|
|
22
|
+
| `quarantined` | the `@quarantined` tag | `// QUARANTINED:` | with `lifecycleTags: true` |
|
|
23
23
|
| `fail`, `slow` | `test.fail(...)`, `test.slow()`, `testInfo.fail()`, `testInfo.slow()` | `// FAIL:`, `// SLOW:` | no |
|
|
24
24
|
| custom | any tag you list in `customStates` | your keyword | when configured |
|
|
25
25
|
|
|
26
|
-
In Jest and Vitest, `xit`, `xdescribe`, Vitest's `{ skip: true }`, `skipIf(true)
|
|
27
|
-
in a test or in `beforeEach`, skip a test too, and `test.failing` and `test.fails` put it in the `fail` state. The
|
|
26
|
+
In Jest and Vitest, `xit`, `xdescribe`, Vitest's `{ skip: true }`, `skipIf(true)`, `context.skip()` and
|
|
27
|
+
`context.skip('note')`, in a test or in `beforeEach`, skip a test too, and `test.failing` and `test.fails` put it in the `fail` state. The
|
|
28
28
|
[README](../../README.md#frameworks) lists every form.
|
|
29
29
|
|
|
30
30
|
Tags are read from the title (`'places a limit order @new'`) and from `{ tag: '@new' }` or `{ tag: ['@new', '@smoke'] }`,
|
|
31
31
|
also when they come from a `const` or an enum declared in the same file, as in `{ tag: TAGS.NEW }`.
|
|
32
32
|
Vitest writes tag names without the `@`, so `{ tags: ['new'] }` is the `@new` tag. Tags on a
|
|
33
|
-
`describe` apply to every test inside it.
|
|
33
|
+
`describe` apply to every test inside it. `states` can rename the tag and marker of `new` and
|
|
34
|
+
`quarantined`, as in `states: { quarantined: { tag: '@flaky', marker: 'FLAKY' } }`.
|
|
34
35
|
|
|
35
36
|
## Marker format
|
|
36
37
|
|
|
@@ -87,7 +88,7 @@ If one test or describe has several markers for the same state, each must be val
|
|
|
87
88
|
Tags can only be read from text: a string or a template literal such as `` `buys ${symbol} @new` ``,
|
|
88
89
|
written in the call or kept in a `const` or an enum in the same file. A title built another way, such
|
|
89
90
|
as `'buys ' + symbol`, and a tag or details object imported from another module may hide a tag. With
|
|
90
|
-
`reportUnreadableTags: true`, the rule reports those whenever a tag state (`@new`, `@
|
|
91
|
+
`reportUnreadableTags: true`, the rule reports those whenever a tag state (`@new`, `@quarantined` or a
|
|
91
92
|
custom state) is on, so no tagged test slips past.
|
|
92
93
|
|
|
93
94
|
<!-- example: invalid settings={"lifecycleTags":true,"reportUnreadableTags":true} -->
|
|
@@ -233,8 +234,8 @@ This rule has no options of its own. It reads the shared options described in th
|
|
|
233
234
|
change what it reports:
|
|
234
235
|
|
|
235
236
|
- `ticket` and `placeholders`: which tickets are accepted.
|
|
236
|
-
- `lifecycleTags`, `states` and `customStates`: which states need a marker,
|
|
237
|
-
called, and, for a state with its own `ticket`, which tickets it accepts.
|
|
237
|
+
- `lifecycleTags`, `states` and `customStates`: which states need a marker, which tag puts a test in
|
|
238
|
+
each, what the marker is called, and, for a state with its own `ticket`, which tickets it accepts.
|
|
238
239
|
- `requireTicketForConditional`: conditional skips need a ticket too.
|
|
239
240
|
- `allowBlankLine`: blank lines may separate a marker from its test.
|
|
240
241
|
- `allowNotes`: text may follow the tickets.
|
package/docs/upgrading.md
CHANGED
|
@@ -1,5 +1,55 @@
|
|
|
1
1
|
# Upgrading
|
|
2
2
|
|
|
3
|
+
## Upgrading to 0.6
|
|
4
|
+
|
|
5
|
+
0.6.0 renames the `unstable` state to `quarantined`. Only projects that turn it on, with
|
|
6
|
+
`lifecycleTags: true` or `states.unstable`, need to change anything.
|
|
7
|
+
|
|
8
|
+
| What changed | Who it affects | What to do |
|
|
9
|
+
| ----------------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------- |
|
|
10
|
+
| [`@unstable` is now `@quarantined`](#the-quarantined-state) | Projects with `lifecycleTags: true` or `states.unstable` | Rename the tags and markers, or keep them with `states` |
|
|
11
|
+
|
|
12
|
+
### The quarantined state
|
|
13
|
+
|
|
14
|
+
The state's tag is now `@quarantined` and its marker `// QUARANTINED:`.
|
|
15
|
+
|
|
16
|
+
Before:
|
|
17
|
+
|
|
18
|
+
```ts
|
|
19
|
+
// UNSTABLE: TRADE-311
|
|
20
|
+
test('streams level 2 quotes @unstable', async ({ page }) => {});
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
After:
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
// QUARANTINED: TRADE-311
|
|
27
|
+
test('streams level 2 quotes @quarantined', async ({ page }) => {});
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
In Vitest, `{ tags: ['unstable'] }` becomes `{ tags: ['quarantined'] }` too. To keep the tags and
|
|
31
|
+
markers you have instead, name them in `states`:
|
|
32
|
+
|
|
33
|
+
```js
|
|
34
|
+
testGovernance.configure({
|
|
35
|
+
framework: 'playwright',
|
|
36
|
+
lifecycleTags: true,
|
|
37
|
+
states: { quarantined: { tag: '@unstable', marker: 'UNSTABLE' } },
|
|
38
|
+
});
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
A config that sets `states.unstable` fails when it loads, so rename it to `states.quarantined`:
|
|
42
|
+
|
|
43
|
+
```text
|
|
44
|
+
eslint-plugin-test-governance: invalid options:
|
|
45
|
+
- states.unstable was renamed to "quarantined"
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
A config that only sets `lifecycleTags: true` still loads, and nothing reports the old tag: an
|
|
49
|
+
`@unstable` test no longer needs a ticket, and its `// UNSTABLE:` marker is an ordinary comment. Before
|
|
50
|
+
you upgrade, list them with `grep -rni unstable tests/` (or wherever your tests are), then rename them
|
|
51
|
+
or keep them as above.
|
|
52
|
+
|
|
3
53
|
## Upgrading to 0.5
|
|
4
54
|
|
|
5
55
|
0.5.0 changes how the plugin is configured, ahead of 1.0. Most projects change one or two lines. A
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "eslint-plugin-test-governance",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "ESLint rules that require a ticket comment above skipped, fixme, todo and tagged Playwright, Jest and Vitest tests.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"eslint",
|
|
@@ -46,8 +46,8 @@
|
|
|
46
46
|
"scripts": {
|
|
47
47
|
"build": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\" && tsc -p tsconfig.build.json",
|
|
48
48
|
"typecheck": "tsc -p tsconfig.json",
|
|
49
|
-
"test": "node --test
|
|
50
|
-
"test:coverage": "node --test --experimental-test-coverage --test-coverage-include
|
|
49
|
+
"test": "node --test \"tests/**/*.test.ts\"",
|
|
50
|
+
"test:coverage": "node --test --experimental-test-coverage --test-coverage-include=\"src/**\" --test-coverage-branches=100 --test-coverage-functions=100 --test-coverage-lines=100 \"tests/**/*.test.ts\"",
|
|
51
51
|
"test:pack": "node scripts/pack-smoke.mjs",
|
|
52
52
|
"docs": "npm run build && eslint-doc-generator",
|
|
53
53
|
"docs:check": "npm run build && eslint-doc-generator --check",
|