eslint-plugin-test-governance 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -1
- package/README.md +31 -24
- package/dist/rules/no-conflicting-states.js +9 -8
- package/dist/rules/require-ticket.d.ts +1 -1
- package/dist/rules/require-ticket.js +25 -3
- package/dist/utils/analyze.d.ts +7 -0
- package/dist/utils/analyze.js +101 -6
- package/dist/utils/frameworks.d.ts +7 -0
- package/dist/utils/frameworks.js +7 -1
- package/dist/utils/options.d.ts +14 -1
- package/dist/utils/options.js +2 -0
- package/dist/utils/schema.js +2 -0
- package/docs/rules/no-conflicting-states.md +15 -1
- package/docs/rules/require-ticket.md +40 -2
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,29 @@ All notable changes to this project are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.13.0] - 2026-11-05
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `requireReasonForConditional`, off by default: a conditional skip, such as `test.skip(isMobile)` or
|
|
14
|
+
`if (isWeekend) this.skip()`, needs a reason but no ticket. A comment above it or at the end of its line, above
|
|
15
|
+
the `if` around it, or above its hook, step or test counts, and so does the reason the call takes where the runner
|
|
16
|
+
has one, as Playwright's `test.skip(condition, 'why')`, Vitest's `context.skip(condition, 'why')`, node:test's
|
|
17
|
+
`t.skip('why')` and `{ skip: isWindows && 'why' }`, and Jasmine's `pending('why')`. `require-ticket` reports the
|
|
18
|
+
skips that give none.
|
|
19
|
+
- `knownTags`, such as `['@smoke', '@regression']`: `no-conflicting-states` gives these tags the case and typo
|
|
20
|
+
checks state tags get, so `@Smoke` is fixed to `@smoke` and `@smok` is reported. A listed tag is never reported as
|
|
21
|
+
a typo of a state tag.
|
|
22
|
+
|
|
23
|
+
## [0.12.1] - 2026-11-04
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- `node:test`'s `tags` option, added in Node 26.2, is read like Vitest's: `{ tags: ['new'] }` puts a test or
|
|
28
|
+
describe in the `@new` state, also when the options come first, as in `test({ tags: ['new'] }, fn)`. The README
|
|
29
|
+
now also says that `fail` exists in `node:test`, where `test.expectFailure` and `{ expectFailure }` already put a
|
|
30
|
+
test in it.
|
|
31
|
+
|
|
9
32
|
## [0.12.0] - 2026-11-04
|
|
10
33
|
|
|
11
34
|
### Added
|
|
@@ -484,7 +507,9 @@ First release.
|
|
|
484
507
|
- Support for ESLint 9 and 10 flat config, JavaScript and TypeScript test files, `.extend()`
|
|
485
508
|
fixtures, describe blocks, runtime skips and tags in titles or the `tag` option.
|
|
486
509
|
|
|
487
|
-
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.
|
|
510
|
+
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.13.0...HEAD
|
|
511
|
+
[0.13.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.12.1...v0.13.0
|
|
512
|
+
[0.12.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.12.0...v0.12.1
|
|
488
513
|
[0.12.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.11.1...v0.12.0
|
|
489
514
|
[0.11.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.11.0...v0.11.1
|
|
490
515
|
[0.11.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.10.1...v0.11.0
|
package/README.md
CHANGED
|
@@ -20,6 +20,10 @@ Without the comment, `test-governance/require-ticket` fails the lint. When the t
|
|
|
20
20
|
skip removed, `no-orphaned-marker` flags the leftover comment. The markers are plain
|
|
21
21
|
comments, so `grep -rn "SKIP:" tests/` lists every disabled test and its ticket.
|
|
22
22
|
|
|
23
|
+
The same check covers tests commented out by hand, `FIXME` and `TODO` comments in test files, and, once
|
|
24
|
+
turned on, retried tests. The ticket can also go in the description of an `eslint-disable` comment, or in
|
|
25
|
+
a Playwright annotation or Vitest `meta`, which reach the test report.
|
|
26
|
+
|
|
23
27
|
> **Status:** early release (0.x). Options may still change before 1.0. Coming from 0.5?
|
|
24
28
|
> [Upgrading to 0.6](docs/upgrading.md#upgrading-to-06) shows what to change.
|
|
25
29
|
|
|
@@ -148,8 +152,9 @@ turn on the same rules. What puts a test in a state in each framework:
|
|
|
148
152
|
| `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | | | | | | | |
|
|
149
153
|
| `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }` | | | | `test.expectFailure`, `expectFailure()`, `{ expectFailure }` | |
|
|
150
154
|
| `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | | | | | | | |
|
|
155
|
+
| `retry` (`// RETRY:`), off by default | `test.describe.configure({ retries })` | `jest.retryTimes()` | `{ retry }` | `this.retries()` in a describe, test or hook written as a `function` | as Mocha, and `{ retries }` | as Mocha | | |
|
|
151
156
|
| conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)` | `(condition ? it.skip : it)(...)` | `skipIf(condition)`, `runIf(condition)`, `{ skip: condition }`, `context.skip(condition)`, `if (...) context.skip()`, `(condition ? it.skip : it)(...)` | `if (...) this.skip()`, `(condition ? it.skip : it)(...)` | as Mocha, and `{ browser: '!webkit' }` | as Mocha | `{ skip: condition }`, `{ todo: condition }`, `if (...) t.skip()`, `(condition ? it.skip : it)(...)` | `if (...) pending()`, `(condition ? xit : it)(...)` |
|
|
152
|
-
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` | the title | the title, `{ tags }` from @cypress/grep | the title | the title
|
|
157
|
+
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` | the title | the title, `{ tags }` from @cypress/grep | the title | the title, `{ tags }`, as in Vitest | the title |
|
|
153
158
|
|
|
154
159
|
Table-driven tests count too: `test.skip.each(table)(...)`, `describe.skip.each` and Vitest's `.for`.
|
|
155
160
|
A test or describe whose function a condition picks, as in `(isMobile ? test.skip : test)('title', fn)`,
|
|
@@ -161,8 +166,9 @@ as is `xit('title')` in Jasmine.
|
|
|
161
166
|
In Mocha, `this.skip()` counts in a test or hook written as a `function`; an arrow function gets no
|
|
162
167
|
Mocha context, so its `this` is not the test's. `this.slow(ms)` only sets Mocha's speed threshold, so
|
|
163
168
|
it isn't the `slow` state.
|
|
164
|
-
Vitest
|
|
165
|
-
tag with an `@` in front: `'flaky'` is `@flaky`, and a custom state with
|
|
169
|
+
Vitest, and `node:test` since Node 26.2, write tag names without the `@`, as in `{ tags: ['flaky'] }`, so a
|
|
170
|
+
name in `tags` counts as the tag with an `@` in front: `'flaky'` is `@flaky`, and a custom state with
|
|
171
|
+
`when: '@flaky'` matches it.
|
|
166
172
|
In `node:test`, an option counts with any value but `false` or `undefined`, as Node reads it, so
|
|
167
173
|
`{ skip: 'broker sandbox is down' }` always skips. `t.skip()` and `t.todo()` take a message, not a
|
|
168
174
|
condition, and in a test or hook written as a `function`, `this.skip()` and `this.todo()` do the same.
|
|
@@ -199,7 +205,7 @@ condition. A function or variable of your own called `pending` is not Jasmine's.
|
|
|
199
205
|
| [marker-matches-state](docs/rules/marker-matches-state.md) | Require the marker keyword to match the test's state | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
|
|
200
206
|
| [no-conflicting-states](docs/rules/no-conflicting-states.md) | Disallow state tags that contradict each other, differ in case or look like typos | | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | 🔧 | |
|
|
201
207
|
| [no-orphaned-marker](docs/rules/no-orphaned-marker.md) | Disallow marker comments that no longer match a test state | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
|
|
202
|
-
| [require-ticket](docs/rules/require-ticket.md) | Require a ticket
|
|
208
|
+
| [require-ticket](docs/rules/require-ticket.md) | Require a ticket for skipped, fixme, todo, retried and tagged tests | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
|
|
203
209
|
| [require-ticket-for-commented-out-tests](docs/rules/require-ticket-for-commented-out-tests.md) | Require a ticket marker on commented-out tests | | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | 💡 |
|
|
204
210
|
| [require-ticket-in-comments](docs/rules/require-ticket-in-comments.md) | Require FIXME and TODO comments in test files to start with a ticket | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
|
|
205
211
|
|
|
@@ -213,7 +219,7 @@ The defaults follow how most teams already work, so each framework config is use
|
|
|
213
219
|
| --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
214
220
|
| States that need a ticket | `skip`, `fixme`; in the other frameworks `skip`, `todo` | Every team skips tests. Chromium DevTools, GitLab and Slack all require a bug link on disabled tests. |
|
|
215
221
|
| `@new`, `@quarantined` | off; `lifecycleTags: true` turns both on | A promotion workflow is not universal. Teams that say `@flaky` instead [rename the tag](#tag-names). |
|
|
216
|
-
| Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true`
|
|
222
|
+
| Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true` asks for a ticket, `requireReasonForConditional: true` for a reason. |
|
|
217
223
|
| `test.fail`, `test.slow`, Jest's `test.failing`, Vitest's `test.fails` | off | They don't remove coverage. Turn them on with `states: { fail: true, slow: true }`. |
|
|
218
224
|
| Retries: `describe.configure({ retries })`, `jest.retryTimes()`, `retry`, `this.retries()` | off | Retries hide flaky tests. `states: { retry: true }` needs a ticket wherever tests retry, even only on CI. |
|
|
219
225
|
| Marker format | `// SKIP: TRADE-123` or `// SKIP: TRADE-123, TRADE-124` | Uppercase keyword and colon, like `TODO:`. Easy to grep. |
|
|
@@ -232,21 +238,23 @@ yourself. `framework` is required, and every other option has a default. The rul
|
|
|
232
238
|
of their own: they all read these shared settings, so they always agree on ticket formats, states
|
|
233
239
|
and markers. To turn a rule off, set it to `'off'` as usual.
|
|
234
240
|
|
|
235
|
-
| Option | Type | Default | Description
|
|
236
|
-
| ----------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------- |
|
|
237
|
-
| `framework` | `'playwright' \| 'jest' \| 'vitest' \| 'mocha' \| 'cypress' \| 'webdriverio' \| 'node-test' \| 'jasmine'` | required | The test framework of the files the config applies to. Each framework's config, such as `configs.playwright`, sets it, and `configure()` needs it. See [Frameworks](#frameworks).
|
|
238
|
-
| `ticket` | `TicketSpec \| TicketSpec[]` | `{ preset: 'any' }` | Accepted ticket formats. An array accepts a ticket that matches any entry. See [Ticket presets](#ticket-presets).
|
|
239
|
-
| `placeholders` | `string[]` | see above | Tickets rejected as placeholders. `*` matches any characters. Case-insensitive; a leading `#` is ignored. Your list replaces the default one, so repeat the defaults you want to keep. Tickets numbered 0 are always rejected.
|
|
240
|
-
| `lifecycleTags` | `boolean` | `false` | Turns on the `new` (`@new`) and `quarantined` (`@quarantined`) states.
|
|
241
|
-
| `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `retry`, `new`, `quarantined`) on or off, renames their tag or marker, or gives them their own ticket format. `fail`, `slow` and `retry` are off by default; a setting for `new` or `quarantined` here wins over `lifecycleTags`. Only `new` and `quarantined` have a tag to rename; see [Tag names](#tag-names). `fixme` and `slow` exist only in Playwright, `todo` in every framework but Playwright, `fail` in Playwright, Jest and
|
|
242
|
-
| `customStates` | `{ [name]: { when, marker, ticket? } }` | `{}` | Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. `when` is a tag and `marker` an uppercase keyword; neither can be shared with another state.
|
|
243
|
-
| `
|
|
244
|
-
| `
|
|
245
|
-
| `
|
|
246
|
-
| `
|
|
247
|
-
| `
|
|
248
|
-
| `
|
|
249
|
-
| `
|
|
241
|
+
| Option | Type | Default | Description |
|
|
242
|
+
| ----------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
243
|
+
| `framework` | `'playwright' \| 'jest' \| 'vitest' \| 'mocha' \| 'cypress' \| 'webdriverio' \| 'node-test' \| 'jasmine'` | required | The test framework of the files the config applies to. Each framework's config, such as `configs.playwright`, sets it, and `configure()` needs it. See [Frameworks](#frameworks). |
|
|
244
|
+
| `ticket` | `TicketSpec \| TicketSpec[]` | `{ preset: 'any' }` | Accepted ticket formats. An array accepts a ticket that matches any entry. See [Ticket presets](#ticket-presets). |
|
|
245
|
+
| `placeholders` | `string[]` | see above | Tickets rejected as placeholders. `*` matches any characters. Case-insensitive; a leading `#` is ignored. Your list replaces the default one, so repeat the defaults you want to keep. Tickets numbered 0 are always rejected. |
|
|
246
|
+
| `lifecycleTags` | `boolean` | `false` | Turns on the `new` (`@new`) and `quarantined` (`@quarantined`) states. |
|
|
247
|
+
| `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `retry`, `new`, `quarantined`) on or off, renames their tag or marker, or gives them their own ticket format. `fail`, `slow` and `retry` are off by default; a setting for `new` or `quarantined` here wins over `lifecycleTags`. Only `new` and `quarantined` have a tag to rename; see [Tag names](#tag-names). `fixme` and `slow` exist only in Playwright, `todo` in every framework but Playwright, `fail` in Playwright, Jest, Vitest and node:test, and `retry` in every framework but node:test and Jasmine, so turning one on elsewhere has no effect. |
|
|
248
|
+
| `customStates` | `{ [name]: { when, marker, ticket? } }` | `{}` | Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. `when` is a tag and `marker` an uppercase keyword; neither can be shared with another state. |
|
|
249
|
+
| `knownTags` | `string[]` | `[]` | Your team's other tags, such as `['@smoke', '@regression']`, written with the `@`. `no-conflicting-states` reports them in the wrong case or misspelled, as it does state tags, and never reports a listed tag as a typo of a state tag. |
|
|
250
|
+
| `requireTicketForConditional` | `boolean` | `false` | Require a ticket for conditional skips too: `test.skip(condition, ...)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)`, a skip in a `catch` block, after other code in a `try` block with a `catch`, or after an early `return` or `throw`. |
|
|
251
|
+
| `requireReasonForConditional` | `boolean` | `false` | Require a reason, but no ticket, for conditional skips: a comment above the skip or at the end of its line, above the `if` around it or above its hook, step or test, or the reason the call takes, as `test.skip(condition, 'why')`. See [require-ticket](docs/rules/require-ticket.md#reasons-for-conditional-skips). |
|
|
252
|
+
| `allowBlankLine` | `boolean` | `false` | Let blank lines separate a marker from its test. |
|
|
253
|
+
| `allowNotes` | `boolean` | `false` | Allow free text after the tickets, e.g. `// SKIP: TRADE-123 flaky on CI`. Applies to markers and to `FIXME` / `TODO` comments. |
|
|
254
|
+
| `comments` | `false \| { keywords?: string[] }` | `{ keywords: ['FIXME', 'TODO'] }` | Comment keywords that must start with a ticket anywhere in a test file, e.g. `// TODO: TRADE-123`. Your keywords replace the default ones. `false` allows free-form comments. |
|
|
255
|
+
| `annotations` | `false \| { type?: string, only?: boolean }` | `{ type: 'issue' }` | Accept the ticket from a test annotation as well: Playwright's `annotation` of the given type or Vitest's `meta` key of that name. One valid annotation covers every state the test is in, and a describe's covers its tests. `only: true` makes it the only place for a test's or describe's ticket. See [Tickets from annotations](#tickets-from-annotations). |
|
|
256
|
+
| `testFunctions` | `string[]` | `['test']` | Names treated as the framework's `test`. Jest and Vitest default to `['test', 'it']`, Mocha, Cypress and WebdriverIO to `['it', 'specify', 'test']`, and Jasmine to `['it']`. Your list replaces the default one, so include those names if you still need them. See below. |
|
|
257
|
+
| `reportUnreadableTags` | `boolean` | `false` | When a tag state is on, report titles that aren't static text, and tags or details that can't be read in the file, such as ones imported from another module, since their tags can't be checked. |
|
|
250
258
|
|
|
251
259
|
### Tag names
|
|
252
260
|
|
|
@@ -470,10 +478,9 @@ The [Frameworks](#frameworks) table lists what each framework's tests can do. In
|
|
|
470
478
|
Not covered in this version: tests generated in loops with dynamic titles, and tags or details imported
|
|
471
479
|
from other modules (see `reportUnreadableTags`), fixtures imported from other modules under a name not in
|
|
472
480
|
`testFunctions`, a Vitest test with only a title, which Vitest runs as a todo but which looks the same as a
|
|
473
|
-
helper called `test`,
|
|
474
|
-
`
|
|
475
|
-
`
|
|
476
|
-
and other runners such as Bun.
|
|
481
|
+
helper called `test`, Cypress tests skipped with @cypress/skip-test's `skipOn()` and `onlyOn()`, `node:test`'s
|
|
482
|
+
`getTestContext()`, Jasmine's `pending()` in a helper function that tests call, Jasmine 5 and 6's
|
|
483
|
+
`require('jasmine-core').noGlobals()`, and other runners such as Bun.
|
|
477
484
|
|
|
478
485
|
## TypeScript
|
|
479
486
|
|
|
@@ -20,7 +20,8 @@ export default createRule({
|
|
|
20
20
|
},
|
|
21
21
|
},
|
|
22
22
|
create(context, analysis, options) {
|
|
23
|
-
|
|
23
|
+
// State tags first, so a tag near both a state tag and a known tag is read as the state's.
|
|
24
|
+
const checkedTags = [...options.states.filter(isTagState).map((s) => s.tag), ...options.knownTags];
|
|
24
25
|
const newTag = options.states.find((s) => s.name === 'new')?.tag;
|
|
25
26
|
const quarantinedTag = options.states.find((s) => s.name === 'quarantined')?.tag;
|
|
26
27
|
for (const subject of analysis.subjects) {
|
|
@@ -53,23 +54,23 @@ export default createRule({
|
|
|
53
54
|
const { tag, range } = occurrence;
|
|
54
55
|
// A Vitest tag name is written without its `@`, so show it, and fix it, the same way.
|
|
55
56
|
const written = (name) => (occurrence.plain ? name.slice(1) : name);
|
|
56
|
-
if (
|
|
57
|
+
if (checkedTags.includes(tag))
|
|
57
58
|
return;
|
|
58
59
|
const lower = tag.toLowerCase();
|
|
59
|
-
const sameLetters =
|
|
60
|
+
const sameLetters = checkedTags.find((checked) => checked.toLowerCase() === lower);
|
|
60
61
|
if (sameLetters) {
|
|
61
62
|
context.report({
|
|
62
63
|
node: occurrence.node,
|
|
63
64
|
messageId: 'tagCase',
|
|
64
|
-
data: { found: written(tag), expected: written(sameLetters
|
|
65
|
-
fix: range && ((fixer) => fixer.replaceTextRange(range, written(sameLetters
|
|
65
|
+
data: { found: written(tag), expected: written(sameLetters) },
|
|
66
|
+
fix: range && ((fixer) => fixer.replaceTextRange(range, written(sameLetters))),
|
|
66
67
|
});
|
|
67
68
|
return;
|
|
68
69
|
}
|
|
69
70
|
// A typo changes, adds, drops or swaps a letter or two. A tag that is two or more letters
|
|
70
71
|
// shorter or longer, like `@unquarantined` next to `@quarantined`, is a different word.
|
|
71
|
-
const near =
|
|
72
|
-
const expected =
|
|
72
|
+
const near = checkedTags.find((checked) => {
|
|
73
|
+
const expected = checked.toLowerCase();
|
|
73
74
|
return (Math.abs(lower.length - expected.length) < 2 &&
|
|
74
75
|
editDistance(lower, expected) <= typoThreshold(expected.length - 1));
|
|
75
76
|
});
|
|
@@ -77,7 +78,7 @@ export default createRule({
|
|
|
77
78
|
context.report({
|
|
78
79
|
node: occurrence.node,
|
|
79
80
|
messageId: 'tagTypo',
|
|
80
|
-
data: { found: written(tag), expected: written(near
|
|
81
|
+
data: { found: written(tag), expected: written(near) },
|
|
81
82
|
});
|
|
82
83
|
}
|
|
83
84
|
}
|
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
declare const _default: import("../utils/create-rule.ts").Rule<"missingMarker" | "missingTicket" | "invalidTicket" | "placeholderTicket" | "extraText" | "removeExtraText" | "markerCase" | "dynamicTitle" | "dynamicTags" | "invalidAnnotation" | "useAnnotation" | "missingAnnotation" | "moveToAnnotation" | "unreadAnnotation">;
|
|
1
|
+
declare const _default: import("../utils/create-rule.ts").Rule<"missingMarker" | "missingTicket" | "invalidTicket" | "placeholderTicket" | "extraText" | "removeExtraText" | "missingReason" | "markerCase" | "dynamicTitle" | "dynamicTags" | "invalidAnnotation" | "useAnnotation" | "missingAnnotation" | "moveToAnnotation" | "unreadAnnotation">;
|
|
2
2
|
export default _default;
|
|
@@ -5,7 +5,7 @@ export default createRule({
|
|
|
5
5
|
meta: {
|
|
6
6
|
type: 'problem',
|
|
7
7
|
docs: {
|
|
8
|
-
description: 'Require a ticket
|
|
8
|
+
description: 'Require a ticket for skipped, fixme, todo, retried and tagged tests',
|
|
9
9
|
recommended: 'error',
|
|
10
10
|
url: 'https://github.com/Huldoser/eslint-plugin-test-governance/blob/main/docs/rules/require-ticket.md',
|
|
11
11
|
},
|
|
@@ -13,6 +13,7 @@ export default createRule({
|
|
|
13
13
|
schema: [],
|
|
14
14
|
messages: {
|
|
15
15
|
missingMarker: "{{subject}} is in the '{{state}}' state but has no `// {{marker}}: <ticket>` comment directly above it, e.g. `// {{marker}}: {{example}}`.",
|
|
16
|
+
missingReason: "{{subject}} is in the '{{state}}' state only under a condition and doesn't say why. Write the reason in a comment above it{{argument}}.",
|
|
16
17
|
missingTicket: '`{{marker}}:` needs a ticket right after the colon, e.g. `// {{marker}}: {{example}}`.',
|
|
17
18
|
invalidTicket: "'{{ticket}}' is not a valid ticket for the '{{state}}' state. Expected {{expected}}.",
|
|
18
19
|
placeholderTicket: "'{{ticket}}' is a placeholder, not a real ticket. Link the ticket that tracks this test.",
|
|
@@ -65,6 +66,24 @@ export default createRule({
|
|
|
65
66
|
data: { subject: describeSubject(subject), state: state.name, type, example },
|
|
66
67
|
});
|
|
67
68
|
};
|
|
69
|
+
/** Reports a conditional state that gives no reason and has no ticket, which would say why as well. */
|
|
70
|
+
const reportReason = (subject, state) => {
|
|
71
|
+
if (evaluate(subject, state).kind === 'ok' || hasAnnotationTicket(subject, state, options))
|
|
72
|
+
return;
|
|
73
|
+
const callee = context.sourceCode.getText(subject.node.callee);
|
|
74
|
+
const first = subject.node.arguments.at(0);
|
|
75
|
+
const where = options.framework.reasonArgument;
|
|
76
|
+
// Vitest's `context.skip()` takes either a reason alone or a condition and then the reason.
|
|
77
|
+
const reasonOnly = where === 'only' ||
|
|
78
|
+
(where === 'either' && (first === undefined || first.type === 'Literal' || first.type === 'TemplateLiteral'));
|
|
79
|
+
const example = reasonOnly ? `${callee}('why')` : `${callee}(condition, 'why')`;
|
|
80
|
+
const argument = subject.kind === 'runtime' && where !== undefined ? `, or pass it to the call: \`${example}\`` : '';
|
|
81
|
+
context.report({
|
|
82
|
+
loc: headLoc(subject),
|
|
83
|
+
messageId: 'missingReason',
|
|
84
|
+
data: { subject: describeSubject(subject), state: state.name, argument },
|
|
85
|
+
});
|
|
86
|
+
};
|
|
68
87
|
for (const subject of analysis.subjects) {
|
|
69
88
|
const needsTicket = subject.states.some((s) => s.required);
|
|
70
89
|
if (options.reportUnreadableTags && hasTagStates) {
|
|
@@ -82,9 +101,12 @@ export default createRule({
|
|
|
82
101
|
}
|
|
83
102
|
}
|
|
84
103
|
const annotationOnly = only && (subject.kind === 'test' || subject.kind === 'describe') ? only : undefined;
|
|
85
|
-
for (const { state, required } of subject.states) {
|
|
86
|
-
if (!required)
|
|
104
|
+
for (const { state, required, reasoned } of subject.states) {
|
|
105
|
+
if (!required) {
|
|
106
|
+
if (options.requireReasonForConditional && reasoned === false)
|
|
107
|
+
reportReason(subject, state);
|
|
87
108
|
continue;
|
|
109
|
+
}
|
|
88
110
|
if (annotationOnly) {
|
|
89
111
|
reportAnnotationOnly(subject, state, annotationOnly);
|
|
90
112
|
continue;
|
package/dist/utils/analyze.d.ts
CHANGED
|
@@ -56,6 +56,11 @@ interface StateSource {
|
|
|
56
56
|
state: StateDef;
|
|
57
57
|
/** False for conditional skips unless `requireTicketForConditional` is set. */
|
|
58
58
|
required: boolean;
|
|
59
|
+
/**
|
|
60
|
+
* Set for a conditional state: whether something says why, a comment above it or its test, the reason its
|
|
61
|
+
* call takes, or one its option gives.
|
|
62
|
+
*/
|
|
63
|
+
reasoned?: boolean;
|
|
59
64
|
}
|
|
60
65
|
export interface Subject {
|
|
61
66
|
kind: 'test' | 'describe' | 'step' | 'runtime';
|
|
@@ -69,6 +74,8 @@ export interface Subject {
|
|
|
69
74
|
inherited: Set<string>;
|
|
70
75
|
/** Names of states added by runtime calls directly inside this test or describe. */
|
|
71
76
|
runtimeStates: Set<string>;
|
|
77
|
+
/** Whether the comment block above this node holds any text, an `eslint-disable` description included. */
|
|
78
|
+
commented: boolean;
|
|
72
79
|
/** Every tag written on this node, known or not. */
|
|
73
80
|
tags: TagOccurrence[];
|
|
74
81
|
/** Tags this node inherits from enclosing describes. */
|
package/dist/utils/analyze.js
CHANGED
|
@@ -841,14 +841,15 @@ const NO_TAGS = {
|
|
|
841
841
|
opaqueDetails: false,
|
|
842
842
|
annotations: [],
|
|
843
843
|
unreadAnnotations: [],
|
|
844
|
+
reasonedOptions: new Set(),
|
|
844
845
|
};
|
|
845
846
|
function collectTags(sourceCode, call, hasTitle, framework, details) {
|
|
846
|
-
if (!hasTitle)
|
|
847
|
-
return NO_TAGS;
|
|
848
847
|
const source = sourceCode.text;
|
|
849
848
|
const tags = [];
|
|
850
849
|
const title = { texts: [], unread: new Set() };
|
|
851
|
-
|
|
850
|
+
// node:test's `test({ tags: ['db'] }, fn)` has no title, only options.
|
|
851
|
+
if (hasTitle)
|
|
852
|
+
readTexts(sourceCode, call.arguments[0], title);
|
|
852
853
|
for (const found of title.texts)
|
|
853
854
|
scanTags(source, found, tags);
|
|
854
855
|
const values = { texts: [], unread: new Set() };
|
|
@@ -987,6 +988,52 @@ function isUnconditional(call, notes) {
|
|
|
987
988
|
const always = first === undefined || booleanOf(first) === true || (notes && isStaticText(first));
|
|
988
989
|
return always && !isGuarded(call);
|
|
989
990
|
}
|
|
991
|
+
/**
|
|
992
|
+
* Whether a comment says anything: a line with text other than a marker keyword alone, such as `SKIP:`. A
|
|
993
|
+
* directive counts only by its description.
|
|
994
|
+
*/
|
|
995
|
+
function hasText(comment) {
|
|
996
|
+
return commentLines(comment).some(({ text }) => /[^\s*]/.test(text) && MARKER_LINE_RE.exec(text)?.[2].trim() !== '');
|
|
997
|
+
}
|
|
998
|
+
/**
|
|
999
|
+
* Whether a reason argument says why: text that isn't blank, or a value known only when the tests run, as
|
|
1000
|
+
* `reason` or `` `${browserName} has no clipboard` ``, which is taken on trust.
|
|
1001
|
+
*/
|
|
1002
|
+
function saysWhy(sourceCode, node) {
|
|
1003
|
+
const text = staticText(sourceCode, node);
|
|
1004
|
+
if (text !== undefined)
|
|
1005
|
+
return text.trim() !== '';
|
|
1006
|
+
const value = unwrap(node);
|
|
1007
|
+
return value.type !== 'Literal' && !(value.type === 'Identifier' && value.name === 'undefined');
|
|
1008
|
+
}
|
|
1009
|
+
/** Whether `node` is text that isn't blank. */
|
|
1010
|
+
function isReasonText(sourceCode, node) {
|
|
1011
|
+
return Boolean(staticText(sourceCode, node)?.trim());
|
|
1012
|
+
}
|
|
1013
|
+
/** Whether a runtime call gives the reason for its state where its framework takes one (see `reasonArgument`). */
|
|
1014
|
+
function callReason(sourceCode, call, where) {
|
|
1015
|
+
const first = call.arguments.at(0);
|
|
1016
|
+
const second = call.arguments.at(1);
|
|
1017
|
+
if (where === 'only')
|
|
1018
|
+
return first !== undefined && saysWhy(sourceCode, first);
|
|
1019
|
+
// Vitest's `context.skip(isMobile)` takes a condition, so only text first is a reason.
|
|
1020
|
+
if (where === 'either' && first !== undefined && staticText(sourceCode, first) !== undefined) {
|
|
1021
|
+
return isReasonText(sourceCode, first);
|
|
1022
|
+
}
|
|
1023
|
+
return second !== undefined && saysWhy(sourceCode, second);
|
|
1024
|
+
}
|
|
1025
|
+
/**
|
|
1026
|
+
* Whether a conditional option gives its reason, as node:test's `{ skip: isWindows && 'no symlinks' }` or
|
|
1027
|
+
* `{ skip: isWindows ? 'no symlinks' : false }`, where the text is what Node reports.
|
|
1028
|
+
*/
|
|
1029
|
+
function optionReason(sourceCode, node) {
|
|
1030
|
+
const value = unwrap(node);
|
|
1031
|
+
if (value.type === 'LogicalExpression')
|
|
1032
|
+
return value.operator === '&&' && isReasonText(sourceCode, value.right);
|
|
1033
|
+
if (value.type !== 'ConditionalExpression')
|
|
1034
|
+
return false;
|
|
1035
|
+
return isReasonText(sourceCode, value.consequent) || isReasonText(sourceCode, value.alternate);
|
|
1036
|
+
}
|
|
990
1037
|
const FUNCTIONS = new Set(['FunctionDeclaration', 'FunctionExpression', 'ArrowFunctionExpression']);
|
|
991
1038
|
/** Whether a statement always leaves the function or block: `return`, `throw`, or a block containing one. */
|
|
992
1039
|
function exits(statement) {
|
|
@@ -1114,6 +1161,33 @@ function runAnalysis(sourceCode, options) {
|
|
|
1114
1161
|
const thisBodies = new Set();
|
|
1115
1162
|
/** Describe bodies written as a `function`, whose `this` is Mocha's suite, which `this.retries()` acts on. */
|
|
1116
1163
|
const suiteBodies = new Set();
|
|
1164
|
+
/**
|
|
1165
|
+
* Whether a statement around `anchor` has a comment that says something above it, as `// why` above an `if`,
|
|
1166
|
+
* up to `owner`, the test or describe the call acts on. The walk goes on through a callback passed to another
|
|
1167
|
+
* call, such as a hook's or a step's, but stops at a function declared some other way.
|
|
1168
|
+
*/
|
|
1169
|
+
const commentedAround = (anchor, owner) => {
|
|
1170
|
+
for (let node = anchor.parent; node; node = node.parent) {
|
|
1171
|
+
if (FUNCTIONS.has(node.type)) {
|
|
1172
|
+
const call = node.parent;
|
|
1173
|
+
if (call?.type !== 'CallExpression' || call === owner || !call.arguments.includes(node)) {
|
|
1174
|
+
return false;
|
|
1175
|
+
}
|
|
1176
|
+
continue;
|
|
1177
|
+
}
|
|
1178
|
+
const container = node.parent;
|
|
1179
|
+
if (!container || !STATEMENT_CONTAINERS.has(container.type))
|
|
1180
|
+
continue;
|
|
1181
|
+
const block = commentBlock(sourceCode, node, options.allowBlankLine);
|
|
1182
|
+
if (block.some((comment) => !commentedOut.comments.has(comment) && hasText(comment)))
|
|
1183
|
+
return true;
|
|
1184
|
+
}
|
|
1185
|
+
return false;
|
|
1186
|
+
};
|
|
1187
|
+
/** Whether a comment that says something ends the line `node` ends on, as `test.skip(isMobile); // why`. */
|
|
1188
|
+
const trailingText = (node) => sourceCode
|
|
1189
|
+
.getCommentsAfter(node)
|
|
1190
|
+
.some((comment) => comment.loc.start.line === node.loc.end.line && hasText(comment));
|
|
1117
1191
|
/**
|
|
1118
1192
|
* `modifiers` maps each state modifier the call has to whether it always applies; a conditional one,
|
|
1119
1193
|
* like `test.skip(isMobile)` or Vitest's `skipIf(...)`, needs a ticket only with `requireTicketForConditional`.
|
|
@@ -1129,6 +1203,14 @@ function runAnalysis(sourceCode, options) {
|
|
|
1129
1203
|
const inRegions = regions.filter((r) => r.start <= call.range[0] && call.range[0] < r.end && !block.includes(r.comment));
|
|
1130
1204
|
for (const comment of [...block, ...inRegions.map((r) => r.comment)])
|
|
1131
1205
|
claimed.add(comment);
|
|
1206
|
+
const commented = [...block, ...inRegions.map((r) => r.comment)].some(hasText);
|
|
1207
|
+
// A runtime call can also say why above the statements around it in its function, above its test, or in
|
|
1208
|
+
// its arguments.
|
|
1209
|
+
const reasonedCall = () => kind === 'runtime' &&
|
|
1210
|
+
((framework.reasonArgument !== undefined && callReason(sourceCode, call, framework.reasonArgument)) ||
|
|
1211
|
+
parent?.commented === true ||
|
|
1212
|
+
trailingText(anchor) ||
|
|
1213
|
+
commentedAround(anchor, parent?.node));
|
|
1132
1214
|
const ownTags = new Set(tagInfo.tags.map((t) => t.tag));
|
|
1133
1215
|
const sources = [];
|
|
1134
1216
|
for (const state of states) {
|
|
@@ -1138,8 +1220,12 @@ function runAnalysis(sourceCode, options) {
|
|
|
1138
1220
|
continue;
|
|
1139
1221
|
}
|
|
1140
1222
|
const always = modifiers.get(state.modifier);
|
|
1141
|
-
if (always
|
|
1142
|
-
|
|
1223
|
+
if (always === undefined)
|
|
1224
|
+
continue;
|
|
1225
|
+
const source = { state, required: always || options.requireTicketForConditional };
|
|
1226
|
+
if (!always)
|
|
1227
|
+
source.reasoned = commented || tagInfo.reasonedOptions.has(state.modifier) || reasonedCall();
|
|
1228
|
+
sources.push(source);
|
|
1143
1229
|
}
|
|
1144
1230
|
const inherited = new Set();
|
|
1145
1231
|
const inheritedTags = new Set();
|
|
@@ -1173,6 +1259,7 @@ function runAnalysis(sourceCode, options) {
|
|
|
1173
1259
|
anchor,
|
|
1174
1260
|
parent,
|
|
1175
1261
|
states: sources,
|
|
1262
|
+
commented,
|
|
1176
1263
|
inherited,
|
|
1177
1264
|
runtimeStates: new Set(),
|
|
1178
1265
|
tags: tagInfo.tags,
|
|
@@ -1332,9 +1419,17 @@ function runAnalysis(sourceCode, options) {
|
|
|
1332
1419
|
const annotations = options.annotations === undefined || framework.annotations === undefined
|
|
1333
1420
|
? { annotations: [], unreadAnnotations: [] }
|
|
1334
1421
|
: collectAnnotations(sourceCode, details, framework.annotations, options.annotations);
|
|
1422
|
+
const reasonedOptions = new Set();
|
|
1423
|
+
if (framework.anyOptionValue) {
|
|
1424
|
+
for (const [key, value] of details.entries) {
|
|
1425
|
+
const modifier = framework.optionEffects.get(key);
|
|
1426
|
+
if (modifier && optionReason(sourceCode, value))
|
|
1427
|
+
reasonedOptions.add(modifier);
|
|
1428
|
+
}
|
|
1429
|
+
}
|
|
1335
1430
|
return {
|
|
1336
1431
|
details,
|
|
1337
|
-
tagInfo: { ...collectTags(sourceCode, call, !untitled, framework, details), ...annotations },
|
|
1432
|
+
tagInfo: { ...collectTags(sourceCode, call, !untitled, framework, details), ...annotations, reasonedOptions },
|
|
1338
1433
|
};
|
|
1339
1434
|
};
|
|
1340
1435
|
const declare = (call, { kind, members, args: conditions }, infos) => {
|
|
@@ -95,6 +95,13 @@ export interface Framework {
|
|
|
95
95
|
* `context.skip('why')`. In Playwright the first argument is always a condition.
|
|
96
96
|
*/
|
|
97
97
|
skipNotes: boolean;
|
|
98
|
+
/**
|
|
99
|
+
* Where a runtime call takes the reason for its state: `after-condition`, as Playwright's
|
|
100
|
+
* `test.skip(condition, 'why')`; `only`, as node:test's `t.skip('why')` and Jasmine's `pending('why')`; or
|
|
101
|
+
* `either`, as Vitest's `context.skip('why')` and `context.skip(condition, 'why')`. Unset when it takes none,
|
|
102
|
+
* as Mocha's `this.skip()`.
|
|
103
|
+
*/
|
|
104
|
+
reasonArgument?: 'after-condition' | 'only' | 'either';
|
|
98
105
|
/**
|
|
99
106
|
* The methods a test or hook written as a `function` can change its state with at runtime through `this`,
|
|
100
107
|
* as Mocha's `this.skip()`. Empty when `this` can't.
|
package/dist/utils/frameworks.js
CHANGED
|
@@ -48,6 +48,7 @@ const playwright = {
|
|
|
48
48
|
subtests: false,
|
|
49
49
|
skipWithoutBody: false,
|
|
50
50
|
skipNotes: false,
|
|
51
|
+
reasonArgument: 'after-condition',
|
|
51
52
|
thisMethods: new Set(),
|
|
52
53
|
todoWithoutBody: false,
|
|
53
54
|
runtimeFunctions: new Map(),
|
|
@@ -138,6 +139,7 @@ const vitest = {
|
|
|
138
139
|
subtests: false,
|
|
139
140
|
skipWithoutBody: true,
|
|
140
141
|
skipNotes: true,
|
|
142
|
+
reasonArgument: 'either',
|
|
141
143
|
thisMethods: new Set(),
|
|
142
144
|
todoWithoutBody: false,
|
|
143
145
|
runtimeFunctions: new Map(),
|
|
@@ -236,12 +238,15 @@ const nodeTest = {
|
|
|
236
238
|
anyOptionValue: true,
|
|
237
239
|
exclusiveStates: ['skip', 'todo', 'fail'],
|
|
238
240
|
optionalTitle: true,
|
|
239
|
-
|
|
241
|
+
// Node 26.2's `{ tags: ['db'] }`: names without their `@`, as Vitest writes them.
|
|
242
|
+
tagKey: 'tags',
|
|
243
|
+
plainTags: true,
|
|
240
244
|
info: { param: 0, methods: NODE_TEST_CONTEXT },
|
|
241
245
|
subtests: true,
|
|
242
246
|
skipWithoutBody: true,
|
|
243
247
|
// `t.skip('why')` takes a message, not a condition.
|
|
244
248
|
skipNotes: true,
|
|
249
|
+
reasonArgument: 'only',
|
|
245
250
|
thisMethods: NODE_TEST_CONTEXT,
|
|
246
251
|
todoWithoutBody: false,
|
|
247
252
|
runtimeFunctions: new Map(),
|
|
@@ -286,6 +291,7 @@ const jasmine = {
|
|
|
286
291
|
skipWithoutBody: true,
|
|
287
292
|
// `pending('why')` takes a reason, not a condition.
|
|
288
293
|
skipNotes: true,
|
|
294
|
+
reasonArgument: 'only',
|
|
289
295
|
thisMethods: new Set(),
|
|
290
296
|
todoWithoutBody: true,
|
|
291
297
|
runtimeFunctions: new Map([
|
package/dist/utils/options.d.ts
CHANGED
|
@@ -44,14 +44,24 @@ export interface GovernanceOptions {
|
|
|
44
44
|
* `skip`, `fixme` and `todo` are on by default, `fail`, `slow` and `retry` off; a setting for `new` or
|
|
45
45
|
* `quarantined` wins over `lifecycleTags`. Only `new` and `quarantined` have a tag, e.g.
|
|
46
46
|
* `{ quarantined: { tag: '@flaky', marker: 'FLAKY' } }`. `fixme` and `slow` exist only in Playwright,
|
|
47
|
-
* `todo` in every framework but Playwright, `fail` in Playwright, Jest and
|
|
47
|
+
* `todo` in every framework but Playwright, `fail` in Playwright, Jest, Vitest and node:test, and `retry` in every
|
|
48
48
|
* framework but node:test and Jasmine.
|
|
49
49
|
*/
|
|
50
50
|
states?: Partial<Record<BuiltinStateName, boolean | StateOverride>>;
|
|
51
51
|
/** Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. */
|
|
52
52
|
customStates?: Record<string, CustomState>;
|
|
53
|
+
/**
|
|
54
|
+
* Your team's other tags, such as `['@smoke', '@regression']`, which get the case and typo checks state tags
|
|
55
|
+
* get: `@Smoke` and `@smok` are reported. Write them with the `@`, also for Vitest's `tags: ['smoke']`.
|
|
56
|
+
*/
|
|
57
|
+
knownTags?: string[];
|
|
53
58
|
/** Require a ticket for conditional skips such as `test.skip(browserName === 'webkit')`. */
|
|
54
59
|
requireTicketForConditional?: boolean;
|
|
60
|
+
/**
|
|
61
|
+
* Require a reason for conditional skips, without a ticket: a comment above the skip, or the reason the
|
|
62
|
+
* call takes, as `test.skip(browserName === 'webkit', 'No clipboard API')`.
|
|
63
|
+
*/
|
|
64
|
+
requireReasonForConditional?: boolean;
|
|
55
65
|
/** Let blank lines separate a marker from the test it belongs to. */
|
|
56
66
|
allowBlankLine?: boolean;
|
|
57
67
|
/**
|
|
@@ -104,7 +114,10 @@ export interface ResolvedOptions {
|
|
|
104
114
|
framework: Framework;
|
|
105
115
|
testFunctions: Set<string>;
|
|
106
116
|
states: StateDef[];
|
|
117
|
+
/** Tags besides the state tags whose case and spelling `no-conflicting-states` checks. */
|
|
118
|
+
knownTags: string[];
|
|
107
119
|
requireTicketForConditional: boolean;
|
|
120
|
+
requireReasonForConditional: boolean;
|
|
108
121
|
allowBlankLine: boolean;
|
|
109
122
|
allowNotes: boolean;
|
|
110
123
|
reportUnreadableTags: boolean;
|
package/dist/utils/options.js
CHANGED
|
@@ -217,7 +217,9 @@ export function compileOptions(options) {
|
|
|
217
217
|
framework,
|
|
218
218
|
testFunctions: new Set(options.testFunctions ?? framework.testFunctions),
|
|
219
219
|
states,
|
|
220
|
+
knownTags: options.knownTags ?? [],
|
|
220
221
|
requireTicketForConditional: options.requireTicketForConditional ?? false,
|
|
222
|
+
requireReasonForConditional: options.requireReasonForConditional ?? false,
|
|
221
223
|
allowBlankLine: options.allowBlankLine ?? false,
|
|
222
224
|
allowNotes: options.allowNotes ?? false,
|
|
223
225
|
reportUnreadableTags: options.reportUnreadableTags ?? false,
|
package/dist/utils/schema.js
CHANGED
|
@@ -96,6 +96,7 @@ export const optionsSchema = {
|
|
|
96
96
|
additionalProperties: false,
|
|
97
97
|
},
|
|
98
98
|
},
|
|
99
|
+
knownTags: { type: 'array', items: { type: 'string', pattern: TAG_RE.source, patternHint: TAG_HINT } },
|
|
99
100
|
comments: {
|
|
100
101
|
anyOf: [
|
|
101
102
|
{ type: 'boolean', enum: [false] },
|
|
@@ -119,6 +120,7 @@ export const optionsSchema = {
|
|
|
119
120
|
],
|
|
120
121
|
},
|
|
121
122
|
requireTicketForConditional: { type: 'boolean' },
|
|
123
|
+
requireReasonForConditional: { type: 'boolean' },
|
|
122
124
|
allowBlankLine: { type: 'boolean' },
|
|
123
125
|
allowNotes: { type: 'boolean' },
|
|
124
126
|
reportUnreadableTags: { type: 'boolean' },
|
|
@@ -14,13 +14,26 @@ rule reports:
|
|
|
14
14
|
- `@new` together with `@quarantined` on the same test, including tags inherited from a `describe`;
|
|
15
15
|
- `@new` on a test that is skipped or fixme'd, since a test that doesn't run can't be promoted;
|
|
16
16
|
- a state tag written in the wrong case, such as `@New` (autofixed to `@new`);
|
|
17
|
-
- a near-miss spelling of a state tag, such as `@quarantiend
|
|
17
|
+
- a near-miss spelling of a state tag, such as `@quarantiend`;
|
|
18
|
+
- the same for the tags listed in `knownTags`, such as `@Smoke` or `@smok` next to `@smoke`.
|
|
18
19
|
|
|
19
20
|
The `@new` checks run when the `new` state is on (`lifecycleTags: true`). Case and typo checks cover
|
|
20
21
|
every tag state that is on, including custom states. A tag kept in a `const` in the same file is
|
|
21
22
|
reported where the test uses it, without an autofix, since other tests may share the constant. Vitest
|
|
22
23
|
tag names are written without the `@`, so `{ tags: ['New'] }` is reported, and fixed, as `'new'`.
|
|
23
24
|
|
|
25
|
+
Tags such as `@smoke` usually pick which tests a CI job runs, so a misspelled one quietly drops a test
|
|
26
|
+
from that job. List your team's tags in `knownTags`, written with the `@`, also for Vitest, to give them
|
|
27
|
+
the same checks. A listed tag is never reported as a typo of a state tag, so `knownTags: ['@quarantine']`
|
|
28
|
+
keeps a tag your team uses next to `@quarantined`. Tags of four letters or fewer after the `@`, such as
|
|
29
|
+
`@e2e` or `@fast`, get only the case check, since one letter off is often another word.
|
|
30
|
+
|
|
31
|
+
<!-- example: invalid settings={"knownTags":["@smoke","@regression"]} -->
|
|
32
|
+
|
|
33
|
+
```js
|
|
34
|
+
test('places a market order @smok', async () => {});
|
|
35
|
+
```
|
|
36
|
+
|
|
24
37
|
## Examples
|
|
25
38
|
|
|
26
39
|
Incorrect:
|
|
@@ -68,4 +81,5 @@ This rule has no options of its own. It reads the shared options described in th
|
|
|
68
81
|
is on, and `@new` together with `@quarantined` is reported when both are on. A tag renamed in
|
|
69
82
|
`states`, such as `@flaky`, is checked in place of the default one.
|
|
70
83
|
- `customStates`: the tags of custom states get the same case and typo checks.
|
|
84
|
+
- `knownTags`: other tags that get the case and typo checks.
|
|
71
85
|
- `framework` and `testFunctions`: which functions count as the framework's `test`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
# Require a ticket
|
|
1
|
+
# Require a ticket for skipped, fixme, todo, retried and tagged tests (`test-governance/require-ticket`)
|
|
2
2
|
|
|
3
|
-
📝 Require a ticket
|
|
3
|
+
📝 Require a ticket for skipped, fixme, todo, retried and tagged tests.
|
|
4
4
|
|
|
5
5
|
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🌼 `jasmine`, 🃏 `jest`, ☕ `mocha`, 🟢 `node-test`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
@@ -109,6 +109,43 @@ title and tags count as usual, and a marker goes above it, or in code without se
|
|
|
109
109
|
If one test or describe has several markers for the same state, each must be valid: in
|
|
110
110
|
`// SKIP: TRADE-1` followed by `// SKIP: nope`, the second line is reported.
|
|
111
111
|
|
|
112
|
+
## Reasons for conditional skips
|
|
113
|
+
|
|
114
|
+
With `requireReasonForConditional: true`, a conditional skip needs a reason but no ticket, so nobody has
|
|
115
|
+
to guess later why it doesn't run. Any of these counts:
|
|
116
|
+
|
|
117
|
+
- a comment above it or at the end of its line, above a statement around it, such as the `if`, or above
|
|
118
|
+
the hook, step or test it is in. Commented-out code, a marker keyword alone such as `// SKIP:`, and an
|
|
119
|
+
`eslint-disable` comment without a description after `--` say nothing, so they don't count;
|
|
120
|
+
- the reason its call takes, where the runner has one: Playwright's `test.skip(condition, 'why')` and
|
|
121
|
+
`testInfo.skip(condition, 'why')`, Vitest's `context.skip(condition, 'why')` or `context.skip('why')`,
|
|
122
|
+
node:test's `t.skip('why')` and Jasmine's `pending('why')`. A reason held in a name or built in a
|
|
123
|
+
template is taken on trust;
|
|
124
|
+
- in node:test, text in the option, as `{ skip: isWindows && 'No symlinks' }`, which Node reports;
|
|
125
|
+
- a ticket, from a marker or an annotation.
|
|
126
|
+
|
|
127
|
+
The same goes for the other states a condition can set, such as `test.fixme(condition)`. With
|
|
128
|
+
`requireTicketForConditional` as well, the ticket is required and the reason alone is not enough.
|
|
129
|
+
|
|
130
|
+
<!-- example: valid settings={"requireReasonForConditional":true} -->
|
|
131
|
+
|
|
132
|
+
```js
|
|
133
|
+
test('copies the order ID', async ({ browserName }) => {
|
|
134
|
+
test.skip(browserName === 'webkit', 'WebKit has no clipboard API in headless mode');
|
|
135
|
+
});
|
|
136
|
+
|
|
137
|
+
// Touch screens have no hover.
|
|
138
|
+
(isMobile ? test.skip : test)('shows the price on hover', async ({ page }) => {});
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
<!-- example: invalid settings={"framework":"mocha","requireReasonForConditional":true} -->
|
|
142
|
+
|
|
143
|
+
```js
|
|
144
|
+
it('places an order', function () {
|
|
145
|
+
if (isWeekend) this.skip();
|
|
146
|
+
});
|
|
147
|
+
```
|
|
148
|
+
|
|
112
149
|
## Titles and tags that can't be read
|
|
113
150
|
|
|
114
151
|
Tags can only be read from text: a string or a template literal such as `` `buys ${symbol} @new` ``,
|
|
@@ -332,6 +369,7 @@ change what it reports:
|
|
|
332
369
|
- `lifecycleTags`, `states` and `customStates`: which states need a marker, which tag puts a test in
|
|
333
370
|
each, what the marker is called, and, for a state with its own `ticket`, which tickets it accepts.
|
|
334
371
|
- `requireTicketForConditional`: conditional skips need a ticket too.
|
|
372
|
+
- `requireReasonForConditional`: conditional skips need a reason, a comment or the call's own reason argument.
|
|
335
373
|
- `allowBlankLine`: blank lines may separate a marker from its test.
|
|
336
374
|
- `allowNotes`: text may follow the tickets.
|
|
337
375
|
- `annotations`: the ticket may also come from a Playwright annotation of the configured type or Vitest's `meta` key of that name. With `only: true`, a test or describe must take it from there, and a marker comment above one is reported with a suggestion that moves it.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "eslint-plugin-test-governance",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "ESLint rules that require a ticket
|
|
3
|
+
"version": "0.13.0",
|
|
4
|
+
"description": "ESLint rules that require a ticket for skipped, fixme, todo, retried, tagged and commented-out Playwright, Jest, Vitest, Mocha, Cypress, WebdriverIO, node:test and Jasmine tests.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"eslint",
|
|
7
7
|
"eslintplugin",
|