eslint-plugin-test-governance 0.6.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +38 -1
- package/README.md +64 -41
- package/dist/index.d.ts +3 -3
- package/dist/index.js +1 -1
- package/dist/utils/analyze.js +122 -29
- package/dist/utils/constants.d.ts +1 -1
- package/dist/utils/constants.js +1 -1
- package/dist/utils/frameworks.d.ts +4 -0
- package/dist/utils/frameworks.js +53 -1
- package/dist/utils/options.d.ts +5 -5
- package/docs/rules/marker-matches-state.md +1 -1
- package/docs/rules/no-conflicting-states.md +1 -1
- package/docs/rules/no-orphaned-marker.md +3 -3
- package/docs/rules/require-ticket-in-comments.md +3 -3
- package/docs/rules/require-ticket.md +56 -7
- package/docs/upgrading.md +2 -2
- package/package.json +5 -2
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,41 @@ All notable changes to this project are documented here. The format follows
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.7.0] - 2026-10-20
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- Mocha, Cypress and WebdriverIO support. `configs.mocha`, `configs.cypress` and `configs.webdriverio`,
|
|
14
|
+
or `configure({ framework })`, check those frameworks' tests with the same rules and markers:
|
|
15
|
+
`it.skip`, `describe.skip`, `context.skip`, `xit`, `xdescribe` and the other `x` forms, the TDD
|
|
16
|
+
interface's `suite.skip` and `test.skip`, with globals or imports from `mocha`, and `this.skip()` in a
|
|
17
|
+
test or hook written as a `function`. Under a condition, as in `if (!broker) this.skip()`, it is a
|
|
18
|
+
conditional skip, so it needs no ticket by default. Cypress and WebdriverIO run their tests on Mocha,
|
|
19
|
+
so their configs read the same calls; `webdriverio` is for WebdriverIO's Mocha framework.
|
|
20
|
+
- A Mocha test with only a title, `it('settles at T+1')`, which Mocha runs as pending, is in the `todo`
|
|
21
|
+
state and needs a `// TODO:` marker, like Jest's `test.todo`. `it.skip('title')` with no body is a
|
|
22
|
+
skip.
|
|
23
|
+
- In Cypress files, tags from @cypress/grep's `{ tags: '@smoke' }` count like tags in the title, and
|
|
24
|
+
`{ browser: '!webkit' }`, which runs the test only in some browsers, is a conditional skip.
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
|
|
28
|
+
- A runtime skip after other code in a `try` block that has a `catch`, as in
|
|
29
|
+
`try { await stat(target); test.skip(); } catch {}`, is a conditional skip: an error before it jumps to
|
|
30
|
+
the `catch`, so the test runs. Before, it needed a ticket like an unconditional skip. A skip that starts
|
|
31
|
+
the `try` block, or one in a `try` with only `finally`, still does. This covers Mocha's `this.skip()` too.
|
|
32
|
+
|
|
33
|
+
## [0.6.2] - 2026-10-14
|
|
34
|
+
|
|
35
|
+
### Fixed
|
|
36
|
+
|
|
37
|
+
- A test or describe whose function a condition picks, as in
|
|
38
|
+
`(process.env.CI ? describe.skip : describe)('order book', fn)`, is now read as a conditional skip in
|
|
39
|
+
Playwright, Jest and Vitest. Before, it wasn't seen at all: a `// SKIP:` marker above it was reported
|
|
40
|
+
as orphaned, tags in its title didn't reach its tests, and `requireTicketForConditional` didn't
|
|
41
|
+
require a ticket for it. In code without semicolons, a marker above the `;` that starts the line
|
|
42
|
+
belongs to it.
|
|
43
|
+
|
|
9
44
|
## [0.6.1] - 2026-10-14
|
|
10
45
|
|
|
11
46
|
### Fixed
|
|
@@ -294,7 +329,9 @@ First release.
|
|
|
294
329
|
- Support for ESLint 9 and 10 flat config, JavaScript and TypeScript test files, `.extend()`
|
|
295
330
|
fixtures, describe blocks, runtime skips and tags in titles or the `tag` option.
|
|
296
331
|
|
|
297
|
-
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.
|
|
332
|
+
[Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.7.0...HEAD
|
|
333
|
+
[0.7.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.6.2...v0.7.0
|
|
334
|
+
[0.6.2]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.6.1...v0.6.2
|
|
298
335
|
[0.6.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.6.0...v0.6.1
|
|
299
336
|
[0.6.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.5.1...v0.6.0
|
|
300
337
|
[0.5.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.5.0...v0.5.1
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
[](https://github.com/Huldoser/eslint-plugin-test-governance/actions/workflows/ci.yml)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
|
-
ESLint rules that keep a Playwright, Jest or
|
|
7
|
+
ESLint rules that keep a Playwright, Jest, Vitest, Mocha, Cypress or WebdriverIO suite honest. Every skipped, fixme, todo, `@new`
|
|
8
8
|
or `@quarantined` test must point at the ticket that tracks it:
|
|
9
9
|
|
|
10
10
|
<!-- example: valid -->
|
|
@@ -34,8 +34,9 @@ Requires ESLint 9 or 10 with flat config, and Node.js 22.12 or later. For TypeSc
|
|
|
34
34
|
|
|
35
35
|
## Usage
|
|
36
36
|
|
|
37
|
-
Pick the config for the framework your tests use, `configs.playwright`, `configs.jest
|
|
38
|
-
`configs.vitest`, and point it at those
|
|
37
|
+
Pick the config for the framework your tests use, `configs.playwright`, `configs.jest`,
|
|
38
|
+
`configs.vitest`, `configs.mocha`, `configs.cypress` or `configs.webdriverio`, and point it at those
|
|
39
|
+
test files with `files`:
|
|
39
40
|
|
|
40
41
|
```js
|
|
41
42
|
// eslint.config.js
|
|
@@ -52,9 +53,14 @@ export default [
|
|
|
52
53
|
```
|
|
53
54
|
|
|
54
55
|
There is no default config. Each framework declares, skips and tags tests its own way, so the rules
|
|
55
|
-
need to know which one a file uses. In a Jest or
|
|
56
|
-
that framework, whether they are globals or imported from `@jest/globals` or `
|
|
57
|
-
Jest and
|
|
56
|
+
need to know which one a file uses. In a Jest, Vitest or Mocha config, `test`, `it` and `describe` belong
|
|
57
|
+
to that framework, whether they are globals or imported from `@jest/globals`, `vitest` or `mocha`, the
|
|
58
|
+
way the Jest, Vitest and Mocha ESLint plugins are set up.
|
|
59
|
+
|
|
60
|
+
Cypress and WebdriverIO run their tests on Mocha, so `configs.cypress` and `configs.webdriverio` read the
|
|
61
|
+
same calls as `configs.mocha`. `webdriverio` means WebdriverIO's Mocha framework; its Jasmine framework
|
|
62
|
+
isn't supported yet. Puppeteer has no test functions of its own, so use the config of the runner its
|
|
63
|
+
tests run on: `configs.jest` for jest-puppeteer, or `configs.vitest` or `configs.mocha`.
|
|
58
64
|
|
|
59
65
|
With ESLint's `defineConfig()` (ESLint 9.22 or later), put the config in `extends`:
|
|
60
66
|
|
|
@@ -125,24 +131,29 @@ export default [
|
|
|
125
131
|
];
|
|
126
132
|
```
|
|
127
133
|
|
|
128
|
-
|
|
129
|
-
`configure()` for the same files keeps that config's options and changes only the framework.
|
|
134
|
+
Each framework's config, such as `configs.playwright`, sets only the framework, so one listed after a
|
|
135
|
+
`configure()` for the same files keeps that config's options and changes only the framework. They all
|
|
130
136
|
turn on the same rules. What puts a test in a state in each framework:
|
|
131
137
|
|
|
132
|
-
| State (marker) | Playwright | Jest | Vitest |
|
|
133
|
-
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
134
|
-
| `skip` (`// SKIP:`) | `test.skip(...)`, `test.describe.skip`, `test.skip()` in a body, `testInfo.skip()`, `test.step.skip`, `step.skip()` | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xtest`, `xdescribe` | `test.skip`, `it.skip`, `describe.skip`, `suite.skip`, `skipIf(true)`, `runIf(false)`, `{ skip: true }`, `context.skip()`, `context.skip('note')` or `skip()` in a body or in `beforeEach` |
|
|
135
|
-
| `todo` (`// TODO:`) | | `test.todo`, `it.todo` | `test.todo`, `describe.todo`, `{ todo: true }` |
|
|
136
|
-
| `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | | |
|
|
137
|
-
| `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }` |
|
|
138
|
-
| `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | | |
|
|
139
|
-
| conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()`
|
|
140
|
-
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` |
|
|
138
|
+
| State (marker) | Playwright | Jest | Vitest | Mocha | Cypress | WebdriverIO |
|
|
139
|
+
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ----------- |
|
|
140
|
+
| `skip` (`// SKIP:`) | `test.skip(...)`, `test.describe.skip`, `test.skip()` in a body, `testInfo.skip()`, `test.step.skip`, `step.skip()` | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xtest`, `xdescribe` | `test.skip`, `it.skip`, `describe.skip`, `suite.skip`, `skipIf(true)`, `runIf(false)`, `{ skip: true }`, `context.skip()`, `context.skip('note')` or `skip()` in a body or in `beforeEach` | `it.skip`, `specify.skip`, `describe.skip`, `context.skip`, `xit`, `xspecify`, `xdescribe`, `xcontext`, TDD `test.skip` and `suite.skip`, `this.skip()` in a test or hook | as Mocha | as Mocha |
|
|
141
|
+
| `todo` (`// TODO:`) | | `test.todo`, `it.todo` | `test.todo`, `describe.todo`, `{ todo: true }` | a test with only a title, `it('title')`, which Mocha runs as pending | as Mocha | as Mocha |
|
|
142
|
+
| `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | | | | | |
|
|
143
|
+
| `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }` | | | |
|
|
144
|
+
| `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | | | | | |
|
|
145
|
+
| conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)` | `(condition ? it.skip : it)(...)` | `skipIf(condition)`, `runIf(condition)`, `{ skip: condition }`, `context.skip(condition)`, `if (...) context.skip()`, `(condition ? it.skip : it)(...)` | `if (...) this.skip()`, `(condition ? it.skip : it)(...)` | as Mocha, and `{ browser: '!webkit' }` | as Mocha |
|
|
146
|
+
| tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` | the title | the title, `{ tags }` from @cypress/grep | the title |
|
|
141
147
|
|
|
142
148
|
Table-driven tests count too: `test.skip.each(table)(...)`, `describe.skip.each` and Vitest's `.for`.
|
|
149
|
+
A test or describe whose function a condition picks, as in `(isMobile ? test.skip : test)('title', fn)`,
|
|
150
|
+
is a conditional skip in every framework.
|
|
143
151
|
A literal condition decides on its own: `skipIf(true)` and `runIf(false)` always skip, and `skipIf(false)`,
|
|
144
152
|
`runIf(true)` and `{ skip: false }` never do.
|
|
145
|
-
In Vitest, `test.skip('title')` and `describe.skip('title')` with no body are skipped too.
|
|
153
|
+
In Vitest and Mocha, `test.skip('title')` and `describe.skip('title')` with no body are skipped too.
|
|
154
|
+
In Mocha, `this.skip()` counts in a test or hook written as a `function`; an arrow function gets no
|
|
155
|
+
Mocha context, so its `this` is not the test's. `this.slow(ms)` only sets Mocha's speed threshold, so
|
|
156
|
+
it isn't the `slow` state.
|
|
146
157
|
Vitest writes tag names without the `@`, as in `{ tags: ['flaky'] }`, so a name in `tags` counts as the
|
|
147
158
|
tag with an `@` in front: `'flaky'` is `@flaky`, and a custom state with `when: '@flaky'` matches it.
|
|
148
159
|
|
|
@@ -152,19 +163,22 @@ tag with an `@` in front: `'flaky'` is `@flaky`, and a custom state with `when:
|
|
|
152
163
|
|
|
153
164
|
💼 [Configurations](https://github.com/Huldoser/eslint-plugin-test-governance#usage) enabled in.\
|
|
154
165
|
⚠️ [Configurations](https://github.com/Huldoser/eslint-plugin-test-governance#usage) set to warn in.\
|
|
166
|
+
🌲 Set in the `cypress` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
155
167
|
🃏 Set in the `jest` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
168
|
+
☕ Set in the `mocha` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
156
169
|
🎭 Set in the `playwright` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
157
170
|
⚡ Set in the `vitest` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
171
|
+
🤖 Set in the `webdriverio` [configuration](https://github.com/Huldoser/eslint-plugin-test-governance#usage).\
|
|
158
172
|
🔧 Automatically fixable by the [`--fix` CLI option](https://eslint.org/docs/user-guide/command-line-interface#--fix).\
|
|
159
173
|
💡 Manually fixable by [editor suggestions](https://eslint.org/docs/latest/use/core-concepts#rule-suggestions).
|
|
160
174
|
|
|
161
|
-
| Name | Description | 💼
|
|
162
|
-
| :--------------------------------------------------------------------- | :-------------------------------------------------------------------------------- |
|
|
163
|
-
| [marker-matches-state](docs/rules/marker-matches-state.md) | Require the marker keyword to match the test's state | 🃏 🎭 ⚡ |
|
|
164
|
-
| [no-conflicting-states](docs/rules/no-conflicting-states.md) | Disallow state tags that contradict each other, differ in case or look like typos |
|
|
165
|
-
| [no-orphaned-marker](docs/rules/no-orphaned-marker.md) | Disallow marker comments that no longer match a test state | 🃏 🎭 ⚡ |
|
|
166
|
-
| [require-ticket](docs/rules/require-ticket.md) | Require a ticket marker comment above skipped, fixme and tagged tests | 🃏 🎭 ⚡ |
|
|
167
|
-
| [require-ticket-in-comments](docs/rules/require-ticket-in-comments.md) | Require FIXME and TODO comments in test files to start with a ticket | 🃏 🎭 ⚡ |
|
|
175
|
+
| Name | Description | 💼 | ⚠️ | 🔧 | 💡 |
|
|
176
|
+
| :--------------------------------------------------------------------- | :-------------------------------------------------------------------------------- | :---------------- | :---------------- | :-- | :-- |
|
|
177
|
+
| [marker-matches-state](docs/rules/marker-matches-state.md) | Require the marker keyword to match the test's state | 🌲 🃏 ☕ 🎭 ⚡ 🤖 | | | 💡 |
|
|
178
|
+
| [no-conflicting-states](docs/rules/no-conflicting-states.md) | Disallow state tags that contradict each other, differ in case or look like typos | | 🌲 🃏 ☕ 🎭 ⚡ 🤖 | 🔧 | |
|
|
179
|
+
| [no-orphaned-marker](docs/rules/no-orphaned-marker.md) | Disallow marker comments that no longer match a test state | 🌲 🃏 ☕ 🎭 ⚡ 🤖 | | | 💡 |
|
|
180
|
+
| [require-ticket](docs/rules/require-ticket.md) | Require a ticket marker comment above skipped, fixme and tagged tests | 🌲 🃏 ☕ 🎭 ⚡ 🤖 | | | 💡 |
|
|
181
|
+
| [require-ticket-in-comments](docs/rules/require-ticket-in-comments.md) | Require FIXME and TODO comments in test files to start with a ticket | 🌲 🃏 ☕ 🎭 ⚡ 🤖 | | | 💡 |
|
|
168
182
|
|
|
169
183
|
<!-- end auto-generated rules list -->
|
|
170
184
|
|
|
@@ -174,7 +188,7 @@ The defaults follow how most teams already work, so each framework config is use
|
|
|
174
188
|
|
|
175
189
|
| Setting | Default | Why |
|
|
176
190
|
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
177
|
-
| States that need a ticket | `skip`, `fixme`; in
|
|
191
|
+
| States that need a ticket | `skip`, `fixme`; in the other frameworks `skip`, `todo` | Every team skips tests. Chromium DevTools, GitLab and Slack all require a bug link on disabled tests. |
|
|
178
192
|
| `@new`, `@quarantined` | off; `lifecycleTags: true` turns both on | A promotion workflow is not universal. Teams that say `@flaky` instead [rename the tag](#tag-names). |
|
|
179
193
|
| Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true` changes this. |
|
|
180
194
|
| `test.fail`, `test.slow`, Jest's `test.failing`, Vitest's `test.fails` | off | They don't remove coverage. Turn them on with `states: { fail: true, slow: true }`. |
|
|
@@ -194,20 +208,20 @@ yourself. `framework` is required, and every other option has a default. The rul
|
|
|
194
208
|
of their own: they all read these shared settings, so they always agree on ticket formats, states
|
|
195
209
|
and markers. To turn a rule off, set it to `'off'` as usual.
|
|
196
210
|
|
|
197
|
-
| Option | Type
|
|
198
|
-
| ----------------------------- |
|
|
199
|
-
| `framework` | `'playwright' \| 'jest' \| 'vitest'`
|
|
200
|
-
| `ticket` | `TicketSpec \| TicketSpec[]`
|
|
201
|
-
| `placeholders` | `string[]`
|
|
202
|
-
| `lifecycleTags` | `boolean`
|
|
203
|
-
| `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }`
|
|
204
|
-
| `customStates` | `{ [name]: { when, marker, ticket? } }`
|
|
205
|
-
| `requireTicketForConditional` | `boolean`
|
|
206
|
-
| `allowBlankLine` | `boolean`
|
|
207
|
-
| `allowNotes` | `boolean`
|
|
208
|
-
| `comments` | `false \| { keywords?: string[] }`
|
|
209
|
-
| `testFunctions` | `string[]`
|
|
210
|
-
| `reportUnreadableTags` | `boolean`
|
|
211
|
+
| Option | Type | Default | Description |
|
|
212
|
+
| ----------------------------- | ----------------------------------------------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
213
|
+
| `framework` | `'playwright' \| 'jest' \| 'vitest' \| 'mocha' \| 'cypress' \| 'webdriverio'` | required | The test framework of the files the config applies to. Each framework's config, such as `configs.playwright`, sets it, and `configure()` needs it. See [Frameworks](#frameworks). |
|
|
214
|
+
| `ticket` | `TicketSpec \| TicketSpec[]` | `{ preset: 'any' }` | Accepted ticket formats. An array accepts a ticket that matches any entry. See [Ticket presets](#ticket-presets). |
|
|
215
|
+
| `placeholders` | `string[]` | see above | Tickets rejected as placeholders. `*` matches any characters. Case-insensitive; a leading `#` is ignored. Your list replaces the default one, so repeat the defaults you want to keep. Tickets numbered 0 are always rejected. |
|
|
216
|
+
| `lifecycleTags` | `boolean` | `false` | Turns on the `new` (`@new`) and `quarantined` (`@quarantined`) states. |
|
|
217
|
+
| `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `new`, `quarantined`) on or off, renames their tag or marker, or gives them their own ticket format. `fail` and `slow` are off by default; a setting for `new` or `quarantined` here wins over `lifecycleTags`. Only `new` and `quarantined` have a tag to rename; see [Tag names](#tag-names). `fixme` and `slow` exist only in Playwright, `todo` in every framework but Playwright, and `fail` in Playwright, Jest and Vitest, so turning one on elsewhere has no effect. |
|
|
218
|
+
| `customStates` | `{ [name]: { when, marker, ticket? } }` | `{}` | Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. `when` is a tag and `marker` an uppercase keyword; neither can be shared with another state. |
|
|
219
|
+
| `requireTicketForConditional` | `boolean` | `false` | Require a ticket for conditional skips too: `test.skip(condition, ...)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)`, a skip in a `catch` block, after other code in a `try` block with a `catch`, or after an early `return` or `throw`. |
|
|
220
|
+
| `allowBlankLine` | `boolean` | `false` | Let blank lines separate a marker from its test. |
|
|
221
|
+
| `allowNotes` | `boolean` | `false` | Allow free text after the tickets, e.g. `// SKIP: TRADE-123 flaky on CI`. Applies to markers and to `FIXME` / `TODO` comments. |
|
|
222
|
+
| `comments` | `false \| { keywords?: string[] }` | `{ keywords: ['FIXME', 'TODO'] }` | Comment keywords that must start with a ticket anywhere in a test file, e.g. `// TODO: TRADE-123`. Your keywords replace the default ones. `false` allows free-form comments. |
|
|
223
|
+
| `testFunctions` | `string[]` | `['test']` | Names treated as the framework's `test`. Jest and Vitest default to `['test', 'it']`, and Mocha, Cypress and WebdriverIO to `['it', 'specify', 'test']`. Your list replaces the default one, so include those names if you still need them. See below. |
|
|
224
|
+
| `reportUnreadableTags` | `boolean` | `false` | When a tag state is on, report titles that aren't static text, and tags or details that can't be read in the file, such as ones imported from another module, since their tags can't be checked. |
|
|
211
225
|
|
|
212
226
|
### Tag names
|
|
213
227
|
|
|
@@ -291,6 +305,14 @@ In Jest and Vitest files:
|
|
|
291
305
|
- Vitest fixtures made with `test.extend()` in the same file.
|
|
292
306
|
- Any name in `testFunctions`, which defaults to `['test', 'it']`.
|
|
293
307
|
|
|
308
|
+
In Mocha, Cypress and WebdriverIO files:
|
|
309
|
+
|
|
310
|
+
- The globals `it`, `specify`, `describe` and `context`, their `x` forms, the TDD names `suite` and
|
|
311
|
+
`test`, and the hooks `before`, `beforeEach`, `after`, `afterEach`, `suiteSetup`, `setup`,
|
|
312
|
+
`suiteTeardown` and `teardown`.
|
|
313
|
+
- The names imported or required from `mocha`, including aliases and `import * as mocha from 'mocha'`.
|
|
314
|
+
- Any name in `testFunctions`, which defaults to `['it', 'specify', 'test']`.
|
|
315
|
+
|
|
294
316
|
A name imported or required from another test runner (Playwright, `vitest`, `@jest/globals`,
|
|
295
317
|
`node:test`, `bun:test`, `mocha`, `ava`, `tap`, `uvu`), or a variable set from one such as
|
|
296
318
|
`const test = require('ava')`, is never treated as the framework's `test`, and neither is a local
|
|
@@ -319,7 +341,8 @@ The [Frameworks](#frameworks) table lists what each framework's tests can do. In
|
|
|
319
341
|
Not covered in this version: tests generated in loops with dynamic titles, and tags or details imported
|
|
320
342
|
from other modules (see `reportUnreadableTags`), fixtures imported from other modules under a name not in
|
|
321
343
|
`testFunctions`, a Vitest test with only a title, which Vitest runs as a todo but which looks the same as a
|
|
322
|
-
helper called `test`, Playwright annotations as markers,
|
|
344
|
+
helper called `test`, Playwright annotations as markers, Cypress tests skipped with @cypress/skip-test's
|
|
345
|
+
`skipOn()` and `onlyOn()`, and other runners such as Jasmine, Bun and `node:test`.
|
|
323
346
|
|
|
324
347
|
## TypeScript
|
|
325
348
|
|
package/dist/index.d.ts
CHANGED
|
@@ -13,8 +13,8 @@ export interface FlatConfig {
|
|
|
13
13
|
settings?: Record<string, unknown>;
|
|
14
14
|
}
|
|
15
15
|
/**
|
|
16
|
-
* One config per framework.
|
|
17
|
-
*
|
|
16
|
+
* One config per framework. Each turns on the same rules for that framework's tests; point each one at its
|
|
17
|
+
* test files with `files`.
|
|
18
18
|
*/
|
|
19
19
|
export type ConfigName = FrameworkName;
|
|
20
20
|
export interface TestGovernancePlugin extends ESLint.Plugin {
|
|
@@ -33,5 +33,5 @@ declare const plugin: TestGovernancePlugin;
|
|
|
33
33
|
* so all rules read the same ticket formats and states.
|
|
34
34
|
*/
|
|
35
35
|
export declare function configure(options: GovernanceOptions): FlatConfig;
|
|
36
|
-
export declare const configs: Record<"playwright" | "jest" | "vitest", FlatConfig>;
|
|
36
|
+
export declare const configs: Record<"playwright" | "jest" | "vitest" | "mocha" | "cypress" | "webdriverio", FlatConfig>;
|
|
37
37
|
export default plugin;
|
package/dist/index.js
CHANGED
|
@@ -52,7 +52,7 @@ for (const framework of FRAMEWORK_NAMES) {
|
|
|
52
52
|
// tools that list the configs don't see it.
|
|
53
53
|
Object.defineProperty(plugin.configs, 'recommended', {
|
|
54
54
|
get() {
|
|
55
|
-
throw new ConfigError('configs.recommended was removed in 0.5.0. Use the config for your framework: configs.playwright, configs.jest or configs.
|
|
55
|
+
throw new ConfigError('configs.recommended was removed in 0.5.0. Use the config for your framework: configs.playwright, configs.jest, configs.vitest, configs.mocha, configs.cypress or configs.webdriverio. See https://github.com/Huldoser/eslint-plugin-test-governance/blob/main/docs/upgrading.md');
|
|
56
56
|
},
|
|
57
57
|
});
|
|
58
58
|
export const configs = plugin.configs;
|
package/dist/utils/analyze.js
CHANGED
|
@@ -247,7 +247,12 @@ function anchorOf(call) {
|
|
|
247
247
|
}
|
|
248
248
|
/** The contiguous comment block directly above `anchor`. */
|
|
249
249
|
function commentBlock(sourceCode, anchor, allowBlankLine) {
|
|
250
|
-
|
|
250
|
+
// Code without semicolons starts a line that opens with `(` with one: `;(isCI ? describe.skip : describe)(...)`.
|
|
251
|
+
const semicolon = sourceCode.getTokenBefore(anchor);
|
|
252
|
+
const leading = semicolon?.value === ';' &&
|
|
253
|
+
semicolon.loc.start.line === anchor.loc.start.line &&
|
|
254
|
+
sourceCode.getTokenBefore(semicolon)?.loc.end.line !== semicolon.loc.start.line;
|
|
255
|
+
const comments = sourceCode.getCommentsBefore(leading ? semicolon : anchor);
|
|
251
256
|
const block = [];
|
|
252
257
|
let nextLine = anchor.loc.start.line;
|
|
253
258
|
for (let i = comments.length - 1; i >= 0; i--) {
|
|
@@ -487,10 +492,17 @@ function collectTags(sourceCode, call, hasTitle, framework, details) {
|
|
|
487
492
|
unread.push(details.unread);
|
|
488
493
|
return { tags, dynamic: title.unread.size > 0, unread, opaqueDetails: details.unread !== undefined };
|
|
489
494
|
}
|
|
495
|
+
/** Whether a title is text, written out or held in a constant declared in this file. */
|
|
496
|
+
function isTextTitle(sourceCode, node) {
|
|
497
|
+
const title = unwrap(node);
|
|
498
|
+
const value = constantValue(sourceCode, title);
|
|
499
|
+
return isStaticText(title) || (value !== undefined && isStaticText(unwrap(value)));
|
|
500
|
+
}
|
|
490
501
|
/**
|
|
491
502
|
* Whether a call declares a test or describe block: a title and a body, as in `test('title', fn)`,
|
|
492
503
|
* `test('title', details, fn)` or `test.skip(title, payWithCard)` with the body defined elsewhere, a
|
|
493
|
-
* describe with only a body, or a `todo` with only a title,
|
|
504
|
+
* describe with only a body, or a `todo` with only a title, in Vitest and Mocha a `skip` with only a
|
|
505
|
+
* title, and in Mocha a test with only a title written out, which it runs as pending.
|
|
494
506
|
* Only Playwright has a runtime `test.skip(isMobile, 'why')`; there a title, written out or held in a
|
|
495
507
|
* constant, or a body written in place tells the two apart.
|
|
496
508
|
*/
|
|
@@ -500,13 +512,22 @@ function isDeclaration(sourceCode, framework, kind, args, members) {
|
|
|
500
512
|
return kind === 'describe';
|
|
501
513
|
if (first === undefined)
|
|
502
514
|
return false;
|
|
503
|
-
if (args.length === 1)
|
|
504
|
-
|
|
515
|
+
if (args.length === 1) {
|
|
516
|
+
if (members.includes('todo') || (framework.skipWithoutBody && members.includes('skip')))
|
|
517
|
+
return true;
|
|
518
|
+
return framework.todoWithoutBody && kind === 'test' && isTextTitle(sourceCode, first);
|
|
519
|
+
}
|
|
505
520
|
if (!framework.playwright)
|
|
506
521
|
return true;
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
522
|
+
return isTextTitle(sourceCode, first) || args.slice(1).some(isFunction);
|
|
523
|
+
}
|
|
524
|
+
/** The function whose `this` a call sees: the nearest one around it that isn't an arrow function. */
|
|
525
|
+
function thisFunction(node) {
|
|
526
|
+
let current = node.parent;
|
|
527
|
+
while (current && (current.type === 'ArrowFunctionExpression' || !FUNCTIONS.has(current.type))) {
|
|
528
|
+
current = current.parent;
|
|
529
|
+
}
|
|
530
|
+
return current;
|
|
510
531
|
}
|
|
511
532
|
/** The value of a `true` or `false` literal; undefined for anything else. */
|
|
512
533
|
function booleanOf(node) {
|
|
@@ -535,8 +556,8 @@ function isEarlyExit(statement) {
|
|
|
535
556
|
}
|
|
536
557
|
/**
|
|
537
558
|
* Whether the call only runs on some paths of its function: under an `if`, a `switch` case, a ternary,
|
|
538
|
-
* the right side of `&&`, `||` or `??`, in a `catch` block,
|
|
539
|
-
* `if (ready) return;`. `if (!enabled) test.skip()` is the same as `test.skip(!enabled)`, and
|
|
559
|
+
* the right side of `&&`, `||` or `??`, in a `catch` block, after other code in a `try` block that has a
|
|
560
|
+
* `catch`, or after an early exit such as `if (ready) return;`. `if (!enabled) test.skip()` is the same as `test.skip(!enabled)`, and
|
|
540
561
|
* `catch { test.skip(true, 'service is down') }` only skips when the service is down.
|
|
541
562
|
*/
|
|
542
563
|
function isGuarded(call) {
|
|
@@ -554,6 +575,10 @@ function isGuarded(call) {
|
|
|
554
575
|
const index = node.body.indexOf(child);
|
|
555
576
|
if (node.body.slice(0, index).some(isEarlyExit))
|
|
556
577
|
return true;
|
|
578
|
+
// `try { await stat(target); test.skip(); } catch {}`: an error before the skip jumps to the catch.
|
|
579
|
+
const owner = node.parent;
|
|
580
|
+
if (index > 0 && owner.type === 'TryStatement' && owner.block === node && owner.handler)
|
|
581
|
+
return true;
|
|
557
582
|
}
|
|
558
583
|
}
|
|
559
584
|
return false;
|
|
@@ -612,6 +637,8 @@ function runAnalysis(sourceCode, options) {
|
|
|
612
637
|
const keys = sourceCode.visitorKeys;
|
|
613
638
|
/** States an options object can put a test in, such as `skip` for Vitest's `{ skip: true }`. */
|
|
614
639
|
const optionModifiers = new Set(framework.optionEffects.values());
|
|
640
|
+
/** Bodies of Mocha tests and hooks written as a `function`, where `this.skip()` skips at runtime. */
|
|
641
|
+
const thisBodies = new Set();
|
|
615
642
|
/**
|
|
616
643
|
* `modifiers` maps each state modifier the call has to whether it always applies; a conditional one,
|
|
617
644
|
* like `test.skip(isMobile)` or Vitest's `skipIf(...)`, needs a ticket only with `requireTicketForConditional`.
|
|
@@ -717,6 +744,22 @@ function runAnalysis(sourceCode, options) {
|
|
|
717
744
|
}
|
|
718
745
|
return modifiers;
|
|
719
746
|
};
|
|
747
|
+
/**
|
|
748
|
+
* The modifiers of a declaration, with `todo` for a Mocha test that has only a title, which Mocha runs as
|
|
749
|
+
* pending: someone has yet to write it. A skipped one, as `it.skip('title')`, stays a skip.
|
|
750
|
+
*/
|
|
751
|
+
const declarationModifiers = (kind, args, modifiers) => {
|
|
752
|
+
if (framework.todoWithoutBody && kind === 'test' && args.length === 1 && !modifiers.has('skip')) {
|
|
753
|
+
modifiers.set('todo', true);
|
|
754
|
+
}
|
|
755
|
+
return modifiers;
|
|
756
|
+
};
|
|
757
|
+
/** Keeps the body of a Mocha test or hook written as a `function`, whose `this` can skip it. */
|
|
758
|
+
const addThisBody = (args) => {
|
|
759
|
+
const body = args.findLast(isFunction);
|
|
760
|
+
if (framework.thisSkip && body?.type === 'FunctionExpression')
|
|
761
|
+
thisBodies.add(body);
|
|
762
|
+
};
|
|
720
763
|
/** What a body can change the test's state with: Playwright's `testInfo` or Vitest's test context. */
|
|
721
764
|
const bodyInfos = (args, path) => {
|
|
722
765
|
const body = args.findLast(isFunction);
|
|
@@ -726,17 +769,65 @@ function runAnalysis(sourceCode, options) {
|
|
|
726
769
|
const param = body.params.at(framework.info.param + (path.includes('for()') ? 1 : 0));
|
|
727
770
|
return infoNames(param, framework.info.methods);
|
|
728
771
|
};
|
|
772
|
+
/**
|
|
773
|
+
* What a callee declares and with which members: `describe` and `['skip']` for `test.describe.skip`.
|
|
774
|
+
* Undefined for a hook.
|
|
775
|
+
*/
|
|
776
|
+
const declarationOf = ({ root, path, args }) => {
|
|
777
|
+
if (root.kind === 'hook')
|
|
778
|
+
return undefined;
|
|
779
|
+
const playwrightDescribe = framework.playwright && path[0] === 'describe';
|
|
780
|
+
const kind = playwrightDescribe ? 'describe' : root.kind;
|
|
781
|
+
return { kind, members: [...root.modifiers, ...(playwrightDescribe ? path.slice(1) : path)], args };
|
|
782
|
+
};
|
|
783
|
+
/** Whether a call with these members and arguments declares a test or describe of this kind. */
|
|
784
|
+
const declares = ({ kind, members }, args) => {
|
|
785
|
+
const allowed = kind === 'test' ? framework.testMembers : framework.describeMembers;
|
|
786
|
+
return members.every((m) => allowed.has(m)) && isDeclaration(sourceCode, framework, kind, args, members);
|
|
787
|
+
};
|
|
788
|
+
/**
|
|
789
|
+
* `(isCI ? describe.skip : describe)('order entry', ...)`: a test or describe whose function is picked by
|
|
790
|
+
* a condition. A state that only one side has depends on the condition, like `test.skip(isCI)`.
|
|
791
|
+
*/
|
|
792
|
+
const pickedDeclaration = (call) => {
|
|
793
|
+
const callee = unwrap(call.callee);
|
|
794
|
+
if (callee.type !== 'ConditionalExpression')
|
|
795
|
+
return undefined;
|
|
796
|
+
const [yes, no] = [callee.consequent, callee.alternate].map((side) => {
|
|
797
|
+
const picked = target(chainOf(unwrap(side), framework.factories));
|
|
798
|
+
return picked && declarationOf(picked);
|
|
799
|
+
});
|
|
800
|
+
const args = call.arguments;
|
|
801
|
+
if (yes === undefined || no === undefined)
|
|
802
|
+
return undefined;
|
|
803
|
+
if (yes.kind !== no.kind || !declares(yes, args) || !declares(no, args))
|
|
804
|
+
return undefined;
|
|
805
|
+
usesTest = true;
|
|
806
|
+
const details = detailsOf(sourceCode, call);
|
|
807
|
+
const tagInfo = collectTags(sourceCode, call, !isFunction(args[0]), framework, details);
|
|
808
|
+
const [ifYes, ifNo] = [yes, no].map(({ members, args: conditions }) => declarationModifiers(yes.kind, args, modifiersOf(members, details.entries, conditions)));
|
|
809
|
+
if (yes.kind === 'test')
|
|
810
|
+
addThisBody(args);
|
|
811
|
+
const modifiers = new Map();
|
|
812
|
+
for (const modifier of new Set([...ifYes.keys(), ...ifNo.keys()])) {
|
|
813
|
+
modifiers.set(modifier, ifYes.get(modifier) === true && ifNo.get(modifier) === true);
|
|
814
|
+
}
|
|
815
|
+
return { subject: makeSubject(yes.kind, call, modifiers, tagInfo), infos: bodyInfos(args, yes.members) };
|
|
816
|
+
};
|
|
729
817
|
const visitCall = (call) => {
|
|
730
818
|
const callee = target(chainOf(call.callee, framework.factories));
|
|
731
819
|
if (callee === undefined)
|
|
732
|
-
return { subject: runtimeCall(call) };
|
|
820
|
+
return pickedDeclaration(call) ?? { subject: runtimeCall(call) };
|
|
733
821
|
usesTest = true;
|
|
734
822
|
const args = call.arguments;
|
|
735
|
-
const {
|
|
736
|
-
|
|
737
|
-
// Vitest's `beforeEach((context) => context.skip())`
|
|
738
|
-
|
|
823
|
+
const { path } = callee;
|
|
824
|
+
const declaration = declarationOf(callee);
|
|
825
|
+
// Vitest's `beforeEach((context) => context.skip())` and Mocha's `before(function () { this.skip(); })`
|
|
826
|
+
// skip the tests they run for.
|
|
827
|
+
if (declaration === undefined) {
|
|
828
|
+
addThisBody(args);
|
|
739
829
|
return { infos: bodyInfos(args, path) };
|
|
830
|
+
}
|
|
740
831
|
if (framework.playwright) {
|
|
741
832
|
// `test.step('fills the order', async (step) => {...})` or `test.step.skip(...)`. The step's body
|
|
742
833
|
// gets a `TestStepInfo` that can skip the step at runtime.
|
|
@@ -749,29 +840,26 @@ function runAnalysis(sourceCode, options) {
|
|
|
749
840
|
}
|
|
750
841
|
}
|
|
751
842
|
const bodyInfo = bodyInfos(args, path);
|
|
752
|
-
|
|
753
|
-
if (
|
|
754
|
-
kind = 'describe';
|
|
755
|
-
path = path.slice(1);
|
|
756
|
-
}
|
|
757
|
-
const members = [...root.modifiers, ...path];
|
|
758
|
-
const allowed = kind === 'test' ? framework.testMembers : framework.describeMembers;
|
|
759
|
-
if (members.every((m) => allowed.has(m)) && isDeclaration(sourceCode, framework, kind, args, members)) {
|
|
843
|
+
const { kind, members } = declaration;
|
|
844
|
+
if (declares(declaration, args)) {
|
|
760
845
|
// A declaration always has a first argument: its title, or the body of a describe without one.
|
|
761
846
|
const details = detailsOf(sourceCode, call);
|
|
762
847
|
const tagInfo = collectTags(sourceCode, call, !isFunction(args[0]), framework, details);
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
};
|
|
848
|
+
const modifiers = declarationModifiers(kind, args, modifiersOf(members, details.entries, callee.args));
|
|
849
|
+
if (kind === 'test')
|
|
850
|
+
addThisBody(args);
|
|
851
|
+
return { subject: makeSubject(kind, call, modifiers, tagInfo), infos: bodyInfo };
|
|
767
852
|
}
|
|
768
853
|
// test.skip(), test.fixme(), test.fail() and test.slow() in a test, a describe or a hook.
|
|
769
|
-
if (framework.playwright &&
|
|
770
|
-
return { subject: runtimeSubject(call,
|
|
854
|
+
if (framework.playwright && members.length === 1 && RUNTIME_MODIFIERS.has(members[0])) {
|
|
855
|
+
return { subject: runtimeSubject(call, members[0]), infos: bodyInfo };
|
|
771
856
|
}
|
|
772
857
|
return { infos: bodyInfo };
|
|
773
858
|
};
|
|
774
|
-
/**
|
|
859
|
+
/**
|
|
860
|
+
* `testInfo.skip()`, `step.skip()`, `test.info().skip()`, Vitest's `context.skip()` or `skip()`, and
|
|
861
|
+
* Mocha's `this.skip()`.
|
|
862
|
+
*/
|
|
775
863
|
const runtimeCall = (call) => {
|
|
776
864
|
const { callee } = call;
|
|
777
865
|
// The innermost parameter with a name is the one the call uses.
|
|
@@ -786,6 +874,11 @@ function runAnalysis(sourceCode, options) {
|
|
|
786
874
|
if (name === undefined || !RUNTIME_MODIFIERS.has(name))
|
|
787
875
|
return undefined;
|
|
788
876
|
const { object } = callee;
|
|
877
|
+
// Mocha's `this.skip()`, in a test or hook whose `function` gives it the test's `this`.
|
|
878
|
+
if (object.type === 'ThisExpression') {
|
|
879
|
+
const owner = name === 'skip' ? thisFunction(call) : undefined;
|
|
880
|
+
return owner !== undefined && thisBodies.has(owner) ? runtimeSubject(call, name) : undefined;
|
|
881
|
+
}
|
|
789
882
|
if (object.type === 'Identifier') {
|
|
790
883
|
const info = innermost(object.name);
|
|
791
884
|
return info?.methods?.has(name) ? runtimeSubject(call, name, info.step) : undefined;
|
|
@@ -8,7 +8,7 @@ export type BuiltinStateName = (typeof BUILTIN_STATE_NAMES)[number];
|
|
|
8
8
|
export declare const TAG_STATE_NAMES: readonly ["new", "quarantined"];
|
|
9
9
|
export type TagStateName = (typeof TAG_STATE_NAMES)[number];
|
|
10
10
|
/** The test frameworks the rules understand. */
|
|
11
|
-
export declare const FRAMEWORK_NAMES: readonly ["playwright", "jest", "vitest"];
|
|
11
|
+
export declare const FRAMEWORK_NAMES: readonly ["playwright", "jest", "vitest", "mocha", "cypress", "webdriverio"];
|
|
12
12
|
export type FrameworkName = (typeof FRAMEWORK_NAMES)[number];
|
|
13
13
|
export declare const TICKET_PRESETS: readonly ["any", "jira", "github", "gitlab", "linear", "azure-devops", "numeric", "pattern"];
|
|
14
14
|
/** Comment keywords that must reference a ticket by default (`// TODO: TRADE-123`). */
|
package/dist/utils/constants.js
CHANGED
|
@@ -6,7 +6,7 @@ export const BUILTIN_STATE_NAMES = ['skip', 'fixme', 'fail', 'slow', 'todo', 'ne
|
|
|
6
6
|
/** The built-in states a tag puts a test in, so the only ones whose tag can be renamed. */
|
|
7
7
|
export const TAG_STATE_NAMES = ['new', 'quarantined'];
|
|
8
8
|
/** The test frameworks the rules understand. */
|
|
9
|
-
export const FRAMEWORK_NAMES = ['playwright', 'jest', 'vitest'];
|
|
9
|
+
export const FRAMEWORK_NAMES = ['playwright', 'jest', 'vitest', 'mocha', 'cypress', 'webdriverio'];
|
|
10
10
|
export const TICKET_PRESETS = [
|
|
11
11
|
'any',
|
|
12
12
|
'jira',
|
|
@@ -62,6 +62,10 @@ export interface Framework {
|
|
|
62
62
|
* `context.skip('why')`. In Playwright the first argument is always a condition.
|
|
63
63
|
*/
|
|
64
64
|
skipNotes: boolean;
|
|
65
|
+
/** Whether a test or hook written as a `function` can skip at runtime with `this.skip()`, as in Mocha. */
|
|
66
|
+
thisSkip: boolean;
|
|
67
|
+
/** Whether a test with only a title written out is a todo, as Mocha's pending `it('title')`. */
|
|
68
|
+
todoWithoutBody: boolean;
|
|
65
69
|
/** Built-in states a test can be put in with this framework. */
|
|
66
70
|
states: ReadonlySet<BuiltinStateName>;
|
|
67
71
|
/** Playwright's own API on `test`: `test.describe`, `test.step`, `test.info()` and runtime `test.skip()`. */
|
package/dist/utils/frameworks.js
CHANGED
|
@@ -32,6 +32,8 @@ const playwright = {
|
|
|
32
32
|
info: { param: 1, methods: RUNTIME_MODIFIERS },
|
|
33
33
|
skipWithoutBody: false,
|
|
34
34
|
skipNotes: false,
|
|
35
|
+
thisSkip: false,
|
|
36
|
+
todoWithoutBody: false,
|
|
35
37
|
states: new Set(['skip', 'fixme', 'fail', 'slow', 'new', 'quarantined']),
|
|
36
38
|
playwright: true,
|
|
37
39
|
};
|
|
@@ -63,6 +65,8 @@ const jest = {
|
|
|
63
65
|
plainTags: false,
|
|
64
66
|
skipWithoutBody: false,
|
|
65
67
|
skipNotes: false,
|
|
68
|
+
thisSkip: false,
|
|
69
|
+
todoWithoutBody: false,
|
|
66
70
|
states: new Set(['skip', 'todo', 'fail', 'new', 'quarantined']),
|
|
67
71
|
playwright: false,
|
|
68
72
|
};
|
|
@@ -103,10 +107,58 @@ const vitest = {
|
|
|
103
107
|
info: { param: 0, methods: new Set(['skip']) },
|
|
104
108
|
skipWithoutBody: true,
|
|
105
109
|
skipNotes: true,
|
|
110
|
+
thisSkip: false,
|
|
111
|
+
todoWithoutBody: false,
|
|
106
112
|
states: new Set(['skip', 'todo', 'fail', 'new', 'quarantined']),
|
|
107
113
|
playwright: false,
|
|
108
114
|
};
|
|
109
|
-
|
|
115
|
+
const SKIPPED_TEST = { kind: 'test', modifiers: ['skip'] };
|
|
116
|
+
const SKIPPED_DESCRIBE = { kind: 'describe', modifiers: ['skip'] };
|
|
117
|
+
/** What the `mocha` module exports, the BDD and TDD names alike. `context` and `specify` are only globals. */
|
|
118
|
+
const MOCHA_EXPORTS = new Map([
|
|
119
|
+
['it', TEST],
|
|
120
|
+
['test', TEST],
|
|
121
|
+
['xit', SKIPPED_TEST],
|
|
122
|
+
['describe', DESCRIBE],
|
|
123
|
+
['suite', DESCRIBE],
|
|
124
|
+
['xdescribe', SKIPPED_DESCRIBE],
|
|
125
|
+
...['before', 'beforeEach', 'after', 'afterEach', 'suiteSetup', 'setup', 'suiteTeardown', 'teardown'].map((name) => [name, HOOK]),
|
|
126
|
+
]);
|
|
127
|
+
/** Mocha, and Cypress and WebdriverIO, which run their tests on Mocha. */
|
|
128
|
+
const mocha = {
|
|
129
|
+
ownsModule: (source) => source === 'mocha',
|
|
130
|
+
exports: MOCHA_EXPORTS,
|
|
131
|
+
globals: new Map([
|
|
132
|
+
...[...MOCHA_EXPORTS].filter(([name]) => name !== 'it' && name !== 'test'),
|
|
133
|
+
['context', DESCRIBE],
|
|
134
|
+
['xcontext', SKIPPED_DESCRIBE],
|
|
135
|
+
['xspecify', SKIPPED_TEST],
|
|
136
|
+
]),
|
|
137
|
+
testFunctions: ['it', 'specify', 'test'],
|
|
138
|
+
testMembers: new Set(['only', 'skip']),
|
|
139
|
+
describeMembers: new Set(['only', 'skip']),
|
|
140
|
+
factories: new Set(),
|
|
141
|
+
effects: effects([['skip', 'skip']]),
|
|
142
|
+
// Cypress's `{ browser: '!webkit' }` runs the test only in some browsers.
|
|
143
|
+
optionEffects: new Map([['browser', 'skip']]),
|
|
144
|
+
// @cypress/grep's `{ tags: '@smoke' }`, written with the `@`.
|
|
145
|
+
tagKey: 'tags',
|
|
146
|
+
plainTags: false,
|
|
147
|
+
skipWithoutBody: true,
|
|
148
|
+
skipNotes: false,
|
|
149
|
+
thisSkip: true,
|
|
150
|
+
todoWithoutBody: true,
|
|
151
|
+
states: new Set(['skip', 'todo', 'new', 'quarantined']),
|
|
152
|
+
playwright: false,
|
|
153
|
+
};
|
|
154
|
+
export const FRAMEWORKS = {
|
|
155
|
+
playwright,
|
|
156
|
+
jest,
|
|
157
|
+
vitest,
|
|
158
|
+
mocha: { name: 'mocha', ...mocha },
|
|
159
|
+
cypress: { name: 'cypress', ...mocha },
|
|
160
|
+
webdriverio: { name: 'webdriverio', ...mocha },
|
|
161
|
+
};
|
|
110
162
|
/** Whether `source` belongs to a test runner other than `framework`, so its names are never tests here. */
|
|
111
163
|
export function isOtherRunner(framework, source) {
|
|
112
164
|
return (isPlaywrightModule(source) || OTHER_RUNNERS.has(source)) && !framework.ownsModule(source);
|
package/dist/utils/options.d.ts
CHANGED
|
@@ -20,14 +20,14 @@ export interface CustomState {
|
|
|
20
20
|
/** Options shared by every rule. Set them once in `settings['test-governance']`. */
|
|
21
21
|
export interface GovernanceOptions {
|
|
22
22
|
/**
|
|
23
|
-
* The test framework of the files the config applies to: `'playwright'`, `'jest'
|
|
24
|
-
* `
|
|
23
|
+
* The test framework of the files the config applies to: `'playwright'`, `'jest'`, `'vitest'`, `'mocha'`,
|
|
24
|
+
* `'cypress'` or `'webdriverio'`. Required; each framework's config, such as `configs.mocha`, sets it.
|
|
25
25
|
*/
|
|
26
26
|
framework: FrameworkName;
|
|
27
27
|
/**
|
|
28
|
-
* Names that are treated as the framework's `test` function. Defaults to `['test']` for Playwright
|
|
29
|
-
* `['test', 'it']` for Jest and Vitest
|
|
30
|
-
* you still need them.
|
|
28
|
+
* Names that are treated as the framework's `test` function. Defaults to `['test']` for Playwright,
|
|
29
|
+
* `['test', 'it']` for Jest and Vitest, and `['it', 'specify', 'test']` for Mocha, Cypress and
|
|
30
|
+
* WebdriverIO. Your list replaces the default one, so include those names if you still need them.
|
|
31
31
|
*/
|
|
32
32
|
testFunctions?: string[];
|
|
33
33
|
/** Accepted ticket formats. Defaults to `{ preset: 'any' }`. */
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
📝 Require the marker keyword to match the test's state.
|
|
4
4
|
|
|
5
|
-
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🃏 `jest`, 🎭 `playwright`, ⚡ `vitest`.
|
|
5
|
+
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🃏 `jest`, ☕ `mocha`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
7
7
|
💡 This rule is manually fixable by [editor suggestions](https://eslint.org/docs/latest/use/core-concepts#rule-suggestions).
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
📝 Disallow state tags that contradict each other, differ in case or look like typos.
|
|
4
4
|
|
|
5
|
-
⚠️ This rule _warns_ in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🃏 `jest`, 🎭 `playwright`, ⚡ `vitest`.
|
|
5
|
+
⚠️ This rule _warns_ in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🃏 `jest`, ☕ `mocha`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
7
7
|
🔧 This rule is automatically fixable by the [`--fix` CLI option](https://eslint.org/docs/latest/user-guide/command-line-interface#--fix).
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
📝 Disallow marker comments that no longer match a test state.
|
|
4
4
|
|
|
5
|
-
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🃏 `jest`, 🎭 `playwright`, ⚡ `vitest`.
|
|
5
|
+
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🃏 `jest`, ☕ `mocha`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
7
7
|
💡 This rule is manually fixable by [editor suggestions](https://eslint.org/docs/latest/use/core-concepts#rule-suggestions).
|
|
8
8
|
|
|
@@ -26,9 +26,9 @@ A `FIXME:` or `TODO:` comment that starts with a valid ticket is treated by wher
|
|
|
26
26
|
leftover marker and is reported, since `// FIXME: TRADE-12` above a test usually means the test was
|
|
27
27
|
fixed and the marker forgotten. Move a work comment into the test body, or onto the line of code it
|
|
28
28
|
is about.
|
|
29
|
-
- Above a
|
|
29
|
+
- Above a test that is not a todo, in every framework but Playwright, a `TODO:` comment is a tracked work comment about
|
|
30
30
|
the test and is allowed. `TODO` is the everyday keyword for such comments, so there it is far more
|
|
31
|
-
often a note than the forgotten marker of a
|
|
31
|
+
often a note than the forgotten marker of a todo test.
|
|
32
32
|
|
|
33
33
|
Other markers that read as a sentence, such as `// SKIP: this one is flaky`, are left alone too: the rule
|
|
34
34
|
counts a comment as prose when its first word is not a valid ticket and more text follows.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
📝 Require FIXME and TODO comments in test files to start with a ticket.
|
|
4
4
|
|
|
5
|
-
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🃏 `jest`, 🎭 `playwright`, ⚡ `vitest`.
|
|
5
|
+
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🃏 `jest`, ☕ `mocha`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
7
7
|
💡 This rule is manually fixable by [editor suggestions](https://eslint.org/docs/latest/use/core-concepts#rule-suggestions).
|
|
8
8
|
|
|
@@ -30,8 +30,8 @@ How comments are read:
|
|
|
30
30
|
style many teams already use. Anything after the parentheses is a note, as it would be after `TODO: TRADE-512`.
|
|
31
31
|
- The tickets follow the same rules as markers: the shared ticket format, placeholders such as `TBD`
|
|
32
32
|
rejected, and no text after the tickets unless `allowNotes` is set.
|
|
33
|
-
- A `// FIXME:` marker directly above a `test.fixme(...)`, or a `// TODO:` marker above a
|
|
34
|
-
|
|
33
|
+
- A `// FIXME:` marker directly above a `test.fixme(...)`, or a `// TODO:` marker above a todo test,
|
|
34
|
+
such as Jest's `test.todo(...)` or Mocha's `it('title')`, is checked by [`require-ticket`](require-ticket.md) instead, so it is
|
|
35
35
|
reported once.
|
|
36
36
|
- Only test files (a test framework import or a call to a test function) are checked.
|
|
37
37
|
Application code is left alone, even if your config applies the plugin to it.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
📝 Require a ticket marker comment above skipped, fixme and tagged tests.
|
|
4
4
|
|
|
5
|
-
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🃏 `jest`, 🎭 `playwright`, ⚡ `vitest`.
|
|
5
|
+
💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🃏 `jest`, ☕ `mocha`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
|
|
6
6
|
|
|
7
7
|
💡 This rule is manually fixable by [editor suggestions](https://eslint.org/docs/latest/use/core-concepts#rule-suggestions).
|
|
8
8
|
|
|
@@ -17,19 +17,21 @@ A test is in a state when it is:
|
|
|
17
17
|
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | -------------------------- |
|
|
18
18
|
| `skip` | `test.skip('title', fn)`, `test.describe.skip`, `test.skip()` in a body, `testInfo.skip()`, `test.step.skip`, `step.skip()` in a step | `// SKIP:` | yes |
|
|
19
19
|
| `fixme` | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | `// FIXME:` | yes |
|
|
20
|
-
| `todo` | Jest's and Vitest's `test.todo`
|
|
20
|
+
| `todo` | Jest's and Vitest's `test.todo`, and a Mocha test with only a title, `it('title')` | `// TODO:` | yes, except in Playwright |
|
|
21
21
|
| `new` | the `@new` tag | `// NEW:` | with `lifecycleTags: true` |
|
|
22
22
|
| `quarantined` | the `@quarantined` tag | `// QUARANTINED:` | with `lifecycleTags: true` |
|
|
23
23
|
| `fail`, `slow` | `test.fail(...)`, `test.slow()`, `testInfo.fail()`, `testInfo.slow()` | `// FAIL:`, `// SLOW:` | no |
|
|
24
24
|
| custom | any tag you list in `customStates` | your keyword | when configured |
|
|
25
25
|
|
|
26
26
|
In Jest and Vitest, `xit`, `xdescribe`, Vitest's `{ skip: true }`, `skipIf(true)`, `context.skip()` and
|
|
27
|
-
`context.skip('note')`, in a test or in `beforeEach`, skip a test too, and `test.failing` and `test.fails` put it in the `fail` state.
|
|
28
|
-
|
|
27
|
+
`context.skip('note')`, in a test or in `beforeEach`, skip a test too, and `test.failing` and `test.fails` put it in the `fail` state. In Mocha, Cypress and WebdriverIO, `xit`, `xdescribe` and
|
|
28
|
+
`this.skip()` in a test or hook skip a test too, and Cypress's `{ browser: '!webkit' }` skips it in other
|
|
29
|
+
browsers, which counts as a conditional skip. The [README](../../README.md#frameworks) lists every form.
|
|
29
30
|
|
|
30
31
|
Tags are read from the title (`'places a limit order @new'`) and from `{ tag: '@new' }` or `{ tag: ['@new', '@smoke'] }`,
|
|
31
32
|
also when they come from a `const` or an enum declared in the same file, as in `{ tag: TAGS.NEW }`.
|
|
32
|
-
Vitest writes tag names without the `@`, so `{ tags: ['new'] }` is the `@new` tag.
|
|
33
|
+
Vitest writes tag names without the `@`, so `{ tags: ['new'] }` is the `@new` tag. Cypress tags from
|
|
34
|
+
@cypress/grep keep it: `{ tags: '@new' }`. Tags on a
|
|
33
35
|
`describe` apply to every test inside it. `states` can rename the tag and marker of `new` and
|
|
34
36
|
`quarantined`, as in `states: { quarantined: { tag: '@flaky', marker: 'FLAKY' } }`.
|
|
35
37
|
|
|
@@ -77,8 +79,28 @@ Conditional skips such as `test.skip(browserName === 'webkit', 'Not supported')`
|
|
|
77
79
|
permanent platform limit, so they need no ticket unless you set `requireTicketForConditional: true`.
|
|
78
80
|
A runtime skip under an `if`, a `switch` case, a ternary or `&&` counts as conditional too, since
|
|
79
81
|
`if (!enabled) test.skip()` does the same as `test.skip(!enabled)`. So does a skip in a `catch` block,
|
|
80
|
-
such as `catch { test.skip(true, 'Market data feed is down') }`,
|
|
81
|
-
|
|
82
|
+
such as `catch { test.skip(true, 'Market data feed is down') }`, a skip after other code in a `try`
|
|
83
|
+
block that has a `catch`, as in `try { await stat(target); test.skip(); } catch {}`, where an error
|
|
84
|
+
skips past it, and a skip after an early exit such as `if (ready) return;` or `if (!ok) throw error;`
|
|
85
|
+
earlier in the same function.
|
|
86
|
+
|
|
87
|
+
A test or describe whose function a condition picks is a conditional skip too, in every framework. Its
|
|
88
|
+
title and tags count as usual, and a marker goes above it, or in code without semicolons above the
|
|
89
|
+
`;` that starts the line.
|
|
90
|
+
|
|
91
|
+
<!-- example: valid -->
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
(isMobile ? test.skip : test)('places an order', async ({ page }) => {});
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
<!-- example: invalid settings={"framework":"jest","requireTicketForConditional":true} -->
|
|
98
|
+
|
|
99
|
+
```js
|
|
100
|
+
(process.env.CI ? describe.skip : describe)('order book', () => {
|
|
101
|
+
it('shows the best bid', () => {});
|
|
102
|
+
});
|
|
103
|
+
```
|
|
82
104
|
|
|
83
105
|
If one test or describe has several markers for the same state, each must be valid: in
|
|
84
106
|
`// SKIP: TRADE-1` followed by `// SKIP: nope`, the second line is reported.
|
|
@@ -173,6 +195,20 @@ it.todo('caps position size at 2% of equity');
|
|
|
173
195
|
test.skipIf(true)('fills a stop order when the market opens', () => {});
|
|
174
196
|
```
|
|
175
197
|
|
|
198
|
+
<!-- example: invalid settings={"framework":"mocha"} -->
|
|
199
|
+
|
|
200
|
+
```js
|
|
201
|
+
it('settles a trade at T+1', function () {
|
|
202
|
+
this.skip();
|
|
203
|
+
});
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
<!-- example: invalid settings={"framework":"cypress","lifecycleTags":true} -->
|
|
207
|
+
|
|
208
|
+
```js
|
|
209
|
+
it('places a stop order', { tags: '@quarantined' }, () => {});
|
|
210
|
+
```
|
|
211
|
+
|
|
176
212
|
Correct:
|
|
177
213
|
|
|
178
214
|
<!-- example: valid -->
|
|
@@ -227,6 +263,19 @@ describe('risk limits', () => {
|
|
|
227
263
|
});
|
|
228
264
|
```
|
|
229
265
|
|
|
266
|
+
<!-- example: valid settings={"framework":"mocha"} -->
|
|
267
|
+
|
|
268
|
+
```js
|
|
269
|
+
describe('settlement', function () {
|
|
270
|
+
// TODO: TRADE-91
|
|
271
|
+
it('settles a trade on a market holiday');
|
|
272
|
+
|
|
273
|
+
it('settles in the clearing house currency', function () {
|
|
274
|
+
if (!process.env.CLEARING_URL) this.skip();
|
|
275
|
+
});
|
|
276
|
+
});
|
|
277
|
+
```
|
|
278
|
+
|
|
230
279
|
## Options
|
|
231
280
|
|
|
232
281
|
This rule has no options of its own. It reads the shared options described in the
|
package/docs/upgrading.md
CHANGED
|
@@ -90,7 +90,7 @@ becomes `extends: ['test-governance/playwright']`.
|
|
|
90
90
|
A config that still reads `configs.recommended` fails when it loads:
|
|
91
91
|
|
|
92
92
|
```text
|
|
93
|
-
eslint-plugin-test-governance: configs.recommended was removed in 0.5.0. Use the config for your framework: configs.playwright, configs.jest or configs.
|
|
93
|
+
eslint-plugin-test-governance: configs.recommended was removed in 0.5.0. Use the config for your framework: configs.playwright, configs.jest, configs.vitest, configs.mocha, configs.cypress or configs.webdriverio. See https://github.com/Huldoser/eslint-plugin-test-governance/blob/main/docs/upgrading.md
|
|
94
94
|
```
|
|
95
95
|
|
|
96
96
|
### The framework option
|
|
@@ -121,7 +121,7 @@ Without it, the config fails when it loads:
|
|
|
121
121
|
|
|
122
122
|
```text
|
|
123
123
|
eslint-plugin-test-governance: invalid options:
|
|
124
|
-
- framework is required: set it to "playwright", "jest" or "
|
|
124
|
+
- framework is required: set it to "playwright", "jest", "vitest", "mocha", "cypress" or "webdriverio", or use configs.playwright, configs.jest, configs.vitest, configs.mocha, configs.cypress or configs.webdriverio
|
|
125
125
|
```
|
|
126
126
|
|
|
127
127
|
If you write `settings['test-governance']` yourself instead of calling `configure()`, add `framework`
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "eslint-plugin-test-governance",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "ESLint rules that require a ticket comment above skipped, fixme, todo and tagged Playwright, Jest and
|
|
3
|
+
"version": "0.7.0",
|
|
4
|
+
"description": "ESLint rules that require a ticket comment above skipped, fixme, todo and tagged Playwright, Jest, Vitest, Mocha, Cypress and WebdriverIO tests.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"eslint",
|
|
7
7
|
"eslintplugin",
|
|
@@ -9,6 +9,9 @@
|
|
|
9
9
|
"playwright",
|
|
10
10
|
"jest",
|
|
11
11
|
"vitest",
|
|
12
|
+
"mocha",
|
|
13
|
+
"cypress",
|
|
14
|
+
"webdriverio",
|
|
12
15
|
"testing"
|
|
13
16
|
],
|
|
14
17
|
"homepage": "https://github.com/Huldoser/eslint-plugin-test-governance#readme",
|