eslint-plugin-test-governance 0.12.1 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -6,6 +6,20 @@ All notable changes to this project are documented here. The format follows
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.13.0] - 2026-11-05
10
+
11
+ ### Added
12
+
13
+ - `requireReasonForConditional`, off by default: a conditional skip, such as `test.skip(isMobile)` or
14
+ `if (isWeekend) this.skip()`, needs a reason but no ticket. A comment above it or at the end of its line, above
15
+ the `if` around it, or above its hook, step or test counts, and so does the reason the call takes where the runner
16
+ has one, as Playwright's `test.skip(condition, 'why')`, Vitest's `context.skip(condition, 'why')`, node:test's
17
+ `t.skip('why')` and `{ skip: isWindows && 'why' }`, and Jasmine's `pending('why')`. `require-ticket` reports the
18
+ skips that give none.
19
+ - `knownTags`, such as `['@smoke', '@regression']`: `no-conflicting-states` gives these tags the case and typo
20
+ checks state tags get, so `@Smoke` is fixed to `@smoke` and `@smok` is reported. A listed tag is never reported as
21
+ a typo of a state tag.
22
+
9
23
  ## [0.12.1] - 2026-11-04
10
24
 
11
25
  ### Fixed
@@ -493,7 +507,8 @@ First release.
493
507
  - Support for ESLint 9 and 10 flat config, JavaScript and TypeScript test files, `.extend()`
494
508
  fixtures, describe blocks, runtime skips and tags in titles or the `tag` option.
495
509
 
496
- [Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.12.1...HEAD
510
+ [Unreleased]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.13.0...HEAD
511
+ [0.13.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.12.1...v0.13.0
497
512
  [0.12.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.12.0...v0.12.1
498
513
  [0.12.0]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.11.1...v0.12.0
499
514
  [0.11.1]: https://github.com/Huldoser/eslint-plugin-test-governance/compare/v0.11.0...v0.11.1
package/README.md CHANGED
@@ -20,6 +20,10 @@ Without the comment, `test-governance/require-ticket` fails the lint. When the t
20
20
  skip removed, `no-orphaned-marker` flags the leftover comment. The markers are plain
21
21
  comments, so `grep -rn "SKIP:" tests/` lists every disabled test and its ticket.
22
22
 
23
+ The same check covers tests commented out by hand, `FIXME` and `TODO` comments in test files, and, once
24
+ turned on, retried tests. The ticket can also go in the description of an `eslint-disable` comment, or in
25
+ a Playwright annotation or Vitest `meta`, which reach the test report.
26
+
23
27
  > **Status:** early release (0.x). Options may still change before 1.0. Coming from 0.5?
24
28
  > [Upgrading to 0.6](docs/upgrading.md#upgrading-to-06) shows what to change.
25
29
 
@@ -148,8 +152,9 @@ turn on the same rules. What puts a test in a state in each framework:
148
152
  | `fixme` (`// FIXME:`) | `test.fixme(...)`, `test.describe.fixme`, `testInfo.fixme()` | | | | | | | |
149
153
  | `fail` (`// FAIL:`), off by default | `test.fail(...)`, `testInfo.fail()` | `test.failing` | `test.fails`, `{ fails: true }` | | | | `test.expectFailure`, `expectFailure()`, `{ expectFailure }` | |
150
154
  | `slow` (`// SLOW:`), off by default | `test.slow()`, `testInfo.slow()` | | | | | | | |
155
+ | `retry` (`// RETRY:`), off by default | `test.describe.configure({ retries })` | `jest.retryTimes()` | `{ retry }` | `this.retries()` in a describe, test or hook written as a `function` | as Mocha, and `{ retries }` | as Mocha | | |
151
156
  | conditional skip, no ticket by default | `test.skip(condition)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)` | `(condition ? it.skip : it)(...)` | `skipIf(condition)`, `runIf(condition)`, `{ skip: condition }`, `context.skip(condition)`, `if (...) context.skip()`, `(condition ? it.skip : it)(...)` | `if (...) this.skip()`, `(condition ? it.skip : it)(...)` | as Mocha, and `{ browser: '!webkit' }` | as Mocha | `{ skip: condition }`, `{ todo: condition }`, `if (...) t.skip()`, `(condition ? it.skip : it)(...)` | `if (...) pending()`, `(condition ? xit : it)(...)` |
152
- | tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` | the title | the title, `{ tags }` from @cypress/grep | the title | the title | the title |
157
+ | tags such as `@new` (`// NEW:`) | the title, `{ tag }` | the title | the title, `{ tags }`, where `'new'` and `'@new'` both mean `@new` | the title | the title, `{ tags }` from @cypress/grep | the title | the title, `{ tags }`, as in Vitest | the title |
153
158
 
154
159
  Table-driven tests count too: `test.skip.each(table)(...)`, `describe.skip.each` and Vitest's `.for`.
155
160
  A test or describe whose function a condition picks, as in `(isMobile ? test.skip : test)('title', fn)`,
@@ -200,7 +205,7 @@ condition. A function or variable of your own called `pending` is not Jasmine's.
200
205
  | [marker-matches-state](docs/rules/marker-matches-state.md) | Require the marker keyword to match the test's state | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
201
206
  | [no-conflicting-states](docs/rules/no-conflicting-states.md) | Disallow state tags that contradict each other, differ in case or look like typos | | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | 🔧 | |
202
207
  | [no-orphaned-marker](docs/rules/no-orphaned-marker.md) | Disallow marker comments that no longer match a test state | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
203
- | [require-ticket](docs/rules/require-ticket.md) | Require a ticket marker comment above skipped, fixme and tagged tests | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
208
+ | [require-ticket](docs/rules/require-ticket.md) | Require a ticket for skipped, fixme, todo, retried and tagged tests | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
204
209
  | [require-ticket-for-commented-out-tests](docs/rules/require-ticket-for-commented-out-tests.md) | Require a ticket marker on commented-out tests | | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | 💡 |
205
210
  | [require-ticket-in-comments](docs/rules/require-ticket-in-comments.md) | Require FIXME and TODO comments in test files to start with a ticket | 🌲 🌼 🃏 ☕ 🟢 🎭 ⚡ 🤖 | | | 💡 |
206
211
 
@@ -214,7 +219,7 @@ The defaults follow how most teams already work, so each framework config is use
214
219
  | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
215
220
  | States that need a ticket | `skip`, `fixme`; in the other frameworks `skip`, `todo` | Every team skips tests. Chromium DevTools, GitLab and Slack all require a bug link on disabled tests. |
216
221
  | `@new`, `@quarantined` | off; `lifecycleTags: true` turns both on | A promotion workflow is not universal. Teams that say `@flaky` instead [rename the tag](#tag-names). |
217
- | Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true` changes this. |
222
+ | Conditional skips (`test.skip(browserName === 'webkit', ...)`, `if (...) test.skip()`, a skip in `catch`) | no ticket needed | These are usually permanent platform limits, not bugs. `requireTicketForConditional: true` asks for a ticket, `requireReasonForConditional: true` for a reason. |
218
223
  | `test.fail`, `test.slow`, Jest's `test.failing`, Vitest's `test.fails` | off | They don't remove coverage. Turn them on with `states: { fail: true, slow: true }`. |
219
224
  | Retries: `describe.configure({ retries })`, `jest.retryTimes()`, `retry`, `this.retries()` | off | Retries hide flaky tests. `states: { retry: true }` needs a ticket wherever tests retry, even only on CI. |
220
225
  | Marker format | `// SKIP: TRADE-123` or `// SKIP: TRADE-123, TRADE-124` | Uppercase keyword and colon, like `TODO:`. Easy to grep. |
@@ -241,7 +246,9 @@ and markers. To turn a rule off, set it to `'off'` as usual.
241
246
  | `lifecycleTags` | `boolean` | `false` | Turns on the `new` (`@new`) and `quarantined` (`@quarantined`) states. |
242
247
  | `states` | `{ [state]: boolean \| { enabled?, tag?, marker?, ticket? } }` | `skip`, `fixme` and `todo` on | Turns built-in states (`skip`, `fixme`, `fail`, `slow`, `todo`, `retry`, `new`, `quarantined`) on or off, renames their tag or marker, or gives them their own ticket format. `fail`, `slow` and `retry` are off by default; a setting for `new` or `quarantined` here wins over `lifecycleTags`. Only `new` and `quarantined` have a tag to rename; see [Tag names](#tag-names). `fixme` and `slow` exist only in Playwright, `todo` in every framework but Playwright, `fail` in Playwright, Jest, Vitest and node:test, and `retry` in every framework but node:test and Jasmine, so turning one on elsewhere has no effect. |
243
248
  | `customStates` | `{ [name]: { when, marker, ticket? } }` | `{}` | Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. `when` is a tag and `marker` an uppercase keyword; neither can be shared with another state. |
249
+ | `knownTags` | `string[]` | `[]` | Your team's other tags, such as `['@smoke', '@regression']`, written with the `@`. `no-conflicting-states` reports them in the wrong case or misspelled, as it does state tags, and never reports a listed tag as a typo of a state tag. |
244
250
  | `requireTicketForConditional` | `boolean` | `false` | Require a ticket for conditional skips too: `test.skip(condition, ...)`, `if (...) test.skip()`, `(condition ? test.skip : test)(...)`, a skip in a `catch` block, after other code in a `try` block with a `catch`, or after an early `return` or `throw`. |
251
+ | `requireReasonForConditional` | `boolean` | `false` | Require a reason, but no ticket, for conditional skips: a comment above the skip or at the end of its line, above the `if` around it or above its hook, step or test, or the reason the call takes, as `test.skip(condition, 'why')`. See [require-ticket](docs/rules/require-ticket.md#reasons-for-conditional-skips). |
245
252
  | `allowBlankLine` | `boolean` | `false` | Let blank lines separate a marker from its test. |
246
253
  | `allowNotes` | `boolean` | `false` | Allow free text after the tickets, e.g. `// SKIP: TRADE-123 flaky on CI`. Applies to markers and to `FIXME` / `TODO` comments. |
247
254
  | `comments` | `false \| { keywords?: string[] }` | `{ keywords: ['FIXME', 'TODO'] }` | Comment keywords that must start with a ticket anywhere in a test file, e.g. `// TODO: TRADE-123`. Your keywords replace the default ones. `false` allows free-form comments. |
@@ -471,10 +478,9 @@ The [Frameworks](#frameworks) table lists what each framework's tests can do. In
471
478
  Not covered in this version: tests generated in loops with dynamic titles, and tags or details imported
472
479
  from other modules (see `reportUnreadableTags`), fixtures imported from other modules under a name not in
473
480
  `testFunctions`, a Vitest test with only a title, which Vitest runs as a todo but which looks the same as a
474
- helper called `test`, Playwright annotations as markers, Cypress tests skipped with @cypress/skip-test's
475
- `skipOn()` and `onlyOn()`, `node:test`'s experimental `tags` option and `getTestContext()`, Jasmine's
476
- `pending()` in a helper function that tests call, Jasmine 5 and 6's `require('jasmine-core').noGlobals()`,
477
- and other runners such as Bun.
481
+ helper called `test`, Cypress tests skipped with @cypress/skip-test's `skipOn()` and `onlyOn()`, `node:test`'s
482
+ `getTestContext()`, Jasmine's `pending()` in a helper function that tests call, Jasmine 5 and 6's
483
+ `require('jasmine-core').noGlobals()`, and other runners such as Bun.
478
484
 
479
485
  ## TypeScript
480
486
 
@@ -20,7 +20,8 @@ export default createRule({
20
20
  },
21
21
  },
22
22
  create(context, analysis, options) {
23
- const tagStates = options.states.filter(isTagState);
23
+ // State tags first, so a tag near both a state tag and a known tag is read as the state's.
24
+ const checkedTags = [...options.states.filter(isTagState).map((s) => s.tag), ...options.knownTags];
24
25
  const newTag = options.states.find((s) => s.name === 'new')?.tag;
25
26
  const quarantinedTag = options.states.find((s) => s.name === 'quarantined')?.tag;
26
27
  for (const subject of analysis.subjects) {
@@ -53,23 +54,23 @@ export default createRule({
53
54
  const { tag, range } = occurrence;
54
55
  // A Vitest tag name is written without its `@`, so show it, and fix it, the same way.
55
56
  const written = (name) => (occurrence.plain ? name.slice(1) : name);
56
- if (tagStates.some((s) => s.tag === tag))
57
+ if (checkedTags.includes(tag))
57
58
  return;
58
59
  const lower = tag.toLowerCase();
59
- const sameLetters = tagStates.find((s) => s.tag.toLowerCase() === lower);
60
+ const sameLetters = checkedTags.find((checked) => checked.toLowerCase() === lower);
60
61
  if (sameLetters) {
61
62
  context.report({
62
63
  node: occurrence.node,
63
64
  messageId: 'tagCase',
64
- data: { found: written(tag), expected: written(sameLetters.tag) },
65
- fix: range && ((fixer) => fixer.replaceTextRange(range, written(sameLetters.tag))),
65
+ data: { found: written(tag), expected: written(sameLetters) },
66
+ fix: range && ((fixer) => fixer.replaceTextRange(range, written(sameLetters))),
66
67
  });
67
68
  return;
68
69
  }
69
70
  // A typo changes, adds, drops or swaps a letter or two. A tag that is two or more letters
70
71
  // shorter or longer, like `@unquarantined` next to `@quarantined`, is a different word.
71
- const near = tagStates.find((s) => {
72
- const expected = s.tag.toLowerCase();
72
+ const near = checkedTags.find((checked) => {
73
+ const expected = checked.toLowerCase();
73
74
  return (Math.abs(lower.length - expected.length) < 2 &&
74
75
  editDistance(lower, expected) <= typoThreshold(expected.length - 1));
75
76
  });
@@ -77,7 +78,7 @@ export default createRule({
77
78
  context.report({
78
79
  node: occurrence.node,
79
80
  messageId: 'tagTypo',
80
- data: { found: written(tag), expected: written(near.tag) },
81
+ data: { found: written(tag), expected: written(near) },
81
82
  });
82
83
  }
83
84
  }
@@ -1,2 +1,2 @@
1
- declare const _default: import("../utils/create-rule.ts").Rule<"missingMarker" | "missingTicket" | "invalidTicket" | "placeholderTicket" | "extraText" | "removeExtraText" | "markerCase" | "dynamicTitle" | "dynamicTags" | "invalidAnnotation" | "useAnnotation" | "missingAnnotation" | "moveToAnnotation" | "unreadAnnotation">;
1
+ declare const _default: import("../utils/create-rule.ts").Rule<"missingMarker" | "missingTicket" | "invalidTicket" | "placeholderTicket" | "extraText" | "removeExtraText" | "missingReason" | "markerCase" | "dynamicTitle" | "dynamicTags" | "invalidAnnotation" | "useAnnotation" | "missingAnnotation" | "moveToAnnotation" | "unreadAnnotation">;
2
2
  export default _default;
@@ -5,7 +5,7 @@ export default createRule({
5
5
  meta: {
6
6
  type: 'problem',
7
7
  docs: {
8
- description: 'Require a ticket marker comment above skipped, fixme and tagged tests',
8
+ description: 'Require a ticket for skipped, fixme, todo, retried and tagged tests',
9
9
  recommended: 'error',
10
10
  url: 'https://github.com/Huldoser/eslint-plugin-test-governance/blob/main/docs/rules/require-ticket.md',
11
11
  },
@@ -13,6 +13,7 @@ export default createRule({
13
13
  schema: [],
14
14
  messages: {
15
15
  missingMarker: "{{subject}} is in the '{{state}}' state but has no `// {{marker}}: <ticket>` comment directly above it, e.g. `// {{marker}}: {{example}}`.",
16
+ missingReason: "{{subject}} is in the '{{state}}' state only under a condition and doesn't say why. Write the reason in a comment above it{{argument}}.",
16
17
  missingTicket: '`{{marker}}:` needs a ticket right after the colon, e.g. `// {{marker}}: {{example}}`.',
17
18
  invalidTicket: "'{{ticket}}' is not a valid ticket for the '{{state}}' state. Expected {{expected}}.",
18
19
  placeholderTicket: "'{{ticket}}' is a placeholder, not a real ticket. Link the ticket that tracks this test.",
@@ -65,6 +66,24 @@ export default createRule({
65
66
  data: { subject: describeSubject(subject), state: state.name, type, example },
66
67
  });
67
68
  };
69
+ /** Reports a conditional state that gives no reason and has no ticket, which would say why as well. */
70
+ const reportReason = (subject, state) => {
71
+ if (evaluate(subject, state).kind === 'ok' || hasAnnotationTicket(subject, state, options))
72
+ return;
73
+ const callee = context.sourceCode.getText(subject.node.callee);
74
+ const first = subject.node.arguments.at(0);
75
+ const where = options.framework.reasonArgument;
76
+ // Vitest's `context.skip()` takes either a reason alone or a condition and then the reason.
77
+ const reasonOnly = where === 'only' ||
78
+ (where === 'either' && (first === undefined || first.type === 'Literal' || first.type === 'TemplateLiteral'));
79
+ const example = reasonOnly ? `${callee}('why')` : `${callee}(condition, 'why')`;
80
+ const argument = subject.kind === 'runtime' && where !== undefined ? `, or pass it to the call: \`${example}\`` : '';
81
+ context.report({
82
+ loc: headLoc(subject),
83
+ messageId: 'missingReason',
84
+ data: { subject: describeSubject(subject), state: state.name, argument },
85
+ });
86
+ };
68
87
  for (const subject of analysis.subjects) {
69
88
  const needsTicket = subject.states.some((s) => s.required);
70
89
  if (options.reportUnreadableTags && hasTagStates) {
@@ -82,9 +101,12 @@ export default createRule({
82
101
  }
83
102
  }
84
103
  const annotationOnly = only && (subject.kind === 'test' || subject.kind === 'describe') ? only : undefined;
85
- for (const { state, required } of subject.states) {
86
- if (!required)
104
+ for (const { state, required, reasoned } of subject.states) {
105
+ if (!required) {
106
+ if (options.requireReasonForConditional && reasoned === false)
107
+ reportReason(subject, state);
87
108
  continue;
109
+ }
88
110
  if (annotationOnly) {
89
111
  reportAnnotationOnly(subject, state, annotationOnly);
90
112
  continue;
@@ -56,6 +56,11 @@ interface StateSource {
56
56
  state: StateDef;
57
57
  /** False for conditional skips unless `requireTicketForConditional` is set. */
58
58
  required: boolean;
59
+ /**
60
+ * Set for a conditional state: whether something says why, a comment above it or its test, the reason its
61
+ * call takes, or one its option gives.
62
+ */
63
+ reasoned?: boolean;
59
64
  }
60
65
  export interface Subject {
61
66
  kind: 'test' | 'describe' | 'step' | 'runtime';
@@ -69,6 +74,8 @@ export interface Subject {
69
74
  inherited: Set<string>;
70
75
  /** Names of states added by runtime calls directly inside this test or describe. */
71
76
  runtimeStates: Set<string>;
77
+ /** Whether the comment block above this node holds any text, an `eslint-disable` description included. */
78
+ commented: boolean;
72
79
  /** Every tag written on this node, known or not. */
73
80
  tags: TagOccurrence[];
74
81
  /** Tags this node inherits from enclosing describes. */
@@ -841,6 +841,7 @@ const NO_TAGS = {
841
841
  opaqueDetails: false,
842
842
  annotations: [],
843
843
  unreadAnnotations: [],
844
+ reasonedOptions: new Set(),
844
845
  };
845
846
  function collectTags(sourceCode, call, hasTitle, framework, details) {
846
847
  const source = sourceCode.text;
@@ -987,6 +988,52 @@ function isUnconditional(call, notes) {
987
988
  const always = first === undefined || booleanOf(first) === true || (notes && isStaticText(first));
988
989
  return always && !isGuarded(call);
989
990
  }
991
+ /**
992
+ * Whether a comment says anything: a line with text other than a marker keyword alone, such as `SKIP:`. A
993
+ * directive counts only by its description.
994
+ */
995
+ function hasText(comment) {
996
+ return commentLines(comment).some(({ text }) => /[^\s*]/.test(text) && MARKER_LINE_RE.exec(text)?.[2].trim() !== '');
997
+ }
998
+ /**
999
+ * Whether a reason argument says why: text that isn't blank, or a value known only when the tests run, as
1000
+ * `reason` or `` `${browserName} has no clipboard` ``, which is taken on trust.
1001
+ */
1002
+ function saysWhy(sourceCode, node) {
1003
+ const text = staticText(sourceCode, node);
1004
+ if (text !== undefined)
1005
+ return text.trim() !== '';
1006
+ const value = unwrap(node);
1007
+ return value.type !== 'Literal' && !(value.type === 'Identifier' && value.name === 'undefined');
1008
+ }
1009
+ /** Whether `node` is text that isn't blank. */
1010
+ function isReasonText(sourceCode, node) {
1011
+ return Boolean(staticText(sourceCode, node)?.trim());
1012
+ }
1013
+ /** Whether a runtime call gives the reason for its state where its framework takes one (see `reasonArgument`). */
1014
+ function callReason(sourceCode, call, where) {
1015
+ const first = call.arguments.at(0);
1016
+ const second = call.arguments.at(1);
1017
+ if (where === 'only')
1018
+ return first !== undefined && saysWhy(sourceCode, first);
1019
+ // Vitest's `context.skip(isMobile)` takes a condition, so only text first is a reason.
1020
+ if (where === 'either' && first !== undefined && staticText(sourceCode, first) !== undefined) {
1021
+ return isReasonText(sourceCode, first);
1022
+ }
1023
+ return second !== undefined && saysWhy(sourceCode, second);
1024
+ }
1025
+ /**
1026
+ * Whether a conditional option gives its reason, as node:test's `{ skip: isWindows && 'no symlinks' }` or
1027
+ * `{ skip: isWindows ? 'no symlinks' : false }`, where the text is what Node reports.
1028
+ */
1029
+ function optionReason(sourceCode, node) {
1030
+ const value = unwrap(node);
1031
+ if (value.type === 'LogicalExpression')
1032
+ return value.operator === '&&' && isReasonText(sourceCode, value.right);
1033
+ if (value.type !== 'ConditionalExpression')
1034
+ return false;
1035
+ return isReasonText(sourceCode, value.consequent) || isReasonText(sourceCode, value.alternate);
1036
+ }
990
1037
  const FUNCTIONS = new Set(['FunctionDeclaration', 'FunctionExpression', 'ArrowFunctionExpression']);
991
1038
  /** Whether a statement always leaves the function or block: `return`, `throw`, or a block containing one. */
992
1039
  function exits(statement) {
@@ -1114,6 +1161,33 @@ function runAnalysis(sourceCode, options) {
1114
1161
  const thisBodies = new Set();
1115
1162
  /** Describe bodies written as a `function`, whose `this` is Mocha's suite, which `this.retries()` acts on. */
1116
1163
  const suiteBodies = new Set();
1164
+ /**
1165
+ * Whether a statement around `anchor` has a comment that says something above it, as `// why` above an `if`,
1166
+ * up to `owner`, the test or describe the call acts on. The walk goes on through a callback passed to another
1167
+ * call, such as a hook's or a step's, but stops at a function declared some other way.
1168
+ */
1169
+ const commentedAround = (anchor, owner) => {
1170
+ for (let node = anchor.parent; node; node = node.parent) {
1171
+ if (FUNCTIONS.has(node.type)) {
1172
+ const call = node.parent;
1173
+ if (call?.type !== 'CallExpression' || call === owner || !call.arguments.includes(node)) {
1174
+ return false;
1175
+ }
1176
+ continue;
1177
+ }
1178
+ const container = node.parent;
1179
+ if (!container || !STATEMENT_CONTAINERS.has(container.type))
1180
+ continue;
1181
+ const block = commentBlock(sourceCode, node, options.allowBlankLine);
1182
+ if (block.some((comment) => !commentedOut.comments.has(comment) && hasText(comment)))
1183
+ return true;
1184
+ }
1185
+ return false;
1186
+ };
1187
+ /** Whether a comment that says something ends the line `node` ends on, as `test.skip(isMobile); // why`. */
1188
+ const trailingText = (node) => sourceCode
1189
+ .getCommentsAfter(node)
1190
+ .some((comment) => comment.loc.start.line === node.loc.end.line && hasText(comment));
1117
1191
  /**
1118
1192
  * `modifiers` maps each state modifier the call has to whether it always applies; a conditional one,
1119
1193
  * like `test.skip(isMobile)` or Vitest's `skipIf(...)`, needs a ticket only with `requireTicketForConditional`.
@@ -1129,6 +1203,14 @@ function runAnalysis(sourceCode, options) {
1129
1203
  const inRegions = regions.filter((r) => r.start <= call.range[0] && call.range[0] < r.end && !block.includes(r.comment));
1130
1204
  for (const comment of [...block, ...inRegions.map((r) => r.comment)])
1131
1205
  claimed.add(comment);
1206
+ const commented = [...block, ...inRegions.map((r) => r.comment)].some(hasText);
1207
+ // A runtime call can also say why above the statements around it in its function, above its test, or in
1208
+ // its arguments.
1209
+ const reasonedCall = () => kind === 'runtime' &&
1210
+ ((framework.reasonArgument !== undefined && callReason(sourceCode, call, framework.reasonArgument)) ||
1211
+ parent?.commented === true ||
1212
+ trailingText(anchor) ||
1213
+ commentedAround(anchor, parent?.node));
1132
1214
  const ownTags = new Set(tagInfo.tags.map((t) => t.tag));
1133
1215
  const sources = [];
1134
1216
  for (const state of states) {
@@ -1138,8 +1220,12 @@ function runAnalysis(sourceCode, options) {
1138
1220
  continue;
1139
1221
  }
1140
1222
  const always = modifiers.get(state.modifier);
1141
- if (always !== undefined)
1142
- sources.push({ state, required: always || options.requireTicketForConditional });
1223
+ if (always === undefined)
1224
+ continue;
1225
+ const source = { state, required: always || options.requireTicketForConditional };
1226
+ if (!always)
1227
+ source.reasoned = commented || tagInfo.reasonedOptions.has(state.modifier) || reasonedCall();
1228
+ sources.push(source);
1143
1229
  }
1144
1230
  const inherited = new Set();
1145
1231
  const inheritedTags = new Set();
@@ -1173,6 +1259,7 @@ function runAnalysis(sourceCode, options) {
1173
1259
  anchor,
1174
1260
  parent,
1175
1261
  states: sources,
1262
+ commented,
1176
1263
  inherited,
1177
1264
  runtimeStates: new Set(),
1178
1265
  tags: tagInfo.tags,
@@ -1332,9 +1419,17 @@ function runAnalysis(sourceCode, options) {
1332
1419
  const annotations = options.annotations === undefined || framework.annotations === undefined
1333
1420
  ? { annotations: [], unreadAnnotations: [] }
1334
1421
  : collectAnnotations(sourceCode, details, framework.annotations, options.annotations);
1422
+ const reasonedOptions = new Set();
1423
+ if (framework.anyOptionValue) {
1424
+ for (const [key, value] of details.entries) {
1425
+ const modifier = framework.optionEffects.get(key);
1426
+ if (modifier && optionReason(sourceCode, value))
1427
+ reasonedOptions.add(modifier);
1428
+ }
1429
+ }
1335
1430
  return {
1336
1431
  details,
1337
- tagInfo: { ...collectTags(sourceCode, call, !untitled, framework, details), ...annotations },
1432
+ tagInfo: { ...collectTags(sourceCode, call, !untitled, framework, details), ...annotations, reasonedOptions },
1338
1433
  };
1339
1434
  };
1340
1435
  const declare = (call, { kind, members, args: conditions }, infos) => {
@@ -95,6 +95,13 @@ export interface Framework {
95
95
  * `context.skip('why')`. In Playwright the first argument is always a condition.
96
96
  */
97
97
  skipNotes: boolean;
98
+ /**
99
+ * Where a runtime call takes the reason for its state: `after-condition`, as Playwright's
100
+ * `test.skip(condition, 'why')`; `only`, as node:test's `t.skip('why')` and Jasmine's `pending('why')`; or
101
+ * `either`, as Vitest's `context.skip('why')` and `context.skip(condition, 'why')`. Unset when it takes none,
102
+ * as Mocha's `this.skip()`.
103
+ */
104
+ reasonArgument?: 'after-condition' | 'only' | 'either';
98
105
  /**
99
106
  * The methods a test or hook written as a `function` can change its state with at runtime through `this`,
100
107
  * as Mocha's `this.skip()`. Empty when `this` can't.
@@ -48,6 +48,7 @@ const playwright = {
48
48
  subtests: false,
49
49
  skipWithoutBody: false,
50
50
  skipNotes: false,
51
+ reasonArgument: 'after-condition',
51
52
  thisMethods: new Set(),
52
53
  todoWithoutBody: false,
53
54
  runtimeFunctions: new Map(),
@@ -138,6 +139,7 @@ const vitest = {
138
139
  subtests: false,
139
140
  skipWithoutBody: true,
140
141
  skipNotes: true,
142
+ reasonArgument: 'either',
141
143
  thisMethods: new Set(),
142
144
  todoWithoutBody: false,
143
145
  runtimeFunctions: new Map(),
@@ -244,6 +246,7 @@ const nodeTest = {
244
246
  skipWithoutBody: true,
245
247
  // `t.skip('why')` takes a message, not a condition.
246
248
  skipNotes: true,
249
+ reasonArgument: 'only',
247
250
  thisMethods: NODE_TEST_CONTEXT,
248
251
  todoWithoutBody: false,
249
252
  runtimeFunctions: new Map(),
@@ -288,6 +291,7 @@ const jasmine = {
288
291
  skipWithoutBody: true,
289
292
  // `pending('why')` takes a reason, not a condition.
290
293
  skipNotes: true,
294
+ reasonArgument: 'only',
291
295
  thisMethods: new Set(),
292
296
  todoWithoutBody: true,
293
297
  runtimeFunctions: new Map([
@@ -50,8 +50,18 @@ export interface GovernanceOptions {
50
50
  states?: Partial<Record<BuiltinStateName, boolean | StateOverride>>;
51
51
  /** Adds a state for a tag, e.g. `{ 'needs-data': { when: '@needs-data', marker: 'NEEDS-DATA' } }`. */
52
52
  customStates?: Record<string, CustomState>;
53
+ /**
54
+ * Your team's other tags, such as `['@smoke', '@regression']`, which get the case and typo checks state tags
55
+ * get: `@Smoke` and `@smok` are reported. Write them with the `@`, also for Vitest's `tags: ['smoke']`.
56
+ */
57
+ knownTags?: string[];
53
58
  /** Require a ticket for conditional skips such as `test.skip(browserName === 'webkit')`. */
54
59
  requireTicketForConditional?: boolean;
60
+ /**
61
+ * Require a reason for conditional skips, without a ticket: a comment above the skip, or the reason the
62
+ * call takes, as `test.skip(browserName === 'webkit', 'No clipboard API')`.
63
+ */
64
+ requireReasonForConditional?: boolean;
55
65
  /** Let blank lines separate a marker from the test it belongs to. */
56
66
  allowBlankLine?: boolean;
57
67
  /**
@@ -104,7 +114,10 @@ export interface ResolvedOptions {
104
114
  framework: Framework;
105
115
  testFunctions: Set<string>;
106
116
  states: StateDef[];
117
+ /** Tags besides the state tags whose case and spelling `no-conflicting-states` checks. */
118
+ knownTags: string[];
107
119
  requireTicketForConditional: boolean;
120
+ requireReasonForConditional: boolean;
108
121
  allowBlankLine: boolean;
109
122
  allowNotes: boolean;
110
123
  reportUnreadableTags: boolean;
@@ -217,7 +217,9 @@ export function compileOptions(options) {
217
217
  framework,
218
218
  testFunctions: new Set(options.testFunctions ?? framework.testFunctions),
219
219
  states,
220
+ knownTags: options.knownTags ?? [],
220
221
  requireTicketForConditional: options.requireTicketForConditional ?? false,
222
+ requireReasonForConditional: options.requireReasonForConditional ?? false,
221
223
  allowBlankLine: options.allowBlankLine ?? false,
222
224
  allowNotes: options.allowNotes ?? false,
223
225
  reportUnreadableTags: options.reportUnreadableTags ?? false,
@@ -96,6 +96,7 @@ export const optionsSchema = {
96
96
  additionalProperties: false,
97
97
  },
98
98
  },
99
+ knownTags: { type: 'array', items: { type: 'string', pattern: TAG_RE.source, patternHint: TAG_HINT } },
99
100
  comments: {
100
101
  anyOf: [
101
102
  { type: 'boolean', enum: [false] },
@@ -119,6 +120,7 @@ export const optionsSchema = {
119
120
  ],
120
121
  },
121
122
  requireTicketForConditional: { type: 'boolean' },
123
+ requireReasonForConditional: { type: 'boolean' },
122
124
  allowBlankLine: { type: 'boolean' },
123
125
  allowNotes: { type: 'boolean' },
124
126
  reportUnreadableTags: { type: 'boolean' },
@@ -14,13 +14,26 @@ rule reports:
14
14
  - `@new` together with `@quarantined` on the same test, including tags inherited from a `describe`;
15
15
  - `@new` on a test that is skipped or fixme'd, since a test that doesn't run can't be promoted;
16
16
  - a state tag written in the wrong case, such as `@New` (autofixed to `@new`);
17
- - a near-miss spelling of a state tag, such as `@quarantiend`.
17
+ - a near-miss spelling of a state tag, such as `@quarantiend`;
18
+ - the same for the tags listed in `knownTags`, such as `@Smoke` or `@smok` next to `@smoke`.
18
19
 
19
20
  The `@new` checks run when the `new` state is on (`lifecycleTags: true`). Case and typo checks cover
20
21
  every tag state that is on, including custom states. A tag kept in a `const` in the same file is
21
22
  reported where the test uses it, without an autofix, since other tests may share the constant. Vitest
22
23
  tag names are written without the `@`, so `{ tags: ['New'] }` is reported, and fixed, as `'new'`.
23
24
 
25
+ Tags such as `@smoke` usually pick which tests a CI job runs, so a misspelled one quietly drops a test
26
+ from that job. List your team's tags in `knownTags`, written with the `@`, also for Vitest, to give them
27
+ the same checks. A listed tag is never reported as a typo of a state tag, so `knownTags: ['@quarantine']`
28
+ keeps a tag your team uses next to `@quarantined`. Tags of four letters or fewer after the `@`, such as
29
+ `@e2e` or `@fast`, get only the case check, since one letter off is often another word.
30
+
31
+ <!-- example: invalid settings={"knownTags":["@smoke","@regression"]} -->
32
+
33
+ ```js
34
+ test('places a market order @smok', async () => {});
35
+ ```
36
+
24
37
  ## Examples
25
38
 
26
39
  Incorrect:
@@ -68,4 +81,5 @@ This rule has no options of its own. It reads the shared options described in th
68
81
  is on, and `@new` together with `@quarantined` is reported when both are on. A tag renamed in
69
82
  `states`, such as `@flaky`, is checked in place of the default one.
70
83
  - `customStates`: the tags of custom states get the same case and typo checks.
84
+ - `knownTags`: other tags that get the case and typo checks.
71
85
  - `framework` and `testFunctions`: which functions count as the framework's `test`.
@@ -1,6 +1,6 @@
1
- # Require a ticket marker comment above skipped, fixme and tagged tests (`test-governance/require-ticket`)
1
+ # Require a ticket for skipped, fixme, todo, retried and tagged tests (`test-governance/require-ticket`)
2
2
 
3
- 📝 Require a ticket marker comment above skipped, fixme and tagged tests.
3
+ 📝 Require a ticket for skipped, fixme, todo, retried and tagged tests.
4
4
 
5
5
  💼 This rule is enabled in the following [configs](https://github.com/Huldoser/eslint-plugin-test-governance#usage): 🌲 `cypress`, 🌼 `jasmine`, 🃏 `jest`, ☕ `mocha`, 🟢 `node-test`, 🎭 `playwright`, ⚡ `vitest`, 🤖 `webdriverio`.
6
6
 
@@ -109,6 +109,43 @@ title and tags count as usual, and a marker goes above it, or in code without se
109
109
  If one test or describe has several markers for the same state, each must be valid: in
110
110
  `// SKIP: TRADE-1` followed by `// SKIP: nope`, the second line is reported.
111
111
 
112
+ ## Reasons for conditional skips
113
+
114
+ With `requireReasonForConditional: true`, a conditional skip needs a reason but no ticket, so nobody has
115
+ to guess later why it doesn't run. Any of these counts:
116
+
117
+ - a comment above it or at the end of its line, above a statement around it, such as the `if`, or above
118
+ the hook, step or test it is in. Commented-out code, a marker keyword alone such as `// SKIP:`, and an
119
+ `eslint-disable` comment without a description after `--` say nothing, so they don't count;
120
+ - the reason its call takes, where the runner has one: Playwright's `test.skip(condition, 'why')` and
121
+ `testInfo.skip(condition, 'why')`, Vitest's `context.skip(condition, 'why')` or `context.skip('why')`,
122
+ node:test's `t.skip('why')` and Jasmine's `pending('why')`. A reason held in a name or built in a
123
+ template is taken on trust;
124
+ - in node:test, text in the option, as `{ skip: isWindows && 'No symlinks' }`, which Node reports;
125
+ - a ticket, from a marker or an annotation.
126
+
127
+ The same goes for the other states a condition can set, such as `test.fixme(condition)`. With
128
+ `requireTicketForConditional` as well, the ticket is required and the reason alone is not enough.
129
+
130
+ <!-- example: valid settings={"requireReasonForConditional":true} -->
131
+
132
+ ```js
133
+ test('copies the order ID', async ({ browserName }) => {
134
+ test.skip(browserName === 'webkit', 'WebKit has no clipboard API in headless mode');
135
+ });
136
+
137
+ // Touch screens have no hover.
138
+ (isMobile ? test.skip : test)('shows the price on hover', async ({ page }) => {});
139
+ ```
140
+
141
+ <!-- example: invalid settings={"framework":"mocha","requireReasonForConditional":true} -->
142
+
143
+ ```js
144
+ it('places an order', function () {
145
+ if (isWeekend) this.skip();
146
+ });
147
+ ```
148
+
112
149
  ## Titles and tags that can't be read
113
150
 
114
151
  Tags can only be read from text: a string or a template literal such as `` `buys ${symbol} @new` ``,
@@ -332,6 +369,7 @@ change what it reports:
332
369
  - `lifecycleTags`, `states` and `customStates`: which states need a marker, which tag puts a test in
333
370
  each, what the marker is called, and, for a state with its own `ticket`, which tickets it accepts.
334
371
  - `requireTicketForConditional`: conditional skips need a ticket too.
372
+ - `requireReasonForConditional`: conditional skips need a reason, a comment or the call's own reason argument.
335
373
  - `allowBlankLine`: blank lines may separate a marker from its test.
336
374
  - `allowNotes`: text may follow the tickets.
337
375
  - `annotations`: the ticket may also come from a Playwright annotation of the configured type or Vitest's `meta` key of that name. With `only: true`, a test or describe must take it from there, and a marker comment above one is reported with a suggestion that moves it.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "eslint-plugin-test-governance",
3
- "version": "0.12.1",
4
- "description": "ESLint rules that require a ticket comment above skipped, fixme, todo and tagged Playwright, Jest, Vitest, Mocha, Cypress, WebdriverIO, node:test and Jasmine tests.",
3
+ "version": "0.13.0",
4
+ "description": "ESLint rules that require a ticket for skipped, fixme, todo, retried, tagged and commented-out Playwright, Jest, Vitest, Mocha, Cypress, WebdriverIO, node:test and Jasmine tests.",
5
5
  "keywords": [
6
6
  "eslint",
7
7
  "eslintplugin",