@assertrx/dino 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +293 -0
- package/extensions/dino.ts +70 -0
- package/extensions/mabl-mcp.ts +306 -0
- package/extensions/playwright-mcp.ts +142 -0
- package/package.json +47 -0
- package/prompts/create-pr.md +110 -0
- package/prompts/generate-bdd-from-url.md +65 -0
- package/prompts/generate-pom-from-url-cli.md +48 -0
- package/prompts/generate-pom-from-url.md +45 -0
- package/prompts/generate-spec-from-feature.md +64 -0
- package/prompts/heal-playwright-failures.md +106 -0
- package/prompts/setup-playwright.md +81 -0
- package/scripts/start-dino.cmd +8 -0
- package/scripts/start-dino.mjs +226 -0
- package/scripts/start-dino.ps1 +9 -0
- package/scripts/start-dino.sh +10 -0
- package/skills/acli/SKILL.md +292 -0
- package/skills/bdd-gherkin/SKILL.md +552 -0
- package/skills/build-review-interface/SKILL.md +96 -0
- package/skills/error-analysis/SKILL.md +164 -0
- package/skills/eval-audit/SKILL.md +183 -0
- package/skills/evaluate-rag/SKILL.md +177 -0
- package/skills/generate-synthetic-data/SKILL.md +131 -0
- package/skills/mabl-mcp/SKILL.md +79 -0
- package/skills/playwright-cli/SKILL.md +246 -0
- package/skills/playwright-cli/advanced-workflows.md +679 -0
- package/skills/playwright-cli/core-commands.md +375 -0
- package/skills/playwright-cli/device-emulation.md +475 -0
- package/skills/playwright-cli/request-mocking.md +373 -0
- package/skills/playwright-cli/running-custom-code.md +553 -0
- package/skills/playwright-cli/screenshots-and-media.md +457 -0
- package/skills/playwright-cli/session-management.md +425 -0
- package/skills/playwright-cli/storage-and-auth.md +487 -0
- package/skills/playwright-cli/test-generation.md +336 -0
- package/skills/playwright-cli/tracing-and-debugging.md +501 -0
- package/skills/playwright-core/SKILL.md +101 -0
- package/skills/playwright-core/accessibility.md +1517 -0
- package/skills/playwright-core/angular.md +1086 -0
- package/skills/playwright-core/api-testing.md +1623 -0
- package/skills/playwright-core/assertions-and-waiting.md +703 -0
- package/skills/playwright-core/auth-flows.md +1081 -0
- package/skills/playwright-core/authentication.md +1409 -0
- package/skills/playwright-core/browser-apis.md +673 -0
- package/skills/playwright-core/browser-extensions.md +319 -0
- package/skills/playwright-core/canvas-and-webgl.md +494 -0
- package/skills/playwright-core/clock-and-time-mocking.md +427 -0
- package/skills/playwright-core/common-pitfalls.md +1318 -0
- package/skills/playwright-core/component-testing.md +1179 -0
- package/skills/playwright-core/configuration.md +731 -0
- package/skills/playwright-core/crud-testing.md +945 -0
- package/skills/playwright-core/debugging.md +827 -0
- package/skills/playwright-core/drag-and-drop.md +992 -0
- package/skills/playwright-core/electron-testing.md +622 -0
- package/skills/playwright-core/error-and-edge-cases.md +1181 -0
- package/skills/playwright-core/error-index.md +1903 -0
- package/skills/playwright-core/file-operations.md +749 -0
- package/skills/playwright-core/file-upload-download.md +1012 -0
- package/skills/playwright-core/fixtures-and-hooks.md +1072 -0
- package/skills/playwright-core/flaky-tests.md +883 -0
- package/skills/playwright-core/forms-and-validation.md +1056 -0
- package/skills/playwright-core/i18n-and-localization.md +622 -0
- package/skills/playwright-core/iframes-and-shadow-dom.md +488 -0
- package/skills/playwright-core/locator-strategy.md +588 -0
- package/skills/playwright-core/locators.md +743 -0
- package/skills/playwright-core/mobile-and-responsive.md +1669 -0
- package/skills/playwright-core/multi-context-and-popups.md +581 -0
- package/skills/playwright-core/multi-user-and-collaboration.md +477 -0
- package/skills/playwright-core/network-mocking.md +1395 -0
- package/skills/playwright-core/nextjs.md +1013 -0
- package/skills/playwright-core/performance-testing.md +608 -0
- package/skills/playwright-core/react.md +1082 -0
- package/skills/playwright-core/search-and-filter.md +1366 -0
- package/skills/playwright-core/security-testing.md +624 -0
- package/skills/playwright-core/service-workers-and-pwa.md +524 -0
- package/skills/playwright-core/test-architecture.md +569 -0
- package/skills/playwright-core/test-data-management.md +1209 -0
- package/skills/playwright-core/test-organization.md +947 -0
- package/skills/playwright-core/third-party-integrations.md +754 -0
- package/skills/playwright-core/trace-analysis.md +161 -0
- package/skills/playwright-core/visual-regression.md +1027 -0
- package/skills/playwright-core/vue.md +1162 -0
- package/skills/playwright-core/websockets-and-realtime.md +623 -0
- package/skills/playwright-core/when-to-mock.md +827 -0
- package/skills/playwright-pom/SKILL.md +94 -0
- package/skills/playwright-pom/page-object-model.md +921 -0
- package/skills/playwright-pom/pom-vs-fixtures-vs-helpers.md +937 -0
- package/skills/validate-evaluator/SKILL.md +215 -0
- package/skills/write-judge-prompt/SKILL.md +144 -0
- package/themes/dino.json +66 -0
|
@@ -0,0 +1,552 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: bdd-gherkin
|
|
3
|
+
description: Generate domain-driven Gherkin/Cucumber BDD scenarios focused on business requirements with minimal technical details when necessary for clarity
|
|
4
|
+
author: Anand Gopalakrishnan
|
|
5
|
+
metadata:
|
|
6
|
+
focus: business-domain
|
|
7
|
+
format: gherkin
|
|
8
|
+
version: cucumber-7+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Skill: bdd-gherkin
|
|
12
|
+
|
|
13
|
+
## What I do
|
|
14
|
+
|
|
15
|
+
- Generate Gherkin feature files with business-focused scenarios
|
|
16
|
+
- Create domain-driven BDD scenarios using ubiquitous language
|
|
17
|
+
- Write scenarios in Given/When/Then format prioritizing business language
|
|
18
|
+
- Parse Jira-style ticket descriptions (Summary, Observed Behavior, Expected Behavior, Conditions/Context, Replication Steps, Acceptance Criteria) and convert them into Gherkin feature files
|
|
19
|
+
- Include Scenario Outlines with Examples tables when explicitly beneficial
|
|
20
|
+
- Add Background sections liberally when scenarios share common preconditions
|
|
21
|
+
- Use minimal tagging - only when explicitly needed
|
|
22
|
+
- Support data tables and doc strings where appropriate
|
|
23
|
+
- Focus on business behavior with minimal technical context when it aids clarity
|
|
24
|
+
- Always ask clarifying questions before generating scenarios, unless the user provides a complete Jira ticket description
|
|
25
|
+
|
|
26
|
+
## When to use me
|
|
27
|
+
|
|
28
|
+
Use this skill when:
|
|
29
|
+
- User asks to create BDD scenarios or feature files
|
|
30
|
+
- User mentions Gherkin, Cucumber, or behavior-driven development
|
|
31
|
+
- User wants to document business requirements as executable specifications
|
|
32
|
+
- User needs acceptance criteria written in Given/When/Then format
|
|
33
|
+
- User requests feature files for user stories
|
|
34
|
+
- User pastes a Jira ticket description containing fields such as Summary, Observed Behavior, Expected Behavior, Conditions/Context, Replication Steps, or Acceptance Criteria
|
|
35
|
+
|
|
36
|
+
## Handling Jira ticket descriptions
|
|
37
|
+
|
|
38
|
+
When the user provides a Jira-style ticket description, treat it as structured input and generate BDD feature files directly from the relevant sections.
|
|
39
|
+
|
|
40
|
+
### Recognized Jira fields
|
|
41
|
+
|
|
42
|
+
| Jira field | How to use it |
|
|
43
|
+
|---|---|
|
|
44
|
+
| **Summary / Title** | Derive the Feature name and the business value statement (`As a ... I want ... So that ...`). |
|
|
45
|
+
| **Observed Behavior** | Capture the current/problem outcome. Turn error messages into negative scenarios or assertion text in `Then` steps. |
|
|
46
|
+
| **Expected Behavior** | Capture the desired outcome. Use as the primary source for `Then` steps and happy-path scenarios. |
|
|
47
|
+
| **Conditions and Context** | Convert environment, scope, and setup details into `Background` or `Given` preconditions. |
|
|
48
|
+
| **Replication Steps** | Convert each meaningful user action into a single high-level `When` step. Move detailed setup into `Given` using data tables or Background. |
|
|
49
|
+
| **Acceptance Criteria** | Primary source of scenarios. If already in Given/When/Then form, adapt it to follow this skill's rules (single-focused scenarios, one `When` per scenario, business language). |
|
|
50
|
+
| **Links / References** | Use only for context; do not include URLs in scenarios unless explicitly needed. |
|
|
51
|
+
|
|
52
|
+
### Rules for Jira input mode
|
|
53
|
+
|
|
54
|
+
1. **Generate scenarios without asking clarifying questions** when the ticket contains at least a Summary and Acceptance Criteria (or clear Expected Behavior). Only ask questions if critical information is missing.
|
|
55
|
+
2. **Produce one feature file per distinct business capability**. If the ticket covers multiple capabilities, split them into separate `.feature` files.
|
|
56
|
+
3. **Create single-focused scenarios** from each acceptance criterion, observed error, and significant replication path.
|
|
57
|
+
4. **Use exactly one `When` action per scenario**. Combine low-level replication steps into a single high-level business action, and move setup details into `Given`.
|
|
58
|
+
5. **Use `Background` liberally** for preconditions shared across scenarios (e.g., project created, SSA enabled, permissions granted).
|
|
59
|
+
6. **Keep business language**; avoid UI field names, API endpoints, or database details unless they are essential for clarity.
|
|
60
|
+
7. **Represent exact error messages from Observed Behavior as `Then` assertions** in negative scenarios, but phrase the scenario itself from the business perspective.
|
|
61
|
+
|
|
62
|
+
### Jira-to-Gherkin example
|
|
63
|
+
|
|
64
|
+
**Input ticket summary:** Publish fails with "User not authorized to access Autodesk project" on newly created projects despite confirmed SSA setup.
|
|
65
|
+
|
|
66
|
+
```gherkin
|
|
67
|
+
Feature: Publish models to newly created Autodesk projects
|
|
68
|
+
As a project team member
|
|
69
|
+
I want to publish models from Revit or AutoCAD to a new project after SSA setup
|
|
70
|
+
So that project data is available in ACC without access errors
|
|
71
|
+
|
|
72
|
+
Background:
|
|
73
|
+
Given a newly created Autodesk/ACC project exists
|
|
74
|
+
And SSA access is enabled for the project
|
|
75
|
+
And the SSA email has View-Download-Upload-Edit permissions on Project Files
|
|
76
|
+
|
|
77
|
+
Scenario: Successful model publish to a new project
|
|
78
|
+
Given Stratus SSA Status confirms "SSA has access to this project"
|
|
79
|
+
And the user is publishing from Revit or AutoCAD
|
|
80
|
+
When the user publishes a model to the new project
|
|
81
|
+
Then the publish succeeds
|
|
82
|
+
And no "User not authorized to access Autodesk project" error occurs
|
|
83
|
+
|
|
84
|
+
Scenario: Publish fails when SSA lacks a project publishing role
|
|
85
|
+
Given the SSA email is added to the project
|
|
86
|
+
But the SSA does not have a project role that allows publishing
|
|
87
|
+
When the user publishes a model to the new project
|
|
88
|
+
Then the publish fails
|
|
89
|
+
And the user sees "User does not have a project role that allows publishing for '<project name>'"
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Save as:
|
|
93
|
+
```
|
|
94
|
+
publish-to-new-autodesk-project.feature
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Core principles I follow
|
|
98
|
+
|
|
99
|
+
### 1. Domain-Driven Language with Pragmatic Clarity
|
|
100
|
+
- Use the **ubiquitous language** of the business domain as primary focus
|
|
101
|
+
- Prioritize business actions, outcomes, and rules
|
|
102
|
+
- Express scenarios from the user's or business stakeholder's perspective
|
|
103
|
+
- Allow minimal technical context when it adds clarity (e.g., "API endpoint", "database record") but avoid implementation specifics
|
|
104
|
+
- Balance business readability with practical clarity
|
|
105
|
+
|
|
106
|
+
**Good (Business-focused):**
|
|
107
|
+
```gherkin
|
|
108
|
+
Given a customer with a premium account
|
|
109
|
+
When they purchase a product worth $500
|
|
110
|
+
Then they receive a 10% loyalty discount
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
**Bad (Too much technical detail):**
|
|
114
|
+
```gherkin
|
|
115
|
+
Given the database has a user with premium_flag = true in users table
|
|
116
|
+
When a POST request is sent to /api/purchase with JSON body {"amount":500}
|
|
117
|
+
Then the HTTP response status code is 200
|
|
118
|
+
And the response contains discount_percentage: 10
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
**Acceptable (Minimal technical context for clarity):**
|
|
122
|
+
```gherkin
|
|
123
|
+
Given a customer record exists with premium status
|
|
124
|
+
When they submit a purchase request for $500
|
|
125
|
+
Then the purchase is confirmed
|
|
126
|
+
And a 10% loyalty discount is applied
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
### 2. Feature Structure
|
|
130
|
+
Every feature file should include:
|
|
131
|
+
- **Feature**: Clear business capability or epic
|
|
132
|
+
- **Business value statement** (recommended): As a [role], I want [feature], So that [benefit]
|
|
133
|
+
- **Background** (use liberally): Common preconditions when 2+ scenarios share setup
|
|
134
|
+
- **Scenarios**: Individual test cases (preferred by default)
|
|
135
|
+
- **Scenario Outlines** (when explicitly beneficial): Use only when truly needed for data variations
|
|
136
|
+
- **Tags** (minimal): Only when explicitly needed for organization
|
|
137
|
+
|
|
138
|
+
### 3. Given/When/Then Best Practices
|
|
139
|
+
|
|
140
|
+
**Given** - Establish context:
|
|
141
|
+
- Describe the initial state
|
|
142
|
+
- Set up preconditions
|
|
143
|
+
- Use past tense or present state ("a customer has...", "the account is...")
|
|
144
|
+
- No actions or events
|
|
145
|
+
|
|
146
|
+
**When** - Trigger action:
|
|
147
|
+
- **CRITICAL**: Single action or event only - exactly ONE When step per scenario
|
|
148
|
+
- Use present tense ("customer places order", "user cancels subscription")
|
|
149
|
+
- Should be one clear action per When
|
|
150
|
+
- **Avoid multiple When/And steps** - consolidate into a single high-level action
|
|
151
|
+
- Put data/parameters in the Given context, not as separate When steps
|
|
152
|
+
|
|
153
|
+
**Then** - Verify outcome:
|
|
154
|
+
- Observable business outcome
|
|
155
|
+
- Use present tense ("order is confirmed", "balance is updated")
|
|
156
|
+
- Focus on business results, not technical state
|
|
157
|
+
|
|
158
|
+
**And/But** - Chain related steps:
|
|
159
|
+
- Use for additional context, actions, or assertions
|
|
160
|
+
- Keep scenarios readable
|
|
161
|
+
|
|
162
|
+
### 4. Scenario Outlines and Examples
|
|
163
|
+
Use Scenario Outlines sparingly - prefer regular Scenarios by default. Only use Outlines when:
|
|
164
|
+
- The same business rule needs validation with multiple data points
|
|
165
|
+
- Data variations are the focus of the test
|
|
166
|
+
- It significantly improves readability over multiple similar scenarios
|
|
167
|
+
|
|
168
|
+
```gherkin
|
|
169
|
+
Scenario Outline: Apply volume discounts based on purchase quantity
|
|
170
|
+
Given a customer is purchasing <quantity> units
|
|
171
|
+
When they complete the checkout
|
|
172
|
+
Then they receive a <discount>% discount
|
|
173
|
+
And the total price is reduced accordingly
|
|
174
|
+
|
|
175
|
+
Examples: Standard volume tiers
|
|
176
|
+
| quantity | discount |
|
|
177
|
+
| 1 | 0 |
|
|
178
|
+
| 10 | 5 |
|
|
179
|
+
| 50 | 10 |
|
|
180
|
+
| 100 | 15 |
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
**When in doubt, use regular Scenarios instead.**
|
|
184
|
+
|
|
185
|
+
### 5. Data Tables
|
|
186
|
+
Use data tables to express structured data in a business-readable way:
|
|
187
|
+
|
|
188
|
+
**For multiple records (horizontal headers):**
|
|
189
|
+
```gherkin
|
|
190
|
+
Given the following products are in the catalog:
|
|
191
|
+
| Product Name | Category | Price |
|
|
192
|
+
| Laptop | Electronics | 1200 |
|
|
193
|
+
| Desk Chair | Furniture | 350 |
|
|
194
|
+
| Monitor | Electronics | 400 |
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
**For single record with multiple fields (vertical key-value pairs):**
|
|
198
|
+
```gherkin
|
|
199
|
+
Given a user wants to add a bookmark with the following details:
|
|
200
|
+
| URL | https://example.com |
|
|
201
|
+
| Title | Example Website |
|
|
202
|
+
| Description | A helpful example site |
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
**When to use data tables in Given steps:**
|
|
206
|
+
- When you have 2+ fields/attributes to specify
|
|
207
|
+
- When the Given step would become too long with inline values
|
|
208
|
+
- When you want to clearly show what fields are present vs. absent
|
|
209
|
+
- For better readability and maintainability
|
|
210
|
+
|
|
211
|
+
**❌ Don't: Use long inline values in Given**
|
|
212
|
+
```gherkin
|
|
213
|
+
Given a user wants to add a bookmark with URL "https://example.com", title "Example Website", and description "A helpful example site for testing"
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
**✓ Do: Use vertical data table for cleaner readability**
|
|
217
|
+
```gherkin
|
|
218
|
+
Given a user wants to add a bookmark with the following details:
|
|
219
|
+
| URL | https://example.com |
|
|
220
|
+
| Title | Example Website |
|
|
221
|
+
| Description | A helpful example site for testing |
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
### 6. Doc Strings
|
|
225
|
+
Use doc strings (triple quotes) for larger text content:
|
|
226
|
+
|
|
227
|
+
```gherkin
|
|
228
|
+
Given a customer service representative receives this inquiry:
|
|
229
|
+
"""
|
|
230
|
+
I ordered product #12345 two weeks ago but haven't
|
|
231
|
+
received any shipping confirmation. Can you help?
|
|
232
|
+
"""
|
|
233
|
+
When they process the inquiry
|
|
234
|
+
Then a shipping status update is sent to the customer
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
### 7. Tags for Organization
|
|
238
|
+
Use tags minimally - only when they provide clear organizational value:
|
|
239
|
+
|
|
240
|
+
```gherkin
|
|
241
|
+
Scenario: Complete purchase with valid payment
|
|
242
|
+
|
|
243
|
+
@wip
|
|
244
|
+
Scenario: Process return for defective item
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Only add tags when:
|
|
248
|
+
- Marking work in progress (@wip)
|
|
249
|
+
- Explicitly requested by the user
|
|
250
|
+
- Required for test execution filtering
|
|
251
|
+
- Necessary for team workflow
|
|
252
|
+
|
|
253
|
+
Avoid over-tagging with @smoke, @regression, @critical unless specifically needed.
|
|
254
|
+
|
|
255
|
+
## Scenario Scope: Default to Single-Focused
|
|
256
|
+
|
|
257
|
+
**IMPORTANT: Always default to single-focused scenarios unless the user explicitly requests journey-based scenarios.**
|
|
258
|
+
|
|
259
|
+
### Single-Focused Scenarios (DEFAULT)
|
|
260
|
+
- Each scenario tests **one specific business rule or behavior**
|
|
261
|
+
- Scenarios are independent and can run in isolation
|
|
262
|
+
- Easier to understand, maintain, and debug
|
|
263
|
+
- Clear pass/fail for specific capabilities
|
|
264
|
+
- Preferred approach for most situations
|
|
265
|
+
|
|
266
|
+
### Journey-Based Scenarios (Only when explicitly requested)
|
|
267
|
+
- Test end-to-end user workflows across multiple capabilities
|
|
268
|
+
- Show how features work together in realistic user flows
|
|
269
|
+
- Used when the user specifically asks for:
|
|
270
|
+
- "End-to-end scenarios"
|
|
271
|
+
- "User journey scenarios"
|
|
272
|
+
- "Full workflow tests"
|
|
273
|
+
- "Integration scenarios"
|
|
274
|
+
|
|
275
|
+
**When in doubt, use single-focused scenarios.**
|
|
276
|
+
|
|
277
|
+
## Questions to ask
|
|
278
|
+
|
|
279
|
+
**ALWAYS ask clarifying questions before generating scenarios, unless the user has provided a complete Jira ticket description.** When the user requests BDD scenarios without a structured Jira ticket, ask:
|
|
280
|
+
|
|
281
|
+
1. **What business capability or feature is this testing?**
|
|
282
|
+
- Helps frame the Feature description
|
|
283
|
+
|
|
284
|
+
2. **Who is the primary actor or user role?**
|
|
285
|
+
- Defines perspective for scenarios (customer, admin, agent, etc.)
|
|
286
|
+
|
|
287
|
+
3. **What business value does this provide?**
|
|
288
|
+
- Helps decide if "As a/I want/So that" statement is appropriate
|
|
289
|
+
|
|
290
|
+
4. **What are the specific business rules or constraints?**
|
|
291
|
+
- Helps identify scenarios and edge cases
|
|
292
|
+
|
|
293
|
+
5. **Are there common preconditions across scenarios?**
|
|
294
|
+
- Determines Background section usage
|
|
295
|
+
|
|
296
|
+
**Note:** Do NOT ask about single-focused vs. journey scenarios - always default to single-focused unless the user explicitly requests journeys.
|
|
297
|
+
|
|
298
|
+
**Jira ticket exception:** If the input is a Jira description containing Summary plus Acceptance Criteria or clear Expected Behavior, generate scenarios directly using the rules in [Handling Jira ticket descriptions](#handling-jira-ticket-descriptions). Only ask questions if critical information (e.g., the action or expected outcome) is missing.
|
|
299
|
+
|
|
300
|
+
Only generate scenarios after receiving answers to relevant questions, or after parsing a complete Jira ticket description.
|
|
301
|
+
|
|
302
|
+
## Standard feature file template
|
|
303
|
+
|
|
304
|
+
```gherkin
|
|
305
|
+
Feature: [Business capability name]
|
|
306
|
+
As a [role/actor]
|
|
307
|
+
I want [feature/capability]
|
|
308
|
+
So that [business value/benefit]
|
|
309
|
+
|
|
310
|
+
Background:
|
|
311
|
+
Given [common precondition for all scenarios]
|
|
312
|
+
And [another common precondition]
|
|
313
|
+
|
|
314
|
+
Scenario: [Happy path scenario name]
|
|
315
|
+
Given [initial context]
|
|
316
|
+
And [additional context]
|
|
317
|
+
When [user action]
|
|
318
|
+
Then [expected business outcome]
|
|
319
|
+
And [additional verification]
|
|
320
|
+
|
|
321
|
+
Scenario: [Edge case or validation scenario]
|
|
322
|
+
Given [specific context for edge case]
|
|
323
|
+
When [action that triggers validation]
|
|
324
|
+
Then [expected business behavior]
|
|
325
|
+
|
|
326
|
+
Scenario: [Another regular scenario]
|
|
327
|
+
Given [context]
|
|
328
|
+
When [action]
|
|
329
|
+
Then [outcome]
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
**Notes:**
|
|
333
|
+
- Business value statement is recommended but optional
|
|
334
|
+
- Use Background liberally when scenarios share setup
|
|
335
|
+
- Prefer regular Scenarios over Scenario Outlines
|
|
336
|
+
- Add tags only when explicitly needed
|
|
337
|
+
- **Default to single-focused scenarios** unless user explicitly requests journeys
|
|
338
|
+
|
|
339
|
+
## Common anti-patterns to avoid
|
|
340
|
+
|
|
341
|
+
### ❌ Don't: Include excessive technical implementation
|
|
342
|
+
```gherkin
|
|
343
|
+
When the user clicks the "Submit" button with id="btn-submit" using Selenium driver
|
|
344
|
+
Then the API returns HTTP status code 200 with JSON schema validation
|
|
345
|
+
And the database transaction commits with ACID properties
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
### ✓ Do: Focus on business action and outcome (minimal tech context OK)
|
|
349
|
+
```gherkin
|
|
350
|
+
When the user submits their application
|
|
351
|
+
Then the application is accepted for review
|
|
352
|
+
And a confirmation is sent
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
### ❌ Don't: Use Scenario Outline unnecessarily
|
|
356
|
+
```gherkin
|
|
357
|
+
Scenario Outline: Process single order
|
|
358
|
+
Given a customer places an order
|
|
359
|
+
When they pay <amount>
|
|
360
|
+
Then order is confirmed
|
|
361
|
+
|
|
362
|
+
Examples:
|
|
363
|
+
| amount |
|
|
364
|
+
| 100 |
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### ✓ Do: Use regular Scenario when data variation isn't the focus
|
|
368
|
+
```gherkin
|
|
369
|
+
Scenario: Process customer order
|
|
370
|
+
Given a customer places an order
|
|
371
|
+
When they complete payment
|
|
372
|
+
Then the order is confirmed
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
### ❌ Don't: Create journey scenarios unless explicitly requested
|
|
376
|
+
```gherkin
|
|
377
|
+
# Only use this pattern when user explicitly asks for journey/end-to-end scenarios
|
|
378
|
+
Scenario: Complete first-time purchase journey
|
|
379
|
+
Given a new customer browses the product catalog
|
|
380
|
+
When they select a product and add it to their cart
|
|
381
|
+
And complete the checkout process with payment
|
|
382
|
+
Then their order is confirmed
|
|
383
|
+
And they receive a welcome discount on their next purchase
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
### ✓ Do: Default to single-focused scenarios (preferred approach)
|
|
387
|
+
```gherkin
|
|
388
|
+
Scenario: Add item to cart
|
|
389
|
+
Given a customer views a product
|
|
390
|
+
When they add it to cart
|
|
391
|
+
Then the cart contains 1 item
|
|
392
|
+
And the product appears in their cart
|
|
393
|
+
|
|
394
|
+
Scenario: Complete checkout with items in cart
|
|
395
|
+
Given a customer has 1 item in their cart
|
|
396
|
+
When they complete the checkout process
|
|
397
|
+
Then their order is confirmed
|
|
398
|
+
And they receive a confirmation email
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
### ❌ Don't: Break high-level actions into multiple When steps
|
|
402
|
+
```gherkin
|
|
403
|
+
Scenario: Add a bookmark with all fields
|
|
404
|
+
Given a user is ready to save a new bookmark
|
|
405
|
+
When they enter "https://example.com" as the URL
|
|
406
|
+
And they enter "Example Website" as the title
|
|
407
|
+
And they enter "A description" as the description
|
|
408
|
+
And they submit the bookmark
|
|
409
|
+
Then the bookmark is saved successfully
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
### ✓ Do: Use a single high-level When action with data table in Given
|
|
413
|
+
```gherkin
|
|
414
|
+
Scenario: Add a complete bookmark
|
|
415
|
+
Given a user wants to add a bookmark with the following details:
|
|
416
|
+
| URL | https://example.com |
|
|
417
|
+
| Title | Example Website |
|
|
418
|
+
| Description | A description |
|
|
419
|
+
When they add the bookmark
|
|
420
|
+
Then the bookmark is saved successfully
|
|
421
|
+
And the bookmark appears in their bookmark list
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
### ❌ Don't: Use long inline values in Given steps
|
|
425
|
+
```gherkin
|
|
426
|
+
Scenario: Add a complete bookmark
|
|
427
|
+
Given a user wants to add a bookmark with URL "https://example.com", title "Example Website", and description "A helpful example site for testing"
|
|
428
|
+
When they add the bookmark
|
|
429
|
+
Then the bookmark is saved successfully
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
### ✓ Do: Use data table when Given has 2+ fields or gets too long
|
|
433
|
+
```gherkin
|
|
434
|
+
Scenario: Add a complete bookmark
|
|
435
|
+
Given a user wants to add a bookmark with the following details:
|
|
436
|
+
| URL | https://example.com |
|
|
437
|
+
| Title | Example Website |
|
|
438
|
+
| Description | A helpful example site for testing |
|
|
439
|
+
When they add the bookmark
|
|
440
|
+
Then the bookmark is saved successfully
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
## Workflow example
|
|
444
|
+
|
|
445
|
+
**User:** "Create BDD scenarios for a customer loyalty program where customers earn points on purchases"
|
|
446
|
+
|
|
447
|
+
**Response:**
|
|
448
|
+
|
|
449
|
+
First, let me ask a few clarifying questions:
|
|
450
|
+
1. What are the specific business rules for earning points? (e.g., 1 point per dollar, different rates for tiers?)
|
|
451
|
+
2. Are there different customer tiers or segments to consider?
|
|
452
|
+
3. What actions should customers be able to do with points? (redeem, expire, transfer?)
|
|
453
|
+
|
|
454
|
+
[After receiving answers:]
|
|
455
|
+
|
|
456
|
+
I'll create a feature file for the customer loyalty program:
|
|
457
|
+
|
|
458
|
+
```gherkin
|
|
459
|
+
Feature: Customer Loyalty Points
|
|
460
|
+
As a customer
|
|
461
|
+
I want to earn points on my purchases
|
|
462
|
+
So that I can receive rewards and discounts on future orders
|
|
463
|
+
|
|
464
|
+
Background:
|
|
465
|
+
Given the loyalty program is active
|
|
466
|
+
And point values are set to 1 point per dollar spent
|
|
467
|
+
|
|
468
|
+
Scenario: Earn points on standard purchase
|
|
469
|
+
Given a customer with a basic loyalty account
|
|
470
|
+
And their current point balance is 0
|
|
471
|
+
When they complete a purchase of $100
|
|
472
|
+
Then they earn 100 loyalty points
|
|
473
|
+
And their new point balance is 100
|
|
474
|
+
|
|
475
|
+
Scenario: Silver tier earns bonus points
|
|
476
|
+
Given a customer with Silver membership
|
|
477
|
+
When they complete a purchase of $100
|
|
478
|
+
Then they earn 150 loyalty points
|
|
479
|
+
|
|
480
|
+
Scenario: Gold tier earns higher bonus points
|
|
481
|
+
Given a customer with Gold membership
|
|
482
|
+
When they complete a purchase of $100
|
|
483
|
+
Then they earn 200 loyalty points
|
|
484
|
+
|
|
485
|
+
Scenario: Redeem points for discount on purchase
|
|
486
|
+
Given a customer has 500 loyalty points
|
|
487
|
+
And they have items worth $80 in their cart
|
|
488
|
+
When they redeem 400 points for a discount
|
|
489
|
+
Then $40 is deducted from their order total
|
|
490
|
+
And their final amount is $40
|
|
491
|
+
And their remaining point balance is 100
|
|
492
|
+
|
|
493
|
+
Scenario: Points expire after 12 months of inactivity
|
|
494
|
+
Given a customer earned 300 points on January 1, 2025
|
|
495
|
+
And they have had no account activity since then
|
|
496
|
+
When the date reaches January 2, 2026
|
|
497
|
+
Then their 300 points are marked as expired
|
|
498
|
+
And their available point balance is 0
|
|
499
|
+
And they receive an expiration notification
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
Would you like me to:
|
|
503
|
+
1. Add more scenarios for edge cases?
|
|
504
|
+
2. Include scenarios for points transfer or gifting?
|
|
505
|
+
3. Add validation scenarios for fraud prevention?
|
|
506
|
+
|
|
507
|
+
## Validation checklist
|
|
508
|
+
|
|
509
|
+
Before finalizing scenarios, verify:
|
|
510
|
+
- [ ] Prioritizes business language with minimal technical details for clarity
|
|
511
|
+
- [ ] Uses domain/business language throughout
|
|
512
|
+
- [ ] **Scenarios are single-focused (default) unless user explicitly requested journeys**
|
|
513
|
+
- [ ] Given/When/Then structure is followed correctly
|
|
514
|
+
- [ ] **Each scenario has exactly ONE When action (no multiple When/And action steps)**
|
|
515
|
+
- [ ] **Data and parameters are in Given context, not spread across multiple When steps**
|
|
516
|
+
- [ ] **Data tables used for Given steps with 2+ fields or long inline values**
|
|
517
|
+
- [ ] Regular Scenarios used by default; Scenario Outlines only when explicitly beneficial
|
|
518
|
+
- [ ] Tags used minimally - only when needed
|
|
519
|
+
- [ ] Background used liberally when scenarios share preconditions
|
|
520
|
+
- [ ] Outcomes are observable business results
|
|
521
|
+
- [ ] Business value statement included when it adds value (recommended but optional)
|
|
522
|
+
- [ ] Scenarios are readable by non-technical stakeholders
|
|
523
|
+
- [ ] **When input is a Jira ticket, Summary, Observed/Expected Behavior, Conditions/Context, Replication Steps, and Acceptance Criteria are mapped to appropriate scenarios**
|
|
524
|
+
|
|
525
|
+
## Tips for success
|
|
526
|
+
|
|
527
|
+
1. **Think like a business analyst**: What does the business care about?
|
|
528
|
+
2. **Balance clarity with domain focus**: Use business language primarily, minimal tech terms for clarity
|
|
529
|
+
3. **Default to single-focused scenarios**: Test one behavior per scenario unless explicitly asked for journeys
|
|
530
|
+
4. **Keep scenarios independent**: Each should run in isolation
|
|
531
|
+
5. **Make scenarios maintainable**: Avoid brittle test data dependencies
|
|
532
|
+
6. **Use declarative style**: Say WHAT should happen, not HOW to do it
|
|
533
|
+
7. **One action per scenario**: Keep exactly ONE When step - move data to Given context
|
|
534
|
+
8. **High-level actions**: Use business-level actions like "add bookmark", not field-by-field steps
|
|
535
|
+
9. **Use data tables liberally**: For 2+ fields or long values in Given steps - improves readability
|
|
536
|
+
10. **Ask questions first**: Always clarify requirements before generating, unless the input is a complete Jira ticket description
|
|
537
|
+
11. **Review with stakeholders**: Scenarios should be readable by product owners
|
|
538
|
+
12. **Use Background liberally**: Share common setup across scenarios
|
|
539
|
+
13. **Prefer regular Scenarios**: Use Outlines only when truly beneficial
|
|
540
|
+
|
|
541
|
+
## File naming conventions
|
|
542
|
+
|
|
543
|
+
Feature files should use descriptive names:
|
|
544
|
+
- `customer-loyalty-program.feature`
|
|
545
|
+
- `order-checkout.feature`
|
|
546
|
+
- `inventory-management.feature`
|
|
547
|
+
- `user-registration.feature`
|
|
548
|
+
|
|
549
|
+
Not:
|
|
550
|
+
- `test1.feature`
|
|
551
|
+
- `scenarios.feature`
|
|
552
|
+
- `cucumber_test.feature`
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: build-review-interface
|
|
3
|
+
description: >
|
|
4
|
+
Build a custom browser-based annotation interface tailored to your data for
|
|
5
|
+
reviewing LLM traces and collecting structured feedback. Use when you need to
|
|
6
|
+
build an annotation tool, review traces, or collect human labels.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Build a Custom Annotation Interface
|
|
10
|
+
|
|
11
|
+
## Overview
|
|
12
|
+
|
|
13
|
+
Build an HTML page that loads traces from a data source (JSON/CSV file), displays one trace at a time with Pass/Fail buttons, a free-text notes field, and Next/Previous navigation. Save labels to a local file (CSV/SQLite/JSON). Then customize to the domain using the guidelines below.
|
|
14
|
+
|
|
15
|
+
## Data Display
|
|
16
|
+
|
|
17
|
+
Format all data in the most human-readable representation for the domain. Emails should look like emails. Code should have syntax highlighting. Markdown should be rendered. Tables should be tables. JSON should be pretty-printed and collapsible.
|
|
18
|
+
|
|
19
|
+
- **Collapse repetitive elements.** If every trace shares the same system prompt, put it in a `<details>` toggle.
|
|
20
|
+
- **Extract and surface key metadata.** If traces contain a property name, client type, or session ID buried in the data, extract it and display it prominently as a header or badge.
|
|
21
|
+
- **Color-code by role or status.** Use left-border colors to distinguish user messages, assistant messages, tool calls, and system prompts at a glance.
|
|
22
|
+
- **Group related elements visually.** Tool calls and their responses should be visually linked (indentation, shared border).
|
|
23
|
+
- **Collapse what doesn't help judgment.** Verbose tool response JSON, intermediate reasoning steps, and debugging context go behind toggles.
|
|
24
|
+
- **Highlight what matters most.** Make the primary content reviewers judge visually dominant. Bold key entities (prices, dates, names). Use font size and spacing to create hierarchy.
|
|
25
|
+
- **Show the full trace.** Include all intermediate steps (tool calls, retrieved context, reasoning), not just the final output. Collapse them by default but keep them accessible.
|
|
26
|
+
- **Sanitize rendered content.** Strip raw HTML from LLM outputs before rendering. Disable images in rendered markdown if they could be tracking pixels.
|
|
27
|
+
|
|
28
|
+
## Feedback Collection
|
|
29
|
+
|
|
30
|
+
Annotate at the trace level. The reviewer judges the whole trace, not individual spans.
|
|
31
|
+
|
|
32
|
+
- Binary Pass/Fail buttons as the primary action.
|
|
33
|
+
- Free-text notes field for the reviewer to describe what went wrong (or right).
|
|
34
|
+
- Defer button for uncertain cases.
|
|
35
|
+
- Auto-save on every action.
|
|
36
|
+
|
|
37
|
+
Once you have established failure categories from error analysis, you can later add predefined failure mode tags as clickable checkboxes, dropdowns or picklists so reviewers can select from known categories in addition to writing notes. But don't add these in the initial build.
|
|
38
|
+
|
|
39
|
+
## Navigation and Status
|
|
40
|
+
|
|
41
|
+
- Next/Previous buttons and keyboard arrow keys.
|
|
42
|
+
- Trace counter showing position and progress ("12 of 87 remaining").
|
|
43
|
+
- Jump to specific trace by ID.
|
|
44
|
+
- Counts of labeled vs unlabeled traces.
|
|
45
|
+
|
|
46
|
+
## Keyboard Shortcuts
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
Arrow keys = Navigate traces
|
|
50
|
+
1 = Pass 2 = Fail
|
|
51
|
+
D = Defer U = Undo last action
|
|
52
|
+
Cmd+S = Save Cmd+Enter = Save and next
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Selecting Traces to Load
|
|
56
|
+
|
|
57
|
+
Build the app to accept traces from any source (JSON/CSV file). Keep sampling logic outside the app in a separate script. Start with random sampling.
|
|
58
|
+
|
|
59
|
+
## Additional Features
|
|
60
|
+
|
|
61
|
+
**Reference panel:** Toggle-able panel showing ground truth, expected answers, or rubric definitions alongside the trace.
|
|
62
|
+
|
|
63
|
+
**Filtering:** Filter traces by metadata dimensions relevant to the product (channel, user type, pipeline version).
|
|
64
|
+
|
|
65
|
+
**Clustering:** Group traces by metadata or semantic similarity. Show representative traces per cluster with drill-down.
|
|
66
|
+
|
|
67
|
+
## Design Checklist
|
|
68
|
+
|
|
69
|
+
- [ ] Same layout, controls, and terminology on every trace
|
|
70
|
+
- [ ] Pass and Fail buttons are visually distinct (color, size)
|
|
71
|
+
- [ ] Keyboard shortcuts work for all primary actions
|
|
72
|
+
- [ ] Full trace accessible even when sections are collapsed
|
|
73
|
+
- [ ] Labels persist automatically without explicit save
|
|
74
|
+
- [ ] Trace-level annotation (not span-level) as the default
|
|
75
|
+
- [ ] All data rendered in its native format (markdown as HTML, code with highlighting, JSON pretty-printed, tables as HTML tables, URLs as clickable links)
|
|
76
|
+
|
|
77
|
+
## Testing
|
|
78
|
+
|
|
79
|
+
After building the interface, verify it with Playwright.
|
|
80
|
+
|
|
81
|
+
**Visual review:** Take screenshots of the interface with representative trace data loaded. Review each screenshot for:
|
|
82
|
+
- Layout and spacing: is the visual hierarchy clear? Can you immediately see what matters?
|
|
83
|
+
- Readability: is all data rendered in its native format? Are there any raw JSON blobs, unrendered markdown, or unstyled content?
|
|
84
|
+
- Aesthetics: does the interface look professional and clean? Would a domain expert use this?
|
|
85
|
+
- Responsiveness: does the layout hold at different window sizes?
|
|
86
|
+
|
|
87
|
+
**Functional test:** Write a Playwright script that performs a full annotation workflow:
|
|
88
|
+
1. Load the app and verify traces are displayed
|
|
89
|
+
2. Click Pass on a trace, verify the label is saved
|
|
90
|
+
3. Click Fail on a trace, add a note, verify both are saved
|
|
91
|
+
4. Click Defer, verify it is recorded
|
|
92
|
+
5. Navigate forward and backward with buttons and keyboard shortcuts
|
|
93
|
+
6. Verify the trace counter updates correctly
|
|
94
|
+
7. Verify auto-save by reloading the page and checking labels persist
|
|
95
|
+
8. Expand collapsed sections (system prompts, tool calls) and verify content is accessible
|
|
96
|
+
9. Test that all keyboard shortcuts trigger the correct actions
|