testdriverai 7.1.3 → 7.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (78) hide show
  1. package/.github/workflows/acceptance.yaml +81 -0
  2. package/.github/workflows/publish.yaml +44 -0
  3. package/.github/workflows/test-init.yml +145 -0
  4. package/agent/index.js +18 -19
  5. package/agent/lib/commander.js +2 -2
  6. package/agent/lib/commands.js +324 -124
  7. package/agent/lib/redraw.js +99 -39
  8. package/agent/lib/sandbox.js +98 -6
  9. package/agent/lib/sdk.js +25 -0
  10. package/agent/lib/system.js +2 -1
  11. package/agent/lib/validation.js +6 -6
  12. package/docs/docs.json +211 -101
  13. package/docs/snippets/tests/type-repeated-replay.mdx +1 -1
  14. package/docs/v7/_drafts/caching-selectors.mdx +24 -0
  15. package/docs/v7/_drafts/migration.mdx +3 -3
  16. package/docs/v7/api/act.mdx +2 -2
  17. package/docs/v7/api/assert.mdx +2 -2
  18. package/docs/v7/api/assertions.mdx +21 -21
  19. package/docs/v7/api/elements.mdx +78 -0
  20. package/docs/v7/api/find.mdx +38 -0
  21. package/docs/v7/api/focusApplication.mdx +2 -2
  22. package/docs/v7/api/hover.mdx +2 -2
  23. package/docs/v7/features/ai-native.mdx +57 -71
  24. package/docs/v7/features/application-logs.mdx +353 -0
  25. package/docs/v7/features/browser-logs.mdx +414 -0
  26. package/docs/v7/features/cache-management.mdx +402 -0
  27. package/docs/v7/features/continuous-testing.mdx +346 -0
  28. package/docs/v7/features/coverage.mdx +508 -0
  29. package/docs/v7/features/data-driven-testing.mdx +441 -0
  30. package/docs/v7/features/easy-to-write.mdx +2 -73
  31. package/docs/v7/features/enterprise.mdx +155 -39
  32. package/docs/v7/features/fast.mdx +63 -81
  33. package/docs/v7/features/managed-sandboxes.mdx +384 -0
  34. package/docs/v7/features/network-monitoring.mdx +568 -0
  35. package/docs/v7/features/observable.mdx +3 -22
  36. package/docs/v7/features/parallel-execution.mdx +381 -0
  37. package/docs/v7/features/powerful.mdx +1 -1
  38. package/docs/v7/features/reports.mdx +414 -0
  39. package/docs/v7/features/sandbox-customization.mdx +229 -0
  40. package/docs/v7/features/scalable.mdx +217 -2
  41. package/docs/v7/features/stable.mdx +106 -147
  42. package/docs/v7/features/system-performance.mdx +616 -0
  43. package/docs/v7/features/test-analytics.mdx +373 -0
  44. package/docs/v7/features/test-cases.mdx +393 -0
  45. package/docs/v7/features/test-replays.mdx +408 -0
  46. package/docs/v7/features/test-reports.mdx +308 -0
  47. package/docs/v7/getting-started/{running-and-debugging.mdx → debugging-tests.mdx} +12 -142
  48. package/docs/v7/getting-started/quickstart.mdx +22 -305
  49. package/docs/v7/getting-started/running-tests.mdx +173 -0
  50. package/docs/v7/overview/readme.mdx +1 -1
  51. package/docs/v7/overview/what-is-testdriver.mdx +2 -14
  52. package/docs/v7/presets/chrome-extension.mdx +147 -122
  53. package/interfaces/cli/commands/init.js +78 -20
  54. package/interfaces/cli/lib/base.js +3 -2
  55. package/interfaces/logger.js +0 -2
  56. package/interfaces/shared-test-state.mjs +0 -5
  57. package/interfaces/vitest-plugin.mjs +69 -42
  58. package/lib/core/Dashcam.js +65 -66
  59. package/lib/vitest/hooks.mjs +42 -50
  60. package/manual/test-init-command.js +223 -0
  61. package/package.json +2 -2
  62. package/schema.json +5 -5
  63. package/sdk-log-formatter.js +351 -176
  64. package/sdk.d.ts +8 -8
  65. package/sdk.js +436 -121
  66. package/setup/aws/cloudformation.yaml +2 -2
  67. package/setup/aws/self-hosted.yml +1 -1
  68. package/test/testdriver/chrome-extension.test.mjs +55 -72
  69. package/test/testdriver/element-not-found.test.mjs +2 -1
  70. package/test/testdriver/hover-image.test.mjs +1 -1
  71. package/test/testdriver/hover-text-with-description.test.mjs +0 -3
  72. package/test/testdriver/scroll-until-text.test.mjs +10 -6
  73. package/test/testdriver/setup/lifecycleHelpers.mjs +19 -24
  74. package/test/testdriver/setup/testHelpers.mjs +18 -23
  75. package/vitest.config.mjs +3 -3
  76. package/.github/workflows/linux-tests.yml +0 -28
  77. package/docs/v7/getting-started/generating-tests.mdx +0 -525
  78. package/test/testdriver/auto-cache-key-demo.test.mjs +0 -56
@@ -1,53 +1,169 @@
1
1
  ---
2
- title: "Enterprise Ready"
3
- description: "Security, compliance, and scale for the largest organizations"
4
- icon: "building"
2
+ title: 'Enterprise'
3
+ description: 'Enterprise-grade testing infrastructure with enhanced security, dedicated support, and custom environments'
4
+ icon: 'building'
5
5
  ---
6
6
 
7
- TestDriver Enterprise provides the security, control, and scalability that large organizations demand.
7
+ ## Overview
8
8
 
9
- ## On-Premise Deployment
9
+ TestDriver Enterprise provides organizations with a comprehensive testing platform that includes advanced security controls, dedicated onboarding, priority support, and custom environment provisioning.
10
10
 
11
- Run TestDriver entirely within your infrastructure:
11
+ ## Security & Infrastructure
12
12
 
13
- <Card title="Deployment Options" icon="server">
14
- - **Air-Gapped Installation** - No external connectivity required
15
- - **Custom VPC** - Deploy in your AWS/Azure/GCP environment
16
- - **Kubernetes** - Helm charts for container orchestration
17
- - **Docker Compose** - Simple self-hosted deployment
18
- - **Bare Metal** - Install on your own hardware
19
- </Card>
13
+ ### Controlled Ingress/Egress
20
14
 
21
- ```yaml docker-compose.yml
22
- version: '3.8'
23
- services:
24
- testdriver-api:
25
- image: testdriver/api:latest
26
- environment:
27
- - LICENSE_KEY=${TESTDRIVER_LICENSE}
28
- - DATABASE_URL=${DATABASE_URL}
29
- - REDIS_URL=${REDIS_URL}
30
- ports:
31
- - "3000:3000"
15
+ Enterprise customers have full control over network traffic in their testing environments:
32
16
 
33
- testdriver-worker:
34
- image: testdriver/worker:latest
35
- environment:
36
- - API_URL=http://testdriver-api:3000
37
- volumes:
38
- - /var/run/docker.sock:/var/run/docker.sock
39
-
40
- testdriver-scheduler:
41
- image: testdriver/scheduler:latest
42
- environment:
43
- - API_URL=http://testdriver-api:3000
44
- ```
17
+ - Configure custom firewall rules and network policies
18
+ - Whitelist specific domains and IP ranges
19
+ - Monitor and audit all network activity
20
+ - Implement zero-trust network architecture
21
+
22
+ ### Bring Your Own Key (BYOK)
23
+
24
+ Maintain complete control over your AI model access:
25
+
26
+ - **AWS Integration**: Use your own AWS credentials and keys
27
+ - **Claude/Anthropic**: Connect using your own Anthropic API keys
28
+ - **OpenAI**: Integrate with your existing OpenAI account
29
+ - All AI requests use your keys, ensuring data sovereignty
30
+
31
+ ### Your Own VPC on AWS
32
+
33
+ Deploy TestDriver infrastructure directly in your AWS environment:
34
+
35
+ - Complete isolation within your own Virtual Private Cloud
36
+ - Full control over networking, security groups, and access policies
37
+ - Compliance with internal security requirements
38
+ - Integration with existing AWS infrastructure
39
+
40
+ ### CloudFormation Deployment
41
+
42
+ Quick deployment using infrastructure as code:
43
+
44
+ - Pre-configured CloudFormation templates for self-hosted setup
45
+ - Automated provisioning of all required resources
46
+ - See our [Self-Hosting Guide](/v6/getting-started/self-hosting) for deployment instructions
47
+ - Customizable templates to match your infrastructure standards
48
+
49
+ ## Onboarding & Training Program
50
+
51
+ ### Guided Onboarding
52
+
53
+ Our team works directly with yours to ensure success:
54
+
55
+ 1. **Initial Setup**: We help configure your first test environment
56
+ 2. **First Tests**: Walk through creating your first automated tests together
57
+ 3. **Best Practices**: Learn TestDriver patterns and techniques specific to your application
58
+ 4. **Knowledge Transfer**: Educational sessions tailored to your team's needs
59
+
60
+ ### Continuous Education
61
+
62
+ Ongoing training as your testing needs evolve:
63
+
64
+ - Regular check-ins and training sessions
65
+ - Access to advanced feature workshops
66
+ - Early access to new capabilities and features
67
+ - Custom training materials for your use cases
68
+
69
+ ## Support & Roadmap Influence
70
+
71
+ ### Priority Support
72
+
73
+ Get help when you need it:
74
+
75
+ - **Private Slack Channels**: Direct access to TestDriver engineers
76
+ - Faster response times for critical issues
77
+ - Dedicated support team familiar with your setup
78
+ - Screen sharing and pair programming sessions
79
+
80
+ ### Roadmap Influence
81
+
82
+ Help shape the future of TestDriver:
45
83
 
46
- [Complete self-hosting guide](/v7/guides/self-hosting)
84
+ - Input on feature prioritization
85
+ - Early access to beta features
86
+ - Custom feature development consideration
87
+ - Quarterly roadmap review sessions
88
+
89
+ ## Pricing Advantages
90
+
91
+ ### Self-Hosted Sandboxes
92
+
93
+ Reduce costs by running tests in your own infrastructure:
94
+
95
+ - No sandbox usage fees for self-hosted environments
96
+ - Unlimited parallel test execution
97
+ - Pay only for your AWS infrastructure costs
98
+
99
+ ### Unlimited Compute Credits
100
+
101
+ No surprise bills based on usage:
102
+
103
+ - Fixed pricing regardless of test volume
104
+ - Unlimited AI model usage for test generation and maintenance
105
+ - Predictable monthly costs
106
+
107
+ ### Per-Parallel Pricing Model
108
+
109
+ Simple, scalable pricing:
110
+
111
+ - Pay based on concurrent test runners, not usage
112
+ - Scale up or down based on your needs
113
+ - No per-test or per-minute charges
114
+
115
+ ## Custom Environments
116
+
117
+ ### Custom VM Configuration
118
+
119
+ We help you create and maintain customized test environments:
120
+
121
+ - Install specific applications and dependencies
122
+ - Configure system settings and preferences
123
+ - Set up authentication and certificates
124
+ - Prepare baseline snapshots
125
+
126
+ ### AWS AMI Management
127
+
128
+ Custom Amazon Machine Images for your testing needs:
129
+
130
+ - We work with you to build custom AMIs
131
+ - Regular updates and security patches
132
+ - Version control for environment changes
133
+ - Rollback capabilities for stability
134
+
135
+ ### Provisioning API
136
+
137
+ Programmatic environment customization:
138
+
139
+ - Use the [provisioning API](/v7/features/sandbox-customization#provisioning-api) to install software
140
+ - Run setup scripts before tests
141
+ - Configure applications dynamically
142
+ - Learn more in our [Sandbox Customization](/v7/features/sandbox-customization) guide
143
+
144
+ ### Environment Templates
145
+
146
+ Reusable environment configurations:
147
+
148
+ - Create templates for different application types
149
+ - Share configurations across teams
150
+ - Version control for environment specs
151
+ - Quick provisioning of standardized environments
152
+
153
+ ## Getting Started
154
+
155
+ To learn more about TestDriver Enterprise:
156
+
157
+ 1. [Book a demo](https://form.typeform.com/to/UECf9rDx?typeform-source=docs.testdriver.ai) with our team
158
+ 2. Join our [Discord community](https://discord.com/invite/cWDFW8DzPm)
159
+ 3. Contact us through your private Slack channel (existing customers)
160
+
161
+ ## Learn More
47
162
 
48
- ## Hardened Security
163
+ - [Managed Sandboxes](/v7/features/managed-sandboxes)
164
+ - [Sandbox Customization](/v7/features/sandbox-customization)
165
+ - [Self-Hosting Guide](/v6/getting-started/self-hosting)
49
166
 
50
- Enterprise-grade security features:
51
167
 
52
168
  <AccordionGroup>
53
169
  <Accordion title="SOC 2 Type II Certified">
@@ -1,32 +1,36 @@
1
1
  ---
2
- title: "Fast Performance"
2
+ title: "Intelligent Caching"
3
3
  description: "1.7x faster test execution with intelligent caching and optimization"
4
4
  icon: "bolt"
5
5
  ---
6
6
 
7
- Speed matters when you're running hundreds or thousands of tests. TestDriver is engineered for performance with intelligent caching that delivers up to **1.7x faster** test execution.
7
+ Speed matters when you're running hundreds or thousands of tests. TestDriver is engineered for performance with intelligent caching that delivers up to **1.7x faster** test execution by skipping redundant AI vision analysis.
8
8
 
9
9
  ## Test Caching Performance
10
10
 
11
11
  Selector caching dramatically reduces test execution time by skipping redundant AI vision analysis:
12
12
 
13
- <Card title="Real Performance Results" icon="chart-line">
14
- **First run (no cache):**
15
- ```
16
- ⏱️ Test duration: 45.2s
17
- 🔍 AI vision calls: 12
18
- 💰 Cost: $0.048
19
- ```
13
+ ```mermaid
14
+ %%{init: {'theme':'base', 'themeVariables': {'xyChart': {'backgroundColor': 'transparent', 'titleColor': '#ffffff', 'xAxisLabelColor': '#ffffff', 'xAxisTitleColor': '#ffffff', 'xAxisTickColor': '#ffffff', 'xAxisLineColor': '#ffffff', 'yAxisLabelColor': '#ffffff', 'yAxisTitleColor': '#ffffff', 'yAxisTickColor': '#ffffff', 'yAxisLineColor': '#ffffff', 'plotColorPalette': '#b3d334, #b3d334'}}}}%%
15
+ xychart-beta
16
+ title "Test Execution Time Comparison"
17
+ x-axis ["First Run (No Cache)", "Cached Run"]
18
+ y-axis "Duration (seconds)" 0 --> 10
19
+ bar [8.2, 4.8]
20
+ ```
21
+
22
+ ```javascript
23
+ // First run: builds cache
24
+ await testdriver.find('submit button');
25
+
26
+ // Second run: exact match
27
+ await testdriver.find('submit button');
28
+ ```
20
29
 
21
- **Cached run:**
22
- ```
23
- ⚡ Test duration: 26.6s (1.7x faster)
24
- ✅ Cache hits: 11/12
25
- 💰 Cost: $0.004 (92% savings)
26
- ```
27
- </Card>
30
+ **Performance Improvement: 1.7x faster** – The more you run your tests, the faster they get. Cache hits are nearly instant, turning slow AI analysis into millisecond lookups.
28
31
 
29
- The more you run your tests, the faster they get. Cache hits are nearly instant, turning slow AI analysis into millisecond lookups.
32
+ - **First run:** 45.2s with 12 AI vision calls
33
+ - **Cached run:** 26.6s with 11/12 cache hits
30
34
 
31
35
  ## Intelligent Cache Strategy
32
36
 
@@ -40,13 +44,6 @@ TestDriver uses a sophisticated three-tier matching system to maximize cache hit
40
44
  - **Accuracy:** 100%
41
45
  - **Use case:** Repeat test runs
42
46
 
43
- ```javascript
44
- // First run: builds cache
45
- await testdriver.find('submit button'); // 2.3s (AI analysis)
46
-
47
- // Second run: exact match
48
- await testdriver.find('submit button'); // 8ms (cache hit) ⚡
49
- ```
50
47
  </Step>
51
48
 
52
49
  <Step title="Pixel Diff Match (Fast)">
@@ -55,13 +52,6 @@ TestDriver uses a sophisticated three-tier matching system to maximize cache hit
55
52
  - **Speed:** Very fast (< 50ms)
56
53
  - **Accuracy:** 95%+ similarity threshold
57
54
  - **Use case:** Minor UI changes (hover states, animations)
58
-
59
- ```javascript
60
- // Cache works even with small visual changes
61
- await testdriver.find('button', {
62
- cacheThreshold: 0.05 // Allow 5% difference
63
- });
64
- ```
65
55
  </Step>
66
56
 
67
57
  <Step title="Template Match (Fallback)">
@@ -70,13 +60,6 @@ TestDriver uses a sophisticated three-tier matching system to maximize cache hit
70
60
  - **Speed:** Fast (< 200ms)
71
61
  - **Accuracy:** 75%+ confidence
72
62
  - **Use case:** Layout changes, responsive design
73
-
74
- ```javascript
75
- // Still uses cache even when layout shifts
76
- await testdriver.find('submit button', {
77
- cacheThreshold: 0.10 // More lenient
78
- });
79
- ```
80
63
  </Step>
81
64
  </Steps>
82
65
 
@@ -102,9 +85,19 @@ test('auto-cached test', async (context) => {
102
85
  ```
103
86
 
104
87
  <Tip>
105
- TestDriver automatically generates cache keys based on your test file hash. Each file has isolated caching, preventing cross-test pollution.
88
+ **File-Based Cache Keys:** TestDriver automatically generates cache keys from a hash of the test file calling the TestDriver methods. This means each test file has its own isolated cache namespace, preventing cache pollution between different tests while enabling fast lookups within the same test file.
106
89
  </Tip>
107
90
 
91
+ ### How Cache Keys Work
92
+
93
+ The cache key is computed from:
94
+ - **File hash**: SHA-256 hash of the test file contents
95
+ - **Selector prompt**: The exact text description passed to `find()`
96
+ - **Screenshot context**: Perceptual hash of the current screen state
97
+ - **Platform**: Operating system and browser version
98
+
99
+ When you modify your test file, the hash changes automatically, invalidating stale cache entries and ensuring fresh AI analysis with your updated test logic.
100
+
108
101
  ## Cache Hit Monitoring
109
102
 
110
103
  Track cache performance in real-time:
@@ -187,47 +180,6 @@ Fine-tune cache behavior for your application:
187
180
  </Tab>
188
181
  </Tabs>
189
182
 
190
- ## Parallel Execution
191
-
192
- Run tests concurrently for maximum throughput:
193
-
194
- ```javascript vitest.config.mjs
195
- import { defineConfig } from 'vitest/config';
196
-
197
- export default defineConfig({
198
- test: {
199
- testTimeout: 120000,
200
- maxConcurrency: 10, // Run 10 tests simultaneously
201
- },
202
- });
203
- ```
204
-
205
- <CardGroup cols={2}>
206
- <Card title="Sequential Execution" icon="arrow-right">
207
- ```
208
- Test 1: 30s
209
- Test 2: 30s
210
- Test 3: 30s
211
- Total: 90s
212
- ```
213
- </Card>
214
-
215
- <Card title="Parallel Execution (3x)" icon="arrows-split-up-and-left">
216
- ```
217
- Test 1: 30s
218
- Test 2: 30s } concurrent
219
- Test 3: 30s
220
- Total: 30s
221
- ```
222
- </Card>
223
- </CardGroup>
224
-
225
- With caching + parallelization:
226
- ```
227
- ✅ Cache-accelerated parallel execution:
228
- Total: 15s (6x faster than sequential, no cache)
229
- ```
230
-
231
183
  ## Cache Persistence
232
184
 
233
185
  Cache entries are stored server-side with intelligent expiration:
@@ -311,7 +263,29 @@ View cache performance in the TestDriver console:
311
263
  Automatic file-based cache keys prevent cross-test issues while enabling caching.
312
264
  </Accordion>
313
265
 
314
- <Accordion title="4. Monitor Cache Hit Rates">
266
+ <Accordion title="4. Use Custom Cache Keys with Variables">
267
+ ```javascript
268
+ // ❌ Without cache key - creates new cache for each variable value
269
+ const email = 'user@example.com';
270
+ await testdriver.find(`input for ${email}`); // Cache miss every time
271
+
272
+ // ✅ With cache key - reuses cache regardless of variable
273
+ const email = 'user@example.com';
274
+ await testdriver.find(`input for ${email}`, {
275
+ cacheKey: 'email-input'
276
+ });
277
+
278
+ // Also useful for dynamic IDs, names, or other changing data
279
+ const orderId = generateOrderId();
280
+ await testdriver.find(`order ${orderId} status`, {
281
+ cacheKey: 'order-status' // Same cache for all orders
282
+ });
283
+ ```
284
+
285
+ Custom cache keys prevent cache pollution when using variables in prompts, dramatically improving cache hit rates.
286
+ </Accordion>
287
+
288
+ <Accordion title="5. Monitor Cache Hit Rates">
315
289
  ```javascript
316
290
  test('track cache performance', async (context) => {
317
291
  const { testdriver } = await chrome(context, { url });
@@ -390,6 +364,14 @@ const testdriver = new TestDriver({
390
364
  ## Learn More
391
365
 
392
366
  <CardGroup cols={2}>
367
+ <Card
368
+ title="Parallel Execution"
369
+ icon="layer-group"
370
+ href="/v7/features/parallel-execution"
371
+ >
372
+ Run tests concurrently for maximum speed
373
+ </Card>
374
+
393
375
  <Card
394
376
  title="Selector Caching Guide"
395
377
  icon="crosshairs"