il-e2e-agent 0.1.0 → 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +155 -63
  3. package/package.json +3 -1
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Vishal Mishra
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -14,6 +14,12 @@ test('a visitor reaches checkout', async ({ app, agent, browser }) => {
14
14
  });
15
15
  ```
16
16
 
17
+ One command starts your dev server, runs the tests and stops the server again:
18
+
19
+ ```bash
20
+ npm run test:e2e
21
+ ```
22
+
17
23
  ## Contents
18
24
 
19
25
  - [How it works](#how-it-works)
@@ -58,114 +64,191 @@ Agent steps count against **your own** Claude plan's usage limits.
58
64
 
59
65
  ## Installation
60
66
 
61
- Install as a dev dependency of your app:
62
-
63
67
  ```bash
64
- npm install --save-dev github:vishalmishraa22/il-e2e-agent
68
+ npm install --save-dev il-e2e-agent
65
69
  npx playwright install chromium
66
70
  ```
67
71
 
68
- Then create this layout in your project:
72
+ Project layout:
69
73
 
70
74
  ```
71
75
  your-app/
72
- ├── package.json add a "test:e2e" script (below)
76
+ ├── package.json il-e2e-agent devDependency + "test:e2e" script
77
+ ├── il-e2e-agent.config.ts dev-server command, network policy
78
+ ├── .gitignore + the two lines below
73
79
  └── e2e/
74
- ├── il-e2e-agent.config.ts app URL and network policy
75
80
  ├── tests/
76
- │ └── *.e2e.ts your tests
77
- ├── support/ optional shared helpers
78
- └── .gitignore
81
+ │ └── *.e2e.ts your tests
82
+ └── support/ optional shared helpers
79
83
  ```
80
84
 
81
- `e2e/.gitignore`. Run output stays local; the replay cache is committed so teammates replay recorded steps:
82
-
83
- ```gitignore
84
- .e2e/*
85
- !.e2e/cache/
86
- ```
87
-
88
- `package.json` scripts:
85
+ `package.json`:
89
86
 
90
87
  ```json
91
88
  {
92
89
  "scripts": {
93
- "test:e2e": "il-e2e-agent run --config e2e/il-e2e-agent.config.ts",
94
- "test:e2e:fresh": "il-e2e-agent run --config e2e/il-e2e-agent.config.ts --no-cache"
90
+ "test:e2e": "il-e2e-agent run",
91
+ "test:e2e:fresh": "il-e2e-agent run --no-cache"
95
92
  }
96
93
  }
97
94
  ```
98
95
 
96
+ `.gitignore`: run output stays local; the replay cache is committed so teammates replay recorded steps:
97
+
98
+ ```gitignore
99
+ .e2e/*
100
+ !.e2e/cache/
101
+ ```
102
+
103
+ `il-e2e-agent run` reads `il-e2e-agent.config.ts` from the directory it runs in, which for an npm
104
+ script is the project root.
105
+
99
106
  ## Framework setup
100
107
 
101
- Tests run against your app's **dev server**. Pick a port for tests (3100 below) so a test run never
102
- collides with the dev server you already use, and start it before running the tests. Alternatively,
103
- let the runner start it with `app.command` (see [Configuration](#configuration)).
108
+ The runner starts your dev server for each run and stops it when the run ends, whether it passed,
109
+ failed or was interrupted. Use `url: 'http://127.0.0.1:0'`: port `0` makes the runner pick a free
110
+ port every time and pass it to your command as `{port}`, so a test run never collides with a dev
111
+ server you already have open. Bind to `127.0.0.1` explicitly, because `localhost` can resolve to IPv6
112
+ only.
113
+
114
+ Avoid a fixed port. Some dev servers silently move to another port when the requested one is taken,
115
+ and the run would then test whatever else answers on it.
116
+
117
+ Every example below also uses `readyUrl`, a page the runner waits for before starting tests, so the
118
+ first compile happens before the first test.
104
119
 
105
120
  ### Next.js
106
121
 
107
- ```json
108
- { "scripts": { "dev:e2e": "next dev -p 3100" } }
122
+ ```ts
123
+ // il-e2e-agent.config.ts
124
+ import { defineConfig } from 'il-e2e-agent';
125
+
126
+ export default defineConfig({
127
+ tests: ['e2e/**/*.e2e.ts'],
128
+ app: {
129
+ url: 'http://127.0.0.1:0',
130
+ command: { executable: 'npx', args: ['next', 'dev', '-H', '127.0.0.1', '-p', '{port}'], startupTimeout: 300_000 },
131
+ readyUrl: 'http://127.0.0.1:{port}/',
132
+ },
133
+ });
109
134
  ```
110
135
 
111
- `next build` type-checks every `.ts` file that `tsconfig.json` includes. Exclude the test folder so
112
- production builds never depend on test code:
136
+ `next build` type-checks every `.ts` file `tsconfig.json` includes. Exclude the test files so production
137
+ builds never depend on them:
113
138
 
114
139
  ```jsonc
115
140
  // tsconfig.json
116
- { "exclude": ["node_modules", "e2e"] }
141
+ { "exclude": ["node_modules", "e2e", "il-e2e-agent.config.ts"] }
117
142
  ```
118
143
 
119
144
  ### Nuxt
120
145
 
121
- ```json
122
- { "scripts": { "dev:e2e": "nuxt dev --port 3100" } }
146
+ ```ts
147
+ // il-e2e-agent.config.ts
148
+ import { defineConfig } from 'il-e2e-agent';
149
+
150
+ export default defineConfig({
151
+ tests: ['e2e/**/*.e2e.ts'],
152
+ app: {
153
+ url: 'http://127.0.0.1:0',
154
+ command: { executable: 'npx', args: ['nuxt', 'dev', '--host', '127.0.0.1', '--port', '{port}'], startupTimeout: 300_000 },
155
+ readyUrl: 'http://127.0.0.1:{port}/',
156
+ },
157
+ });
123
158
  ```
124
159
 
125
- Add the test folder to `.nuxtignore` so the dev server's file watcher skips it (test runs write files
126
- under `e2e/.e2e/`):
160
+ Keep the dev server's file watcher away from the tests and their output. In `.nuxtignore`:
127
161
 
128
162
  ```gitignore
129
163
  e2e/**
164
+ .e2e/**
130
165
  ```
131
166
 
132
- If CI runs `nuxi typecheck`, also exclude `e2e/` there through `typescript.tsConfig.exclude` in
133
- `nuxt.config.ts`.
167
+ If CI runs `nuxi typecheck`, exclude `e2e/` and `il-e2e-agent.config.ts` there through
168
+ `typescript.tsConfig.exclude` in `nuxt.config.ts`.
134
169
 
135
170
  ### React (Vite)
136
171
 
137
- ```json
138
- { "scripts": { "dev:e2e": "vite --port 3100 --strictPort" } }
172
+ ```ts
173
+ // il-e2e-agent.config.ts
174
+ import { defineConfig } from 'il-e2e-agent';
175
+
176
+ export default defineConfig({
177
+ tests: ['e2e/**/*.e2e.ts'],
178
+ app: {
179
+ url: 'http://127.0.0.1:0',
180
+ command: { executable: 'npx', args: ['vite', '--host', '127.0.0.1', '--port', '{port}', '--strictPort'] },
181
+ readyUrl: 'http://127.0.0.1:{port}/',
182
+ },
183
+ });
139
184
  ```
140
185
 
141
- No further changes are needed with the default Vite templates: `tsconfig.app.json` only includes
142
- `src/`, and Vite's watcher only reacts to files your app imports.
186
+ No other changes are needed with the default Vite templates: their TypeScript configs only include
187
+ `src/` and `vite.config.ts`.
143
188
 
144
189
  ### Create React App
145
190
 
146
- ```json
147
- { "scripts": { "dev:e2e": "PORT=3100 BROWSER=none react-scripts start" } }
191
+ ```ts
192
+ // il-e2e-agent.config.ts
193
+ import { defineConfig } from 'il-e2e-agent';
194
+
195
+ export default defineConfig({
196
+ tests: ['e2e/**/*.e2e.ts'],
197
+ app: {
198
+ url: 'http://127.0.0.1:0',
199
+ command: {
200
+ executable: 'npx',
201
+ args: ['react-scripts', 'start'],
202
+ env: { HOST: '127.0.0.1', PORT: '{port}', BROWSER: 'none' },
203
+ startupTimeout: 300_000,
204
+ },
205
+ readyUrl: 'http://127.0.0.1:{port}/',
206
+ },
207
+ });
208
+ ```
209
+
210
+ ### Using your own dev script
211
+
212
+ To run the dev server through an npm script, for example to set environment flags, pass the host and
213
+ port after `--`:
214
+
215
+ ```ts
216
+ command: { executable: 'npm', args: ['run', 'dev', '--', '--host', '127.0.0.1', '--port', '{port}'] },
148
217
  ```
149
218
 
219
+ The command runs without a shell and inherits only `PATH`, `HOME` and the temp-directory variables.
220
+ Pass anything else it needs through `command.env`.
221
+
150
222
  ### Server-side tracking
151
223
 
152
- The network policy below controls what the **test browser** can reach. Calls your **server** makes
153
- (for example a Next.js route handler or Nuxt server route forwarding to an analytics or CRM API)
154
- are outside its reach. If your app does this in development, turn it off in the `dev:e2e` script
155
- with your app's own environment flags, and block the forwarding routes with `blockPaths`.
224
+ The [network policy](#network-policy) controls what the **test browser** can reach. Calls your
225
+ **server** makes (for example a route handler forwarding events to an analytics or CRM API) are outside
226
+ its reach. If your app does this in development, switch it off for test runs with your app's own
227
+ environment flags in `command.env`, and block the forwarding routes with `blockPaths`.
156
228
 
157
229
  ## Configuration
158
230
 
159
- `e2e/il-e2e-agent.config.ts`:
231
+ A complete `il-e2e-agent.config.ts`:
160
232
 
161
233
  ```ts
162
234
  import { defineConfig } from 'il-e2e-agent';
163
235
 
164
236
  export default defineConfig({
165
- app: { url: 'http://localhost:3100' },
237
+ tests: ['e2e/**/*.e2e.ts'],
166
238
  timeout: 600_000,
167
239
  actionTimeout: 8_000,
168
240
  maxSteps: 40,
241
+ app: {
242
+ url: 'http://127.0.0.1:0',
243
+ command: {
244
+ executable: 'npx',
245
+ args: ['nuxt', 'dev', '--host', '127.0.0.1', '--port', '{port}'],
246
+ env: { ANALYTICS_ENABLED: 'false' },
247
+ startupTimeout: 300_000,
248
+ log: '.e2e/logs/dev-server.log',
249
+ },
250
+ readyUrl: 'http://127.0.0.1:{port}/',
251
+ },
169
252
  network: {
170
253
  allow: ['*.stripe.com', '*.stripe.network', 'fonts.googleapis.com', 'fonts.gstatic.com'],
171
254
  readOnly: ['api-staging.example.com'],
@@ -180,13 +263,16 @@ except `targets` and `agents`, which it builds for you, plus:
180
263
 
181
264
  | Option | Default | Description |
182
265
  | --- | --- | --- |
183
- | `app` | required | `url` of the app under test. Optionally `command` to let the runner start it, e.g. `{ executable: 'npm', args: ['run', 'dev:e2e'], cwd: '..', reuseExisting: true }`. |
266
+ | `app` | required | `url` of the app, `command` that starts it, and `readyUrl` to wait for. See [Framework setup](#framework-setup). |
267
+ | `tests` | `e2e/**/*.e2e.ts`, `tests/**/*.e2e.ts` | Test file globs, relative to the config file. |
184
268
  | `model` | `'sonnet'` | Claude model for agent steps: `'sonnet'`, `'haiku'` or `'opus'`. |
185
269
  | `effort` | `'low'` | Reasoning effort per agent step. Higher values think longer on every turn. |
186
270
  | `maxSteps` | `25` | Actions a single `agent.act` may take. Raise it for long multi-screen steps. |
187
271
  | `browser` | e2e defaults | Playwright web-engine options, e.g. `{ viewport: { width: 1280, height: 900 } }`. |
188
272
  | `network` | localhost only | Which hosts the test browser may reach. `false` disables the guard. |
189
273
 
274
+ `command.log` keeps the dev server's output in a file instead of discarding it.
275
+
190
276
  ### Network policy
191
277
 
192
278
  Every request the test browser makes is checked, and anything not explicitly allowed is aborted.
@@ -245,30 +331,29 @@ documented at [e2e.tester.army/docs](https://e2e.tester.army/docs).
245
331
  ## Running tests
246
332
 
247
333
  ```bash
248
- npm run dev:e2e # terminal 1: the app on the test port
249
-
250
- npm run test:e2e # terminal 2: all tests
251
- npm run test:e2e -- e2e/tests/checkout.e2e.ts # one file
252
- npm run test:e2e -- --headed # watch the browser
253
- npm run test:e2e -- --reporter list,markdown # writes e2e/.e2e/summary.md and failure pages
254
- npm run test:e2e:fresh # ignore recordings and run every step live
334
+ npm run test:e2e # all tests
335
+ npm run test:e2e -- e2e/tests/checkout.e2e.ts # one file, path relative to the project root
336
+ npm run test:e2e -- --headed # watch the browser
337
+ npm run test:e2e -- --reporter list,markdown # also writes .e2e/summary.md and failure pages
338
+ npm run test:e2e:fresh # ignore recordings and run every step live
255
339
  ```
256
340
 
257
- Failure details, screenshots and traces are written to `e2e/.e2e/`. Start with
258
- `e2e/.e2e/failures/` when a test fails.
341
+ Failure details, screenshots and traces are written to `.e2e/`. Start with `.e2e/failures/` when a
342
+ test fails, and `.e2e/logs/dev-server.log` (if `command.log` is set) when the app didn't start.
259
343
 
260
344
  The CLI sets `E2E_TELEMETRY_DISABLED=1` unless you set it yourself.
261
345
 
262
346
  ## Replay cache
263
347
 
264
- - **First run of a step:** Claude performs it live, and the actions are recorded in `e2e/.e2e/cache/`.
348
+ - **First run of a step:** Claude performs it live, and the actions are recorded in `.e2e/cache/`.
265
349
  - **Later runs:** the recording is replayed with no model call.
266
350
  - **The page changes:** the replay hands over to Claude where it stopped matching, and the step is
267
351
  recorded again.
268
- - **Changing a test's name, an instruction or its params** starts that step from scratch.
352
+ - **Renaming or moving a test file, changing a test's name, an instruction or its params** starts that
353
+ step from scratch. Recordings are keyed by the test's path relative to the config file.
269
354
 
270
- **Commit `e2e/.e2e/cache/`** so everyone on the team replays the same recordings instead of running
271
- every step live on their own plan. Review cache changes in pull requests like any other test data.
355
+ **Commit `.e2e/cache/`** so everyone on the team replays the same recordings instead of running every
356
+ step live on their own plan. Review cache changes in pull requests like any other test data.
272
357
 
273
358
  ## Keeping it out of production builds
274
359
 
@@ -281,6 +366,8 @@ npm pkg delete devDependencies.il-e2e-agent && npm install --include=dev
281
366
  ```
282
367
 
283
368
  This changes the build's working copy only. Verified with both `npm install` and `npm ci` (npm 11).
369
+ Also exclude `il-e2e-agent.config.ts` and `e2e/` from any type-check your build runs (see
370
+ [Next.js](#nextjs)), since the package isn't installed there.
284
371
 
285
372
  AWS Amplify (`amplify.yml`):
286
373
 
@@ -309,12 +396,13 @@ GitHub Actions:
309
396
  | Symptom | Cause and fix |
310
397
  | --- | --- |
311
398
  | `Claude Code was not found` | Install Claude Code and run `claude auth login`, or set `IL_E2E_CLAUDE_PATH`. |
312
- | `il-e2e-agent.config.ts not found` | Pass `--config e2e/il-e2e-agent.config.ts`, as the scripts above do, or run from the folder that holds it. |
313
- | `APP_UNREACHABLE` | The app isn't running on the configured URL. Start `npm run dev:e2e` first, or configure `app.command`. |
399
+ | `il-e2e-agent.config.ts not found` | Run from the project root (npm scripts do), or pass `--config <path>`. |
400
+ | `APP_UNREACHABLE` | The dev server didn't answer `readyUrl` within `startupTimeout`. Read `command.log`; raise `startupTimeout` for slow first compiles. |
401
+ | Tests run against the wrong app | The config uses a fixed port that something else was already using. Use `url: 'http://127.0.0.1:0'` with `{port}`. |
314
402
  | A page renders broken or incomplete | A resource it needs is blocked. Run with `E2E_NETWORK_LOG=1` and add the host to `network.allow`. |
315
403
  | A step takes long or exhausts its steps | Split the instruction into smaller `agent.act` calls, or raise `maxSteps`. |
316
- | A recorded step keeps re-running live | Its instruction, params or test name changed, or a value differs on every run. Use `unique()` for those. |
317
- | The dev server misbehaves while tests run | Exclude `e2e/` from its file watcher (`.nuxtignore` for Nuxt). |
404
+ | A recorded step keeps re-running live | Its instruction, params, test name or file path changed, or a value differs on every run. Use `unique()` for those. |
405
+ | The dev server misbehaves while tests run | Exclude `e2e/` and `.e2e/` from its file watcher (`.nuxtignore` for Nuxt). |
318
406
 
319
407
  ## Limitations
320
408
 
@@ -326,6 +414,10 @@ GitHub Actions:
326
414
  - **`agent.assert`, `waitFor` and `extract` always run live**, so they use your plan on every run.
327
415
  - **Pre-1.0 foundations.** e2e is still before 1.0, and this package pins exact versions of it.
328
416
 
417
+ ## License
418
+
419
+ [MIT](LICENSE). The license covers this package's own code; its dependencies keep their own licenses.
420
+
329
421
  ## Disclaimer
330
422
 
331
423
  il-e2e-agent is an independent project. It is not affiliated with, endorsed by or sponsored by
package/package.json CHANGED
@@ -1,7 +1,9 @@
1
1
  {
2
2
  "name": "il-e2e-agent",
3
- "version": "0.1.0",
3
+ "version": "0.1.2",
4
4
  "description": "Natural-language browser tests (tester-army/e2e) run by the Claude Code installed on each developer's machine, on their own login",
5
+ "license": "MIT",
6
+ "author": "Vishal Mishra",
5
7
  "type": "module",
6
8
  "repository": {
7
9
  "type": "git",