joycraft 0.5.7 → 0.5.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -348
- package/dist/{chunk-G342HURJ.js → chunk-A2CQG5J5.js} +495 -61
- package/dist/{chunk-G342HURJ.js.map → chunk-A2CQG5J5.js.map} +1 -1
- package/dist/cli.js +3 -3
- package/dist/index.d.ts +6 -0
- package/dist/index.js +78 -7
- package/dist/index.js.map +1 -1
- package/dist/{init-7FUTURUY.js → init-QXG5BT4Y.js} +82 -11
- package/dist/init-QXG5BT4Y.js.map +1 -0
- package/dist/{init-autofix-K4B5BD5V.js → init-autofix-Y5DQOFEU.js} +2 -2
- package/dist/{upgrade-2Y7D2HCD.js → upgrade-VUOSXPR5.js} +2 -2
- package/package.json +1 -1
- package/dist/init-7FUTURUY.js.map +0 -1
- /package/dist/{init-autofix-K4B5BD5V.js.map → init-autofix-Y5DQOFEU.js.map} +0 -0
- /package/dist/{upgrade-2Y7D2HCD.js.map → upgrade-VUOSXPR5.js.map} +0 -0
|
@@ -1874,13 +1874,59 @@ is required, though you can add one via the GitHub Checks API if you prefer.
|
|
|
1874
1874
|
|
|
1875
1875
|
---
|
|
1876
1876
|
|
|
1877
|
+
## Testing by Stack Type
|
|
1878
|
+
|
|
1879
|
+
The scenario agent selects the appropriate test format based on the project's
|
|
1880
|
+
testing backbone. Each backbone tests the same holdout principle \u2014 observable
|
|
1881
|
+
behavior only, no source imports \u2014 but uses different tools.
|
|
1882
|
+
|
|
1883
|
+
### Web Apps (Playwright)
|
|
1884
|
+
|
|
1885
|
+
For Next.js, Vite, Nuxt, Remix, and other web frameworks. Tests run against a
|
|
1886
|
+
dev server or preview URL using a headless browser.
|
|
1887
|
+
|
|
1888
|
+
- **Template:** \`example-scenario-web.spec.ts\`
|
|
1889
|
+
- **Config:** \`playwright.config.ts\`
|
|
1890
|
+
- **Package:** \`package-web.json\` (use instead of \`package.json\` for web projects)
|
|
1891
|
+
- **Run:** \`npx playwright test\`
|
|
1892
|
+
|
|
1893
|
+
### Mobile Apps (Maestro)
|
|
1894
|
+
|
|
1895
|
+
For React Native, Flutter, and native iOS/Android. Tests are declarative YAML
|
|
1896
|
+
flows that interact with a running app on a simulator.
|
|
1897
|
+
|
|
1898
|
+
- **Template:** \`example-scenario-mobile.yaml\`
|
|
1899
|
+
- **Login sub-flow:** \`example-scenario-mobile-login.yaml\`
|
|
1900
|
+
- **Setup guide:** \`README-mobile.md\`
|
|
1901
|
+
- **Run:** \`maestro test example-scenario-mobile.yaml\`
|
|
1902
|
+
|
|
1903
|
+
### API Backends (HTTP)
|
|
1904
|
+
|
|
1905
|
+
For Express, FastAPI, Django, and other API-only backends. Tests send HTTP
|
|
1906
|
+
requests using Node.js built-in \`fetch\`.
|
|
1907
|
+
|
|
1908
|
+
- **Template:** \`example-scenario-api.test.ts\`
|
|
1909
|
+
- **Run:** \`npx vitest run\`
|
|
1910
|
+
|
|
1911
|
+
### CLI Tools & Libraries (native)
|
|
1912
|
+
|
|
1913
|
+
For CLI tools, npm packages, and non-UI projects. Tests invoke the built
|
|
1914
|
+
binary via \`spawnSync\` and assert on stdout/stderr.
|
|
1915
|
+
|
|
1916
|
+
- **Template:** \`example-scenario.test.ts\`
|
|
1917
|
+
- **Run:** \`npx vitest run\`
|
|
1918
|
+
|
|
1919
|
+
---
|
|
1920
|
+
|
|
1877
1921
|
## Adding scenarios
|
|
1878
1922
|
|
|
1879
1923
|
### Rules
|
|
1880
1924
|
|
|
1881
|
-
|
|
1882
|
-
|
|
1883
|
-
|
|
1925
|
+
These rules apply to ALL backbones:
|
|
1926
|
+
|
|
1927
|
+
1. **Behavioral, not structural.** Test what the app does from a user's
|
|
1928
|
+
perspective. For web: navigate and assert on content. For CLI: run commands
|
|
1929
|
+
and check output. For API: send requests and check responses.
|
|
1884
1930
|
|
|
1885
1931
|
2. **End-to-end.** Each test should represent something a real user would
|
|
1886
1932
|
actually do. If you would not put it in a demo or docs example, reconsider
|
|
@@ -1890,9 +1936,8 @@ is required, though you can add one via the GitHub Checks API if you prefer.
|
|
|
1890
1936
|
see source code. Any \`import\` that reaches into \`../main-repo/src\` breaks
|
|
1891
1937
|
the pattern.
|
|
1892
1938
|
|
|
1893
|
-
4. **Independent.** Each test must be able to run in isolation.
|
|
1894
|
-
|
|
1895
|
-
state between tests.
|
|
1939
|
+
4. **Independent.** Each test must be able to run in isolation. No shared
|
|
1940
|
+
mutable state between tests.
|
|
1896
1941
|
|
|
1897
1942
|
5. **Deterministic.** Avoid network calls, timestamps, or random values in
|
|
1898
1943
|
assertions unless the feature under test genuinely involves them.
|
|
@@ -1901,31 +1946,25 @@ is required, though you can add one via the GitHub Checks API if you prefer.
|
|
|
1901
1946
|
|
|
1902
1947
|
\`\`\`
|
|
1903
1948
|
$SCENARIOS_REPO/
|
|
1904
|
-
\u251C\u2500\u2500 example-scenario.test.ts
|
|
1949
|
+
\u251C\u2500\u2500 example-scenario.test.ts # CLI/binary scenario template
|
|
1950
|
+
\u251C\u2500\u2500 example-scenario-web.spec.ts # Web app scenario template (Playwright)
|
|
1951
|
+
\u251C\u2500\u2500 example-scenario-api.test.ts # API backend scenario template
|
|
1952
|
+
\u251C\u2500\u2500 example-scenario-mobile.yaml # Mobile app scenario template (Maestro)
|
|
1953
|
+
\u251C\u2500\u2500 example-scenario-mobile-login.yaml # Reusable login sub-flow
|
|
1954
|
+
\u251C\u2500\u2500 playwright.config.ts # Playwright config (web projects)
|
|
1955
|
+
\u251C\u2500\u2500 package.json # Default (vitest for CLI/API)
|
|
1956
|
+
\u251C\u2500\u2500 package-web.json # Alternative (Playwright for web)
|
|
1957
|
+
\u251C\u2500\u2500 README-mobile.md # Mobile testing setup guide
|
|
1905
1958
|
\u251C\u2500\u2500 workflows/
|
|
1906
|
-
\u2502 \
|
|
1907
|
-
\
|
|
1959
|
+
\u2502 \u251C\u2500\u2500 run.yml # CI workflow (do not rename)
|
|
1960
|
+
\u2502 \u2514\u2500\u2500 generate.yml # Scenario generation workflow
|
|
1961
|
+
\u251C\u2500\u2500 prompts/
|
|
1962
|
+
\u2502 \u2514\u2500\u2500 scenario-agent.md # Scenario agent instructions
|
|
1908
1963
|
\u2514\u2500\u2500 README.md
|
|
1909
1964
|
\`\`\`
|
|
1910
1965
|
|
|
1911
|
-
|
|
1912
|
-
|
|
1913
|
-
|
|
1914
|
-
### Example structure
|
|
1915
|
-
|
|
1916
|
-
\`\`\`ts
|
|
1917
|
-
import { spawnSync } from "node:child_process";
|
|
1918
|
-
import { join } from "node:path";
|
|
1919
|
-
|
|
1920
|
-
const CLI = join(__dirname, "..", "main-repo", "dist", "cli.js");
|
|
1921
|
-
|
|
1922
|
-
it("init creates a CLAUDE.md file", () => {
|
|
1923
|
-
const tmp = mkdtempSync(join(tmpdir(), "scenario-"));
|
|
1924
|
-
const { status } = spawnSync("node", [CLI, "init", tmp], { encoding: "utf8" });
|
|
1925
|
-
expect(status).toBe(0);
|
|
1926
|
-
expect(existsSync(join(tmp, "CLAUDE.md"))).toBe(true);
|
|
1927
|
-
});
|
|
1928
|
-
\`\`\`
|
|
1966
|
+
Use the template that matches your project's stack. Remove the ones you
|
|
1967
|
+
don't need.
|
|
1929
1968
|
|
|
1930
1969
|
---
|
|
1931
1970
|
|
|
@@ -1937,6 +1976,7 @@ it("init creates a CLAUDE.md file", () => {
|
|
|
1937
1976
|
| Visible to agent | Yes | No |
|
|
1938
1977
|
| What they test | Units, modules, logic | End-to-end behavior |
|
|
1939
1978
|
| Import source code | Yes | Never |
|
|
1979
|
+
| Test method | Unit test framework | Depends on backbone (Playwright/Maestro/vitest/fetch) |
|
|
1940
1980
|
| Run on every push | Yes | Yes (via dispatch) |
|
|
1941
1981
|
| Purpose | Catch regressions fast | Validate real behavior |
|
|
1942
1982
|
|
|
@@ -2085,6 +2125,304 @@ describe("CLI: init command (example \u2014 replace with your real scenarios)",
|
|
|
2085
2125
|
}
|
|
2086
2126
|
}
|
|
2087
2127
|
`,
|
|
2128
|
+
"scenarios/package-web.json": `{
|
|
2129
|
+
"name": "$SCENARIOS_REPO",
|
|
2130
|
+
"version": "0.0.1",
|
|
2131
|
+
"private": true,
|
|
2132
|
+
"type": "module",
|
|
2133
|
+
"scripts": {
|
|
2134
|
+
"test": "playwright test"
|
|
2135
|
+
},
|
|
2136
|
+
"devDependencies": {
|
|
2137
|
+
"@playwright/test": "^1.50.0"
|
|
2138
|
+
}
|
|
2139
|
+
}
|
|
2140
|
+
`,
|
|
2141
|
+
"scenarios/playwright.config.ts": `import { defineConfig } from '@playwright/test';
|
|
2142
|
+
|
|
2143
|
+
/**
|
|
2144
|
+
* Playwright configuration for holdout scenario tests.
|
|
2145
|
+
*
|
|
2146
|
+
* BASE_URL can be set to test against a preview deployment URL
|
|
2147
|
+
* or defaults to http://localhost:3000 for local dev server testing.
|
|
2148
|
+
*/
|
|
2149
|
+
export default defineConfig({
|
|
2150
|
+
testDir: '.',
|
|
2151
|
+
testMatch: '**/*.spec.ts',
|
|
2152
|
+
timeout: 60_000,
|
|
2153
|
+
retries: 0,
|
|
2154
|
+
use: {
|
|
2155
|
+
baseURL: process.env.BASE_URL || 'http://localhost:3000',
|
|
2156
|
+
headless: true,
|
|
2157
|
+
screenshot: 'only-on-failure',
|
|
2158
|
+
},
|
|
2159
|
+
projects: [
|
|
2160
|
+
{ name: 'chromium', use: { browserName: 'chromium' } },
|
|
2161
|
+
],
|
|
2162
|
+
});
|
|
2163
|
+
`,
|
|
2164
|
+
"scenarios/example-scenario-web.spec.ts": `/**
|
|
2165
|
+
* Example Web Scenario Test (Playwright)
|
|
2166
|
+
*
|
|
2167
|
+
* This file is a template for scenario tests against web applications.
|
|
2168
|
+
* The holdout pattern applies: test the running app through its UI,
|
|
2169
|
+
* never import source code from the main repo.
|
|
2170
|
+
*
|
|
2171
|
+
* The main repo is available at ../main-repo and is already built.
|
|
2172
|
+
* Tests run against either:
|
|
2173
|
+
* - A dev server started from ../main-repo (default)
|
|
2174
|
+
* - A preview deployment URL (set BASE_URL env var)
|
|
2175
|
+
*
|
|
2176
|
+
* DO:
|
|
2177
|
+
* - Navigate to pages, click elements, fill forms, assert on visible content
|
|
2178
|
+
* - Use page.locator() with accessible selectors (role, text, test-id)
|
|
2179
|
+
* - Keep each test fully independent
|
|
2180
|
+
*
|
|
2181
|
+
* DON'T:
|
|
2182
|
+
* - Import from ../main-repo/src \u2014 that defeats the holdout
|
|
2183
|
+
* - Test internal implementation details
|
|
2184
|
+
* - Rely on specific CSS classes or DOM structure (use accessible selectors)
|
|
2185
|
+
*/
|
|
2186
|
+
|
|
2187
|
+
import { test, expect } from '@playwright/test';
|
|
2188
|
+
import { spawn, type ChildProcess } from 'node:child_process';
|
|
2189
|
+
import { join } from 'node:path';
|
|
2190
|
+
|
|
2191
|
+
const MAIN_REPO = join(__dirname, '..', 'main-repo');
|
|
2192
|
+
let serverProcess: ChildProcess | undefined;
|
|
2193
|
+
|
|
2194
|
+
/**
|
|
2195
|
+
* Wait for a URL to become reachable.
|
|
2196
|
+
*/
|
|
2197
|
+
async function waitForServer(url: string, timeoutMs = 60_000): Promise<void> {
|
|
2198
|
+
const start = Date.now();
|
|
2199
|
+
while (Date.now() - start < timeoutMs) {
|
|
2200
|
+
try {
|
|
2201
|
+
const res = await fetch(url);
|
|
2202
|
+
if (res.ok || res.status < 500) return;
|
|
2203
|
+
} catch {
|
|
2204
|
+
// Server not ready yet
|
|
2205
|
+
}
|
|
2206
|
+
await new Promise(r => setTimeout(r, 1000));
|
|
2207
|
+
}
|
|
2208
|
+
throw new Error(\`Server at \${url} did not become ready within \${timeoutMs}ms\`);
|
|
2209
|
+
}
|
|
2210
|
+
|
|
2211
|
+
test.beforeAll(async () => {
|
|
2212
|
+
// If BASE_URL is set, skip starting a dev server \u2014 test against the provided URL
|
|
2213
|
+
if (process.env.BASE_URL) return;
|
|
2214
|
+
|
|
2215
|
+
serverProcess = spawn('npm', ['run', 'dev'], {
|
|
2216
|
+
cwd: MAIN_REPO,
|
|
2217
|
+
stdio: 'pipe',
|
|
2218
|
+
env: { ...process.env, PORT: '3000' },
|
|
2219
|
+
});
|
|
2220
|
+
|
|
2221
|
+
await waitForServer('http://localhost:3000');
|
|
2222
|
+
});
|
|
2223
|
+
|
|
2224
|
+
test.afterAll(async () => {
|
|
2225
|
+
if (serverProcess) {
|
|
2226
|
+
serverProcess.kill('SIGTERM');
|
|
2227
|
+
serverProcess = undefined;
|
|
2228
|
+
}
|
|
2229
|
+
});
|
|
2230
|
+
|
|
2231
|
+
// ---------------------------------------------------------------------------
|
|
2232
|
+
// Example scenarios \u2014 replace with real tests for your application
|
|
2233
|
+
// ---------------------------------------------------------------------------
|
|
2234
|
+
|
|
2235
|
+
test.describe('Home page', () => {
|
|
2236
|
+
test('loads successfully and shows main heading', async ({ page }) => {
|
|
2237
|
+
await page.goto('/');
|
|
2238
|
+
// Replace with your app's actual heading or key element
|
|
2239
|
+
await expect(page.locator('h1')).toBeVisible();
|
|
2240
|
+
});
|
|
2241
|
+
|
|
2242
|
+
test('navigates to a subpage', async ({ page }) => {
|
|
2243
|
+
await page.goto('/');
|
|
2244
|
+
// Replace with your app's actual navigation
|
|
2245
|
+
// await page.click('text=About');
|
|
2246
|
+
// await expect(page).toHaveURL(/\\/about/);
|
|
2247
|
+
// await expect(page.locator('h1')).toContainText('About');
|
|
2248
|
+
});
|
|
2249
|
+
});
|
|
2250
|
+
`,
|
|
2251
|
+
"scenarios/example-scenario-api.test.ts": `/**
|
|
2252
|
+
* Example API Scenario Test
|
|
2253
|
+
*
|
|
2254
|
+
* This file is a template for scenario tests against API-only backends.
|
|
2255
|
+
* The holdout pattern applies: test the running server via HTTP requests,
|
|
2256
|
+
* never import route handlers or source code from the main repo.
|
|
2257
|
+
*
|
|
2258
|
+
* The main repo is available at ../main-repo and is already built.
|
|
2259
|
+
* Tests run against either:
|
|
2260
|
+
* - A server started from ../main-repo (default)
|
|
2261
|
+
* - A deployed URL (set BASE_URL env var)
|
|
2262
|
+
*
|
|
2263
|
+
* Uses Node.js built-in fetch \u2014 no additional HTTP client dependencies.
|
|
2264
|
+
*
|
|
2265
|
+
* DO:
|
|
2266
|
+
* - Send HTTP requests to endpoints, assert on status codes and response bodies
|
|
2267
|
+
* - Test realistic user actions (create, read, update, delete flows)
|
|
2268
|
+
* - Keep each test fully independent
|
|
2269
|
+
*
|
|
2270
|
+
* DON'T:
|
|
2271
|
+
* - Import from ../main-repo/src \u2014 that defeats the holdout
|
|
2272
|
+
* - Use supertest or similar tools that import the app directly
|
|
2273
|
+
* - Test internal implementation details
|
|
2274
|
+
*/
|
|
2275
|
+
|
|
2276
|
+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
|
2277
|
+
import { spawn, type ChildProcess } from 'node:child_process';
|
|
2278
|
+
import { join } from 'node:path';
|
|
2279
|
+
|
|
2280
|
+
const MAIN_REPO = join(__dirname, '..', 'main-repo');
|
|
2281
|
+
const BASE_URL = process.env.BASE_URL || 'http://localhost:3000';
|
|
2282
|
+
let serverProcess: ChildProcess | undefined;
|
|
2283
|
+
|
|
2284
|
+
/**
|
|
2285
|
+
* Wait for a URL to become reachable.
|
|
2286
|
+
*/
|
|
2287
|
+
async function waitForServer(url: string, timeoutMs = 60_000): Promise<void> {
|
|
2288
|
+
const start = Date.now();
|
|
2289
|
+
while (Date.now() - start < timeoutMs) {
|
|
2290
|
+
try {
|
|
2291
|
+
const res = await fetch(url);
|
|
2292
|
+
if (res.ok || res.status < 500) return;
|
|
2293
|
+
} catch {
|
|
2294
|
+
// Server not ready yet
|
|
2295
|
+
}
|
|
2296
|
+
await new Promise(r => setTimeout(r, 1000));
|
|
2297
|
+
}
|
|
2298
|
+
throw new Error(\`Server at \${url} did not become ready within \${timeoutMs}ms\`);
|
|
2299
|
+
}
|
|
2300
|
+
|
|
2301
|
+
beforeAll(async () => {
|
|
2302
|
+
// If BASE_URL is set externally, skip starting a server
|
|
2303
|
+
if (process.env.BASE_URL) return;
|
|
2304
|
+
|
|
2305
|
+
serverProcess = spawn('npm', ['start'], {
|
|
2306
|
+
cwd: MAIN_REPO,
|
|
2307
|
+
stdio: 'pipe',
|
|
2308
|
+
env: { ...process.env, PORT: '3000' },
|
|
2309
|
+
});
|
|
2310
|
+
|
|
2311
|
+
await waitForServer(BASE_URL);
|
|
2312
|
+
}, 90_000);
|
|
2313
|
+
|
|
2314
|
+
afterAll(() => {
|
|
2315
|
+
if (serverProcess) {
|
|
2316
|
+
serverProcess.kill('SIGTERM');
|
|
2317
|
+
serverProcess = undefined;
|
|
2318
|
+
}
|
|
2319
|
+
});
|
|
2320
|
+
|
|
2321
|
+
// ---------------------------------------------------------------------------
|
|
2322
|
+
// Example scenarios \u2014 replace with real tests for your API
|
|
2323
|
+
// ---------------------------------------------------------------------------
|
|
2324
|
+
|
|
2325
|
+
describe('API health', () => {
|
|
2326
|
+
it('GET / returns a success status', async () => {
|
|
2327
|
+
const res = await fetch(\`\${BASE_URL}/\`);
|
|
2328
|
+
expect(res.status).toBeLessThan(500);
|
|
2329
|
+
});
|
|
2330
|
+
});
|
|
2331
|
+
|
|
2332
|
+
describe('API endpoints', () => {
|
|
2333
|
+
it('GET /api/example returns JSON', async () => {
|
|
2334
|
+
const res = await fetch(\`\${BASE_URL}/api/example\`);
|
|
2335
|
+
// Replace with your actual endpoint
|
|
2336
|
+
// expect(res.status).toBe(200);
|
|
2337
|
+
// const body = await res.json();
|
|
2338
|
+
// expect(body).toHaveProperty('data');
|
|
2339
|
+
});
|
|
2340
|
+
|
|
2341
|
+
it('POST /api/example creates a resource', async () => {
|
|
2342
|
+
// Replace with your actual endpoint and payload
|
|
2343
|
+
// const res = await fetch(\\\`\\\${BASE_URL}/api/example\\\`, {
|
|
2344
|
+
// method: 'POST',
|
|
2345
|
+
// headers: { 'Content-Type': 'application/json' },
|
|
2346
|
+
// body: JSON.stringify({ name: 'test' }),
|
|
2347
|
+
// });
|
|
2348
|
+
// expect(res.status).toBe(201);
|
|
2349
|
+
// const body = await res.json();
|
|
2350
|
+
// expect(body).toHaveProperty('id');
|
|
2351
|
+
});
|
|
2352
|
+
|
|
2353
|
+
it('returns 404 for unknown routes', async () => {
|
|
2354
|
+
const res = await fetch(\`\${BASE_URL}/api/does-not-exist\`);
|
|
2355
|
+
expect(res.status).toBe(404);
|
|
2356
|
+
});
|
|
2357
|
+
});
|
|
2358
|
+
`,
|
|
2359
|
+
"scenarios/example-scenario-mobile.yaml": `# Example Mobile Scenario Test (Maestro)
|
|
2360
|
+
#
|
|
2361
|
+
# This file is a template for scenario tests against mobile applications.
|
|
2362
|
+
# The holdout pattern applies: test the running app through its UI,
|
|
2363
|
+
# never reference source code from the main repo.
|
|
2364
|
+
#
|
|
2365
|
+
# Maestro tests are declarative YAML flows that interact with a running
|
|
2366
|
+
# app on a simulator/emulator. Install Maestro:
|
|
2367
|
+
# curl -Ls "https://get.maestro.mobile.dev" | bash
|
|
2368
|
+
#
|
|
2369
|
+
# Run this flow:
|
|
2370
|
+
# maestro test example-scenario-mobile.yaml
|
|
2371
|
+
#
|
|
2372
|
+
# DO:
|
|
2373
|
+
# - Tap elements, fill inputs, assert on visible text
|
|
2374
|
+
# - Use runFlow for reusable sub-flows (e.g., login)
|
|
2375
|
+
# - Use assertWithAI for natural-language assertions
|
|
2376
|
+
#
|
|
2377
|
+
# DON'T:
|
|
2378
|
+
# - Reference source code paths or internal identifiers
|
|
2379
|
+
# - Depend on exact pixel positions (use text and accessibility labels)
|
|
2380
|
+
|
|
2381
|
+
appId: com.example.myapp # Replace with your app's bundle identifier
|
|
2382
|
+
name: "Core User Journey"
|
|
2383
|
+
tags:
|
|
2384
|
+
- smoke
|
|
2385
|
+
- holdout
|
|
2386
|
+
---
|
|
2387
|
+
# Step 1: Launch the app
|
|
2388
|
+
- launchApp
|
|
2389
|
+
|
|
2390
|
+
# Step 2: Login (using a reusable sub-flow)
|
|
2391
|
+
- runFlow: example-scenario-mobile-login.yaml
|
|
2392
|
+
|
|
2393
|
+
# Step 3: Verify the main screen loaded
|
|
2394
|
+
- assertVisible: "Home"
|
|
2395
|
+
|
|
2396
|
+
# Step 4: Navigate to a feature
|
|
2397
|
+
# - tapOn: "Settings"
|
|
2398
|
+
# - assertVisible: "Account"
|
|
2399
|
+
|
|
2400
|
+
# Step 5: AI-powered assertion (natural language)
|
|
2401
|
+
# - assertWithAI: "The main dashboard is visible with navigation tabs at the bottom"
|
|
2402
|
+
|
|
2403
|
+
# Step 6: Go back
|
|
2404
|
+
# - back
|
|
2405
|
+
# - assertVisible: "Home"
|
|
2406
|
+
`,
|
|
2407
|
+
"scenarios/example-scenario-mobile-login.yaml": `# Reusable Login Sub-Flow (Maestro)
|
|
2408
|
+
#
|
|
2409
|
+
# This flow handles authentication. Other flows include it via:
|
|
2410
|
+
# - runFlow: example-scenario-mobile-login.yaml
|
|
2411
|
+
#
|
|
2412
|
+
# Replace the selectors and credentials with your app's actual login flow.
|
|
2413
|
+
|
|
2414
|
+
appId: com.example.myapp
|
|
2415
|
+
name: "Login"
|
|
2416
|
+
---
|
|
2417
|
+
- assertVisible: "Sign In"
|
|
2418
|
+
- tapOn: "Email"
|
|
2419
|
+
- inputText: "test@example.com"
|
|
2420
|
+
- tapOn: "Password"
|
|
2421
|
+
- inputText: "testpassword123"
|
|
2422
|
+
- tapOn: "Log In"
|
|
2423
|
+
- assertVisible: "Home" # Verify login succeeded
|
|
2424
|
+
`,
|
|
2425
|
+
"scenarios/README-mobile.md": '# Mobile Scenario Testing with Maestro\n\nThis guide explains how to set up and run mobile holdout scenario tests using [Maestro](https://maestro.dev/).\n\n## Prerequisites\n\n- **Maestro CLI:** `curl -Ls "https://get.maestro.mobile.dev" | bash`\n- **Java 17+** (required by Maestro)\n- **Simulator/Emulator:**\n - iOS: Xcode with iOS Simulator (macOS only)\n - Android: Android Studio with an AVD configured\n\n> **Important:** Joycraft does not install Maestro or manage simulators. This is your responsibility.\n\n## Running Tests Locally\n\n```bash\n# Boot your simulator/emulator first, then:\nmaestro test example-scenario-mobile.yaml\n\n# Run all flows in a directory:\nmaestro test .maestro/\n```\n\n## Writing Flows\n\nMaestro flows are declarative YAML. Core commands:\n\n| Command | Purpose |\n|---------|--------|\n| `launchApp` | Start or restart the app |\n| `tapOn: "text"` | Tap an element by visible text or test ID |\n| `inputText: "value"` | Type into a focused field |\n| `assertVisible: "text"` | Assert an element is on screen |\n| `assertNotVisible: "text"` | Assert an element is NOT on screen |\n| `scroll` | Scroll down |\n| `back` | Press the back button |\n| `runFlow: file.yaml` | Run a reusable sub-flow |\n| `assertWithAI: "description"` | Natural-language assertion (AI-powered) |\n\n## CI Options\n\n### Option A: Maestro Cloud (paid, easiest)\n\nUpload your app binary and flows to Maestro Cloud. No simulator management.\n\n```yaml\n- uses: mobile-dev-inc/action-maestro-cloud@v2\n with:\n api-key: ${{ secrets.MAESTRO_API_KEY }}\n app-file: app.apk # or app.ipa\n workspace: .\n```\n\n### Option B: Self-hosted emulator (free, more setup)\n\nSpin up an Android emulator on a Linux runner or iOS simulator on a macOS runner.\n\n> **Cost note:** macOS GitHub Actions runners are ~10x more expensive than Linux runners.\n\n## The Holdout Pattern\n\nThese tests live in the scenarios repo, separate from the main codebase. The scenario agent generates them from specs. They test observable behavior through the app\'s UI \u2014 never referencing source code or internal implementation.\n',
|
|
2088
2426
|
"scenarios/prompts/scenario-agent.md": `You are a QA engineer working in a holdout test repository. You CANNOT access the main repository's source code. Your job is to write or update behavioral scenario tests based on specs that are pushed from the main repo.
|
|
2089
2427
|
|
|
2090
2428
|
## What You Have Access To
|
|
@@ -2092,7 +2430,23 @@ describe("CLI: init command (example \u2014 replace with your real scenarios)",
|
|
|
2092
2430
|
- This scenarios repository (test files, \`specs/\` mirror, \`package.json\`)
|
|
2093
2431
|
- The incoming spec (provided below)
|
|
2094
2432
|
- A list of existing test files and spec mirrors (provided below)
|
|
2095
|
-
- The main repo is available at \`../main-repo\` and is already built
|
|
2433
|
+
- The main repo is available at \`../main-repo\` and is already built
|
|
2434
|
+
- The testing strategy for this project (provided below)
|
|
2435
|
+
|
|
2436
|
+
## Testing Strategy
|
|
2437
|
+
|
|
2438
|
+
This project uses the **$TESTING_BACKBONE** testing backbone.
|
|
2439
|
+
|
|
2440
|
+
Select the correct test format based on the backbone:
|
|
2441
|
+
|
|
2442
|
+
| Backbone | Tool | Test Format | File Extension | How to Test |
|
|
2443
|
+
|----------|------|-------------|---------------|-------------|
|
|
2444
|
+
| \`playwright\` | Playwright | Browser-based E2E | \`.spec.ts\` | Navigate pages, click elements, assert on visible content |
|
|
2445
|
+
| \`maestro\` | Maestro | YAML flows | \`.yaml\` | Tap elements, fill inputs, assert on screen state |
|
|
2446
|
+
| \`api\` | fetch (Node.js built-in) | HTTP requests | \`.test.ts\` | Send requests to endpoints, assert on responses |
|
|
2447
|
+
| \`native\` | vitest + spawnSync | CLI/binary invocation | \`.test.ts\` | Run commands, assert on stdout/stderr/exit codes |
|
|
2448
|
+
|
|
2449
|
+
If the backbone is not provided or unrecognized, default to \`native\`.
|
|
2096
2450
|
|
|
2097
2451
|
## Triage Decision Tree
|
|
2098
2452
|
|
|
@@ -2111,7 +2465,7 @@ If you SKIP, write a brief comment in the relevant test file (or a new one) expl
|
|
|
2111
2465
|
- A new output format or file that gets generated
|
|
2112
2466
|
- A new user-facing behavior that doesn't map to any existing test file
|
|
2113
2467
|
|
|
2114
|
-
Name the file after the feature area
|
|
2468
|
+
Name the file after the feature area using the correct extension for the backbone.
|
|
2115
2469
|
|
|
2116
2470
|
### UPDATE \u2014 Modify an existing test file if the spec:
|
|
2117
2471
|
- Changes behavior that is already tested
|
|
@@ -2122,25 +2476,20 @@ Match to the most relevant existing test file by feature area.
|
|
|
2122
2476
|
|
|
2123
2477
|
**If you are unsure whether a spec is user-facing, err on the side of writing a test.**
|
|
2124
2478
|
|
|
2125
|
-
## Test Writing Rules
|
|
2126
|
-
|
|
2127
|
-
1. **Behavioral only.** Test observable output \u2014 stdout, stderr, exit codes, files created/modified on disk. Never test internal implementation details or import source modules.
|
|
2128
|
-
|
|
2129
|
-
2. **Use \`execSync\` or \`spawnSync\`.** Invoke the built binary at \`../main-repo/dist/cli.js\` (or whatever the main repo's entry point is). Check \`../main-repo/package.json\` to find the correct entry point if unsure.
|
|
2479
|
+
## Test Writing Rules (All Backbones)
|
|
2130
2480
|
|
|
2131
|
-
|
|
2481
|
+
1. **Behavioral only.** Test observable behavior \u2014 what a real user would see. Never test internal implementation details or import source modules.
|
|
2482
|
+
2. **Each test is fully independent.** No shared mutable state between tests.
|
|
2483
|
+
3. **Assert on realistic user actions.** Write tests that reflect what a real user would do.
|
|
2484
|
+
4. **Never import from the parent repo's source.** If you find yourself writing \`import { ... } from '../main-repo/src/...'\`, stop \u2014 that defeats the holdout.
|
|
2132
2485
|
|
|
2133
|
-
|
|
2486
|
+
## Backbone: native (CLI/Binary)
|
|
2134
2487
|
|
|
2135
|
-
|
|
2136
|
-
|
|
2137
|
-
6. **Never import from the parent repo's source.** If you find yourself writing \`import { ... } from '../main-repo/src/...'\`, stop \u2014 that defeats the holdout.
|
|
2138
|
-
|
|
2139
|
-
## Test File Template
|
|
2488
|
+
Use when the project is a CLI tool, library, or has no web/mobile UI.
|
|
2140
2489
|
|
|
2141
2490
|
\`\`\`typescript
|
|
2142
|
-
import {
|
|
2143
|
-
import {
|
|
2491
|
+
import { spawnSync } from 'node:child_process';
|
|
2492
|
+
import { mkdtempSync, rmSync } from 'node:fs';
|
|
2144
2493
|
import { tmpdir } from 'node:os';
|
|
2145
2494
|
import { join } from 'node:path';
|
|
2146
2495
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
|
@@ -2153,39 +2502,122 @@ function runCLI(args: string[], cwd?: string) {
|
|
|
2153
2502
|
cwd: cwd ?? process.cwd(),
|
|
2154
2503
|
env: { ...process.env, NO_COLOR: '1' },
|
|
2155
2504
|
});
|
|
2156
|
-
return {
|
|
2157
|
-
stdout: result.stdout ?? '',
|
|
2158
|
-
stderr: result.stderr ?? '',
|
|
2159
|
-
status: result.status ?? 1,
|
|
2160
|
-
};
|
|
2505
|
+
return { stdout: result.stdout ?? '', stderr: result.stderr ?? '', status: result.status ?? 1 };
|
|
2161
2506
|
}
|
|
2162
2507
|
|
|
2163
|
-
describe('[feature area]
|
|
2508
|
+
describe('[feature area]', () => {
|
|
2164
2509
|
let tmpDir: string;
|
|
2510
|
+
beforeEach(() => { tmpDir = mkdtempSync(join(tmpdir(), 'scenarios-')); });
|
|
2511
|
+
afterEach(() => { rmSync(tmpDir, { recursive: true, force: true }); });
|
|
2165
2512
|
|
|
2166
|
-
|
|
2167
|
-
|
|
2513
|
+
it('[observable behavior]', () => {
|
|
2514
|
+
const { stdout, status } = runCLI(['command', 'args'], tmpDir);
|
|
2515
|
+
expect(status).toBe(0);
|
|
2516
|
+
expect(stdout).toContain('expected output');
|
|
2168
2517
|
});
|
|
2518
|
+
});
|
|
2519
|
+
\`\`\`
|
|
2169
2520
|
|
|
2170
|
-
|
|
2171
|
-
|
|
2521
|
+
## Backbone: playwright (Web Apps)
|
|
2522
|
+
|
|
2523
|
+
Use when the project is a web application (Next.js, Vite, Nuxt, etc.).
|
|
2524
|
+
|
|
2525
|
+
\`\`\`typescript
|
|
2526
|
+
import { test, expect } from '@playwright/test';
|
|
2527
|
+
|
|
2528
|
+
// Tests run against BASE_URL (configured in playwright.config.ts)
|
|
2529
|
+
// The dev server is started automatically or BASE_URL points to a preview deploy
|
|
2530
|
+
|
|
2531
|
+
test.describe('[feature area]', () => {
|
|
2532
|
+
test('[observable behavior]', async ({ page }) => {
|
|
2533
|
+
await page.goto('/');
|
|
2534
|
+
await expect(page.locator('h1')).toBeVisible();
|
|
2172
2535
|
});
|
|
2173
2536
|
|
|
2174
|
-
|
|
2175
|
-
|
|
2176
|
-
|
|
2177
|
-
|
|
2537
|
+
test('[user interaction]', async ({ page }) => {
|
|
2538
|
+
await page.goto('/login');
|
|
2539
|
+
await page.fill('[name="email"]', 'test@example.com');
|
|
2540
|
+
await page.click('button[type="submit"]');
|
|
2541
|
+
await expect(page).toHaveURL(/dashboard/);
|
|
2542
|
+
});
|
|
2543
|
+
});
|
|
2544
|
+
\`\`\`
|
|
2545
|
+
|
|
2546
|
+
## Backbone: api (API Backends)
|
|
2547
|
+
|
|
2548
|
+
Use when the project is an API-only backend (Express, FastAPI, etc.).
|
|
2549
|
+
|
|
2550
|
+
\`\`\`typescript
|
|
2551
|
+
import { describe, it, expect } from 'vitest';
|
|
2552
|
+
|
|
2553
|
+
const BASE_URL = process.env.BASE_URL || 'http://localhost:3000';
|
|
2554
|
+
|
|
2555
|
+
describe('[feature area]', () => {
|
|
2556
|
+
it('[endpoint behavior]', async () => {
|
|
2557
|
+
const res = await fetch(\\\`\\\${BASE_URL}/api/endpoint\\\`);
|
|
2558
|
+
expect(res.status).toBe(200);
|
|
2559
|
+
const body = await res.json();
|
|
2560
|
+
expect(body).toHaveProperty('data');
|
|
2561
|
+
});
|
|
2562
|
+
|
|
2563
|
+
it('[error handling]', async () => {
|
|
2564
|
+
const res = await fetch(\\\`\\\${BASE_URL}/api/not-found\\\`);
|
|
2565
|
+
expect(res.status).toBe(404);
|
|
2178
2566
|
});
|
|
2179
2567
|
});
|
|
2180
2568
|
\`\`\`
|
|
2181
2569
|
|
|
2570
|
+
## Backbone: maestro (Mobile Apps)
|
|
2571
|
+
|
|
2572
|
+
Use when the project is a mobile application (React Native, Flutter, native iOS/Android).
|
|
2573
|
+
|
|
2574
|
+
\`\`\`yaml
|
|
2575
|
+
appId: com.example.myapp
|
|
2576
|
+
name: "[feature area]: [behavior being tested]"
|
|
2577
|
+
tags:
|
|
2578
|
+
- holdout
|
|
2579
|
+
---
|
|
2580
|
+
- launchApp
|
|
2581
|
+
- tapOn: "Sign In"
|
|
2582
|
+
- inputText: "test@example.com"
|
|
2583
|
+
- tapOn: "Submit"
|
|
2584
|
+
- assertVisible: "Welcome"
|
|
2585
|
+
# Use assertWithAI for complex visual assertions:
|
|
2586
|
+
# - assertWithAI: "The dashboard shows a list of recent items"
|
|
2587
|
+
\`\`\`
|
|
2588
|
+
|
|
2589
|
+
## Graceful Degradation
|
|
2590
|
+
|
|
2591
|
+
If the primary backbone tool is not available in this repo, fall back to the next deepest testable layer:
|
|
2592
|
+
|
|
2593
|
+
| Layer | What's Tested | When to Use |
|
|
2594
|
+
|-------|-------------|-------------|
|
|
2595
|
+
| **Layer 4: UI** | Full user flows through browser/simulator | \`@playwright/test\` or Maestro is installed |
|
|
2596
|
+
| **Layer 3: API** | HTTP requests against running server | Server can be started from \`../main-repo\` |
|
|
2597
|
+
| **Layer 2: Logic** | Unit tests via test runner | Test runner (vitest/jest) is available |
|
|
2598
|
+
| **Layer 1: Static** | Build, typecheck, lint | Build toolchain is available |
|
|
2599
|
+
|
|
2600
|
+
**Fallback rules:**
|
|
2601
|
+
- If backbone is \`playwright\` but \`@playwright/test\` is NOT in this repo's \`package.json\`: fall back to \`api\` (fetch-based HTTP tests)
|
|
2602
|
+
- If backbone is \`maestro\` but no simulator context is available: fall back to \`api\` if a server can be started, else \`native\`
|
|
2603
|
+
- If backbone is \`api\` but no server start script exists: fall back to \`native\`
|
|
2604
|
+
- \`native\` is always available as the floor
|
|
2605
|
+
|
|
2606
|
+
Start each test file with a comment indicating the testing layer:
|
|
2607
|
+
\`// Testing Layer: [4|3|2|1] - [UI|API|Logic|Static]\`
|
|
2608
|
+
|
|
2609
|
+
If you fell back from the intended backbone, note this in your commit message:
|
|
2610
|
+
\`scenarios: [action] for [spec] (layer: [N], reason: [why])\`
|
|
2611
|
+
|
|
2182
2612
|
## Checklist Before Committing
|
|
2183
2613
|
|
|
2184
2614
|
- [ ] Decision: SKIP / NEW / UPDATE (and why)
|
|
2615
|
+
- [ ] Correct backbone selected (or fallback justified)
|
|
2185
2616
|
- [ ] Tests assert on observable behavior, not implementation
|
|
2186
2617
|
- [ ] No imports from \`../main-repo/src\`
|
|
2187
|
-
- [ ] Each test
|
|
2188
|
-
- [ ] File
|
|
2618
|
+
- [ ] Each test is independent (own temp dir, own state)
|
|
2619
|
+
- [ ] File uses the correct extension for the backbone
|
|
2620
|
+
- [ ] Testing layer comment at top of file
|
|
2189
2621
|
`,
|
|
2190
2622
|
"scenarios/workflows/generate.yml": `# Scenario Generation Workflow
|
|
2191
2623
|
#
|
|
@@ -2285,7 +2717,9 @@ jobs:
|
|
|
2285
2717
|
## Context
|
|
2286
2718
|
|
|
2287
2719
|
Existing test files in this repo: \${{ steps.context.outputs.existing_tests }}
|
|
2288
|
-
Existing spec mirrors: \${{ steps.context.outputs.existing_specs }}
|
|
2720
|
+
Existing spec mirrors: \${{ steps.context.outputs.existing_specs }}
|
|
2721
|
+
|
|
2722
|
+
Testing backbone: \${{ github.event.client_payload.testing_backbone || 'native' }}"
|
|
2289
2723
|
|
|
2290
2724
|
# \u2500\u2500 7. Commit any changes the agent made \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500
|
|
2291
2725
|
- name: Commit scenario changes
|
|
@@ -2861,4 +3295,4 @@ export {
|
|
|
2861
3295
|
SKILLS,
|
|
2862
3296
|
TEMPLATES
|
|
2863
3297
|
};
|
|
2864
|
-
//# sourceMappingURL=chunk-
|
|
3298
|
+
//# sourceMappingURL=chunk-A2CQG5J5.js.map
|