@checkly/playwright-reporter 0.1.10 → 1.0.0

package/README.md

@@ -0,0 +1,266 @@
+# @checkly/playwright-reporter
+
+> **Technical Preview** - This reporter is currently in technical preview. Features and APIs may change.
+
+Official Playwright reporter for Checkly. Automatically upload test results, screenshots, videos, and traces to gain visibility into your end-to-end tests.
+
+**Fully compatible with Playwright's native JSON reporter** - use it as a drop-in replacement that adds Checkly integration.
+
+## Installation
+
+```bash
+npm install --save-dev @checkly/playwright-reporter
+```
+
+**Requirements:**
+- Node.js >= 18.0.0
+- Playwright >= 1.40.0
+- A [Checkly account](https://app.checklyhq.com)
+
+## Quick Start
+
+### 1. Get Your Credentials
+
+1. Go to **Account Settings > General** to find your Account ID
+2. Go to **User Settings > API Keys** to create an API key
+
+### 2. Configure Playwright
+
+Add the reporter to your `playwright.config.ts`:
+
+```typescript
+import { defineConfig } from '@playwright/test';
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+export default defineConfig({
+  reporter: [
+    ['list'],
+    createChecklyReporter(),
+  ],
+});
+```
+
+> **Tip:** Using `createChecklyReporter()` provides full intellisense for configuration options. Alternatively, use the array syntax: `['@checkly/playwright-reporter', { ... }]`
+
+### 3. Set Environment Variables
+
+```bash
+export CHECKLY_API_KEY=your_api_key
+export CHECKLY_ACCOUNT_ID=your_account_id
+```
+
+Or pass them inline:
+
+```bash
+CHECKLY_API_KEY=... CHECKLY_ACCOUNT_ID=... npx playwright test
+```
+
+### 4. Run Tests
+
+```bash
+npx playwright test
+```
+
+You'll see a session URL in the output:
+
+```
+View test results: https://app.checklyhq.com/test-sessions/abc123
+
+[...test execution...]
+
+✓ Check the results: https://app.checklyhq.com/test-sessions/abc123
+```
+
+## Configuration Options
+
+```typescript
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+createChecklyReporter({
+  outputDir: 'test-results',
+  sessionName: 'My Test Suite',
+  dryRun: false,
+  verbose: false,
+})
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `outputDir` | `string` | Playwright's `outputDir` | Directory for assets, JSON, and ZIP |
+| `sessionName` | `string \| function` | Auto-generated | Custom session name |
+| `dryRun` | `boolean` | `false` | Create ZIP without uploading |
+| `verbose` | `boolean` | `false` | Enable debug logging |
+
+**Output files** (written to `outputDir`):
+- `checkly-report.json` - JSON test report
+- `checkly-report.zip` - ZIP archive with report and assets
+
+**Deprecated options** (will be removed in next major version):
+| Option | Migration |
+|--------|-----------|
+| `outputFile` | JSON now at `{outputDir}/checkly-report.json` |
+| `testResultsDir` | Use `outputDir` |
+| `outputPath` | ZIP now at `{outputDir}/checkly-report.zip` |
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `CHECKLY_API_KEY` | Your Checkly API key |
+| `CHECKLY_ACCOUNT_ID` | Your Checkly account ID |
+| `CHECKLY_REPORTER_VERBOSE` | Set to `true` for detailed debug output |
+
+## What Gets Uploaded
+
+- Test results and status (passed/failed/flaky)
+- Execution duration and timing
+- Screenshots (on failure or explicit capture)
+- Video recordings
+- Playwright traces
+- Console logs and network requests (extracted from traces)
+
+## Flaky Test Detection
+
+The reporter automatically detects flaky tests:
+
+- **Flaky test**: A test that failed initially but passed after retry
+- **Degraded status**: Set when there are flaky tests but no complete failures
+- **Overall status**: `passed` if all tests eventually pass, `failed` otherwise
+
+## Running Locally (No Upload)
+
+For local development without uploading to Checkly:
+
+```bash
+# Simply don't set CHECKLY_API_KEY or CHECKLY_ACCOUNT_ID
+npx playwright test
+```
+
+What happens:
+- Reporter creates `checkly-report.zip` locally
+- Upload is automatically skipped
+- Tests run normally
+- You can inspect the ZIP file for debugging
+
+## Dry Run Mode
+
+Create local JSON and ZIP files without uploading:
+
+```typescript
+createChecklyReporter({
+  dryRun: true,
+})
+```
+
+Conditional dry run (skip uploads when no credentials):
+
+```typescript
+createChecklyReporter({
+  dryRun: !process.env.CHECKLY_API_KEY,
+})
+```
+
+## CI/CD Integration
+
+### GitHub Actions
+
+```yaml
+name: Playwright Tests
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps
+
+      - name: Run Playwright tests
+        env:
+          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
+          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
+        run: npx playwright test
+```
+
+### GitLab CI
+
+```yaml
+test:
+  image: mcr.microsoft.com/playwright:v1.57.0-jammy
+  stage: test
+  script:
+    - npm ci
+    - npx playwright test
+  variables:
+    CHECKLY_API_KEY: $CHECKLY_API_KEY
+    CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID
+```
+
+**Setting up variables:**
+1. Go to your GitLab project
+2. Navigate to Settings > CI/CD > Variables
+3. Add `CHECKLY_API_KEY` (check "Mask variable")
+4. Add `CHECKLY_ACCOUNT_ID` (check "Mask variable")
+
+## Multiple Reporters
+
+Combine with other Playwright reporters:
+
+```typescript
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+export default defineConfig({
+  reporter: [
+    createChecklyReporter(),
+    ['html', { outputFolder: 'playwright-report' }],
+    ['list'],
+    ['junit', { outputFile: 'test-results/junit.xml' }],
+  ],
+});
+```
+
+## Understanding Test Sessions
+
+The reporter creates **suite-level test sessions**, not individual test file results.
+
+When you run `npx playwright test`:
+
+1. **One test session** is created, named after your directory
+2. **One test result** for the entire run
+3. **All assets uploaded together** in a single ZIP file
+
+Benefits:
+- Consolidated view of your entire test suite
+- Efficient storage
+- Track overall pass/fail rates over time
+
+## Security
+
+Always use environment variables for credentials. Never commit API keys to version control.
+
+```typescript
+// DON'T DO THIS
+createChecklyReporter({
+  apiKey: 'chk_api_...',  // Hardcoded credentials!
+})
+
+// DO THIS
+createChecklyReporter()  // Reads from environment
+```
+
+## Documentation
+
+For detailed documentation, visit [checklyhq.com/docs/detect/testing/playwright-reporter](https://www.checklyhq.com/docs/detect/testing/playwright-reporter/).
+
+## License
+
+Apache 2.0