@checkly/playwright-reporter 0.1.9 → 1.1.0

package/CHANGELOG.md

@@ -0,0 +1,114 @@
+# Changelog
+
+All notable changes to `@checkly/playwright-reporter` will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+## 1.1.0 (2026-01-20)
+
+> **Note:** This minor version contains breaking changes. Due to options API configurations in 1.0.x that could lead to broken states, we decided to publish this as a minor version and unpublish previous 1.0.x releases to prevent issues. If you were using 1.0.x, please update your configuration as described below.
+
+### Breaking Changes
+
+- **Removed deprecated options** - The following options are no longer supported and will throw an error if used:
+  - `outputFile` - Use `outputDir` instead
+  - `testResultsDir` - Use `outputDir` instead
+  - `outputPath` - Use `outputDir` instead
+
+### Changed
+
+- **Simplified options API** - The reporter now has a cleaner configuration:
+  - `outputDir` - Directory for all output (JSON report and ZIP assets)
+  - `verbose` - Enable debug logging
+  - `apiKey` - Checkly API key (or use `CHECKLY_API_KEY` env var)
+  - `accountId` - Checkly account ID (or use `CHECKLY_ACCOUNT_ID` env var)
+  - `sessionName` - Custom session name (string or function)
+  - `dryRun` - Generate report without uploading
+- Reduced package size
+
+### Migration from 1.0.x
+
+If you were using any deprecated options, update your configuration:
+
+```diff
+// playwright.config.ts
+export default defineConfig({
+  reporter: [
+-   ['@checkly/playwright-reporter', {
+-     outputFile: 'results/report.json',
+-     testResultsDir: 'test-results',
+-   }],
++   ['@checkly/playwright-reporter', {
++     outputDir: 'results'
++   }],
+  ],
+})
+```
+
+## 1.0.4 (2026-01-20)
+
+### Fixed
+
+- Fixed module loading issues in certain Playwright configurations.
+- Reduced package size.
+
+## 1.0.3 (2026-01-20)
+
+### Fixed
+
+- Fixed compatibility with Playwright's TypeScript configuration loading mechanism.
+- Reporter version now correctly displays in test session summary.
+- `CHECKLY_ACCOUNT_ID` environment variable is now properly read for all features.
+
+## 1.0.2 (2026-01-20)
+
+### Added
+
+- Anonymous telemetry to understand reporter usage patterns. Disable with `telemetry: false` option.
+
+### Fixed
+
+- Test session creation is now properly awaited before uploading results, fixing intermittent upload failures in CI environments.
+- Sharded test execution now works correctly when using `playwright merge-reports` command.
+- Default projects now preserve their empty name correctly.
+
+### Changed
+
+- Output types now re-export from `@playwright/test/reporter` with Checkly augmentations for `_checkly` property.
+
+## 1.0.0 (2025-01-19)
+
+First stable release of the rewritten Playwright reporter with extension-based architecture.
+
+### Added
+
+- **Extension-based architecture** - Modular design allows for composable functionality
+- **Broad Playwright support** - Compatible with Playwright versions 1.40 through 1.57+
+- **Drop-in JSON replacement** - Fully compatible with Playwright's native JSON reporter output format
+- **Trace extraction** - Automatically extracts console messages and network requests from Playwright trace files
+- **Checkly integration** - Automatic upload to Checkly Test Sessions when credentials are configured
+- **New `createChecklyReporter()` function** - Better intellisense support in `playwright.config.ts`
+- **Unified `outputDir` option** - Single option for all output files (JSON report, ZIP assets)
+- **Verbose logging** - Debug mode for troubleshooting report generation
+
+### Changed
+
+- `outputFile` option is deprecated - use `outputDir` instead (JSON written to `{outputDir}/checkly-report.json`)
+- `testResultsDir` option is deprecated - use `outputDir` instead
+- `outputPath` option is deprecated - ZIP written to `{outputDir}/checkly-report.zip`
+
+### Migration from 0.x
+
+```diff
+// playwright.config.ts
++import { createChecklyReporter } from '@checkly/playwright-reporter'
+
+export default defineConfig({
+  reporter: [
+-   ['@checkly/playwright-reporter', { outputFile: 'results/report.json' }],
++   createChecklyReporter({ outputDir: 'results' }),
+  ],
+})
+```
+
+The legacy 0.x reporter remains available via `npm install @checkly/playwright-reporter@legacy`.

package/README.md

@@ -0,0 +1,266 @@
+# @checkly/playwright-reporter
+
+> **Technical Preview** - This reporter is currently in technical preview. Features and APIs may change.
+
+Official Playwright reporter for Checkly. Automatically upload test results, screenshots, videos, and traces to gain visibility into your end-to-end tests.
+
+**Fully compatible with Playwright's native JSON reporter** - use it as a drop-in replacement that adds Checkly integration.
+
+## Installation
+
+```bash
+npm install --save-dev @checkly/playwright-reporter
+```
+
+**Requirements:**
+- Node.js >= 18.0.0
+- Playwright >= 1.40.0
+- A [Checkly account](https://app.checklyhq.com)
+
+## Quick Start
+
+### 1. Get Your Credentials
+
+1. Go to **Account Settings > General** to find your Account ID
+2. Go to **User Settings > API Keys** to create an API key
+
+### 2. Configure Playwright
+
+Add the reporter to your `playwright.config.ts`:
+
+```typescript
+import { defineConfig } from '@playwright/test';
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+export default defineConfig({
+  reporter: [
+    ['list'],
+    createChecklyReporter(),
+  ],
+});
+```
+
+> **Tip:** Using `createChecklyReporter()` provides full intellisense for configuration options. Alternatively, use the array syntax: `['@checkly/playwright-reporter', { ... }]`
+
+### 3. Set Environment Variables
+
+```bash
+export CHECKLY_API_KEY=your_api_key
+export CHECKLY_ACCOUNT_ID=your_account_id
+```
+
+Or pass them inline:
+
+```bash
+CHECKLY_API_KEY=... CHECKLY_ACCOUNT_ID=... npx playwright test
+```
+
+### 4. Run Tests
+
+```bash
+npx playwright test
+```
+
+You'll see a session URL in the output:
+
+```
+View test results: https://app.checklyhq.com/test-sessions/abc123
+
+[...test execution...]
+
+✓ Check the results: https://app.checklyhq.com/test-sessions/abc123
+```
+
+## Configuration Options
+
+```typescript
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+createChecklyReporter({
+  outputDir: 'test-results',
+  sessionName: 'My Test Suite',
+  dryRun: false,
+  verbose: false,
+})
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `outputDir` | `string` | Playwright's `outputDir` | Directory for assets, JSON, and ZIP |
+| `sessionName` | `string \| function` | Auto-generated | Custom session name |
+| `dryRun` | `boolean` | `false` | Create ZIP without uploading |
+| `verbose` | `boolean` | `false` | Enable debug logging |
+
+**Output files** (written to `outputDir`):
+- `checkly-report.json` - JSON test report
+- `checkly-report.zip` - ZIP archive with report and assets
+
+**Deprecated options** (will be removed in next major version):
+| Option | Migration |
+|--------|-----------|
+| `outputFile` | JSON now at `{outputDir}/checkly-report.json` |
+| `testResultsDir` | Use `outputDir` |
+| `outputPath` | ZIP now at `{outputDir}/checkly-report.zip` |
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `CHECKLY_API_KEY` | Your Checkly API key |
+| `CHECKLY_ACCOUNT_ID` | Your Checkly account ID |
+| `CHECKLY_REPORTER_VERBOSE` | Set to `true` for detailed debug output |
+
+## What Gets Uploaded
+
+- Test results and status (passed/failed/flaky)
+- Execution duration and timing
+- Screenshots (on failure or explicit capture)
+- Video recordings
+- Playwright traces
+- Console logs and network requests (extracted from traces)
+
+## Flaky Test Detection
+
+The reporter automatically detects flaky tests:
+
+- **Flaky test**: A test that failed initially but passed after retry
+- **Degraded status**: Set when there are flaky tests but no complete failures
+- **Overall status**: `passed` if all tests eventually pass, `failed` otherwise
+
+## Running Locally (No Upload)
+
+For local development without uploading to Checkly:
+
+```bash
+# Simply don't set CHECKLY_API_KEY or CHECKLY_ACCOUNT_ID
+npx playwright test
+```
+
+What happens:
+- Reporter creates `checkly-report.zip` locally
+- Upload is automatically skipped
+- Tests run normally
+- You can inspect the ZIP file for debugging
+
+## Dry Run Mode
+
+Create local JSON and ZIP files without uploading:
+
+```typescript
+createChecklyReporter({
+  dryRun: true,
+})
+```
+
+Conditional dry run (skip uploads when no credentials):
+
+```typescript
+createChecklyReporter({
+  dryRun: !process.env.CHECKLY_API_KEY,
+})
+```
+
+## CI/CD Integration
+
+### GitHub Actions
+
+```yaml
+name: Playwright Tests
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps
+
+      - name: Run Playwright tests
+        env:
+          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
+          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
+        run: npx playwright test
+```
+
+### GitLab CI
+
+```yaml
+test:
+  image: mcr.microsoft.com/playwright:v1.57.0-jammy
+  stage: test
+  script:
+    - npm ci
+    - npx playwright test
+  variables:
+    CHECKLY_API_KEY: $CHECKLY_API_KEY
+    CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID
+```
+
+**Setting up variables:**
+1. Go to your GitLab project
+2. Navigate to Settings > CI/CD > Variables
+3. Add `CHECKLY_API_KEY` (check "Mask variable")
+4. Add `CHECKLY_ACCOUNT_ID` (check "Mask variable")
+
+## Multiple Reporters
+
+Combine with other Playwright reporters:
+
+```typescript
+import { createChecklyReporter } from '@checkly/playwright-reporter';
+
+export default defineConfig({
+  reporter: [
+    createChecklyReporter(),
+    ['html', { outputFolder: 'playwright-report' }],
+    ['list'],
+    ['junit', { outputFile: 'test-results/junit.xml' }],
+  ],
+});
+```
+
+## Understanding Test Sessions
+
+The reporter creates **suite-level test sessions**, not individual test file results.
+
+When you run `npx playwright test`:
+
+1. **One test session** is created, named after your directory
+2. **One test result** for the entire run
+3. **All assets uploaded together** in a single ZIP file
+
+Benefits:
+- Consolidated view of your entire test suite
+- Efficient storage
+- Track overall pass/fail rates over time
+
+## Security
+
+Always use environment variables for credentials. Never commit API keys to version control.
+
+```typescript
+// DON'T DO THIS
+createChecklyReporter({
+  apiKey: 'chk_api_...',  // Hardcoded credentials!
+})
+
+// DO THIS
+createChecklyReporter()  // Reads from environment
+```
+
+## Documentation
+
+For detailed documentation, visit [checklyhq.com/docs/detect/testing/playwright-reporter](https://www.checklyhq.com/docs/detect/testing/playwright-reporter/).
+
+## License
+
+Apache 2.0

package/dist/index.d.ts

@@ -1,219 +1,64 @@
-export { Asset, AssetCollector, ZipEntry, ZipResult, Zipper, ZipperOptions } from '@checkly/reporter-utils';
-import { FullConfig, Suite, JSONReport as JSONReport$1, JSONReportSuite as JSONReportSuite$1, JSONReportSpec as JSONReportSpec$1, JSONReportTest as JSONReportTest$1, JSONReportTestResult as JSONReportTestResult$1, Reporter, TestCase, TestResult, TestError } from '@playwright/test/reporter';
+import { JSONReportTestResult as JSONReportTestResult$1, Reporter } from '@playwright/test/reporter';
+export { JSONReport, JSONReportError, JSONReportSTDIOEntry, JSONReportSpec, JSONReportSuite, JSONReportTest, JSONReportTestStep } from '@playwright/test/reporter';
 
-/**
- * Checkly environment for API endpoints
- */
-type ChecklyEnvironment = 'local' | 'development' | 'staging' | 'production';
-/**
- * Context passed to sessionName callback function
- */
-interface SessionNameContext {
-    /**
-     * The directory name where tests are running
-     */
-    directoryName: string;
-    /**
-     * Playwright's full configuration object
-     */
-    config: FullConfig;
-    /**
-     * The root test suite containing all tests
-     */
-    suite: Suite;
+interface ChecklyConsoleLog {
+    type: 'log' | 'warn' | 'error' | 'info' | 'debug';
+    message: string;
+    timestamp: string;
 }
+interface ChecklyNetworkRequest {
+    url: string;
+    method: string;
+    status?: number;
+    duration?: number;
+    timestamp: string;
+}
+interface ChecklyData {
+    consoleLogs?: ChecklyConsoleLog[];
+    networkRequests?: ChecklyNetworkRequest[];
+}
+interface JSONReportTestResult extends JSONReportTestResult$1 {
+    _checkly?: ChecklyData;
+}
+
 /**
- * Session name can be a static string or a function that returns a string
- */
-type SessionNameOption = string | ((context: SessionNameContext) => string);
-/**
- * Configuration options for ChecklyReporter
+ * Configuration options for the Checkly Playwright Reporter.
  */
 interface ChecklyReporterOptions {
-    /**
-     * Checkly account ID for uploading test results
-     * Required for Phase 2 (upload functionality)
-     */
-    accountId?: string;
-    /**
-     * Checkly API key for authentication
-     * Required for Phase 2 (upload functionality)
-     */
+    /** Directory for report output. Defaults to Playwright's outputDir. */
+    outputDir?: string;
+    /** Enable verbose logging. */
+    verbose?: boolean;
+    /** Checkly API key. Can also be set via CHECKLY_API_KEY env var. */
     apiKey?: string;
-    /**
-     * Checkly environment to use
-     * Can also be set via CHECKLY_ENV environment variable
-     * @default 'production'
-     */
-    environment?: ChecklyEnvironment;
-    /**
-     * Output path for the generated ZIP file (for testing/debugging)
-     * @default 'checkly-report.zip'
-     * @internal
-     */
-    outputPath?: string;
-    /**
-     * Path to the JSON report file generated by Playwright's json reporter
-     * @default 'test-results/playwright-test-report.json'
-     * @internal
-     */
-    jsonReportPath?: string;
-    /**
-     * Directory containing test results and assets
-     * @default 'test-results'
-     * @internal
-     */
-    testResultsDir?: string;
-    /**
-     * Dry run mode - skips API calls and only creates local ZIP file
-     * @default false
-     */
+    /** Checkly account ID. Can also be set via CHECKLY_ACCOUNT_ID env var. */
+    accountId?: string;
+    /** Session name for the test run. Can be a string or a function. */
+    sessionName?: string | ((context: {
+        directoryName: string;
+        config: unknown;
+        suite: unknown;
+    }) => string);
+    /** Dry run mode - generate report but don't upload. */
     dryRun?: boolean;
-    /**
-     * Custom name for the test session
-     * Can be a string or a callback function for dynamic names
-     * @default 'Playwright Test Session: {directoryName}'
-     * @example
-     * // Static string
-     * sessionName: 'My E2E Tests'
-     *
-     * // Dynamic with callback
-     * sessionName: ({ directoryName, config }) => `E2E: ${directoryName} (${config.projects.length} projects)`
-     */
-    sessionName?: SessionNameOption;
-}
-/**
- * Warning types that can be attached to test results
- */
-type ChecklyWarningType = 'trace-off' | 'trace-retained-on-failure' | 'trace-first-retry-only' | 'trace-retries-only' | 'trace-retained-on-first-failure' | 'trace-missing';
-/**
- * A warning attached to a test result
- */
-interface ChecklyWarning {
-    /**
-     * Type of warning for programmatic handling
-     */
-    type: ChecklyWarningType;
-    /**
-     * Human-readable warning message
-     */
-    message: string;
-}
-/**
- * Checkly-specific extensions added to JSONReportTestResult
- */
-interface ChecklyTestResultExtensions {
-    /**
-     * Warnings about the test result (e.g., missing traces)
-     */
-    warnings?: ChecklyWarning[];
 }
-/**
- * Extended JSONReportTestResult with Checkly-specific data
- */
-type JSONReportTestResult = JSONReportTestResult$1 & {
-    _checkly?: ChecklyTestResultExtensions;
-};
-/**
- * Extended JSONReportTest with Checkly-extended results
- */
-type JSONReportTest = Omit<JSONReportTest$1, 'results'> & {
-    results: JSONReportTestResult[];
-};
-/**
- * Extended JSONReportSpec with Checkly-extended tests
- */
-type JSONReportSpec = Omit<JSONReportSpec$1, 'tests'> & {
-    tests: JSONReportTest[];
-};
-/**
- * Extended JSONReportSuite with Checkly-extended specs and nested suites
- */
-type JSONReportSuite = Omit<JSONReportSuite$1, 'suites' | 'specs'> & {
-    suites?: JSONReportSuite[];
-    specs: JSONReportSpec[];
-};
-/**
- * Extended JSONReport with Checkly-extended suites
- */
-type JSONReport = Omit<JSONReport$1, 'suites'> & {
-    suites: JSONReportSuite[];
-};
+type ChecklyReporterClass = new (options?: ChecklyReporterOptions) => Reporter;
+declare const _default: ChecklyReporterClass;
 
 /**
- * Checkly Playwright Reporter
+ * Create a Checkly reporter configuration with full intellisense support.
  *
- * Creates a ZIP archive containing the JSON report and all test assets.
- * Designed to work alongside Playwright's built-in JSONReporter.
+ * Use this instead of the array-based syntax for better IDE experience.
  *
  * @example
- * // playwright.config.ts
+ * import { createChecklyReporter } from '@checkly/playwright-reporter'
+ *
  * export default defineConfig({
  *   reporter: [
- *     ['json', { outputFile: 'test-results/playwright-test-report.json' }],
- *     ['@checkly/playwright-reporter', {
- *       apiKey: process.env.CHECKLY_API_KEY,
- *       accountId: process.env.CHECKLY_ACCOUNT_ID,
- *     }]
- *   ]
- * });
+ *     createChecklyReporter({ outputDir: 'test-results' }),
+ *   ],
+ * })
  */
-declare class ChecklyReporter implements Reporter {
-    private options;
-    private assetCollector;
-    private zipper;
-    private testResults?;
-    private testSession?;
-    private startTime?;
-    private testCounts;
-    private stepsMap;
-    private warningsMap;
-    constructor(options?: ChecklyReporterOptions);
-    /**
-     * Resolves the session name from options
-     * Supports string, callback function, or falls back to default
-     */
-    private resolveSessionName;
-    /**
-     * Checks if test result has a trace attachment and adds context-aware warning if missing
-     * The warning type depends on the trace configuration and test result state
-     */
-    private checkTraceAttachment;
-    /**
-     * Called once before running tests
-     * Creates test session in Checkly if credentials provided
-     */
-    onBegin(config: FullConfig, suite: Suite): void;
-    /**
-     * Called for each test when it completes
-     * Captures steps and warnings, tracks test results for final status calculation
-     */
-    onTestEnd(test: TestCase, result: TestResult): void;
-    /**
-     * Called after all tests have completed
-     * This is where we create the ZIP archive and upload results
-     */
-    onEnd(): Promise<void>;
-    private printSummary;
-    /**
-     * Injects captured steps and warnings into the JSON report
-     * Traverses the report structure and matches by test ID + retry
-     */
-    private injectDataIntoReport;
-    /**
-     * Reconstructs config.projects and test.projectId from test data
-     * This is necessary for blob merge scenarios where Playwright's JSON reporter
-     * doesn't populate projects array or projectId fields
-     */
-    private reconstructProjectsFromTests;
-    /**
-     * Uploads test results to Checkly API
-     */
-    private uploadResults;
-    /**
-     * Called when a global error occurs
-     */
-    onError(error: TestError): void;
-}
+declare function createChecklyReporter(options?: ChecklyReporterOptions): ['@checkly/playwright-reporter', ChecklyReporterOptions];
 
-export { ChecklyReporter, type ChecklyReporterOptions, type ChecklyTestResultExtensions, type ChecklyWarning, type ChecklyWarningType, type JSONReport, type JSONReportSpec, type JSONReportSuite, type JSONReportTest, type JSONReportTestResult, ChecklyReporter as default };
+export { type ChecklyConsoleLog, type ChecklyData, type ChecklyNetworkRequest, type ChecklyReporterOptions, type JSONReportTestResult, createChecklyReporter, _default as default };