creevey 0.10.0-beta.9 → 0.10.0-rc.1

package/docs/migration-0.9-to-0.10.md

@@ -0,0 +1,182 @@
+# Migrating from Creevey 0.9 to 0.10
+
+This guide outlines the key changes and steps required to update your Creevey setup from version 0.9 to 0.10. Version 0.10 introduces several significant updates, including Playwright support, a new test context API, and changes to configuration.
+
+---
+
+## Breaking Changes
+
+### 1. CLI Commands Structure (Breaking Change)
+
+Creevey 0.10 introduces a command-based CLI structure. You can no longer run Creevey without specifying a command.
+
+**Before (0.9):**
+
+```bash
+# Running tests
+creevey --ui
+creevey -s --ui
+creevey --debug
+
+# Update mode (approving images all at once with no way to review them)
+creevey --update
+
+# Viewing reports
+# Doesn't have this feature
+```
+
+**After (0.10):**
+
+```bash
+# Running tests
+creevey test --ui
+creevey test -s --ui
+creevey test --debug
+
+# Viewing reports (approving images)
+creevey report
+creevey report ./custom-report-dir
+```
+
+**Key Changes:**
+
+- All test execution now requires the `test` command
+- Image approval/update mode uses the `report` command instead of `-u/--update` flags
+- Report viewing uses the `report` command
+
+### 2. Browser Configuration in `creevey.config.ts`
+
+The way browsers are configured has been updated to better distinguish between Selenium and Playwright settings.
+
+- The new `webdriver` field lets you choose your WebDriver implementation. While it defaults to Selenium for now, this may change. We recommend explicitly setting it by importing and using either `SeleniumWebdriver` or `PlaywrightWebdriver`.
+- Selenium-specific capabilities (e.g., `browserVersion`, `platformName`) must now be nested under a `seleniumCapabilities` object.
+- Playwright-specific options should be placed under a `playwrightOptions` object.
+
+**Before (0.9):**
+
+```javascript
+// creevey.config.js
+export default {
+  browsers: {
+    chrome: {
+      browserName: 'chrome',
+      browserVersion: '90.0',
+      platformName: 'linux',
+    },
+  },
+};
+```
+
+**After (0.10):**
+
+```typescript
+// creevey.config.ts
+import type { CreeveyConfig } from 'creevey';
+import { SeleniumWebdriver } from 'creevey/selenium';
+// or
+// import { PlaywrightWebdriver } from 'creevey/playwright';
+
+const config: CreeveyConfig = {
+  webdriver: SeleniumWebdriver, // or PlaywrightWebdriver
+  browsers: {
+    chrome: {
+      // For Selenium
+      browserName: 'chrome',
+      seleniumCapabilities: {
+        browserVersion: '90.0',
+        platformName: 'linux',
+      },
+    },
+    firefoxPlaywright: {
+      // For Playwright
+      browserName: 'firefox', // 'chromium', 'firefox', or 'webkit' for Playwright
+      playwrightOptions: {
+        headless: true,
+      },
+    },
+  },
+};
+
+export default config;
+```
+
+### 3. Mocha Removal and New `CreeveyTestContext` API
+
+Creevey no longer uses the Mocha testing framework. Tests now use a `CreeveyTestContext` object passed as an argument to the test function.
+
+- Replace `this` with the `context` parameter in your tests.
+- Image matching methods (e.g., `matchImage`, `takeScreenshot`) are now called on the `context` object.
+- The `context.webdriver` property gives you direct access to the configured WebDriver instance (Selenium or Playwright) for advanced browser interactions.
+
+**Before (0.9 - Mocha style):**
+
+```javascript
+it('should match the image', async function () {
+  this.expect(await this.takeScreenshot()).to.matchImage('example');
+});
+```
+
+**After (0.10 - New context style):**
+
+```javascript
+it('should match the image', async (context) => {
+  await context.matchImage(await context.takeScreenshot(), 'example');
+});
+
+// Example using context.webdriver with Selenium WebDriver API
+it('should interact with an element using Selenium', async (context) => {
+  const seleniumWebDriver = context.webdriver; // Assuming SeleniumWebdriver is configured
+  const element = await seleniumWebDriver.findElement({ css: '#myElement' });
+  await element.click();
+  // ... more Selenium interactions
+  await context.matchImage(await context.takeScreenshot(), 'selenium-interaction');
+});
+
+// Example using context.webdriver with Playwright API
+it('should interact with an element using Playwright', async (context) => {
+  const playwrightPage = context.webdriver; // Assuming PlaywrightWebdriver is configured
+  await playwrightPage.click('#myElement');
+  // ... more Playwright interactions
+  await context.matchImage(await context.takeScreenshot(), 'playwright-interaction');
+});
+```
+
+### 4. Reporter Configuration
+
+Reporter setup has moved from command-line options to the `creevey.config.ts` file.
+
+**Example (0.10):**
+
+```typescript
+// creevey.config.ts
+import type { CreeveyConfig } from 'creevey';
+
+const config: CreeveyConfig = {
+  reporter: 'junit', // or 'teamcity', 'creevey'
+  reporterOptions: {
+    // For JUnit: { outputFile: 'report.xml' }
+    // For Creevey HTML reporter (default): { reportDir: './report' }
+  },
+};
+
+export default config;
+```
+
+---
+
+## Key New Features & Updates
+
+- **Creevey Playwright Reporter:** Integrate Creevey's visual testing into your existing Playwright test suites. See `docs/playwright-reporter.md` for details.
+- **Approve Tests from Report UI (Update Mode):** Approve visual changes directly in the Creevey report UI. Run with `creevey report` to use the web UI for reviewing and approving screenshots.
+- **ODiff Image Comparison:** Use `odiff` for image comparison with the `--odiff` flag and `odiffOptions` in config.
+- **Easier Storybook Autostart:** Use `creevey test -s` or `creevey test --storybook-start` to automatically start Storybook. Creevey will find a free port if necessary.
+
+---
+
+## Other Notable Changes
+
+- **`--no-docker` Option:** A command-line flag to disable Docker, overriding `useDocker: true` in your config.
+
+---
+
+We recommend reviewing the full announcement for details on all new features and improvements. After updating, test your suite thoroughly to ensure a smooth transition.

package/docs/playwright-reporter.md

@@ -0,0 +1,170 @@
+# Playwright Reporter for Creevey
+
+Creevey provides a Playwright-compatible reporter that allows you to use Creevey's image comparison, reporting, and UI features with Playwright tests.
+
+## Installation
+
+The reporter is included with the Creevey package. Install Playwright and Creevey in your project:
+
+```bash
+npm install --save-dev @playwright/test creevey
+```
+
+## Configuration
+
+Add the Creevey reporter to your Playwright configuration file:
+
+```typescript
+// playwright.config.ts
+import { defineConfig, devices } from '@playwright/test';
+
+export default defineConfig({
+  testDir: './tests',
+  testMatch: 'visual.spec.ts',
+  globalSetup: 'creevey/playwright/setup',
+  // Configure reporters
+  reporter: [
+    ['list'], // Standard Playwright reporter for console output
+    ['creevey/playwright/reporter'],
+  ],
+  // Reference images and report will be saved in these directories
+  snapshotDir: './images',
+  outputDir: './report',
+  use: {
+    viewport: { width: 1280, height: 720 },
+    baseURL: 'http://localhost:6006',
+  },
+  projects: [
+    {
+      name: 'chromium',
+      use: { ...devices['Desktop Chrome'] },
+      metadata: {
+        // Define storybook globals here (optional)
+        storybookGlobals: {
+          theme: 'dark',
+        },
+      },
+    },
+  ],
+  webServer: {
+    command: 'yarn storybook dev --ci -p 6006',
+    url: 'http://localhost:6006',
+    reuseExistingServer: !process.env.CI,
+  },
+});
+```
+
+### Required Configuration
+
+- **Global Setup**: `globalSetup: 'creevey/playwright/setup'` - Required for proper initialization
+- **Reporter**: `['creevey/playwright/reporter', options]` - The Creevey reporter
+- **Output Directories**: Use standard Playwright `snapshotDir` and `outputDir` options
+
+## Usage
+
+### Basic Test Setup
+
+Create a visual test file using Creevey's `definePlaywrightTests()` function:
+
+```typescript
+// tests/visual.spec.ts
+import { definePlaywrightTests } from 'creevey/playwright';
+
+definePlaywrightTests();
+```
+
+This simple setup will automatically:
+
+- Discover your Storybook stories
+- Generate Playwright tests for each story
+- Take screenshots and compare them with reference images
+- Integrate with the Creevey UI for reviewing differences
+
+### Storybook Globals Configuration
+
+You can configure Storybook globals for each browser project:
+
+```typescript
+projects: [
+  {
+    name: 'chromium-light',
+    use: { ...devices['Desktop Chrome'] },
+    metadata: {
+      storybookGlobals: {
+        theme: 'light',
+      },
+    },
+  },
+  {
+    name: 'chromium-dark',
+    use: { ...devices['Desktop Chrome'] },
+    metadata: {
+      storybookGlobals: {
+        theme: 'dark',
+      },
+    },
+  },
+];
+```
+
+## Running Tests
+
+Run your visual tests with Playwright:
+
+```bash
+# Run all tests
+yarn playwright test
+
+# Run with debug output
+yarn playwright test --debug
+
+
+# Run tests for specific browser
+yarn playwright test --project=chromium
+```
+
+### Viewing Results
+
+After running your tests, you can run `yarn creevey report` to start the Creevey UI to review results:
+
+1. The reporter will output the URL where you can view results
+2. Navigate to the provided URL in your browser
+3. Browse test results, view image comparisons, and approve changes
+
+### Approving Screenshots
+
+You can approve screenshots directly from the Creevey UI by:
+
+1. Selecting the test with the failed comparison
+2. Reviewing the differences between expected and actual images
+3. Clicking the "Approve" button to update the reference image
+
+## Troubleshooting
+
+### Common Issues
+
+**Tests not discovering stories**:
+
+- Ensure your Storybook server is running on the configured URL
+- Check that `globalSetup: 'creevey/playwright/setup'` is included in your config
+- Verify your `baseURL` points to your Storybook instance
+
+**Screenshots not being captured**:
+
+- Make sure you're using the `definePlaywrightTests()` function
+- Check that `snapshotDir` is properly configured and writable
+
+**Port conflicts**:
+
+- If Storybook fails to start, ensure the port isn't already in use
+- Configure a different port in the `webServer` configuration
+
+### Storybook Issues
+
+If tests can't connect to Storybook:
+
+1. Ensure Storybook starts successfully with `yarn storybook dev`
+2. Check that the URL in `webServer.url` matches your Storybook instance
+3. Verify your `baseURL` is correctly configured
+
+For more help, please check the main Creevey documentation or open an issue on GitHub.

package/docs/storybook.md

@@ -0,0 +1,60 @@
+## Creevey Storybook Parameters
+
+Creevey allows you to customize how stories will be captured. You could define parameters on `global`, `kind` or `story` levels. All these parameters are deeply merged by Storybook for each story.
+
+```ts
+// .storybook/preview.tsx
+export const parameters = {
+  creevey: {
+    // Global parameters are applied to all stories
+    captureElement: '#storybook-root',
+  },
+};
+```
+
+```ts
+// stories/MyModal.stories.tsx
+import React from 'react';
+import { Meta, StoryObj } from '@storybook/react';
+import { CreeveyMeta, CreeveyStory } from 'creevey';
+import MyModal from './src/components/MyModal';
+
+const Kind: Meta<typeof MyModal> = {
+  title: 'MyModal',
+  component: MyModal,
+  parameters: {
+    creevey: {
+      // It's possible to add additional delay before capturing screenshot
+      delay: 1000,
+
+      // For capturing the whole browser viewport, you can define `null` instead of css selector.
+      captureElement: null,
+
+      // You can skip some stories from capturing, by defining `skip` option:
+      // skip: { "The reason why story is skipped": { in: 'chrome', stories: 'Loading' } }
+      // - `in` - browser name, regex or array of browser names, which are defined in the Creevey config
+      // - `stories` - story name, regex or array of story names
+      // - `tests` - test name, regex or array of test names
+      // NOTE: If you try to skip stories by story name, the storybook name format will be used
+      // For more info see [storybook-export-vs-name-handling](https://storybook.js.org/docs/formats/component-story-format/#storybook-export-vs-name-handling))
+      skip: {
+        "`MyModal` doesn't support ie11": { in: 'ie11' },
+        'Loading stories are flaky in firefox': { in: 'firefox', stories: 'Loading' },
+        "`MyModal` hovering doesn't work correctly": {
+          in: ['firefox', 'chrome'],
+          tests: /.*hover$/,
+        },
+      },
+    },
+  },
+};
+
+export default Kind;
+
+export const Basic: StoryObj<typeof MyModal> = {
+  creevey: {
+    // Lastly some elements can be ignored in capturing screenshot
+    ignoreElements: ['button', '.local-time'],
+  },
+};
+```

package/docs/tests.md

@@ -1,63 +1,68 @@
-## Write tests
+## Write interactive screenshot tests
 
-By default Creevey generate for each story very simple screenshot test. In most cases it would be enough to test your UI. But you may want to do some interactions and capture one or multiple screenshots with different states of your story. For this case you could write custom tests, like this
+In most cases following Storybook's ideology of [writing stories](https://storybook.js.org/docs/get-started/whats-a-story) is enough to test your UI components. Where each component has a separate stories file with its different states. But sometimes you might have pretty complicated components with a lot of interactions and internal states. In this case, you can write tests for your stories.
 
-```tsx
-import React from 'react';
-import { Story } from '@storybook/react';
-import { CreeveyStory } from 'creevey';
-import MyComponent from './src/components/MyComponent';
+There are two different ways how to write interactive tests with Creevey:
 
-export default { title: 'MyComponent' };
+### Write tests in `*.creevey.ts` files
 
-export const Basic: Story & CreeveyStory = () => <MyComponent />;
-Basic.parameters = {
-  creevey: {
-    captureElement: '#storybook-root',
-    tests: {
-      async click() {
-        await this.browser.actions().click(this.captureElement).perform();
+It's the recommended way to write tests. It allows you to run these tests by Creevey itself and utilize webdriver benefits. The crucial part of it is webdriver action calls are more close to real user interactions and mitigate flakiness and false-negative results. Here is a simple example of how to write tests in `*.creevey.ts` files
 
-        await this.expect(await this.takeScreenshot()).to.matchImage('clicked component');
-      },
-    },
-  },
-};
+```ts
+// stories/MyComponent.creevey.ts
+import { kind, story, test } from 'creevey';
+
+kind('MyComponent', () => {
+  story('Story', ({ setStoryParameters }) => {
+    // It's possible to pass Creevey parameters to story
+    setStoryParameters({
+      captureElement: 'span[data-test-id~="x"]',
+      ignoreElements: [],
+    });
+
+    test('idle', async (context) => {
+      await context.matchImage(await context.takeScreenshot());
+    });
+
+    test('input', async (context) => {
+      await context.webdriver.keyboard.press('Tab');
+      const focus = await context.takeScreenshot();
+      await context.webdriver.keyboard.type('Hello Creevey');
+      const input = await context.takeScreenshot();
+      await context.matchImages({ focus, input });
+    });
+  });
+});
 ```
 
-NOTE: Here you define story parameters with simple test `click`. Where you setup capturing element `#storybook-root` then click on that element and taking screenshot to assert it. `this.browser` allow you to access to native selenium webdriver instance you could check [API here](https://www.selenium.dev/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html).
+In the example above, we used Playwright API to interact with the story. But Creevey also supports Selenium webdriver. And in that case `context.webdriver` will be an instance of Selenium webdriver. Obviously Selenium API is different from Playwright.
 
-You also could write more powerful tests with asserting multiple screenshots
+### Using Storybook's `play` function
 
-```tsx
-import React from 'react';
-import { CSFStory } from 'creevey';
-import MyForm from './src/components/MyForm';
+Storybook allows you to write tests in the story file itself by using [`play` function](https://storybook.js.org/docs/writing-tests/component-testing). It's a good way to write simple tests. But there are couple drawbacks of this approach:
 
-export default { title: 'MyForm' };
+- You can have only one test per story. Which is not a big deal, but sometimes you might not want to have multiple stories with the same markup.
+- Tests are running in browser environment and use https://testing-library.com API under the hood. It's good for unit tests, but might not be suitable for visual regression tests, because testing-library relies on DOM API and not even close to real user interactions. For example, you might have a button that could be visible for user, but it's covered by some other transparent element. With testing-library the button easily accessible and clickable, but the user can't interact with it.
 
-export const Basic: CSFStory<JSX.Element> = () => <MyForm />;
-Basic.story = {
-  parameters: {
-    creevey: {
-      captureElement: '#storybook-root',
-      delay: 1000,
-      tests: {
-        async submit() {
-          const input = await this.browser.findElement({ css: '.my-input' });
+Here is an example of how to write tests using Storybook's `play` function:
 
-          const empty = await this.takeScreenshot();
+```tsx
+// stories/MyComponent.stories.tsx
+import React from 'react';
+import { Meta, StoryObj } from '@storybook/react';
+import { fireEvent, within } from '@storybook/test';
+import MyComponent from './src/components/MyComponent';
 
-          await this.browser.actions().click(input).sendKeys('Hello Creevey').sendKeys(this.keys.ENTER).perform();
+export default {
+  title: 'MyComponent',
+  component: MyComponent,
+};
 
-          const submitted = await this.takeScreenshot();
+export const Basic: StoryObj<typeof MyComponent> = {
+  play: async ({ canvasElement }) => {
+    const slider = await within(canvasElement).findByTestId('slider');
 
-          await this.expect({ empty, submitted }).to.matchImages();
-        },
-      },
-    },
+    await fireEvent.change(slider, { target: { value: 50 } });
   },
 };
 ```
-
-NOTE: In this example I fill some simple form and submit it. Also as you could see, I taking two different screenshots `empty` and `submitted` and assert these in the end.