@public-ui/visual-tests 1.7.0-rc.8 → 1.7.0

package/README.md

@@ -1,168 +1,53 @@
-# KoliBri - CLI-Tools
+# KoliBri - Visual Tests
 
 ## Motivation
 
-The `KoliBri` CLI-Tools are a collection of tools to support the development with `KoliBri` components.
+The `KoliBri` Visual Tests provide a way to add visual regression testing to **theme** modules.  
+It takes screenshots of every component defined in the [React Sample App](https://github.com/public-ui/kolibri/tree/develop/packages/samples/react) with the theme applied and compares them to their references.
 
 ## Installation
 
-You can install the `KoliBri` CLI-Tools with `npm`, `pnpm` or `yarn`.
+You can install the `KoliBri` Visual Tests with `npm`, `pnpm` or `yarn`:
 
 ```bash
-npm i -g @public-ui/kolibri-cli
-pnpm i -g @public-ui/kolibri-cli
-yarn add -g @public-ui/kolibri-cli
+npm i -D @public-ui/visual-tests
+pnpm i -D @public-ui/visual-tests
+yarn add -D @public-ui/visual-tests
 ```
 
 ## Usage
 
-The `KoliBri` CLI is intended to be executed in your project root directory. Use the `kolibri` command to start the CLI.
+Add the following npm scripts to the theme's `package.json`:
 
-```bash
-kolibri --help
-```
-
-```bash
-,--. ,--.         ,--. ,--. ,-----.           ,--.
-|  .'   /  ,---.  |  | `--' |  |) /_  ,--.--. `--'
-|  .   '  | .-. | |  | ,--. |  .-.  \ |  .--' ,--.
-|  |\   \ | '-' | |  | |  | |  '--' / |  |    |  |
-`--' `--´  `---´  `--' `--' `------´  `--'    `--'
-🚹 The accessible HTML-Standard | 👉 https://public-ui.github.io
-
-Usage: kolibri [options] [command]
-
-CLI for executing some helpful commands for KoliBri projects.
-
-Options:
-  -V, --version               output the version number
-  -h, --help                  display help for command
-
-Commands:
-  migrate [options] <string>  This command migrates KoliBri code to the current version.
-  help [command]              display help for command
+```json
+{
+	"scripts": {
+		"test": "THEME_MODULE=src/index THEME_EXPORT=THEME_NAME kolibri-visual-test",
+		"test-update": "THEME_MODULE=src/index THEME_EXPORT=THEME_NAME kolibri-visual-test --update-snapshots"
+	}
+}
 ```
 
-### Migrate
-
-With the `migrate` command you can migrate your project to the latest version of `KoliBri`.
-
-Actually the following migrations are available:
-
-- Component renaming ✓
-- Component removal ⏰
-- Property renaming ✓
-- Property removal ✓
-- Property type change ⏰
-- Logic refactoring ⏰
-- Expert-Slot refactoring ⏰
-- `.vscode/settings.json` add IntelliSense for HTML ⏰
-- `.gitignore` exclude `.kolibri.migrate.json` ⏰
-- `.tsconfig` add `@public-ui/components` to `types` array ⏰
-
-#### How does it work?
-
-1. The migration command will check your project for clear `git history` and the `installed version` of `KoliBri`. Now it loads all available migration tasks.
-2. Tasks in the correct version range will be executed one by one. Otherwise they will be skipped.
-3. After that the `package.json` will be updated with the new version of `KoliBri` and execute the `npm install` command.
-4. If there are any pending tasks, the migration command will be executed again. Otherwise the migration is finished.
-5. Now you can check the result and commit the changes.
+- `THEME_MODULE` defines the relative path to the TypeScript module containing the the theme definitions. Without file extension.
+- `THEME_EXPERT` defines the name of the export within the the module. (e.g., `export const THEME_NAME = {/**/};`) Defaults to `default`.
 
-> **Note:** You can reset the migration with `git reset --hard HEAD~1` or by discarding the affected files.
+Run the tests with `npm test`. The first time, this will create a new folder `snapshots` which is supposed to be committed to the repository.
+In the following runs, new screenshots will be compared to this reference.
 
-#### Help
+To update the reference screenshots call `npm run test-update`.
 
-```bash
-kolibri migrate --help
-```
-
-#### Execute
+It's recommended to add the `test-results` folder to the `.gitignore` file:
 
 ```bash
-kolibri migrate <path>
+/test-results/
 ```
 
-#### Options
-
-| Option                         | Description                               |         Type         | Default  |
-| ------------------------------ | ----------------------------------------- | :------------------: | :------: |
-| `--ignore-uncommitted-changes` | Allows execution with uncommitted changes |       boolean        |  false   |
-| `--remove-mode`                | Prefix property name or delete property   | `delete` \| `prefix` | `prefix` |
-
-#### Configuration
-
-You can configure the migration with the `.kolibri.config.json` file in your project root folder. This file will be created automatically after the first migration.
-
-**Troubleshooting:** If you have problems with migration, you can exclude some tasks with the configuration by setting the `false` flag (see `kol-select`).
+It's also recommended to automatically run the tests before packing/publishing the module:
 
 ```json
 {
-	"migrate": {
-		"tasks": {
-			"kol-abbr-rename-property-_title-to-_label": true,
-			"kol-accordion-rename-property-_heading-to-_label": true,
-			"kol-badge-rename-property-_icon-only-to-_hide-label": true,
-			"kol-badge-remove-property-_hide-label": true,
-			"kol-badge-remove-property-_icon-only": true,
-			"kol-breadcrumb-rename-property-_aria-label-to-_label": true,
-			"kol-button-link-remove-property-_aria-current": true,
-			"kol-button-link-remove-property-_aria-label": true,
-			"kol-button-link-rename-property-_icon-only-to-_hide-label": true,
-			"kol-button-remove-property-_aria-current": true,
-			"kol-button-remove-property-_aria-label": true,
-			"kol-button-rename-property-_icon-only-to-_hide-label": true,
-			"kol-card-rename-property-_heading-to-_label": true,
-			"kol-card-rename-property-_headline-to-_label": true,
-			"kol-details-rename-property-_summary-to-_label": true,
-			"kol-icon-remove-property-_part": true,
-			"kol-icon-rename-property-_aria-label-to-_label": true,
-			"kol-checkbox-rename-property-_type-to-_variant": true,
-			"kol-color-rename-property-_list-to-_suggestions": true,
-			"kol-date-rename-property-_list-to-_suggestions": true,
-			"kol-input-email-rename-property-_list-to-_suggestions": true,
-			"kol-input-number-rename-property-_list-to-_suggestions": true,
-			"kol-input-radio-rename-property-_list-to-_options": true,
-			"kol-input-range-rename-property-_list-to-_suggestions": true,
-			"kol-input-text-rename-property-_list-to-_suggestions": true,
-			"kol-link-group-rename-property-_aria-label-to-_label": true,
-			"kol-link-group-remove-property-_ordered": true,
-			"kol-nav-rename-property-_aria-label-to-_label": true,
-			"kol-nav-rename-property-_compact-to-_hide-label": true,
-			"kol-nav-remove-property-_has-compact-button": true,
-			"kol-pagination-rename-property-_count-to-_total": true,
-			"kol-progress-rename-property-_type-to-_variant": true,
-			"kol-quote-rename-property-_caption-to-_label": true,
-			"kol-select-rename-property-_height-to-_rows": false,
-			"kol-select-rename-property-_list-to-_options": false,
-			"kol-skip-nav-rename-property-_aria-label-to-_label": true,
-			"kol-span-rename-property-_icon-only-to-_hide-label": true,
-			"kol-split-button-remove-property-_aria-label": true,
-			"kol-table-rename-property-_caption-to-_label": true,
-			"kol-tabs-rename-property-_aria-label-to-_label": true,
-			"kol-tabs-rename-property-_tab-align-to-_align": true,
-			"kol-toast-rename-property-_heading-to-_label": true,
-			"kol-version-rename-property-_version-to-_label": true
-		}
+	"scripts": {
+		"prepack": "npm test"
 	}
 }
 ```
-
-## Troubleshooting
-
-If the migration failed, you can reset the migration with `git reset --hard HEAD~1`.
-
-Use the configuration (`.kolibri.config.json`) to exclude some tasks.
-
-If there are multiple obsolete properties that have been migrated to just one new property, the new property may appear multiple times in the tag. You can then decide which variant to use and remove all other variants accordingly.
-
-Maybe it can help to prepare your code in the tricky places for migration.
-
-Please give us feedback, if you have problems with the migration: [GitHub Issues](https://github.com/public-ui/kolibri/issues/new?assignees=&labels=useful+hint&projects=&template=7_feedback.md&title=%F0%9F%92%A1+CLI%3A+)
-
-## Dry run
-
-You have always the possibility of a dry run. Because before the migration will be executed, you need a clean git history of you project.
-
-After the migration you can check the result with `git status` and `git diff`.
-
-Is anything wrong, you can reset the migration with `git reset --hard HEAD~1` or by discarding the affected files.

package/package.json

@@ -1,46 +1,49 @@
 {
-	"name": "@public-ui/visual-tests",
-	"version": "1.7.0-rc.8",
-	"license": "EUPL-1.2",
-	"homepage": "https://public-ui.github.io",
-	"repository": "https://github.com/public-ui/kolibri",
-	"bugs": {
-		"url": "https://github.com/public-ui/kolibri/issues",
-		"email": "kolibri@itzbund.de"
-	},
-	"author": {
-		"name": "Informationstechnikzentrum Bund",
-		"email": "kolibri@itzbund.de"
-	},
-	"type": "module",
-	"sideEffects": false,
-	"description": "Provides utility to run visual regression tests for themes.",
-	"scripts": {
-		"depcheck": "depcheck --ignores=@public-ui/sample-react,http-server,tslib",
-		"format": "prettier --check src",
-		"lint": "eslint src",
-		"unused": "knip"
-	},
-	"bin": {
-		"kolibri-visual-test": "src/index.js"
-	},
-	"dependencies": {
-		"@playwright/test": "1.37.1",
-		"@public-ui/sample-react": "1.7.0-rc.8",
-		"http-server": "14.1.1",
-		"portfinder": "1.0.32"
-	},
-	"devDependencies": {
-		"@babel/eslint-parser": "7.22.15",
-		"depcheck": "1.4.6",
-		"eslint": "8.49.0",
-		"eslint-plugin-no-loops": "0.3.0",
-		"knip": "2.24.0",
-		"prettier": "3.0.3"
-	},
-	"files": [
-		"playwright.config.js",
-		"src",
-		"tests"
-	]
-}
+  "name": "@public-ui/visual-tests",
+  "version": "1.7.0",
+  "license": "EUPL-1.2",
+  "homepage": "https://public-ui.github.io",
+  "repository": "https://github.com/public-ui/kolibri",
+  "bugs": {
+    "url": "https://github.com/public-ui/kolibri/issues",
+    "email": "kolibri@itzbund.de"
+  },
+  "author": {
+    "name": "Informationstechnikzentrum Bund",
+    "email": "kolibri@itzbund.de"
+  },
+  "type": "module",
+  "sideEffects": false,
+  "description": "Provides utility to run visual regression tests for themes.",
+  "bin": {
+    "kolibri-visual-test": "src/index.js"
+  },
+  "dependencies": {
+    "@playwright/test": "1.38.1",
+    "@public-ui/sample-react": "1.7.0",
+    "portfinder": "1.0.32",
+    "serve": "14.2.1"
+  },
+  "devDependencies": {
+    "@babel/eslint-parser": "7.22.15",
+    "@babel/plugin-syntax-import-attributes": "7.22.5",
+    "@babel/preset-env": "7.22.20",
+    "axe-playwright": "1.2.3",
+    "depcheck": "1.4.6",
+    "eslint": "8.50.0",
+    "eslint-plugin-no-loops": "0.3.0",
+    "knip": "2.30.0",
+    "prettier": "3.0.3"
+  },
+  "files": [
+    "playwright.config.js",
+    "src",
+    "tests"
+  ],
+  "scripts": {
+    "depcheck": "depcheck --ignores=@babel/plugin-syntax-import-attributes,@babel/preset-env,@public-ui/sample-react,portfinder,serve",
+    "format": "prettier --check src",
+    "lint": "eslint src",
+    "unused": "knip"
+  }
+}

package/playwright.config.js

@@ -0,0 +1,59 @@
+import { defineConfig, devices } from '@playwright/test';
+import * as path from 'path';
+import * as process from 'process';
+
+const PORT = Number(process.env.KOLIBRI_VISUAL_TEST_PORT);
+const URL = `http://127.0.0.1:${PORT}`;
+
+console.log('Serving React Sample app:', URL);
+
+/**
+ * See https://playwright.dev/docs/test-configuration.
+ */
+export default defineConfig({
+	testDir: './tests',
+	snapshotDir: path.join(process.env.KOLIBRI_CWD, 'snapshots'),
+	// snapshotPathTemplate: '',
+	outputDir: path.join(process.env.KOLIBRI_CWD, 'test-results'),
+	/* Run tests in files in parallel */
+	fullyParallel: true,
+	/* Fail the build on CI if you accidentally left test.only in the source code. */
+	forbidOnly: !!process.env.CI,
+	/* Retry on CI only */
+	retries: process.env.CI ? 2 : 0,
+	/* Opt out of parallel tests on CI. */
+	workers: process.env.CI ? 1 : undefined,
+	/* Reporter to use. See https://playwright.dev/docs/test-reporters */
+	reporter: 'line',
+	/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
+	use: {
+		/* Base URL to use in actions like `await page.goto('/')`. */
+		baseURL: URL,
+
+		/* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
+		trace: 'on-first-retry',
+	},
+
+	/* Configure projects for major browsers */
+	projects: [
+		// {
+		// 	name: 'chrome',
+		// 	use: { ...devices['Desktop Chrome'] },
+		// },
+		// {
+		// 	name: 'edge',
+		// 	use: { ...devices['Desktop Edge'] },
+		// },
+		{
+			name: 'firefox',
+			use: { ...devices['Desktop Firefox'] },
+		},
+	],
+
+	/* Run your local dev server before starting the tests */
+	webServer: {
+		command: `serve ${process.env.KOLIBRI_VISUAL_TESTS_BUILD_PATH} -p ${PORT}`,
+		url: URL,
+		reuseExistingServer: false,
+	},
+});

package/src/index.js

@@ -0,0 +1,80 @@
+import child_process from 'node:child_process';
+import path from 'node:path';
+import { fileURLToPath } from 'url';
+import * as crypto from 'crypto';
+import * as fs from 'fs';
+import portfinder from 'portfinder';
+import * as process from 'process';
+
+process.env.KOLIBRI_CWD = process.cwd();
+const tempDir = process.env.RUNNER_TEMP || process.env.TMPDIR; // TODO: Check on Windows
+
+if (!process.env.THEME_MODULE) {
+	throw new Error('Environment variable THEME_MODULE not specified.');
+}
+if (!tempDir) {
+	throw new Error('Neither environment variable RUNNER_TEMP or TMPDIR specified.');
+}
+
+process.env.THEME_MODULE = path.join(process.cwd(), process.env.THEME_MODULE); // Use current working directory (i.e. the theme folder) to complete module path
+const visualsTestModulePath = fileURLToPath(new URL('..', import.meta.url));
+const binaryPath = fileURLToPath(new URL('../node_modules/.bin', import.meta.url));
+
+let sampleReactPath = '../node_modules/@public-ui/sample-react';
+let backSteps = ``;
+let workingDir = null;
+do {
+	const url = new URL(backSteps + sampleReactPath, import.meta.url);
+	workingDir = fileURLToPath(url);
+	backSteps += `../`;
+} while (!fs.existsSync(workingDir) && path.resolve(process.cwd(), backSteps) !== '/');
+
+if (!fs.existsSync(workingDir)) {
+	throw new Error('Could not find React Sample App package. Please install it with "npm install @public-ui/sample-react".');
+}
+
+const buildPath = path.join(tempDir, `kolibri-visual-testing-build-${crypto.randomUUID()}`);
+const packageJsonPath = await import(new URL(`${workingDir}/package.json`, import.meta.url), {
+	assert: { type: 'json' },
+});
+
+process.env.KOLIBRI_VISUAL_TESTS_BUILD_PATH = buildPath;
+
+console.log(`
+Building React Sample App (v${packageJsonPath?.default?.version ?? '#.#.#'}) …`);
+child_process.execFileSync('npm', ['run', 'build', '--', `--output-path=${buildPath}`], {
+	cwd: workingDir,
+	encoding: 'utf-8',
+});
+
+console.log(`React Sample App build finished. Directory:`, buildPath);
+
+void (async () => {
+	process.env.KOLIBRI_VISUAL_TEST_PORT = String(await portfinder.getPortPromise());
+
+	const playwright = child_process.spawn(path.join(binaryPath, 'playwright'), ['test', ...process.argv.slice(2)], {
+		cwd: visualsTestModulePath,
+	});
+
+	playwright.stdout.on('data', (data) => {
+		console.log('Playwright: ' + data.toString());
+	});
+
+	playwright.stderr.on('data', (data) => {
+		console.log('Playwright stderr: ' + data.toString());
+	});
+
+	playwright.on('exit', (code) => {
+		console.log(`Playwright test finished with exit code ${code}.`);
+
+		if (process.env.KOLIBRI_CLEANUP === '0') {
+			console.log('Skipping cleanup up build folder.');
+			console.log(`You can serve this build with "npx serve ${buildPath}".`);
+		} else {
+			console.log('Cleaning up build folder …');
+			fs.rmSync(buildPath, { recursive: true, force: true });
+			console.log('Cleaning up finished successfully.');
+		}
+		process.exit(code ?? 1);
+	});
+})();

package/tests/axe-snapshots.spec.js

@@ -0,0 +1,82 @@
+import { test } from '@playwright/test';
+import { checkA11y, injectAxe } from 'axe-playwright';
+import path from 'path';
+import { ROUTES } from './sample-app.routes.js';
+
+const rename = (snapshotName) => {
+	const result = snapshotName
+
+		// Remove browser name from snapshot name
+		// .replace('-chromium', '')
+		// .replace('-firefox', '')
+
+		// Remove os name from snapshot name
+		// .replace('-darwin', '')
+		// .replace('-linux', '')
+		// .replace('-windows', '')
+
+		// Remove test counter from snapshot name
+		.replace('-1-', '-')
+
+		// Make different snapshot folder for different themes
+		.replace('theme-snapshots.spec.js', `axe-${process.env.THEME_EXPORT.toLocaleLowerCase()}`)
+		.replace('-snapshots', '');
+	return result;
+};
+
+// https://playwright.dev/docs/emulation
+test.use({
+	colorScheme: 'light',
+	locale: 'de-DE',
+	isMobile: false,
+	timezoneId: 'Europe/Berlin',
+	viewport: {
+		width: 800,
+		height: 0,
+	},
+});
+
+/**
+ * @todo stabilize and re-enable test
+ */
+const blocklist = [];
+
+ROUTES.forEach((options, route) => {
+	if (blocklist.includes(route)) {
+		return;
+	}
+	test(`snapshot for ${route}`, async ({ page }, testInfo) => {
+		const outputPath = rename(testInfo.outputDir);
+		await page.goto(`/#${route}?hideMenus`, { waitUntil: 'networkidle' });
+		if (options?.viewportSize) {
+			await page.setViewportSize(options.viewportSize);
+		}
+		if (options?.waitForTimeout) {
+			await page.waitForTimeout(options.waitForTimeout);
+		}
+		await injectAxe(page);
+		await checkA11y(
+			page,
+			undefined,
+			{
+				axeOptions: {
+					runOnly: {
+						type: 'tag',
+						values: ['best-practices', 'wcag2a', 'wcag2aa', 'wcag21aa'],
+					},
+				},
+				detailedReport: true,
+				detailedReportOptions: {
+					html: true,
+				},
+			},
+			options?.axe?.skipFailures ?? true,
+			'html',
+			{
+				outputDirPath: outputPath.replace(/\/[^/]+$/, ''),
+				outputDir: `axe-${process.env.THEME_EXPORT.toLocaleLowerCase()}`,
+				reportFileName: `${route.replace('/', '-')}.html`,
+			},
+		);
+	});
+});