dappress 0.3.0 → 0.3.2

package/README.md

@@ -95,12 +95,14 @@ Add the command types to `tsconfig.json`:
 
 JavaScript projects use the same files with a `.js` extension and `module.exports`.
 
-Run the tests in a headed browser that supports extensions:
+Run the tests in a browser that supports extensions:
 
 ```bash
 npx cypress run --browser chrome-for-testing --headed
 ```
 
+Without `--headed`, the run is headless and works the same, except for the wallet cache, which needs a headed browser.
+
 ## Usage
 
 ```ts
@@ -157,7 +159,7 @@ Settings normally live in `cypress/wallet.setup.ts`. Secrets go in `cypress.env.
 | `network` | | None. The dapp stays on the network MetaMask starts on, Ethereum mainnet. |
 | `backupAndSync` | | `false`. Dappress turns off MetaMask's backup and sync while importing the wallet, so that an account a test adds is not restored by the next import of the same seed phrase. With `true`, MetaMask keeps saving and restoring the accounts and contacts of that phrase. |
 | `autoSetup` | | `true`. Dappress imports or unlocks the wallet before the first test of each spec. Set to `false` to call `cy.setupMetaMask()` yourself. |
-| `cache` | | `false`. The wallet is imported in every run, about fifteen seconds. With `true`, and a seed phrase of your own, it is imported once, in a browser Dappress opens before the run, and the resulting profile is reused by later runs, which then start by unlocking the wallet. |
+| `cache` | | `false`. The wallet is imported in every run, about fifteen seconds. With `true`, and a seed phrase of your own, it is imported once, in a browser Dappress opens before the run, and the resulting profile is reused by later runs, which then start by unlocking the wallet. Headed runs only: a headless run imports the wallet as usual. |
 | `timeout` | | `20000` ms. The time allowed for MetaMask to display a request before a command fails. |
 
 ### Security
@@ -171,11 +173,15 @@ With `cache: true`, the profile under `~/.cache/dappress/profiles` holds the wal
 The suite runs one test per command against [MetaMask's test dapp](https://metamask.github.io/test-dapp/). Each run makes a wallet nobody has used, with Foundry's `cast`, and starts a local [Anvil](https://getfoundry.sh) node that funds it. It requires Chrome for Testing and Foundry.
 
 ```bash
-npm run conformance               # default MetaMask version
-npm run conformance -- 13.51.0    # a specific release
+npm run conformance                            # default MetaMask version, side panel
+npm run conformance -- 13.51.0                 # a specific release
+DAPPRESS_MODE=headless npm run conformance     # without a browser window
+DAPPRESS_MODE=popup npm run conformance        # wallet from the profile cache: requests in the popup
 ```
 
-Each run writes `reports/metamask-<version>.json`. The GitHub workflow runs the suite daily against the latest MetaMask release that has no report yet.
+A mode is where MetaMask shows the dapp's requests: `sidepanel` (the default), `headless`, or `popup`. Each run writes `reports/metamask-<version>-<mode>.json`, and `npm run matrix -- reports .` turns the reports into `MATRIX.md` and `badge.json`, a [shields.io endpoint](https://shields.io/badges/endpoint-badge) for the latest version.
+
+The GitHub workflow runs the suite daily, in the three modes, against the latest MetaMask release that has no report yet, and publishes the reports, the matrix and the badge on the [`conformance-reports`](https://github.com/blassaut/dappress/tree/conformance-reports) branch: [MATRIX.md](https://github.com/blassaut/dappress/blob/conformance-reports/MATRIX.md).
 
 ### Updating to a new MetaMask release
 
@@ -187,7 +193,9 @@ Each run writes `reports/metamask-<version>.json`. The GitHub workflow runs the
 
 Cypress executes tests inside the dapp's tab and has no access to the extension. For each command, Dappress connects Puppeteer to the browser Cypress launched, through the debugging URL Cypress provides, locates the MetaMask page displaying the request and interacts with it.
 
-MetaMask displays requests in its side panel when the panel is open, and in its popup window otherwise. Without the cache, the import of the wallet ends by opening the side panel, so requests appear there. With the cache, the wallet was imported in another browser, the panel is closed, and requests appear in the popup. Dappress handles both, and the conformance suite covers both.
+MetaMask displays requests in its side panel when the panel is open, and in its popup window otherwise. Without the cache, the import of the wallet ends by opening the side panel, so requests appear there. With the cache, the wallet was imported in another browser, the panel is closed, and requests appear in the popup. Dappress handles both, and the conformance suite runs in both: its `popup` mode turns the cache on.
+
+Cypress leaves extensions out of a headless launch, so Dappress passes MetaMask to Chrome itself. Headless Chrome displays the side panel but does not open the popup window, which is why the cache, whose requests appear in the popup, is limited to headed runs.
 
 With `cache: true`, Dappress imports the wallet once before the run, in a browser of its own, and keeps the profile; MetaMask's storage is copied from it into the profile Cypress is about to launch, so each run starts from a wallet that has never seen the dapp.
 
@@ -217,7 +225,7 @@ Selectors are MetaMask's `data-testid` attributes, each with the button's Englis
 
 ## Status
 
-Verified with MetaMask 13.49.0 and 13.50.0, Cypress 16 and Chrome for Testing 154, in headed mode, on macOS and on GitHub's Linux runners. Headless execution is not validated yet.
+Verified with MetaMask 13.49.0 and 13.50.0, Cypress 16 and Chrome for Testing 154: in headed mode on macOS and on GitHub's Linux runners, and in headless mode on macOS.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dappress",
-  "version": "0.3.0",
+  "version": "0.3.2",
   "description": "MetaMask automation for Cypress: load the real extension, import a test wallet, answer dapp requests with cy.* commands. Tracks the latest MetaMask release.",
   "license": "MIT",
   "author": "Benjamin Lassaut",
@@ -51,6 +51,8 @@
   },
   "scripts": {
     "conformance": "node scripts/conformance.js",
+    "matrix": "node scripts/matrix.js",
+    "test": "node --test scripts/*.test.js",
     "check-types": "tsc -p tsconfig.json"
   },
   "peerDependencies": {

package/src/index.js

@@ -20,9 +20,12 @@ function configureDappress(on, config, userOptions = {}) {
     console.warn('[dappress] chromeWebSecurity is off: MetaMask cannot start its snaps, so adding or importing an account will hang');
   }
 
+  // A cached wallet gets its requests in MetaMask's popup window, which headless Chrome doesn't open
+  const usesCache = (browser) => options.cache && !browser.isHeadless;
+
   // Build the wallet profile before the run, out of the time Cypress allows a browser to come up
   on('before:run', async ({ browser }) => {
-    if (options.cache && browser) await prepareProfile({ browserPath: browser.path, extensionDir: await prepareExtension(options), options });
+    if (browser && usesCache(browser)) await prepareProfile({ browserPath: browser.path, extensionDir: await prepareExtension(options), options });
   });
 
   on('before:browser:launch', async (browser, launchOptions) => {
@@ -31,7 +34,12 @@ function configureDappress(on, config, userOptions = {}) {
     }
     const extensionDir = await prepareExtension(options);
     launchOptions.extensions.push(extensionDir);
-    if (options.cache) {
+    if (browser.isHeadless) {
+      // Cypress leaves extensions out of a headless launch, so ask Chrome directly
+      launchOptions.args.push(`--load-extension=${extensionDir}`);
+      if (options.cache) console.warn('[dappress] The wallet cache needs a headed browser: importing the wallet in this run instead');
+    }
+    if (usesCache(browser)) {
       const profileDir = await prepareProfile({ browserPath: browser.path, extensionDir, options });
       await installProfile(profileDir, browser, config.isTextTerminal);
     }

package/src/metamask-pages.js

@@ -74,7 +74,7 @@ async function getConfirmationPage(browser, extensionId, timeout) {
   while (Date.now() < deadline) {
     for (const pathname of CONFIRMATION_PATHS) {
       const [target] = pagesOf(browser, extensionId, pathname).filter((candidate) => !isHomeRoute(candidate.url()));
-      if (target) return target.page();
+      if (target) return pathname === '/notification.html' ? withUsableWindow(await target.page()) : target.page();
     }
     await sleep(250);
   }
@@ -88,4 +88,28 @@ async function getConfirmationPage(browser, extensionId, timeout) {
   );
 }
 
-module.exports = { findExtensionId, getHomePage, getConfirmationPage };
+// MetaMask asks Chrome for a 400x620 popup. Under Xvfb on a GitHub runner the
+// window it gets is 1x1: nothing is laid out, the pane of a confirmation that
+// must be read to the end has no height, so it can never be scrolled there and
+// its Confirm button stays disabled. The popup is given its size back first.
+const POPUP = { width: 400, height: 620 };
+
+async function withUsableWindow(page) {
+  const session = await page.target().createCDPSession();
+  try {
+    const { windowId, bounds } = await session.send('Browser.getWindowForTarget');
+    if (bounds.windowState === 'normal' && bounds.width >= 200 && bounds.height >= 200) return page;
+    console.log(`[dappress] MetaMask's popup is ${bounds.width}x${bounds.height} (${bounds.windowState}); resizing it to ${POPUP.width}x${POPUP.height}`);
+    if (bounds.windowState !== 'normal') await session.send('Browser.setWindowBounds', { windowId, bounds: { windowState: 'normal' } });
+    await session.send('Browser.setWindowBounds', { windowId, bounds: { ...POPUP } });
+    // Let the page lay itself out at its new size before anything is looked for in it
+    await sleep(300);
+  } catch (error) {
+    console.warn(`[dappress] Could not check the popup window's size: ${error.message}`);
+  } finally {
+    await session.detach().catch(() => {});
+  }
+  return page;
+}
+
+module.exports = { findExtensionId, getHomePage, getConfirmationPage, withUsableWindow };

package/src/metamask.js

@@ -66,7 +66,10 @@ const selectors = {
   confirmation: {
     confirm: [testId('confirm-footer-button'), buttonText('Confirm'), buttonText('Approve')],
     cancel: [testId('confirm-footer-cancel-button'), buttonText('Cancel')],
-    scrollToBottom: testId('confirm-scroll-to-bottom'),
+    // The arrow of the newer confirmations has a class and no test id
+    scrollToBottom: `${testId('confirm-scroll-to-bottom')}, .confirm-scroll-to-bottom__button`,
+    // The pane the content scrolls in, by its inline style: it has no test id
+    scrollPane: '[style*="overflow: auto"]',
   },
   // The older footer, still used by the permission update a network switch
   // asks for, by "Add suggested tokens", and by template confirmations
@@ -83,6 +86,8 @@ const selectors = {
   // sends many requests in a row, and it swallows the clicks
   modal: {
     content: '.mm-modal-content',
+    // The cross of the modal's header, when it has one
+    close: '.mm-modal-content button[aria-label="Close"]',
     lastButton: 'xpath/(.//div[contains(@class, "mm-modal-content")]//button)[last()]',
   },
 };
@@ -220,7 +225,7 @@ async function reloadHome(page) {
  * is pressed again when no account shows up.
  */
 async function addAccount(page) {
-  await click(page, selectors.home.accountMenu);
+  await openAccountList(page);
   const before = await accountNames(page);
   for (let attempt = 0; attempt < 3; attempt++) {
     const added = (await newAccount(page, before, 0)) || (await pressAddAccount(page, before));
@@ -255,19 +260,42 @@ async function accountNames(page) {
 
 /** Make `name` the selected account of the wallet. */
 async function switchAccount(page, name) {
-  await click(page, selectors.home.accountMenu);
+  await openAccountList(page);
   await selectAccount(page, name);
 }
 
-// Picking an account closes the list and shows it in the home header
+// A click on the menu is lost while the home screen settles, as with the
+// buttons in the list, or swallowed by a modal MetaMask shows over the home
+// screen after some activity (the Transaction Shield offer, for one): any
+// modal is dismissed first, and the menu is pressed again until the list shows.
+async function openAccountList(page) {
+  for (let attempt = 0; attempt < 3; attempt++) {
+    // Back in front: a tab something opened meanwhile would hide this page and stall its rendering
+    await page.bringToFront().catch(() => {});
+    await dismissModal(page);
+    await click(page, selectors.home.accountMenu);
+    if (await isVisible(page, selectors.accounts.name, 10000)) return;
+  }
+  throw await failure(page, 'MetaMask did not open the account list');
+}
+
+// Picking an account closes the list and shows it in the home header. As with
+// adding one, a lost click leaves the list open, so the account is pressed again.
 async function selectAccount(page, name) {
-  await click(page, selectors.accounts.cell(name));
-  const deadline = Date.now() + 15000;
+  for (let attempt = 0; attempt < 3; attempt++) {
+    await click(page, selectors.accounts.cell(name));
+    if (await isSelected(page, name, 5000)) return;
+  }
+  throw await failure(page, `MetaMask did not select "${name}"`);
+}
+
+async function isSelected(page, name, timeout) {
+  const deadline = Date.now() + timeout;
   while (Date.now() < deadline) {
-    if ((await selectedAccount(page)) === name) return;
+    if ((await selectedAccount(page)) === name) return true;
     await sleep(250);
   }
-  throw await failure(page, `MetaMask did not select "${name}"`);
+  return false;
 }
 
 async function selectedAccount(page) {
@@ -315,13 +343,10 @@ function approveNetworkChange(page, timeout) {
 async function pressAndWaitForDismissal(page, button, timeout) {
   await waitFor(page, button, { timeout });
   await dismissModal(page);
-  if (await isVisible(page, selectors.confirmation.scrollToBottom, 500)) {
-    await click(page, selectors.confirmation.scrollToBottom);
-  }
   const logged = recordErrors(page);
   const request = page.url();
   try {
-    await clickWhenEnabled(page, button);
+    await clickWhenEnabled(page, button, { whileDisabled: () => scrollContentToEnd(page) });
     for (let attempt = 0; attempt < 3; attempt++) {
       if (await isVisible(page, selectors.alert.acknowledge, 500)) await click(page, selectors.alert.acknowledge);
       if (await isGone(page, button, 3000)) return;
@@ -355,11 +380,47 @@ function recordErrors(page) {
   };
 }
 
-/** Close a modal shown over the confirmation (Escape, or its last button: "Cancel"). */
+/**
+ * MetaMask keeps the button of a confirmation disabled until its content was
+ * read to the end: scrolled to the bottom, or not scrollable at all. Some
+ * screens offer a button for that, most don't, so the content is scrolled
+ * with the wheel, as a reader would. The popup, being small, needs it often.
+ */
+async function scrollContentToEnd(page) {
+  const scrollButton = await page.$(selectors.confirmation.scrollToBottom);
+  if (scrollButton) {
+    await scrollButton.click().catch(() => {});
+    return;
+  }
+  // The pane itself, to its end: that fires the scroll event MetaMask listens to
+  await page
+    .$$eval(selectors.confirmation.scrollPane, (panes) => {
+      for (const pane of panes) {
+        if (pane.scrollHeight > pane.clientHeight) pane.scrollTop = pane.scrollHeight;
+      }
+    })
+    .catch(() => {});
+  // And the wheel over the content, for a pane styled otherwise
+  const [width, height] = await page.evaluate(() => [document.documentElement.clientWidth, document.documentElement.clientHeight]);
+  await page.mouse.move(Math.round(width / 2), Math.round(height / 2));
+  await page.mouse.wheel({ deltaY: 10000 });
+}
+
+/**
+ * Close a modal shown over the screen: Escape, then the cross of its header,
+ * then its last button, which is "Cancel" on the "multiple requests" modal.
+ * The last button is the last resort: on an offer such as Transaction
+ * Shield's, it is a call to action that opens a page, which hides this one.
+ */
 async function dismissModal(page) {
   if (!(await isVisible(page, selectors.modal.content, 300))) return;
   await page.keyboard.press('Escape');
   if (await isGone(page, selectors.modal.content, 1000)) return;
+  const close = await page.$(selectors.modal.close);
+  if (close) {
+    await close.click().catch(() => {});
+    if (await isGone(page, selectors.modal.content, 1000)) return;
+  }
   await dispatchClick(page, selectors.modal.lastButton);
 }
 

package/src/page-helpers.js

@@ -55,21 +55,80 @@ async function isGone(page, selector, timeout = 1500) {
 }
 
 async function click(page, selector, options) {
-  const element = await waitFor(page, selector, options);
-  await waitForStill(element);
-  await element.click();
+  await clickFresh(page, selector, options);
 }
 
-/** Wait until the element is both visible and enabled, then click it. */
-async function clickWhenEnabled(page, selector, { timeout = DEFAULT_TIMEOUT } = {}) {
-  const element = await waitFor(page, selector, { timeout });
+/**
+ * Wait until the element is both visible and enabled, then click it. The
+ * element is taken again at each turn: MetaMask re-renders its screens, and
+ * a handle on a replaced element reads its last state forever.
+ */
+async function clickWhenEnabled(page, selector, { timeout = DEFAULT_TIMEOUT, whileDisabled } = {}) {
   const deadline = Date.now() + timeout;
-  while (await element.evaluate((el) => el.disabled)) {
-    if (Date.now() > deadline) throw await failure(page, `"${describe(selector)}" stayed disabled for ${timeout}ms`);
+  let nudged = 0;
+  for (;;) {
+    const element = await waitFor(page, selector, { timeout });
+    if (!(await element.evaluate((el) => el.disabled))) break;
+    if (Date.now() > deadline) {
+      throw await failure(page, `"${describe(selector)}" stayed disabled for ${timeout}ms. ${await describeDisabled(page, element)}`);
+    }
+    // Something the page wants done before it enables the button, every half second
+    if (whileDisabled && Date.now() - nudged > 500) {
+      await whileDisabled();
+      nudged = Date.now();
+    }
     await sleep(100);
   }
-  await waitForStill(element);
-  await element.click();
+  await clickFresh(page, selector, { timeout });
+}
+
+// What may keep a button disabled: the button itself (a spinner, a class),
+// a pane left to scroll, the page not being the focused one
+async function describeDisabled(page, element) {
+  try {
+    const state = await page.evaluate((button) => {
+      const panes = [...document.querySelectorAll('[style*="overflow"]')]
+        .filter((pane) => pane.scrollHeight > pane.clientHeight)
+        .map((pane) => `${pane.scrollTop}+${pane.clientHeight}/${pane.scrollHeight}`);
+      return {
+        button: button.outerHTML.replace(/\s+/g, ' ').slice(0, 300),
+        panesToScroll: panes,
+        viewport: `${document.documentElement.clientWidth}x${document.documentElement.clientHeight}`,
+        focused: document.hasFocus(),
+        visibility: document.visibilityState,
+      };
+    }, element);
+    state.window = await windowBounds(page);
+    return `Button: ${JSON.stringify(state)}.`;
+  } catch {
+    return '';
+  }
+}
+
+// The size and state of the window the page is in, as "400x620 normal"
+async function windowBounds(page) {
+  const session = await page.target().createCDPSession();
+  try {
+    const { bounds } = await session.send('Browser.getWindowForTarget');
+    return `${bounds.width}x${bounds.height} ${bounds.windowState}`;
+  } catch {
+    return 'unknown';
+  } finally {
+    await session.detach().catch(() => {});
+  }
+}
+
+// Find the element and click it; once more if a re-render replaced it between the two
+async function clickFresh(page, selector, options) {
+  for (let attempt = 0; ; attempt++) {
+    const element = await waitFor(page, selector, options);
+    await waitForStill(element);
+    try {
+      return await element.click();
+    } catch (error) {
+      if (attempt > 0 || !/detached/.test(error.message)) throw error;
+    }
+  }
 }
 
 /** Click by dispatching the event on the element itself, for buttons something may cover. */
@@ -97,16 +156,34 @@ async function fill(page, selector, text, options) {
   await element.type(text);
 }
 
-/** The error for something that went wrong on `page`, with a screenshot to look at. */
+/**
+ * The error for something that went wrong on `page`: what the page shows, in
+ * words for the log and as a screenshot to look at.
+ */
 async function failure(page, message) {
   const prefix = `[dappress] ${message} on ${page.url()}`;
+  const shown = await pageText(page);
   try {
     fs.mkdirSync(SCREENSHOTS_DIR, { recursive: true });
     const file = path.join(SCREENSHOTS_DIR, `${Date.now()}.png`);
     await page.screenshot({ path: file });
-    return new Error(`${prefix}. Screenshot: ${file}`);
+    return new Error(`${prefix}. ${shown}Screenshot: ${file}`);
+  } catch {
+    return new Error(`${prefix}. ${shown}`.trim());
+  }
+}
+
+// The visible text of the page, shortened: enough to tell which MetaMask
+// screen it is from the log alone, when the screenshot is out of reach
+async function pageText(page) {
+  try {
+    const text = await Promise.race([
+      page.evaluate(() => document.body.innerText.replace(/\s+/g, ' ').trim()),
+      sleep(2000).then(() => null),
+    ]);
+    return text ? `The page shows: "${text.length > 500 ? `${text.slice(0, 500)}…` : text}". ` : '';
   } catch {
-    return new Error(prefix);
+    return '';
   }
 }
 

package/src/profile.js

@@ -31,7 +31,14 @@ async function buildProfile(dir, { browserPath, extensionDir, options }) {
       executablePath: browserPath,
       headless: false,
       userDataDir: dir,
-      args: [`--load-extension=${extensionDir}`, `--disable-extensions-except=${extensionDir}`, '--no-first-run'],
+      args: [
+        `--load-extension=${extensionDir}`,
+        `--disable-extensions-except=${extensionDir}`,
+        '--no-first-run',
+        // No sandbox on Linux: Ubuntu 23.10+ (GitHub's ubuntu-latest) blocks the
+        // user namespaces it needs, and this browser only imports a test wallet
+        ...(process.platform === 'linux' ? ['--no-sandbox'] : []),
+      ],
     });
 
   await withMetaMask(launch, (home) => metamask.onboard(home, options));
@@ -49,13 +56,29 @@ async function buildProfile(dir, { browserPath, extensionDir, options }) {
 async function withMetaMask(launch, action) {
   const browser = await launch();
   try {
-    await action(await getHomePage(browser, await findExtensionId(browser)));
+    const home = await getHomePage(browser, await findExtensionId(browser));
+    await keepInFront(browser, home);
+    await action(home);
     await new Promise((resolve) => setTimeout(resolve, 3000));
   } finally {
     await browser.close();
   }
 }
 
+// Chrome stops rendering a tab that isn't the active one, and Puppeteer waits
+// on rendering to find an element or to click it: a hidden tab never answers.
+// So the home page is made the active tab, and again whenever MetaMask opens
+// a tab of its own, as it does for its onboarding right after the install.
+async function keepInFront(browser, home) {
+  const toFront = () => home.bringToFront().catch(() => {});
+  await toFront();
+  browser.on('targetcreated', (target) => {
+    if (target.type() !== 'page') return;
+    toFront();
+    setTimeout(toFront, 500);
+  });
+}
+
 // Where Cypress is about to create the browser profile:
 // <app data>/Cypress/cy/<env>/browsers/<name>-<channel>/run-<pid of the Cypress process>,
 // or .../interactive in open mode. The pid is our parent's: Cypress runs

package/types/index.d.ts

@@ -33,7 +33,7 @@ export interface DappressOptions extends WalletSetup {
   backupAndSync?: boolean;
   /** Run cy.setupMetaMask() before the first test of each spec. Default: true. */
   autoSetup?: boolean;
-  /** Import the wallet once and reuse the profile across runs. Keeps the vault under cacheDir. Default: false. */
+  /** Import the wallet once and reuse the profile across runs. Keeps the vault under cacheDir. Headed runs only. Default: false. */
   cache?: boolean;
   /** Time allowed for MetaMask to display a request, in ms. Default: 20000. */
   timeout?: number;