http-snapshotter 0.3.0 → 0.4.0-beta.2

package/README.md

@@ -11,10 +11,10 @@ To have predictable inputs to external requests there are 2 popular approaches:
 However stubs / fakes take quite a while to write. And a mock service is an additional piece to deploy and maintain. 
 
 Presenting you another solution:
+
 3. Create snapshots of the requests automatically the first time you run your test and then replay the snapshot responses on future runs of the test.
-Additionally with the approach, with predictability and speed in mind, one wouldn't want any real network request from being made; and if it does happen, then the test should fail.
 
-WARNING: This module isn't concurrent or thread safe yet. You can only use it on serial test runners like `tape`. If you use `ava`, you need to convert tests to run serially with `test.serial()`.
+Additionally with the approach, with predictability and speed in mind, one wouldn't want any real network request from being made; and if it does happen, then the test should fail.
 
 Example (test.js):
 
@@ -22,19 +22,18 @@ Example (test.js):
 import test from "tape";
 import { fileURLToPath } from "node:url";
 import { resolve, dirname } from "node:path";
-import { start } from "http-snapshotter";
-
-const __filename = fileURLToPath(import.meta.url);
-const __dirname = dirname(__filename);
-const snapshotDirectory = resolve(__dirname, "http-snapshots");
+import { start, startTestCase, endTestCase } from "http-snapshotter";
 
-start({ snapshotDirectory });
+const __dirname = dirname(fileURLToPath(import.meta.url));
+start({ snapshotDirectory: resolve(__dirname, "http-snapshots") });
 
 test("Latest XKCD comic (ESM)", async (t) => {
+  startTestCase('test-case-1');
   const res = await fetch("https://xkcd.com/info.0.json");
   const json = await res.json();
 
   t.deepEquals(json.title, "Iceberg Efficiency", "must be equal");
+  endTestCase();
 });
 ```
 
@@ -49,11 +48,11 @@ You will see a file named `get-xkcd-com-info-0-arAlFb5gfcr9aCN.json` created in
 Then onwards running: `node test.js` or `SNAPSHOT=read node test.js` will ensure HTTP network calls are all read from a snapshot file.
 In this mode, http-snapshotter will prevent any real HTTP calls from happening by failing the test (if it didn't have a snapshot file) and print out the request details and the snapshot file name it should have had.
 
-There is also a `SNAPSHOT=ignore` option to neither read nor write from snapshot files and do real network requests instead. This could be useful while writing a new test.
+For adding new snapshots without touching existing snapshots use `SNAPSHOT=append`. There is also a `SNAPSHOT=ignore` option to neither read nor write from snapshot files and do real network requests instead. These could be useful while writing a new test.
 
-Tip: When you do `SNAPSHOT=update` to create snapshots, run it against a single test, so you know what exact snapshots that one test created/updated.
+Tip: When you do `SNAPSHOT=update` or `SNAPHOT=append` to create snapshots, run it against a single test, so you know what exact snapshots that one test created/updated.
 
-Log read/saved snapshots by setting LOG_SNAPSHOT=1 env variable. Log requests with LOG_REQ=1 or LOG_REQ=summary (to just print request HTTP method, url and snapshot file that it would use).
+Log read/saved snapshots by setting LOG_SNAPSHOT=1 or LOG_SNAPSHOT=summary env variable. It prints the HTTP method, url and snapshot file that it would use. If you want even more details in the logs use LOG_REQ=detailed.
 
 Once you are done writing your tests, run your test runner on all your tests and then take a look at `<snapshots directory>/unused-snapshots.log` file to see which snapshot files haven't been used by your final test suite. You can delete unused snapshot files.
 
@@ -62,7 +61,7 @@ The tests of this library uses this library itself, check the `tests/` directory
 ## About snapshot files and its names
 
 A snapshot file name uniquely identifies a request. By default it is a combination of HTTP method + URL + body that makes a request unique (headers are ignored).
-For example, take the filename `get-xkcd-com-info-0-arAlFb5gfcr9aCN.json` - The prefix `get-xkcd-com-info-0` is added just for readability, and the suffix `arAlFb5gfcr9aCN` is a SHA256 hash of concatenated HTTP method + URL + body string that makes the file name unique.
+For example, take the filename `get-xkcd-com-info-0-arAlFb5gfcr9aCN.json` - The prefix `get-xkcd-com-info-0` is added just for readability, and the suffix `arAlFb5gfcr9aCN` is a SHA256 hash of concatenated HTTP method + URL + body of request that makes the file name unique.
 
 However you may want to specially handle some requests. e.g. DynamoDB calls also need the `x-amz-target` header to uniquely identify the request,
 because the header affects the response data. You can add logic to create better snapshot files for this case:
@@ -189,3 +188,11 @@ test('Test behavior on a free account', async (t) => {
 ```
 
 Now when you run `SNAPHOT=update node test2.js` you will get a snapshot file with `free-account-test-` as prefix. You can now edit the JSON response for this test.
+
+## Concurrency
+
+WARNING: This module isn't concurrent or thread safe. Make sure that:
+
+1. within one worker only one test is being executed at a time. e.g. If you use `ava`, and you have multiple `test()` blocks in one file, you need to change it to run serially with `test.serial()`.
+
+2. parallel tests don't update the same snapshot file at the same time (i.e. while you run with SNAPSHOT=update). Regardless, updating snapshots of multiple tests at the same time is not a great idea in my opinion, because reviewing the snapshots files are a pain, escpecially if you have a shared snapshot files.

package/index.d.ts

@@ -40,6 +40,15 @@ export type ReadSnapshotReturnType = Promise<{
 }>;
 export type ClientRequestInterceptorType = import('@mswjs/interceptors/ClientRequest').ClientRequestInterceptor;
 export type FetchInterceptorType = import('@mswjs/interceptors/fetch').FetchInterceptor;
+/**
+ * Write/read snapshots to/from a sub directory. This isolates snapshots for a test.
+ * @param {string} directoryName Directory name relative to snapshot directory. It will be created if it doesn't exist.
+ */
+export function startTestCase(directoryName: string): void;
+/**
+ * Reset the directory to the root directory
+ */
+export function endTestCase(): void;
 /**
  * @param {Request} request
  */

package/index.js

@@ -10,7 +10,9 @@
  * SNAPSHOT=update <test runner command>
  * e.g. SNAPSHOT=update tape tests/**\/*.js | tap-diff
  *
- * Here onwards run test runner without SNAPSHOT env variable or SNAPSHOT=read
+ * Here onwards run test runner without SNAPSHOT env variable or SNAPSHOT=read.
+ * For adding new snapshots without touching existing snapshots use SNAPSHOT=append.
+ *
  * You can use SNAPSHOT=ignore to neither read not write snapshots, for testing on real
  * network operations.
  * 
@@ -19,8 +21,8 @@
  * Unused snapshot files will be written into a log file named 'unused-snapshots.log'.
  * You can delete those files manually.
  * 
- * Log requests with LOG_REQ=1 or LOG_REQ=summary (to just print summary) env variable
- * or node.js built-in NODE_DEBUG=http,http2
+ * Log requests with LOG_REQ=1 or LOG_REQ=summary (to just print summary) or LOG_REQ=detailed
+ * (to print request details) env variable or node.js built-in NODE_DEBUG=http,http2
  *
  * More docs at the end of this file, find the exported methods.
  */
@@ -31,9 +33,9 @@ const { FetchInterceptor } = require('@mswjs/interceptors/fetch');
 const slugify = require('@sindresorhus/slugify');
 const { createHash } = require('node:crypto');
 const { promises: fs } = require('node:fs');
-const { resolve } = require('node:path');
+const { resolve, dirname, relative } = require('node:path');
 
-// Environment variable SNAPSHOT = update / ignore / read (default)
+// Environment variable SNAPSHOT = update / append / ignore / read (default)
 const SNAPSHOT = process.env.SNAPSHOT || 'read';
 const { LOG_REQ, LOG_SNAPSHOT } = process.env;
 const unusedSnapshotsLogFile = 'unused-snapshots.log';
@@ -79,18 +81,30 @@ let snapshotDirectory = null;
  * @typedef {SnapshotText | SnapshotJson} Snapshot
  */
 
+const dynamodbHostNameRegex = /^dynamodb\..+\.amazonaws\.com$/;
 
 const defaultKeyDerivationProps = ['method', 'url', 'body'];
 /**
  * @param {Request} request 
  */
 async function defaultSnapshotFileNameGenerator(request) {
+  let filePrefix;
+
   const url = new URL(request.url);
-  const filePrefix = [
-    request.method.toLowerCase(),
-    slugify(url.hostname),
-    slugify(url.pathname.replace('.json', '')),
-  ].filter(Boolean).join('-');
+  if (dynamodbHostNameRegex.test(url.hostname)) {
+    filePrefix = [
+      'dynamodb',
+      // slugify(url.hostname), // FIXME: uncomment this next release
+      slugify(request.headers?.get?.('x-amz-target')?.split?.('.')?.pop?.() || ''),
+      slugify(JSON.parse(await request.clone().text())?.TableName),
+    ].filter(Boolean).join('-');
+  } else {
+    filePrefix = [
+      request.method.toLowerCase(),
+      slugify(url.hostname),
+      slugify(url.pathname.replace('.json', '')),
+    ].filter(Boolean).join('-');
+  }
 
   // Input data
   const dataList = await Promise.all(
@@ -114,6 +128,7 @@ async function defaultSnapshotFileNameGenerator(request) {
  * @type {(req: Request) => Promise<{ filePrefix: string, fileSuffixKey: string }>}
  */
 let snapshotFileNameGenerator = defaultSnapshotFileNameGenerator;
+let snapshotSubDirectory = '';
 
 /**
  * @param {Request} request
@@ -127,7 +142,7 @@ async function getSnapshotFileName(request) {
     .digest('base64url')
     .slice(0, 15);
 
-  const fileName = `${filePrefix}-${hash}.json`;
+  const fileName = `${snapshotSubDirectory ? `${snapshotSubDirectory}/` : ''}${filePrefix}-${hash}.json`;
 
   return {
     absoluteFilePath: resolve(/** @type {string} */ (snapshotDirectory), fileName),
@@ -148,14 +163,23 @@ async function getSnapshotFileName(request) {
 /** @type {Map<string, ReadSnapshotReturnType>} */
 const alreadyWrittenFiles = new Map();
 const readFiles = new Set();
+const existingSubDirectories = new Set();
 
 /**
- * @param {Request} request
- * @param {Response} response
+ * @param {object} param
+ * @param {Request} param.request
+ * @param {Response} param.response
+ * @param {string} param.absoluteFilePath
+ * @param {string} param.fileName
+ * @param {string} param.fileSuffixKey
  */
-async function saveSnapshot(request, response) {
-  const { absoluteFilePath, fileName, fileSuffixKey } = await getSnapshotFileName(request);
-
+async function saveSnapshot({
+  request,
+  response,
+  absoluteFilePath,
+  fileName,
+  fileSuffixKey,
+}) {
   // Prevent multiple tests from having same snapshot
   if (alreadyWrittenFiles.has(absoluteFilePath)) {
     return /** @type {ReadSnapshotReturnType} */ (alreadyWrittenFiles.get(absoluteFilePath));
@@ -220,6 +244,11 @@ async function saveSnapshot(request, response) {
       fileSuffixKey,
     };
     const json = JSON.stringify(snapshot, null, 2);
+    const dir = dirname(absoluteFilePath);
+    if (!existingSubDirectories.has(dir)) {
+      existingSubDirectories.add(dir);
+      await fs.mkdir(dir, { recursive: true });
+    }
     await fs.writeFile(absoluteFilePath, json, 'utf-8');
     return { snapshot, absoluteFilePath, fileName };
   };
@@ -249,6 +278,7 @@ async function readSnapshot(request) {
       // Fail any test that fires a real network request (without snapshot)
       // @ts-ignore
       if (err.code === 'ENOENT') {
+        if (SNAPSHOT === 'append') return {};
         const reqBody = await request.clone().text();
         console.error('No network snapshot found for request with cache keys:', {
           request: {
@@ -312,16 +342,10 @@ async function sendResponse(request, snapshot) {
  */
 async function readSnapshotAndSendResponse(request) {
   const { snapshot } = await readSnapshot(request);
-  return sendResponse(request, snapshot);
-}
-
-/**
- * @param {Request} request
- * @param {Response} response
- */
-async function saveSnapshotAndSendResponse(request, response) {
-  const { snapshot } = await saveSnapshot(request, response);
-  return sendResponse(request, snapshot);
+  if (snapshot) {
+    return sendResponse(request, snapshot);
+  }
+  return undefined;
 }
 
 /** @typedef {import('@mswjs/interceptors/ClientRequest').ClientRequestInterceptor} ClientRequestInterceptorType */
@@ -336,15 +360,18 @@ let unusedFiles;
 process.on('beforeExit', async () => {
   if (SNAPSHOT === 'read' && !beforeExitEventSeen) {
     beforeExitEventSeen = true;
+    const dir = /** @type {string} */(snapshotDirectory);
+    /** @type {import('node:fs').Dirent[]} */
     let files;
     try {
-      // @ts-ignore
-      files = await fs.readdir(snapshotDirectory);
+      files = await fs.readdir(dir, { recursive: true, withFileTypes: true });
     } catch (err) {
       return;
     }
-    let dir = /** @type {string} */(snapshotDirectory);
-    unusedFiles = files.filter((file) => !readFiles.has(file) && file !== unusedSnapshotsLogFile);
+    unusedFiles = files
+      .filter((file) => file.isFile())
+      .map((file) => relative(dir, resolve(file.path, file.name)))
+      .filter((file) => (!readFiles.has(file) && file !== unusedSnapshotsLogFile)); 
     if (unusedFiles.length) {
       await fs
         .writeFile(
@@ -365,6 +392,23 @@ process.on('beforeExit', async () => {
   }
 });
 
+/**
+ * Write/read snapshots to/from a sub directory. This isolates snapshots for a test.
+ * @param {string} directoryName Directory name relative to snapshot directory. It will be created if it doesn't exist.
+ */
+function startTestCase(directoryName) {
+  if (snapshotSubDirectory) {
+    throw new Error(`Cannot start test case '${directoryName}' as test case '${snapshotSubDirectory}' is already running.`); 
+  }
+  snapshotSubDirectory = directoryName;
+}
+/**
+ * Reset the directory to the root directory
+ */
+function endTestCase() {
+  snapshotSubDirectory = '';
+}
+
 /**
  * Attach snapshot filename generator function
  * 
@@ -421,7 +465,7 @@ function start({
 
   // @ts-ignore
   interceptor.on('request', async ({ request }) => {
-    if (SNAPSHOT === 'read') {
+    if (['read', 'append'].includes(SNAPSHOT)) {
       await readSnapshotAndSendResponse(request);
     }
   });
@@ -430,12 +474,12 @@ function start({
     'response',
     /** @type {(params: { request: Request, response: Response }) => Promise<void>} */
     async ({ request, response }) => {
+      const { absoluteFilePath, fileName, fileSuffixKey } = await getSnapshotFileName(request);
       if (LOG_REQ) {
-        const { fileName, fileSuffixKey } = await getSnapshotFileName(request);
         const summary = `----------\n${request.method} ${request.url}\nWould use file name: ${fileName}`;
-        if (LOG_REQ === 'summary') {
+        if (LOG_REQ === '1' || LOG_REQ === 'summary') {
           console.debug(summary);
-        } else {
+        } else if (LOG_REQ === 'detailed') {
           console.debug(`${summary}\n----------\n`, {
             request: {
               url: request.url,
@@ -453,12 +497,14 @@ function start({
           });
         }
       }
-      if (SNAPSHOT === 'update') {
+      if (SNAPSHOT === 'update' || (SNAPSHOT === 'append' && !readFiles.has(fileName))) {
         if (!dirCreatePromise) {
           dirCreatePromise = fs.mkdir( /** @type {string} */(snapshotDirectory), { recursive: true });
         }
         await dirCreatePromise;
-        await saveSnapshotAndSendResponse(request, response);
+        await saveSnapshot({
+          request, response, absoluteFilePath, fileName, fileSuffixKey,
+        });
       }
     },
   );
@@ -475,6 +521,8 @@ function stop() {
 
 // Singleton - as it makes sense only one interceptor be active at any given moment.
 module.exports = {
+  startTestCase,
+  endTestCase,
   defaultSnapshotFileNameGenerator,
   attachSnapshotFilenameGenerator,
   resetSnapshotFilenameGenerator,

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "http-snapshotter",
-  "version": "0.3.0",
+  "version": "0.4.0-beta.2",
   "description": "Snapshot HTTP requests for tests (node.js)",
-  "main": "index.cjs",
+  "main": "index.js",
   "types": "index.d.ts",
   "exports": {
     "import": "./index.mjs",
@@ -10,7 +10,7 @@
     "types": "./index.d.ts"
   },
   "scripts": {
-    "test": "tape tests/**/*.test.* | tap-diff",
+    "test": "tape tests/**/*.test.* | tap-arc",
     "tsc": "tsc"
   },
   "keywords": [
@@ -29,7 +29,7 @@
   },
   "devDependencies": {
     "@types/node": "^20.6.2",
-    "tap-diff": "^0.1.1",
+    "tap-arc": "^1.3.2",
     "tape": "^5.6.6",
     "typescript": "^5.2.2"
   }