gatsby-source-filesystem 5.5.0-next.0 → 5.5.0-next.1

package/CHANGELOG.md

@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+## [5.4.0](https://github.com/gatsbyjs/gatsby/commits/gatsby-source-filesystem@5.4.0/packages/gatsby-source-filesystem) (2023-01-10)
+
+[🧾 Release notes](https://www.gatsbyjs.com/docs/reference/release-notes/v5.4)
+
+#### Chores
+
+- update babel monorepo [#37386](https://github.com/gatsbyjs/gatsby/issues/37386) ([b941876](https://github.com/gatsbyjs/gatsby/commit/b94187633d94d0f0071b38ffe93380dd802ec70f))
+
 ### [5.3.1](https://github.com/gatsbyjs/gatsby/commits/gatsby-source-filesystem@5.3.1/packages/gatsby-source-filesystem) (2022-12-14)
 
 **Note:** Version bump only for package gatsby-source-filesystem

package/README.md

@@ -1,35 +1,28 @@
 # gatsby-source-filesystem
 
-A Gatsby source plugin for sourcing data into your Gatsby application
-from your local filesystem.
+A Gatsby source plugin for sourcing data into your Gatsby application from your local filesystem.
 
-The plugin creates `File` nodes from files. The various "transformer"
-plugins can transform `File` nodes into various other types of data e.g.
-`gatsby-transformer-json` transforms JSON files into JSON data nodes and
-`gatsby-transformer-remark` transforms markdown files into `MarkdownRemark`
-nodes from which you can query an HTML representation of the markdown.
+The plugin creates `File` nodes from files. The various "transformer" plugins can transform `File` nodes into various other types of data e.g. [`gatsby-transformer-json`](https://www.gatsbyjs.com/plugins/gatsby-transformer-json/) transforms JSON files into JSON data nodes and [`gatsby-transformer-remark`](https://www.gatsbyjs.com/plugins/gatsby-transformer-remark/) transforms markdown files into `MarkdownRemark` nodes from which you can query an HTML representation of the markdown.
 
 ## Install
 
-`npm install gatsby-source-filesystem`
+```shell
+npm install gatsby-source-filesystem
+```
 
 ## How to use
 
-```javascript
-// In your gatsby-config.js
+You can have multiple instances of this plugin to read source nodes from different locations on your filesystem. Be sure to give each instance a unique `name`.
+
+```js:title=gatsby-config.js
 module.exports = {
   plugins: [
-    // You can have multiple instances of this plugin
-    // to read source nodes from different locations on your
-    // filesystem.
-    //
-    // The following sets up the Jekyll pattern of having a
-    // "pages" directory for Markdown files and a "data" directory
-    // for `.json`, `.yaml`, `.csv`.
     {
       resolve: `gatsby-source-filesystem`,
       options: {
+        // The unique name for each instance
         name: `pages`,
+        // Path to the directory
         path: `${__dirname}/src/pages/`,
       },
     },
@@ -38,7 +31,10 @@ module.exports = {
       options: {
         name: `data`,
         path: `${__dirname}/src/data/`,
-        ignore: [`**/\.*`], // ignore files starting with a dot
+        // Ignore files starting with a dot
+        ignore: [`**/\.*`],
+        // Use "mtime" and "inode" to fingerprint files (to check if file has changed)
+        fastHash: true,
       },
     },
   ],
@@ -47,9 +43,23 @@ module.exports = {
 
 ## Options
 
-In addition to the name and path parameters you may pass an optional `ignore` array of file globs to ignore.
+### name
+
+**Required**
+
+A unique name for the `gatsby-source-filesytem` instance. This name will also be a key on the `File` node called `sourceInstanceName`. You can use this e.g. for filtering.
+
+### path
+
+**Required**
+
+Path to the folder that should be sourced. Ideally an absolute path.
 
-They will be added to the following default list:
+### ignore
+
+**Optional**
+
+Array of file globs to ignore. They will be added to the following default list:
 
 ```text
 **/*.un~
@@ -62,8 +72,24 @@ They will be added to the following default list:
 ../**/dist/**
 ```
 
+### fastHash
+
+**Optional**
+
+By default, `gatsby-source-filesystem` creates an MD5 hash of each file to determine if it has changed between sourcing. However, on sites with many large files this can lead to a significant slowdown. Thus you can enable the `fastHash` setting to use an alternative hashing mechanism.
+
+`fastHash` uses the `mtime` and `inode` to fingerprint the files. On a modern OS this can be considered a robust solution to determine if a file has changed, however on older systems it can be unreliable. Therefore it's not enabled by default.
+
+### Environment variables
+
 To prevent concurrent requests overload of `processRemoteNode`, you can adjust the `200` default concurrent downloads, with `GATSBY_CONCURRENT_DOWNLOAD` environment variable.
 
+In case that due to spotty network, or slow connection, some remote files fail to download. Even after multiple retries and adjusting concurrent downloads, you can adjust timeout and retry settings with these environment variables:
+
+- `GATSBY_STALL_RETRY_LIMIT`, default: `3`
+- `GATSBY_STALL_TIMEOUT`, default: `30000`
+- `GATSBY_CONNECTION_TIMEOUT`, default: `30000`
+
 ## How to query
 
 You can query file nodes like the following:
@@ -263,7 +289,7 @@ The `createFileNodeFromBuffer` helper accepts a `Buffer`, caches its contents to
 
 The name of the file can be passed to the `createFileNodeFromBuffer` helper. If no name is given, the content hash will be used to determine the name.
 
-## Example usage
+#### Example usage
 
 The following example is adapted from the source of [`gatsby-source-mysql`](https://github.com/malcolm-kee/gatsby-source-mysql):
 
@@ -338,11 +364,3 @@ function createMySqlNodes({ name, __sql, idField, keys }, results, ctx) {
 
 module.exports = createMySqlNodes
 ```
-
-## Troubleshooting
-
-In case that due to spotty network, or slow connection, some remote files fail to download. Even after multiple retries and adjusting concurrent downloads, you can adjust timeout and retry settings with these environment variables:
-
-- `GATSBY_STALL_RETRY_LIMIT`, default: `3`
-- `GATSBY_STALL_TIMEOUT`, default: `30000`
-- `GATSBY_CONNECTION_TIMEOUT`, default: `30000`

package/create-file-node.js

@@ -4,12 +4,12 @@ const path = require(`path`);
 const fs = require(`fs-extra`);
 const mime = require(`mime`);
 const prettyBytes = require(`pretty-bytes`);
-const md5File = require(`md5-file`);
 const {
   createContentDigest,
-  slash
+  slash,
+  md5File
 } = require(`gatsby-core-utils`);
-exports.createFileNode = async (pathToFile, createNodeId, pluginOptions = {}) => {
+exports.createFileNode = async (pathToFile, createNodeId, pluginOptions = {}, cache = null) => {
   const slashed = slash(pathToFile);
   const parsedSlashed = path.parse(slashed);
   const slashedFile = {
@@ -31,7 +31,19 @@ exports.createFileNode = async (pathToFile, createNodeId, pluginOptions = {}) =>
       description: `Directory "${path.relative(process.cwd(), slashed)}"`
     };
   } else {
-    const contentDigest = await md5File(slashedFile.absolutePath);
+    const key = stats.mtimeMs.toString() + stats.ino.toString();
+    let contentDigest;
+    if (pluginOptions.fastHash) {
+      // Skip hashing.
+      contentDigest = key;
+    } else {
+      // Generate a hash, but only if the file has changed.
+      contentDigest = cache && (await cache.get(key));
+      if (!contentDigest) {
+        contentDigest = await md5File(slashedFile.absolutePath);
+        if (cache) await cache.set(key, contentDigest);
+      }
+    }
     const mediaType = mime.getType(slashedFile.ext);
     internal = {
       contentDigest,

package/gatsby-node.js

@@ -32,10 +32,11 @@ const createFSMachine = ({
   },
   getNode,
   createNodeId,
-  reporter
+  reporter,
+  cache
 }, pluginOptions) => {
   const createAndProcessNode = path => {
-    const fileNodePromise = createFileNode(path, createNodeId, pluginOptions).then(fileNode => {
+    const fileNodePromise = createFileNode(path, createNodeId, pluginOptions, cache).then(fileNode => {
       createNode(fileNode);
       return null;
     });
@@ -190,6 +191,7 @@ exports.pluginOptionsSchema = ({
 }) => Joi.object({
   name: Joi.string(),
   path: Joi.string(),
+  fastHash: Joi.boolean().default(false),
   ignore: Joi.array().items(Joi.string(), Joi.object().regex(), Joi.function())
 });
 exports.sourceNodes = (api, pluginOptions) => {

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "gatsby-source-filesystem",
   "description": "Gatsby source plugin for building websites from local data. Markdown, JSON, images, YAML, CSV, and dozens of other data types supported.",
-  "version": "5.5.0-next.0",
+  "version": "5.5.0-next.1",
   "author": "Kyle Mathews <mathews.kyle@gmail.com>",
   "bugs": {
     "url": "https://github.com/gatsbyjs/gatsby/issues"
@@ -10,10 +10,9 @@
     "@babel/runtime": "^7.20.7",
     "chokidar": "^3.5.3",
     "file-type": "^16.5.4",
-    "fs-extra": "^10.1.0",
-    "gatsby-core-utils": "^4.5.0-next.0",
-    "md5-file": "^5.0.0",
-    "mime": "^2.6.0",
+    "fs-extra": "^11.1.0",
+    "gatsby-core-utils": "^4.5.0-next.1",
+    "mime": "^3.0.0",
     "pretty-bytes": "^5.6.0",
     "valid-url": "^1.0.9",
     "xstate": "^4.34.0"
@@ -47,5 +46,5 @@
   "engines": {
     "node": ">=18.0.0"
   },
-  "gitHead": "4ef5d775d18329abc5d4e78d1e0a99e904f00cd1"
+  "gitHead": "f8f084ae96bd105509903216daa766325ec65b9e"
 }