svelteesp32 1.0.4 → 1.1.0

package/README.md

@@ -10,6 +10,8 @@ In order to be able to easily update OTA, it is important - from the users' poin
 
 This npm package provides a solution for **inserting any JS client application into the ESP32 web server** (PsychicHttp is my favorite, faster than ESPAsyncWebServer). For this, JS, html, css, font, assets, etc. files must be converted to binary byte array. Npm mode is easy to use and easy to **integrate into your CI/CD pipeline**.
 
+> Starting with version v1.1.0, the ETag header is also supported.
+
 ### Usage
 
 **Install package** as devDependency (it is practical if the package is part of the project so that you always receive updates)
@@ -21,7 +23,7 @@ npm install -D svelteesp32
 After a successful Svelte build (rollup/webpack/vite) **create an includeable c++ header** file
 
 ```bash
-npx svelteesp32 -s ../svelteapp/dist -o ../esp32project/svelteesp32.h
+npx svelteesp32 -s ../svelteapp/dist -o ../esp32project/svelteesp32.h --etag
 ```
 
 During the **translation process**, the processed file details are visible, and at the end, the result shows the ESP32's memory allocation (gzip size)
@@ -73,10 +75,17 @@ The content of **generated file** (do not edit, just use)
 ```c
 const uint8_t data0[12547] = {0x1f, 0x8b, 0x8, 0x0, ...
 const uint8_t data1[5368] = {0x1f, 0x8b, 0x8, 0x0, 0x0, ...
+const char * etag0 = "387b88e345cc56ef9091...";
 
 void initSvelteStaticFiles(PsychicHttpServer * server) {
   server->on("/assets/index-KwubEIf-.js", HTTP_GET, [](PsychicRequest * request)
   {
+    if (request->hasHeader("If-None-Match") && ...) {
+      PsychicResponse response304(request);
+      response304.setCode(304);
+      return response304.send();
+    }
+
     PsychicResponse response(request);
     response.setContentType("application/javascript");
     response.addHeader("Content-Encoding", "gzip");
@@ -86,6 +95,10 @@ void initSvelteStaticFiles(PsychicHttpServer * server) {
 
   server->on("/assets/index-Soe6cpLA.css", HTTP_GET, [](PsychicRequest * request)
   {
+    if (request->hasHeader("If-None-Match") && ...) {
+      ...
+    }
+
     PsychicResponse response(request);
     response.setContentType("text/css");
     response.addHeader("Content-Encoding", "gzip");
@@ -101,23 +114,32 @@ void initSvelteStaticFiles(PsychicHttpServer * server) {
 
 All modern browsers have been able to handle gzip-compressed content for years. For this reason, there is no question that the easily compressed JS and CSS files are stored compressed in the ESP32 and sent to the browser.
 
-During the translation process, data in gzip format is generated and will be used if the size is greater than 100 bytes and we experience a reduction of at least 15%. In such a case, the compressed data is unconditionally sent to the browser with the appropriate header information.
+During the translation process, data in gzip format is generated and will be used if the **size is greater than 100 bytes** and we experience a **reduction of at least 15%**. In such a case, the compressed data is unconditionally sent to the browser with the appropriate **Content-Encoding** header information.
 
 Automatic compression can be turned off with the `--no-gzip` option.
 
+### ETag
+
+The ETag HTTP header can be used to significantly reduce network traffic. If the server sends ETag information, the client can check the integrity of the file by sending back this ETag (in `If-None-Match` header) without sending the data back again. All browsers use this function, the 304 HTTP response code is clearly visible in the network traffic.
+
+Since ESP32 data traffic is moderately expensive, it is an individual decision whether to use the ETag or not. We **recommend using ETag**, which adds a bit more code (about 1-3%) but results in a much cleaner operation.
+
+The use of ETag is **not enabled by default**, this can be achieved with the `--etag` switch.
+
 ### Main entry point - index.html
 
 Typically, the entry point for web applications is the **index.htm or index.html** file. This does not need to be listed in the browser's address bar because web servers know that this file should be served by default. Svelteesp32 also does this: if there is an index.htm or index.html file, it sets it as the main file to be served. So using `http://esp32_xxx.local` or just entering the `http://x.y.w.z/` IP address will serve this main file.
 
 ### Command line options
 
-| Option        | Mandatory | Description                                    | default                 |
-| ------------- | :-------: | ---------------------------------------------- | ----------------------- |
-| `-s`          |     x     | Source dist folder contains compiled web files |                         |
-| `-o`          |     x     | Generated output file with path                | `svelteesp32.h`         |
-| `--no-gzip`   |           | Do not compress content with gzip              |                         |
-| `--espmethod` |     x     | Name of generated method                       | `initSvelteStaticFiles` |
-| `-h`          |           | Show help                                      |                         |
+| Option        | Required | Description                                    | default                 |
+| ------------- | :------: | ---------------------------------------------- | ----------------------- |
+| `-s`          |    x     | Source dist folder contains compiled web files |                         |
+| `-o`          |    x     | Generated output file with path                | `svelteesp32.h`         |
+| `--etag`      |          | Use ETag header for cache                      | false                   |
+| `--no-gzip`   |          | Do not compress content with gzip              | false -> gzip used      |
+| `--espmethod` |    x     | Name of generated method                       | `initSvelteStaticFiles` |
+| `-h`          |          | Show help                                      |                         |
 
 ### Q&A
 

package/dist/commandLine.d.ts

@@ -3,6 +3,7 @@ interface ICopyFilesArguments {
     outputfile: string;
     espmethod: string;
     'no-gzip': boolean;
+    etag: boolean;
     help?: boolean;
 }
 export declare const cmdLine: ICopyFilesArguments;

package/dist/commandLine.js

@@ -20,6 +20,11 @@ exports.cmdLine = (0, ts_command_line_args_1.parse)({
         description: 'Do not compress content with gzip',
         defaultValue: false
     },
+    etag: {
+        type: Boolean,
+        description: 'Use ETAG header for cache',
+        defaultValue: false
+    },
     espmethod: {
         type: String,
         description: 'Name of generated method',

package/dist/cppCode.d.ts

@@ -5,5 +5,6 @@ export type cppCodeSource = {
     mime: string;
     content: Buffer;
     isGzip: boolean;
+    md5: string;
 };
 export declare const getCppCode: (sources: cppCodeSource[]) => string;

package/dist/cppCode.js

@@ -7,12 +7,20 @@ const replaceAll = (s, from, to) => {
         s = s.replace(from, to);
     return s;
 };
+const blockTemplateETag = `if (request->hasHeader("If-None-Match") && request->header("If-None-Match") == String(etag$index$)) {
+      PsychicResponse response304(request);
+      response304.setCode(304);
+      return response304.send();
+    }
+`;
 const blockTemplate = `
   $default$server->on("/$filename$", HTTP_GET, [](PsychicRequest * request)
   {
+    ${commandLine_1.cmdLine.etag ? blockTemplateETag : ''}
     PsychicResponse response(request);
     response.setContentType("$mime$");
     response.addHeader("Content-Encoding", "$encoding$");
+    ${commandLine_1.cmdLine.etag ? `response.addHeader("ETag", etag$index$);` : ''}
     response.setContent(data$index$, sizeof(data$index$));
     return response.send();
   });
@@ -27,22 +35,34 @@ const getCppBlock = (source) => {
     result = replaceAll(result, '$encoding$', source.isGzip ? 'gzip' : 'identity');
     return result;
 };
-const fileTemplate = `
-$arrays$
+const fileTemplate = `//cmdline:  $commandline$
+//created:  $now$
+//files:    $filecount$
+//memory:   $filesize$
+
+$filedataarrays$
+$hashaarrays$
 
 void $method$(PsychicHttpServer * server) {
 $code$
 }`;
 const getCppCode = (sources) => {
-    const arrays = [];
+    const fileDataArrays = [];
+    const hashArrays = [];
     const blocks = [];
     for (const source of sources) {
         const bytesString = [...source.content].map((v) => `0x${v.toString(16)}`).join(', ');
-        arrays.push(`const uint8_t data${source.index}[${source.content.length}] = {${bytesString}};`);
+        fileDataArrays.push(`const uint8_t data${source.index}[${source.content.length}] = {${bytesString}};`);
+        hashArrays.push(`const char * etag${source.index} = "${source.md5}";`);
         blocks.push(getCppBlock(source));
     }
     let result = fileTemplate;
-    result = replaceAll(result, '$arrays$', arrays.join('\n'));
+    result = replaceAll(result, '$commandline$', process.argv.slice(2).join(' '));
+    result = replaceAll(result, '$now$', `${new Date().toLocaleDateString()} ${new Date().toLocaleTimeString()}`);
+    result = replaceAll(result, '$filecount$', sources.length.toString());
+    result = replaceAll(result, '$filesize$', sources.reduce((previous, current) => previous + current.content.length, 0).toString());
+    result = replaceAll(result, '$filedataarrays$', fileDataArrays.join('\n'));
+    result = replaceAll(result, '$hashaarrays$', commandLine_1.cmdLine.etag ? hashArrays.join('\n') : '');
     result = replaceAll(result, '$method$', commandLine_1.cmdLine.espmethod);
     result = replaceAll(result, '$code$', blocks.join(''));
     return result;

package/dist/index.js

@@ -1,5 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
+const node_crypto_1 = require("node:crypto");
 const node_fs_1 = require("node:fs");
 const node_path_1 = require("node:path");
 const node_zlib_1 = require("node:zlib");
@@ -19,6 +20,7 @@ for (const file of (0, file_1.getFiles)()) {
     summary.filecount++;
     console.log(`[${file}]`);
     const rawContent = (0, node_fs_1.readFileSync)((0, node_path_1.join)(commandLine_1.cmdLine.sourcepath, file), { flag: 'r' });
+    const md5 = (0, node_crypto_1.createHash)('md5').update(rawContent).digest('hex');
     summary.size += rawContent.length;
     if (commandLine_1.cmdLine['no-gzip']) {
         sources.push({
@@ -26,7 +28,8 @@ for (const file of (0, file_1.getFiles)()) {
             filename: file,
             content: rawContent,
             isGzip: false,
-            mime
+            mime,
+            md5
         });
         console.log(`- gzip skipped (${rawContent.length})`);
     }
@@ -39,7 +42,8 @@ for (const file of (0, file_1.getFiles)()) {
                 filename: file,
                 content: zipContent,
                 isGzip: true,
-                mime
+                mime,
+                md5
             });
             console.log(`✓ gzip used (${rawContent.length} -> ${zipContent.length})`);
         }
@@ -49,7 +53,8 @@ for (const file of (0, file_1.getFiles)()) {
                 filename: file,
                 content: rawContent,
                 isGzip: false,
-                mime
+                mime,
+                md5
             });
             console.log(`x gzip unused (${rawContent.length} -> ${zipContent.length})`);
         }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "svelteesp32",
-  "version": "1.0.4",
+  "version": "1.1.0",
   "description": "Convert Svelte (or any frontend) JS application to serve it from ESP32 webserver (PsychicHttp)",
   "author": "BCsabaEngine",
   "license": "ISC",
@@ -28,7 +28,8 @@
   },
   "homepage": "https://github.com/BCsabaEngine/svelteesp32",
   "scripts": {
-    "dev": "nodemon src/index.ts -- -s ./svelte/dist -o ../../../Arduino/EspSvelte/svelteesp32.h",
+    "dev": "nodemon src/index.ts -- -s ./svelte/dist -o ../../../Arduino/EspSvelte/svelteesp32.h --etag",
+    "dev:local": "nodemon src/index.ts -- -s ./svelte/dist -o ./svelteesp32.h --etag",
     "clean": "tsc --build --clean",
     "build": "tsc --build --clean && tsc --build --force",
     "format:check": "prettier --check .",
@@ -47,7 +48,7 @@
   ],
   "devDependencies": {
     "@types/mime-types": "^2.1.4",
-    "@types/node": "^20.11.17",
+    "@types/node": "^20.11.19",
     "@typescript-eslint/eslint-plugin": "^7.0.1",
     "@typescript-eslint/parser": "^7.0.1",
     "eslint": "^8.56.0",