@akc42/server-utils 3.4.2 → 4.0.0

package/README.md

@@ -1,50 +1,109 @@
 # server-utils
 A Set of Utilities that I generally use on SPA projects for the server side of the project
 
-It consists of 4 separate packages 6 entry points.
+It consists of 4 separate packages 8 entry points.
 
 The packages are:-
 
-`logger` provides a logging service for the app.  It is controlled by three environment variables LOG_NONE prevents it from logging anything.
-This is designed to be used during testing of the server side of the app so that nothing is logged.  LOG_NO_DATE omits the date and time from
-the logged output.  This is generally used when another logger (e.g PM2 log output) is also adding date/time.  Finally LOG_HIDDEN_IP is used
-to say  to try and anonomise client ip addresses (see below).  `logger` is called so `logger([clientip,] level, ...messages);`.  
+## The Debug Suite
 
-`Responder` is a class to provide the ability to stream JSON responses to a node js http request. It is instanciated
-with `new Responder(response);` and the resultant object has three methods;
+This package is a complete debugging solution.  The basic concept is that messages get written to a database to be
+available for examination in the future. Messages can be immediately written to the console (that is called `logging`),
+Also if the message was a "crash", the previous `<n>` (where `<n>` is determined from the `DEBUG_CACHE_SIZE` environment
+if it exists, else 100) is re-read from the database and also written to the console.
 
-- `addSection(name [,value])` creates a new section in the response of the given name, with an optional value (which should
-   be the entirety of a section).
-- `write` allows you add an array row to an existing open section (one where `addSection` is called without a value). It will return a 
-  promise which resolves when any blockage is cleared.
-- `end` signifies the end of stream.  Any attempt to call the other two methods after this has been called will throw an error.
-
-`Version` provides an async function with a single parameter, the path to you
-project root) that ultimately resolves to an object which has two
-fields.  `version` which is the version string and `year` which is the copyright
-year.  The project root is where either the `.git` directory exists (in which case
-`version` will ask git for the version and calculate the copyright year from the
-last git log entry) or where a `release.info` file is sitting (in which case
-`version` will expect that to contain a version string and have a modification
-time from which the copyright year can be derived).  If neither of those
-possibilities exist it will try to get the version info from the `package.json` file.
-
-`Debug` module provides three entry points, `Debug`, `dumpDebugCache` and
-`setDebugConfig`. The `Debug` entry point is the main one, the user calls this a
-string representing the topic for the debug stream and we return a function that
-will allow him to call with string arguments which will be concatenated (with a
-space separator) to form a debug string.  If `setDebugConfig` has already been
-called to specify that the topic is to be logged (by providing a colon
-separated list of topics to be logged), then this is output. Regardless, all
-debug calls are stored in a 50 line cache, and will be output (newest first) on a call
-to `dumpDebugCache`
-
-Breaking change as of 3.0.0  logger is now an async function returning a promise fulfilled when (if set) a log file entry is made
-
-both `Debug` and `logger` have had their file logging removed as its causing more issues that its worth
-
-
-These are installed with as many of few of the items that you want like so:-
-```
-import {logger,Responder,Debug} from '@akc42/server-utils';
+The **COLOURS** constant is an object contains a set of predefined colours (using the `npm chalk` package) for its properties of `app`,`db`,`api`,`client`, `log`, `mail`, `auth` and `error`.  A *colourspec* is defined as one of those predefined colours or a hex string (hex digits preceeded by a `#`) or an rgb value (a string of three comma separated numbers between 0 and 255) and that will be used to colour the message itself when (and if) is is written to the console. 
+
+If the *shortdate* is defined (defaults to false) then the message, when eventually written to the console, is formatted as "YYYY-MM-DD hh:mm" otherwise it is
+formatted as "YYYY-MM-DD hh:mm:ss.sss" (ie to millisecond accuracy).
+
+The *immediate* parameter says to immediately, after logging to the database, to format the message and output it.
+
+When called the **Debug** function returns a function which is the actual *logger*.  This function can then be called with any number of parameters.  The first three, if present, are checked to match the requirement, but if not are assumed not to be present.  These are
+
+- *crash*  the literal string "crash" - see above for its meaning.  In this case the *colourspec* is ignored and the
+  message is printed in white on a red background.
+- *logtime* A unix timestamp with millisecond accuracy (e.g the result from `Date.now()`), Its only considered valid if
+  it is for today, although it can be earlier than the current time.
+- *ipaddress* An ipv4 address as a string.
+- *...messages* Any number of parameters following which are joined together with a space.
+
+Formally **Debug** is called like this:-
+
+```javascript
+  const debug = Debug(topic,colourspec, shortdate, immediate); 
+
+  debug([crash,][,logtime][ipaddress,]...messages);
 ```
+
+This `debug` instance also remembers the time between calls and this time is logged (and subsequently printed) as a "gap". This is printed in milliseconds unless it was a "shortdate" in which case it is printed in minutes.
+
+**Logger** is a function that is a wrapper for *Debug* where `shortdate` and `immediate` are both true. 
+
+**messageFormatter** is the routine that formats the raw message that has been written to the database.
+
+It is called with the following parameters in order:-
+
+- *logid* The `logid` (the primary key) of the message in the database (only used in the message output if the item was
+  a crash).
+- *logtime* This is either a unix timestamp *or* a string with the date and/or time in it.  It should be in the same
+  format as being formatted (see above).
+- *crash* a 0 or 1 dependant on if this message was a crash or not.
+- *shortdate* a 0 or 1 dependant on if this message has a short date or not,
+- *ipaddress* should be a valid ip address or `null`.
+- *topic*
+- *message* Just a single string
+- *colourspec*
+- *gap* Gap in milliseconds (this routine does the conversion to minutes if a `shortdate`).
+
+It returns an Object with 4 properties
+
+- *dayoutput* If the first message of the day, text with the date (only) in it, otherwise a zero length string.
+- *message* The complete formatted message
+- *logid* The `logid` the formatter was called with.
+- *ip* The `ipaddress` the formatter was called with.
+
+**DebugHelper** is a helper function for *Debug* and performs most of its work.   It is called with the same parameters as *Debug* plus an additional one; *writer*.  *writer* should be a callback function that can do something with the message and then return the return object that a debug call does.  In the use by *Debug* this function is the one writes the data to the database, but other writers can be provided. For instance the *Debug* function is the `@akc42/app-utils` package uses the writer to send the message from the client to the server.
+
+*writer* is called with the following parameters (all described above for *debug*, although in this case they *must* be supplied)
+
+- *logtime* (unix timestamp)
+- *crash* (0 or 1)
+- *shortdate* (0 or 1)
+- *ipaddress* (or `null`)
+- *topic*
+- *colourspec*
+- *gap*
+- *immediate* (`true` or `false`)
+
+## Responder
+
+**Responder** is a class to provide the ability to stream JSON responses to a node js http request. It is instanciated
+with `new Responder(response);` and the resultant object has three methods;
+
+- *addSection* called like
+  
+  ```javascript
+  addSection(name [,value])
+  ```
+  which creates a new *Object property* in the response of the given name, with an optional value (which should be the entirety of a section).
+
+- *write* allows you add an array row to an existing open section (one where *addSection* is called without a value). It
+  will return a promise which resolves when any blockage is cleared. It is recommended to use this when an array of
+  database rows will return more than a very limited number.
+- *end* when signifies the end of stream.  Any attempt to call the other two methods after this has been called will throw an error.
+  
+## Version
+
+**getVersion** is an async function with a single parameter, the path to your project root.
+
+It ultimately resolves to an object which has two fields.  `version` which is the version string and `year` which is the
+copyright year.  The project root is where either the `.git` directory exists (in which case `version` will ask git for
+the version and calculate the copyright year from the last git log entry) or where a `release.info` file is sitting (in
+which case `version` will expect that to contain a version string and have a modification time from which the copyright
+year can be derived).  If neither of those possibilities exist it will try to get the version info from the
+`package.json` file.
+
+## Utils
+
+**nullif0len** is the only function currently in this package.  It is a function that takes a single parameter.  If that parameters is either undefined or a string that has zero length it returns null. Otherwise it returns what was input.

package/base32-utils.js

@@ -0,0 +1,89 @@
+/**
+    @licence
+    Copyright (c) 2025 Alan Chandler, all rights reserved
+
+    This file is part of PASv5, an implementation of the Patient Administration
+    System used to support Accuvision's Laser Eye Clinics.
+
+    PASv5 is licenced to Accuvision (and its successors in interest) free of royality payments
+    and in perpetuity in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
+    implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Accuvision
+    may modify, or employ an outside party to modify, any of the software provided that
+    this modified software is only used as part of Accuvision's internal business processes.
+
+    The software may be run on either Accuvision's own computers or on external computing
+    facilities provided by a third party, provided that the software remains soley for use
+    by Accuvision (or by potential or existing customers in interacting with Accuvision).
+*/
+export function hexToBase32(input) {
+  let output = '';
+  let next = 0;
+  let lshift = 0;
+  for(let i = 0; i< input.length; i++) {
+    let digit = parseInt(input.charAt(i), 16);
+    if (!Number.isInteger(digit)) throw new RangeError('Expected string to include only HEX digits');
+    if (lshift > 0) {
+      const rshift = 4 - lshift;
+      const remainder = digit & (2**rshift - 1);
+      next |= digit >>> rshift;
+      output += alphabet.charAt(next);
+      lshift = (lshift + 1) % 5;
+      next = remainder << lshift;
+    } else {
+      lshift++;
+      next = digit << lshift;
+    }
+  }
+  if (lshift > 0) output += alphabet.charAt(next);
+  return output;
+}
+
+/*
+
+The code below was obtained from elsewhere with the attached notice
+
+MIT License
+
+Copyright (c) 2016-2021 Linus Unnebäck
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+*/
+
+
+const alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
+
+export function base32ToHex (input) {
+  input = input.replace(/=+$/, '')
+  const length = input.length
+  let bits = 0;
+  let value = 0;
+  let index = 0;
+  const output = new Uint8Array(Math.ceil(length * 5 / 8) | 0)
+  for (var i = 0; i < length; i++) {
+    value = (value << 5) | alphabet.indexOf(input[i].toUpperCase())
+    bits += 5
+    if (bits >= 8) {
+      output[index++] = (value >>> (bits - 8)) & 255
+      bits -= 8
+    }
+  }
+  return Buffer.from(output).toString('hex');
+
+}
+

package/debug-helper.js

@@ -0,0 +1,60 @@
+/**
+    @licence
+    Copyright (c) 2025 Alan Chandler, all rights reserved
+
+    This file is part of PASv5, an implementation of the Patient Administration
+    System used to support Accuvision's Laser Eye Clinics.
+
+    PASv5 is licenced to Accuvision (and its successors in interest) free of royality payments
+    and in perpetuity in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
+    implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Accuvision
+    may modify, or employ an outside party to modify, any of the software provided that
+    this modified software is only used as part of Accuvision's internal business processes.
+
+    The software may be run on either Accuvision's own computers or on external computing
+    facilities provided by a third party, provided that the software remains soley for use
+    by Accuvision (or by potential or existing customers in interacting with Accuvision).
+*/
+
+export function DebugHelper(topic, colourspec, shortdate, immediate, writer) {
+  const t = topic;
+  const cs = (colourspec in COLOURS.COLOURS) || COLOURS.hexmatch.test(colourspec) || COLOURS.rgbmatch.test(colourspec) ? colourspec : null;
+  const sd = shortdate? 1:0;
+  let timestamp = Date.now();
+  const i = immediate;
+  return function (c, logtime ,ip, ...args) {
+    let crash = 1;
+    if (c !== 'crash') {
+      if (ip !== undefined) {
+        if (Array.isArray(ip)) args = ip.concat(args); else args.unshift(ip);        
+      }
+      ip = logtime;
+      logtime = c;
+      crash = 0;
+    }
+
+    const fromDate = new Date(); //logtime is only possible if later than midnight last night.
+    const from = fromDate.setHours(0,0,0,0)
+    if (!(Number.isInteger(logtime) && logtime > from)) {
+      if (ip !== undefined) {
+        if (Array.isArray(ip)) args = ip.concat(args); else args.unshift(ip);        
+      }
+      ip = logtime;
+      logtime = Date.now();
+    } 
+    if (!ipmatch.test(ip)) {
+      if (ip !== undefined){
+        if (Array.isArray(ip)) args = ip.concat(args); else args.unshift(ip);        
+      }
+      ip = null
+    }
+    const now = Date.now();
+    const gap = now - timestamp;
+    timestamp = now;
+    const message = args.reduce((cum, arg) => {
+      if (arg === undefined) return cum;
+      return `${cum} ${arg}`.trim();
+    },'');
+    return writer(logtime, crash, sd, ip, t, message, cs, gap,i)
+  }
+};

package/debug.js

@@ -1,85 +1,264 @@
 /**
-@licence
-    Copyright (c) 2021 Alan Chandler, all rights reserved
+    @licence
+    Copyright (c) 2025 Alan Chandler, all rights reserved
 
-    This file is part of Server Utils.
+    This file is part of PASv5, an implementation of the Patient Administration
+    System used to support Accuvision's Laser Eye Clinics.
 
-    Server Utils is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
+    PASv5 is licenced to Accuvision (and its successors in interest) free of royality payments
+    and in perpetuity in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the
+    implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Accuvision
+    may modify, or employ an outside party to modify, any of the software provided that
+    this modified software is only used as part of Accuvision's internal business processes.
 
-    Server Utils is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
+    The software may be run on either Accuvision's own computers or on external computing
+    facilities provided by a third party, provided that the software remains soley for use
+    by Accuvision (or by potential or existing customers in interacting with Accuvision).
+*/
+
+import { EventEmitter } from 'node:events';
+import { setTimeout } from 'node:timers/promises';
+import chalk from "chalk";
+import {openDatabase} from '@akc42/sqlite-db';
+import { DebugHelper, messageFormatter, COLOURS } from './debug-utils.js';
 
-    You should have received a copy of the GNU General Public License
-    along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
+class DebugLogEvents extends EventEmitter {}
+/*
+  The following events are emitted by the debug log
+  
+  'log-write': (<formatted message>, <day boundary>, <logid>, <crash>, <ipaddress>, <topic>, <colourspec> )
+    <logid> is current message unless <day boundary> is true, in which case it is the logid of the last message of the
+            previous day, or 0 if its the first message since startup. If <day-boundary> is true the remaining parameters will be
+            null,
+    <crash> a true of false setting which says if it was crash or not,
+    <ipaddress> if not null is the ip address of the client that caused the message (as a string)
+    <topic> the topic of the message.
+    <colourspec> One of name of standard colors [app,db,api,client,log,mail,auth,error] else ''
+  
+  'log-raw' (<log object>)
+    <log-object> contains the following fields 
+      logid,      This is the same logid as <logid> above, except the <day-boundary> message is not emitted in the raw feed
+      logtime,    The number of milliseconds since Unix Epoch, or as a datetime string with optional milliseconds
+      logmin,     The number of millisecons since Unix Epoch of logtime rounded to the minute boundary before it.
+      crash,      If 1, this entry is a crash report, otherwise its a normal message
+      shortdate,  If 1. the request is only to log to the nearest minute, rather than in milliseconds
+      ipaddress,  If not null, is the ip address of the client that caused the message (as a string)
+      topic,      The topic of this message
+      message,    The text of the message is self
+      colourspec, One of name of standard colors [app,db,api,client,log,mail,auth,error], a hex color string, an rgb, 
+                  comma seperated, string of three values 0-255 
+      gap         gap in milliseconds since the last message of the same topic.
 */
+export const DebugLog = new DebugLogEvents();
+
+const db = await openDatabase(`${process.env.SQLITE_DB_NAME}-log`)
+db.exec(`
+    CREATE TABLE IF NOT EXISTS Log (
+      logid INTEGER PRIMARY KEY,
+      logtime DATETIME NOT NULL DEFAULT (datetime('now','subsec')),
+      logmin INTEGER AS (CAST(round((unixepoch(logtime)/60) - 0.5) AS INT)) STORED, 
+      crash BOOLEAN NOT NULL DEFAULT 0,
+      shortdate BOOLEAN NOT NULL DEFAULT 0,
+      ipaddress TEXT,
+      topic TEXT,
+      message TEXT,
+      colourspec TEXT,
+      gap INTEGER
+  );
+  CREATE INDEX IF NOT EXISTS IX_Log_topic ON Log(topic);
+  CREATE INDEX IF NOT EXISTS IX_Log_logmin ON Log(logmin);
+  CREATE INDEX IF NOT EXISTS IX_Log_crash ON Log(crash);
+  PRAGMA journal_mode=WAL;
+  `);
 
-import chalk from 'chalk';
-let config = process.env.DEBUG || '';
-let cache = [];
-let cachelimit = 50;
- 
-export function Debug (topic) {
-    const t = topic;
-    let timestamp = new Date().getTime();
-    return function (...args) {
-      let enabled = false;
-      if (config) {
-        const topics = config.split(':');
-        if (topics.includes(t)) enabled = true;
+
+export async function awaitTransaction() {
+  return new Promise((resolve) => {
+    if (db.inTransaction) {
+      if (timerAbort.signal.aborted) {
+        setTimeout(200,'awaittransaction').then(resolve);
+      } else {
+        timerAbort.signal.once('abort',resolve);
+        timerAbort.abort();
       }
-      const now = new Date().getTime();
-      const gap = now - timestamp;
-      timestamp = now;
-      const message = args.reduce((cum, arg) => {
-        return `${cum} ${arg}`.trim();
-      }, '');      
-      const output = `${chalk.greenBright(topic)} ${chalk.cyan(message)} ${chalk.whiteBright(`(${gap}ms)`)}`
-      if (enabled) {
-        let logLine = '';
-        if (typeof process.env.LOG_NO_DATE === 'undefined') {
-          if (typeof process.env.LOG_ISO_DATE !== 'undefined') {
-            logLine += new Date().toISOString() + ': ';
-          } else {
-            logLine += new Date().toISOString().substring(0,10) + ' ' + new Date().toLocaleTimeString() + ': ';
-          }
-        }
-        logLine += output;
-        //eslint-disable-next-line no-console
-        console.log(logLine);
+    } else {
+     resolve();
+    }
+  });
+}
+
+let timerAbort = new AbortController();
+
+const insertLogTime = db.prepare(`INSERT INTO Log (logtime,crash,shortdate,ipaddress, topic,message,colourspec,gap) VALUES 
+  (datetime(?,'unixepoch','subsec'),?,?,?,?,?,?,?)`);
+
+const insertLogNoTime = db.prepare(`INSERT INTO LOG(crash,shortdate,ipaddress, topic,message,colourspec,gap) VALUES (?,?,?,?,?,?,?)`);
+
+const getLogTime = db.prepare('SELECT logtime FROM Log WHERE logid = ?');
+
+/*
+  Writes the output to the log, pretty much assumed to be raw
+  logtime is in milliseconds since epoch (ie what date gives from Date.getTime())
+*/
+let shuttingDown = false;
+
+export function logWriter(logtime, crash, shortdate,ipaddress, topic, message, colourspec,gap) {
+  if (shuttingDown) return messageFormatter(0,logtime, crash, shortdate,ipaddress, topic, message, colourspec,gap);
+  if (!db.isOpen) db.open();
+  if (!db.inTransaction) {
+    db.exec('BEGIN TRANSACTION;');
+    timerAbort = new AbortController();
+    setTimeout(1000,'logwriter',timerAbort.signal).then(() => {
+      if (db.inTransaction) {
+        db.exec('COMMIT;');
       }
-      const time = chalk.whiteBright(new Date().toISOString().substring(11,23)); //store millisecond timing
-      cache.push(`${time} ${output}`);
-      if (cache.length > cachelimit) cache.splice(0,cache.length - cachelimit); //prevent it getting too big  
+    })
   }
-};
-export function dumpDebugCache() {
-  const time = chalk.whiteBright(new Date().toISOString());
-  const output = chalk.white.bgBlue('Above are all the debug calls (most recent first) which lead up to, and then followed on from, the error above');
-  cache.reverse();
-  for(const line of cache) {
-    //eslint-disable-next-line no-console
-    console.log(line);
+
+  let logid;
+  
+  if (logtime) {
+    const {lastInsertRowid} = insertLogTime.run((logtime/1000), crash, shortdate,ipaddress, topic, message, colourspec,gap);
+    logid = lastInsertRowid;
+
+  } else {
+    const {lastInsertRowid} = insertLogNoTime.run(crash, shortdate,ipaddress, topic, message, colourspec,gap);
+    logid = lastInsertRowid;
+    const logtimereq = getLogTime.get(logid);
+    logtime = logtimereq.logtime;
   }
-  cache.reverse();
-    //eslint-disable-next-line no-console
-  console.log(`${time} - ${output}`);
+  if (crash === 1) timerAbort.abort(); //force the transaction to close;
+  
+  const output = messageFormatter(logid,logtime, crash, shortdate,ipaddress, topic, message, colourspec,gap);
+  if (output.dayoutput.length > 0) {
+    DebugLog.emit('log-day', output.dayoutput);
+  }
+  delete output.dayoutput;
+  DebugLog.emit(`log-write`,output.message, logid, (crash === 1), ipaddress, topic, (colourspec in COLOURS.COLOURS)? colourspec: '');
+  return output;
+
 };
-export function setDebugConfig(con, limit = 50) {
-  cachelimit = limit;
-  if (con !== config) {
-    config = con;
-    const output = `${chalk.greenBright('debug server config')} ${chalk.redBright(`new server config "${config}"`)}`
-    //eslint-disable-next-line no-console
-    console.log(output);
+
+function logWrapper(logtime, crash, shortdate,ipaddress, topic, message, colourspec,gap, i) {
+  const output = logWriter(logtime, crash, shortdate, ipaddress, topic, message, colourspec, gap);
+  if (i) console.log(output.message);
+  return output;
+}
+
+/*
+  getDebugLog
+
+    will read the debug log that occured just before the provided logid, and for each row returned will call the callback
+    function. The callback function may be asynchronous. the "no" parameter is how many to fetch. The logger uses the
+    DEBUG_CACHE_SIZE environment variable to specify this.
+*/
+
+async function getDebugLog(callback, loid, no, ip) {
+  const lid = loid;
+  const limit = no
+  const ipadd = ip;
+  db.open();
+  try {
+    await awaitTransaction(); //make sure we have all the info committed before looking for it.
+    //we are looking from log entries from the crash backwards in time
+    const getLogtime = db.prepare(`SELECT unixepoch(logtime,'subsec') AS logtime FROM Log WHERE logid = ?`)
+    const {logtime:lt } = getLogtime.get(lid)??{logtime:0}
+    if (lt > 0) {
+      const fetchRecords = db.prepare(`SELECT logid, logtime,crash,shortdate,ipaddress, topic,message,colourspec,gap FROM Log 
+        WHERE (unixepoch(logtime,'subsec')) * 1000 <= ? AND logid <> ? AND ipaddress = ? ORDER BY unixepoch(logtime,'subsec') DESC LIMIT ?`)
+      if (!db.inTransaction) db.exec('BEGIN TRANSACTION');
+      for (const {logid,logtime,crash,shortdate,ipaddress,topic,message,colourspec,gap} of fetchRecords.iterate(lt, lid, ipadd??null, limit)) {
+        const output = messageFormatter(logid,logtime,crash,shortdate,ipaddress,topic,message,colourspec,gap)
+        await callback(output.logid, output.message);
+      }
+      
+    } else {
+      console.log(chalk.white.bgBlue('The transaction provided has not yet cleared its transaction, so no records to list.'));
+    }
+  } catch(e) {
+    console.log(chalk.white.bgRed('failed with error'), e.stack);
+    throw e;
+  } finally {
+    if (db.inTransaction) db.exec('ROLLBACK'); //make sure callback hasn't changed anything
+    db.close();
   }
+}
+
+
+
+/*
+  Debug creates an instance of a debug function 
+
+  parameters:
+    topic       - a value that can be searched for. Useful for dividing into different sections
+    colourspec  - One of name of standard colors [app,db,api,client,log,mail,auth,error], a hex color string, an rgb, 
+                  comma seperated, string of three values 0-255 
+    shortdate   - if true, then dates will be output as YYYY-MM-DD hh:mm else YYYY-MM-DD hh:mm:ss.ms
+
+    immediate   - if set, the message is output (formatted) to the console.
+  
+  Returns a function that will write a row into the log, using the parameters above and some optional extra values
+  these extra parameters are
+
+    crash       - the literal word "crash".  if set, then this will be highlighted in the output. Don't provide this as
+                  the first parameter if a normal call
+    logtime     - a unix millisecond timestamp.  If provided if must be for today, otherwise it will be as
+                  though it were not provided. If provided it will be the logtime, otherwise "Now" will be used.
+    ipaddress   - an optional parameter container a string representation of an ip address. Ignored if not
+                  a valid adddress. If provided its value will be highlighted and surrounded in "[]"
+    ...messages - As many parameters containing parts of the message.  The message will be joined together
+                  with a space separation and displayed with the colourspec parameter.
+*/
+
+export function Debug (topic, colourspec, shortdate, immediate = false) {
+  return DebugHelper(topic, colourspec, shortdate, immediate, logWrapper);
 };
 
 
 
+/*
+  Logger is like Debug (indeed its a wrapper for it) except
+
+  - It doesn't need short date, or immediate parameters as thats whats assumed
+
+  - If a crash, all the messages just before the crash (especially the debug ones that were not output before), are also
+    printed to the consolein reverse order
+*/
+
+export function Logger(topic, colourspec) {
+  const debug = Debug(topic, colourspec, 1,1);
+  return function(c, ip, ...args) {
+    const crash = (c === 'crash');
+    const output = debug(c,ip,args);
+    console.log(output.message);
+    if (crash) {
+      let lt;
+      getDebugLog(async(logid,message) => {
+        if (lt === undefined) lt=message.substring(0,24);
+        console.log(chalk.whiteBright(logid), message)
+      },output.logid,Number(process.env.DEBUG_CACHE_SIZE??100), output.ip).then(() => {
+        console.log(chalk.whiteBright(lt),chalk.white.bgBlue('Above are all the debug calls (most recent first) which lead up to the error above') )
+
+      });
+    }
+  }
+}
+
+/*
+  Closes the database and ensures any partial transactions are written
+*/
+
+export function close() {
+  shuttingDown = true;
+  DebugLog.emit('close');
+  if (db.inTransaction) {
+    db.exec('COMMIT;');
+  }
+  if(db.isOpen) db.close();
+  
+
+};
+
+
 
 

package/message-formatter.js

@@ -0,0 +1,84 @@
+/**
+@licence
+    Copyright (c) 2026 Alan Chandler, all rights reserved
+
+    This file is part of Server Utils.
+
+    Server Utils is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    Server Utils is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
+*/
+import chalk from "chalk";
+export const COLOURS = {
+  hexmatch: /^#(?:[0-9a-fA-F]{3}){1,2}$/,
+  rgbmatch: /^(\d{1,3}), ?(\d{1,3}), ?(\d{1,3})$/,
+  COLOURS: {
+    app: chalk.rgb(255, 136, 0).bold, //orange,
+    db: chalk.greenBright,
+    api: chalk.magentaBright,
+    client: chalk.redBright,
+    log: chalk.hex('#ff651d'),
+    mail: chalk.cyanBright,
+    //error like have backGround colouring
+    auth: chalk.whiteBright.bgBlue,
+    error: chalk.whiteBright.bgHex('#ff1165')
+  }
+ };
+const ipmatch = /^((25[0-5]|2[0-4]\d|1\d{2}|[1-9]?\d)(\.|$)){4}$/;
+const datematch =/^(\d{4})-([01]\d)-([0-3]\d) ([0-2]\d):([0-5]\d)(:([0-5]\d)(\.(\d{1,3}))?)?$/;  
+let lastdate = '';
+
+export function messageFormatter(logid,logtime, crash, shortdate, ipaddress, topic, message, colourspec, gap) {
+  let matches;
+  if (typeof logtime === 'string') {
+    matches = logtime.match(datematch);
+  } else {
+    matches = [logtime];
+  }
+  const logdate = new Date(matches[0]);
+  logdate.setMinutes(logdate.getMinutes() + logdate.getTimezoneOffset());
+  const displaydate = `${logdate.getFullYear()}-${(logdate.getMonth() + 1).toString().padStart(2,'0')}-${logdate.getDate().toString().padStart(2,'0')}`;
+  const displaytime = `${logdate.getHours().toString().padStart(2,'0')}:${logdate.getMinutes().toString().padStart(2,'0')}:${
+    logdate.getSeconds().toString().padStart(2,'0')}.${
+    (typeof logtime === 'string')? (matches[9]??'').toString().padStart(3,'0') : logdate.getMilliseconds().toString().padStart(3,'0')}`;
+  const d = chalk.blueBright(`${displaydate} ${shortdate === 1? displaytime.slice(0,-7): displaytime}`);
+  const ip = ipmatch.test(ipaddress)? chalk.red(` [${ipaddress}]`) : '';
+  const t = chalk.greenBright(`(${topic})`);
+  let m;
+  let l = '';
+  if (crash === 1) {
+    l = chalk.whiteBright(` ${logid}`)
+    m = chalk.white.bgRed(message);
+  } else if (colourspec in COLOURS.COLOURS) {
+    m  = COLOURS.COLOURS[colourspec](message);
+  } else if (COLOURS.hexmatch.test(colourspec)) {
+    m = chalk.hex(colourspec)(message);
+  } else if (COLOURS.rgbmatch.test(colourspec)) {
+    const matches = COLOURS.rgbmatch.exec(colourspec);
+    m = chalk.rgb(matches[1], matches[2], matches[3])(message);
+  } else {
+    m = chalk.cyan(message)
+  }
+  const g = Number.isInteger(gap)?chalk.whiteBright(` gap: ${shortdate? Math.round(gap/60000) + ' mins': gap + 'ms'}`):'';
+  let dayoutput = '';
+  if (lastdate !== displaydate) {
+    dayoutput = `${chalk.whiteBright(displaydate)}:\n`
+    lastdate = displaydate;
+  }
+
+  return {
+    dayoutput: dayoutput,
+    message:`${d}${l}${ip} ${t} ${m}${g}`,
+    logid: logid,
+    ip: ipaddress
+  }
+};

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "@akc42/server-utils",
-  "version": "3.4.2",
+  "version": "4.0.0",
   "type": "module",
   "description": "A Set of Utilities to use on the Server Side of a Project",
-  "main": "./utils.js",
+  "main": "./server-utils.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"
   },
@@ -22,6 +22,7 @@
   },
   "homepage": "https://github.com/akc42/server-utils#readme",
   "dependencies": {
-    "chalk": "^5.3.0"
+    "@akc42/sqlite-db": "3.0.6",
+    "chalk": "^5.6.2"
   }
 }

package/responder.js

@@ -18,13 +18,8 @@
     along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
 */
 
- import Debug from 'debug';
-
-const debug = Debug('responder');
-  
-export default  class Responder {
+export class Responder {
   constructor(response) {
-    debug('Starting responder');
     this.response = response;
     this.doneFirstRow = false;
     this.doneFirstSection = false;
@@ -50,11 +45,9 @@ export default  class Responder {
       if (typeof value !== 'undefined') {
         this.response.write(JSON.stringify(value));
         this.inSection = false;
-        debug('Write conplete section',name, 'with' , JSON.stringify(value));
       } else {
         this.response.write('[');
         this.inSection = true;
-        debug('Started Section',name);
       }
       this.doneFirstSection = true;
       this.doneFirstRow = false;
@@ -80,15 +73,12 @@ export default  class Responder {
       if (reply) {
         return Promise.resolve();
       }
-      debug('False reply from write so need return the promise of a drain');
       if (!this.awaitingDrain) {
         this.awaitingDrain = true;
         const self = this;
-        debug('create a drain promise as we do not have one');
         this.drainPromise = new Promise(resolve => {
           self.response.once('drain', () => {
             self.awaitingDrain = false;
-            debug('drained so resolve promise of drain');
             resolve();
           });
         });
@@ -98,7 +88,6 @@ export default  class Responder {
     return Promise.reject(); //mark as blocked
   }
   end() {
-    debug('End Responder');
     if (!this.ended) {
       if (this.inSection) {
         this.response.write(']');

package/server-utils.js

@@ -0,0 +1,38 @@
+/**
+@licence
+		Copyright (c) 2026 Alan Chandler, all rights reserved
+
+		This file is part of Server Utils.
+
+		Server Utils is free software: you can redistribute it and/or modify
+		it under the terms of the GNU General Public License as published by
+		the Free Software Foundation, either version 3 of the License, or
+		(at your option) any later version.
+
+		Server Utils is distributed in the hope that it will be useful,
+		but WITHOUT ANY WARRANTY; without even the implied warranty of
+		MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+		GNU General Public License for more details.
+
+		You should have received a copy of the GNU General Public License
+		along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+
+
+import {getVersion}from './version.js';
+import {Responder} from './responder.js';
+import { messageFormatter, COLOURS } from './message-formatter.js';
+import { DebugHelper } from './debug-helper.js';
+import { Debug, Logger} from './debug.js';
+import { nullif0len } from './utils.js';
+export {
+  COLOURS,
+  Debug,
+  DebugHelper,
+  getVersion,
+  Logger,
+  messageFormatter,
+  nullif0len,
+  Responder
+};

package/utils.js

@@ -1,35 +1,27 @@
 /**
 @licence
-		Copyright (c) 2021 Alan Chandler, all rights reserved
+    Copyright (c) 2026 Alan Chandler, all rights reserved
 
-		This file is part of Server Utils.
+    This file is part of Server Utils.
 
-		Server Utils is free software: you can redistribute it and/or modify
-		it under the terms of the GNU General Public License as published by
-		the Free Software Foundation, either version 3 of the License, or
-		(at your option) any later version.
+    Server Utils is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
 
-		Server Utils is distributed in the hope that it will be useful,
-		but WITHOUT ANY WARRANTY; without even the implied warranty of
-		MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-		GNU General Public License for more details.
+    Server Utils is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
 
-		You should have received a copy of the GNU General Public License
-		along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
+    You should have received a copy of the GNU General Public License
+    along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
 */
 
 
-import logger from './logger.js';
-import Version from './version.js';
-import Responder from './responder.js';
-import { Debug, dumpDebugCache, setDebugConfig} from './debug.js';
+export function nullif0len (str) {
+  if (typeof str === 'undefined') return null;
+  if (typeof str === 'string' && str.length === 0) return null;
+  return str;
+};
 
-
-export {
-	logger,
-	Version,
-	Responder,
-	Debug,
-	dumpDebugCache,
-	setDebugConfig
-}

package/version.js

@@ -1,6 +1,6 @@
 /**
 @licence
-    Copyright (c) 2021 Alan Chandler, all rights reserved
+    Copyright (c) 2026 Alan Chandler, all rights reserved
 
     This file is part of Server Utils.
 
@@ -17,16 +17,11 @@
     You should have received a copy of the GNU General Public License
     along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
 */
-
-import {Debug}  from './debug.js';
 import {access, readFile, stat} from 'node:fs/promises';
 import { resolve } from 'node:path';
 import { exec } from 'node:child_process';
 
-const debug = Debug('version');
-
 async function shCmd(cmd, root) {
-  debug('About to execute Command ', cmd);
   return new Promise(async (accept, reject) => {
     exec(cmd, { cwd: root }, (err, stdout, stderr) => {
       if (stderr) {
@@ -34,21 +29,16 @@ async function shCmd(cmd, root) {
         reject(err);
       } else {
         const out = stdout.trim();
-        debug('Command ', cmd, 'Success with ', out);
         accept(out);
       }
     });
   });
 }
-export default async function(root) {
-
+export async function getVersion(root) {
   let version;
   let vtime;
-
   try {
-    debug('Look for git')
     await access(resolve(root, '.git'));
-    debug('Git found, so use it to get data')
     //we get here if there is a git directory, so we can look up version and latest commit from them
     version = await shCmd('git describe --abbrev=0 --tags');
     //git is installed and we found a tag
@@ -60,7 +50,6 @@ export default async function(root) {
   } catch (e) {
     //no git, or no tag, so we must look for a version file
     try {
-      debug('Git approach failed, so look for release info');
       version = await readFile(resolve(root, 'release.info'), 'utf8');
       try {
         const { mtime } = await stat(resolve(root, 'release.info'));
@@ -88,7 +77,6 @@ export default async function(root) {
   } finally {
     const finalversion = version.replace(/\s+/g, ' ').trim(); //trim out new lines and multiple spaces just one.
     const copyrightTime = new Date(vtime);
-    debug('Resolving with Git copyright Year is ', copyrightTime.getUTCFullYear());
     return({ version: finalversion, year: copyrightTime.getUTCFullYear() });
   }
 };

package/logger.js

@@ -1,77 +0,0 @@
-/**
-@licence
-    Copyright (c) 2021 Alan Chandler, all rights reserved
-
-    This file is part of Server Utils.
-
-    Server Utils is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation, either version 3 of the License, or
-    (at your option) any later version.
-
-    Server Utils is distributed in the hope that it will be useful,
-    but WITHOUT ANY WARRANTY; without even the implied warranty of
-    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
-
-    You should have received a copy of the GNU General Public License
-    along with Server Utils.  If not, see <http://www.gnu.org/licenses/>.
-*/
-
-import chalk from 'chalk';
-import { isIP } from 'node:net';
-
-const COLOURS = {
-  app: chalk.rgb(255, 136, 0).bold, //orange,
-  db: chalk.greenBright,
-  api: chalk.magentaBright,
-  client: chalk.redBright,
-  log: chalk.yellowBright,
-  mail: chalk.cyanBright,
-  //error like have backGround colouring
-  auth: chalk.black.bgCyan,
-  err: chalk.white.bgBlue,
-  error: chalk.white.bgRed
-
-};
-function cyrb53 (str, seed = 0) {
-    let h1 = 0xdeadbeef ^ seed, h2 = 0x41c6ce57 ^ seed;
-    for (let i = 0, ch; i < str.length; i++) {
-      ch = str.charCodeAt(i);
-      h1 = Math.imul(h1 ^ ch, 2654435761);
-      h2 = Math.imul(h2 ^ ch, 1597334677);
-    }
-    h1 = Math.imul(h1 ^ h1 >>> 16, 2246822507) ^ Math.imul(h2 ^ h2 >>> 13, 3266489909);
-    h2 = Math.imul(h2 ^ h2 >>> 16, 2246822507) ^ Math.imul(h1 ^ h1 >>> 13, 3266489909);
-    return 4294967296 * (2097151 & h2) + (h1 >>> 0);
-}
-
-export default function logger(ip,level, ...messages) {
-  if (process.env.LOG_NONE === undefined) {
-    let logLine = '';
-    if (typeof process.env.LOG_NO_DATE === 'undefined') {
-      if (typeof process.env.LOG_ISO_DATE !== 'undefined') {
-        logLine += new Date().toISOString() + ': ';
-      } else {
-        logLine += new Date().toISOString().substring(0,10) + ' ' + new Date().toLocaleTimeString() + ': ';
-      }
-    }
-    let message;
-    let logcolor;
-    if (isIP(ip) === 0 ) {
-      logcolor = ip;
-      message = level + messages.join(' ');
-    } else {
-      const client = typeof process.env.LOG_IP_HIDDEN !== 'undefined' ? cyrb53(ip): ip;
-      logLine += COLOURS.client(client + ': ');
-      logcolor = level
-      message = messages.join(' ');
-    }
-    logLine += COLOURS[logcolor](message);
-    //eslint-disable-next-line no-console
-      console.log(logLine.trim());
-  }
-}
-
-
-