sos-inventory-db 1.0.0

package/LICENSE

@@ -0,0 +1,23 @@
+# MIT License
+
+MIT License
+
+Copyright (c) 2024–2026 Chris Rowley
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,226 @@
+# **sos-inventory-db**
+
+A schema‑driven relational database model for **SOS Inventory**, including:
+
+- Fully documented API contract definitions
+- Deterministic ingestion of *all* SOS objects
+- Multi‑engine database support (SQLite, PostgreSQL, MariaDB, MySQL, SQL Server)
+- A single, stable entry point: **`downloadSOS()`**
+- Clean, normalized relational tables generated from real SOS payloads
+
+This package gives you a **local, queryable mirror** of your SOS Inventory data — built from the *actual* API payloads, not the inconsistent documentation.
+
+---
+
+## **Features**
+
+- 🚀 **One function**: `downloadSOS(params)`
+- 🧱 **Schema‑driven**: every table is defined in `db/definitions.js`
+- 🔄 **Deterministic ingestion**: same results across all engines
+- 🗄️ **Multi‑database support**:
+  - SQLite (default, zero config)
+  - PostgreSQL
+  - MariaDB
+  - MySQL
+  - SQL Server
+- 📦 **Dual‑module build**: CommonJS + ESM
+- 🧪 **Optional DB drivers** — install only what you need
+
+---
+
+## **Installation**
+
+```bash
+npm install sos-inventory-db
+```
+
+Optional: install the driver for your database engine:
+
+```bash
+npm install better-sqlite3
+npm install pg
+npm install mariadb
+npm install mysql2
+npm install mssql
+```
+
+---
+
+## **Quickstart**
+
+### ESM
+
+```js
+import downloadSOS from 'sos-inventory-db'
+// or: const downloadSOS = require('sos-inventory-db')
+
+await downloadSOS({
+  database: {
+    engine: 'sqlite',
+    filename: './sos.db'
+  },
+  sosAuthorization: process.env.SOS_AUTH
+})
+```
+
+### CommonJS
+
+```js
+const downloadSOS = require('sos-inventory-db')
+
+downloadSOS({
+  database: {
+    engine: 'sqlite',
+    filename: './sos.db'
+  },
+  sosAuthorization: process.env.SOS_AUTH
+})
+```
+
+This will:
+
+- Create all tables defined in `db/definitions.js`
+- Fetch all SOS Inventory objects
+- Normalize them into relational tables
+- Insert them into your chosen database engine
+
+---
+
+## **Engine Configuration**
+
+`sos-inventory-db` accepts a `database` object describing the engine and connection parameters.
+
+Here is the full interface:
+
+```js
+const testDatabases = {
+  sqlite: {
+    engine: 'sqlite',
+    filename: 'db/wdm.db'
+  },
+  mariadb: {
+    engine: 'mariadb',
+    host: process.env.MARIADB_HOST,
+    port: process.env.MARIADB_PORT,
+    user: process.env.MARIADB_USER,
+    password: process.env.MARIADB_PASSWORD,
+    database: process.env.MARIADB_DATABASE
+  },
+  postgres: {
+    engine: 'postgres',
+    host: process.env.POSTGRESQL_HOST,
+    port: process.env.POSTGRESQL_PORT,
+    user: process.env.POSTGRESQL_USER,
+    password: process.env.POSTGRESQL_PASSWORD,
+    database: process.env.POSTGRESQL_DATABASE
+  },
+  mssql: {
+    engine: 'mssql',
+    server: process.env.MSSQL_SERVER,
+    port: Number(process.env.MSSQL_PORT),
+    authentication: {
+      type: 'default',
+      options: {
+        userName: process.env.MSSQL_USER,
+        password: process.env.MSSQL_PASSWORD
+      }
+    },
+    options: {
+      database: process.env.MSSQL_DATABASE,
+      encrypt: true,
+      trustServerCertificate: true,
+      enableArithAbort: true
+    }
+  }
+}
+```
+
+Pass one of these objects to `downloadSOS()`:
+
+```js
+await downloadSOS({
+  database: testDatabases.postgres,
+  sosAuthorization: process.env.SOS_AUTH
+})
+```
+
+---
+
+## **Authentication**
+
+You must provide a valid SOS Inventory API token:
+
+```js
+await downloadSOS({
+  database,
+  sosAuthorization: 'Bearer <your-token-here>'
+})
+```
+
+Or via environment variable:
+
+```bash
+export SOS_AUTH="Bearer <token>"
+```
+
+---
+
+## **What Gets Created**
+
+`sos-inventory-db` builds a complete relational mirror of your SOS Inventory account, including:
+
+- Items
+- Item BOMs
+- Customers
+- Vendors
+- Sales Orders
+- Purchase Orders
+- Invoices
+- Payments
+- Locations
+- Categories
+- Adjustments
+- And all supporting lookup tables
+
+Every table is defined in:
+
+```js
+db/definitions.js
+```
+
+This file is the single source of truth for:
+
+- Table names
+- Field types
+- Nullability
+- Primary keys
+- Reference mappings
+- Read‑only fields
+- API contract definitions
+
+---
+
+## **Why This Exists**
+
+SOS Inventory’s API documentation is:
+
+- incomplete
+- inconsistent
+- sometimes incorrect
+- and often out of sync with real payloads
+
+This package solves that by:
+
+- capturing the *actual* API responses
+- documenting the real schema
+- normalizing everything into relational tables
+- providing a deterministic ingestion engine
+- supporting multiple SQL backends
+
+The result is a stable, queryable, local mirror of your SOS data.
+
+---
+
+## **License**
+
+MIT

package/dist/index.cjs

@@ -0,0 +1,216 @@
+"use strict";
+const openDb = require("./db/database.js");
+const createTable = require("./db/create-table.js");
+const insertRow = require("./db/insert-row.js");
+const { tables, supportTables } = require("./db/definitions.js");
+function formatRuntime(ms) {
+  const totalSeconds = Math.floor(ms / 1e3);
+  const hours = Math.floor(totalSeconds / 3600);
+  const minutes = Math.floor(totalSeconds % 3600 / 60);
+  const seconds = totalSeconds % 60;
+  return `${hours.toString().padStart(2, "0")}:${minutes.toString().padStart(2, "0")}:${seconds.toString().padStart(2, "0")}`;
+}
+async function sosApi(url, method, authorization, retries = 5) {
+  console.log(url);
+  for (let attempt = 1; attempt <= retries; attempt++) {
+    try {
+      const res = await fetch(url, {
+        method,
+        headers: {
+          "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+          Accept: "*/*",
+          "Accept-Encoding": "gzip, deflate, br",
+          "Content-Type": "application/x-www-form-urlencoded",
+          Host: "api.sosinventory.com",
+          Authorization: authorization
+        }
+      });
+      if (!res.ok) {
+        throw new Error(`HTTP ${res.status} ${res.statusText}`);
+      }
+      return await res.json();
+    } catch (err) {
+      if (attempt === retries) throw err;
+      const delay = 250 * attempt;
+      console.log(`Retry ${attempt}/${retries} after error: ${err.message}`);
+      console.log(url);
+      await new Promise((resolve) => setTimeout(resolve, delay));
+    }
+  }
+}
+async function downloadTable(params, engine, table) {
+  const ctResult = await createTable(engine, table);
+  if (!ctResult.ok) return ctResult;
+  const retries = params.retries ?? 5;
+  let offset = 1;
+  await engine.begin();
+  while (true) {
+    const result = await sosApi(
+      `https://api.sosinventory.com${table.api.query.endpoint}?start=${offset}&maxresults=200`,
+      table.api.query.method,
+      params.sosAuthorization,
+      retries
+    );
+    if (result.status !== "ok") {
+      await engine.rollback();
+      return { ok: false, message: result.message || "SOS API returned non-ok status" };
+    }
+    if (!Array.isArray(result.data)) {
+      await engine.rollback();
+      return { ok: false, message: "SOS API returned invalid data array" };
+    }
+    if (typeof result.count !== "number" || typeof result.totalCount !== "number") {
+      await engine.rollback();
+      return { ok: false, message: "SOS API returned invalid pagination fields" };
+    }
+    for (const record of result.data) {
+      delete record.keys;
+      delete record.values;
+      const irResult = await insertRow(engine, table, record);
+      if (!irResult.ok) {
+        await engine.rollback();
+        return { ok: false, message: irResult };
+      }
+    }
+    offset += result.count;
+    if (offset > result.totalCount) break;
+  }
+  await engine.commit();
+  return { ok: true };
+}
+async function handleSupportTable(engine, table) {
+  await createTable(engine, table);
+  const supportMap = /* @__PURE__ */ new Map();
+  for (const sourceTable of table.sourceTables) {
+    const rows = await engine.query(
+      `SELECT DISTINCT ${engine.q(table.sourceField)} FROM ${engine.q(sourceTable)} WHERE ${engine.q(table.sourceField)} IS NOT NULL`,
+      [],
+      true
+    );
+    for (const row of rows) {
+      let obj;
+      try {
+        obj = row[table.sourceField];
+        if (typeof obj !== "object") obj = JSON.parse(obj);
+      } catch (err) {
+        console.error(`Invalid JSON in ${table.name}:`, row[table.sourceField]);
+        continue;
+      }
+      if (!obj || typeof obj !== "object") continue;
+      const key = obj.id ?? obj.value ?? obj.code ?? obj.name;
+      if (!key) continue;
+      if (!supportMap.has(key)) {
+        supportMap.set(key, {
+          id: obj.id ?? obj.value ?? obj.code,
+          name: obj.name ?? obj.code ?? obj.value
+        });
+      }
+    }
+  }
+  for (const value of supportMap.values()) {
+    await insertRow(engine, table, value);
+  }
+  return { ok: true };
+}
+async function handleItemBoms(params, engine, table) {
+  await createTable(engine, table);
+  const items = await engine.query("select id from items where type IN ('Assembly', 'Item Group')", [], true);
+  const retries = params.retries ?? 5;
+  for (const item of items) {
+    const result = await sosApi(`https://api.sosinventory.com/api/v2/item/${item.id}/bom`, "GET", params.sosAuthorization, retries);
+    if (!result.data) continue;
+    if (result.data.lines.length === 0) continue;
+    const lines = result.data.lines;
+    for (const line of lines) {
+      const id = Number(line.lineId);
+      delete line.lineId;
+      const record = {
+        id,
+        itemId: item.id,
+        ...line
+      };
+      await insertRow(engine, table, record);
+    }
+  }
+  return { ok: true };
+}
+async function downloadSOS(params) {
+  if (!params.database.engine) {
+    return {
+      ok: false,
+      message: "Missing required parameter: params.database.engine",
+      error: new Error("params.database.engine is required"),
+      extra: null
+    };
+  }
+  if (!params.sosAuthorization) {
+    return {
+      ok: false,
+      message: "Missing required parameter: params.sosAuthorization",
+      error: new Error("params.sosAuthorization is required"),
+      extra: null
+    };
+  }
+  const start = Date.now();
+  const engine = await openDb(params.database);
+  const referenceTables = tables.filter((table) => table.reference === true);
+  for (const table of referenceTables) {
+    const result = await downloadTable(params, engine, table);
+    if (!result.ok) {
+      if (engine.close) await engine.close();
+      return {
+        ok: false,
+        message: `downloadTable failed for ${table.name}`,
+        error: result.error || new Error("Unknown downloadTable error"),
+        extra: { table: table.name, result }
+      };
+    }
+  }
+  const primaryTables = tables.filter((table) => table.primary === true);
+  for (const table of primaryTables) {
+    const result = await downloadTable(params, engine, table);
+    if (!result.ok) {
+      if (engine.close) await engine.close();
+      return {
+        ok: false,
+        message: `downloadTable failed for ${table.name}`,
+        error: result.error || new Error("Unknown downloadTable error"),
+        extra: { table: table.name, result }
+      };
+    }
+  }
+  for (const table of supportTables) {
+    if (table.name === "itemBoms") {
+      const bomResult = await handleItemBoms(params, engine, table);
+      if (!bomResult.ok) {
+        if (engine.close) await engine.close();
+        return {
+          ok: false,
+          message: "handleItemBoms failed",
+          error: bomResult.error || new Error("Unknown item BOM handler error"),
+          extra: { table: table.name, result: bomResult }
+        };
+      }
+    } else {
+      const refResult = await handleSupportTable(engine, table);
+      if (!refResult.ok) {
+        if (engine.close) await engine.close();
+        return {
+          ok: false,
+          message: "handleSupportTable failed",
+          error: refResult.error || new Error("Unknown support table handler error"),
+          extra: { table: table.name, result: refResult }
+        };
+      }
+    }
+  }
+  if (engine.close) await engine.close();
+  const runtime = formatRuntime(Date.now() - start);
+  return {
+    ok: true,
+    message: "downloadSOS completed successfully",
+    error: null,
+    extra: { runtime }
+  };
+}
+module.exports = downloadSOS;

package/dist/index.mjs

@@ -0,0 +1,224 @@
+var __getOwnPropNames = Object.getOwnPropertyNames;
+var __commonJS = (cb, mod) => function __require() {
+  return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
+};
+var require_index = __commonJS({
+  "index.js"(exports, module) {
+    const openDb = require("./db/database.js");
+    const createTable = require("./db/create-table.js");
+    const insertRow = require("./db/insert-row.js");
+    const { tables, supportTables } = require("./db/definitions.js");
+    function formatRuntime(ms) {
+      const totalSeconds = Math.floor(ms / 1e3);
+      const hours = Math.floor(totalSeconds / 3600);
+      const minutes = Math.floor(totalSeconds % 3600 / 60);
+      const seconds = totalSeconds % 60;
+      return `${hours.toString().padStart(2, "0")}:${minutes.toString().padStart(2, "0")}:${seconds.toString().padStart(2, "0")}`;
+    }
+    async function sosApi(url, method, authorization, retries = 5) {
+      console.log(url);
+      for (let attempt = 1; attempt <= retries; attempt++) {
+        try {
+          const res = await fetch(url, {
+            method,
+            headers: {
+              "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
+              Accept: "*/*",
+              "Accept-Encoding": "gzip, deflate, br",
+              "Content-Type": "application/x-www-form-urlencoded",
+              Host: "api.sosinventory.com",
+              Authorization: authorization
+            }
+          });
+          if (!res.ok) {
+            throw new Error(`HTTP ${res.status} ${res.statusText}`);
+          }
+          return await res.json();
+        } catch (err) {
+          if (attempt === retries) throw err;
+          const delay = 250 * attempt;
+          console.log(`Retry ${attempt}/${retries} after error: ${err.message}`);
+          console.log(url);
+          await new Promise((resolve) => setTimeout(resolve, delay));
+        }
+      }
+    }
+    async function downloadTable(params, engine, table) {
+      const ctResult = await createTable(engine, table);
+      if (!ctResult.ok) return ctResult;
+      const retries = params.retries ?? 5;
+      let offset = 1;
+      await engine.begin();
+      while (true) {
+        const result = await sosApi(
+          `https://api.sosinventory.com${table.api.query.endpoint}?start=${offset}&maxresults=200`,
+          table.api.query.method,
+          params.sosAuthorization,
+          retries
+        );
+        if (result.status !== "ok") {
+          await engine.rollback();
+          return { ok: false, message: result.message || "SOS API returned non-ok status" };
+        }
+        if (!Array.isArray(result.data)) {
+          await engine.rollback();
+          return { ok: false, message: "SOS API returned invalid data array" };
+        }
+        if (typeof result.count !== "number" || typeof result.totalCount !== "number") {
+          await engine.rollback();
+          return { ok: false, message: "SOS API returned invalid pagination fields" };
+        }
+        for (const record of result.data) {
+          delete record.keys;
+          delete record.values;
+          const irResult = await insertRow(engine, table, record);
+          if (!irResult.ok) {
+            await engine.rollback();
+            return { ok: false, message: irResult };
+          }
+        }
+        offset += result.count;
+        if (offset > result.totalCount) break;
+      }
+      await engine.commit();
+      return { ok: true };
+    }
+    async function handleSupportTable(engine, table) {
+      await createTable(engine, table);
+      const supportMap = /* @__PURE__ */ new Map();
+      for (const sourceTable of table.sourceTables) {
+        const rows = await engine.query(
+          `SELECT DISTINCT ${engine.q(table.sourceField)} FROM ${engine.q(sourceTable)} WHERE ${engine.q(table.sourceField)} IS NOT NULL`,
+          [],
+          true
+        );
+        for (const row of rows) {
+          let obj;
+          try {
+            obj = row[table.sourceField];
+            if (typeof obj !== "object") obj = JSON.parse(obj);
+          } catch (err) {
+            console.error(`Invalid JSON in ${table.name}:`, row[table.sourceField]);
+            continue;
+          }
+          if (!obj || typeof obj !== "object") continue;
+          const key = obj.id ?? obj.value ?? obj.code ?? obj.name;
+          if (!key) continue;
+          if (!supportMap.has(key)) {
+            supportMap.set(key, {
+              id: obj.id ?? obj.value ?? obj.code,
+              name: obj.name ?? obj.code ?? obj.value
+            });
+          }
+        }
+      }
+      for (const value of supportMap.values()) {
+        await insertRow(engine, table, value);
+      }
+      return { ok: true };
+    }
+    async function handleItemBoms(params, engine, table) {
+      await createTable(engine, table);
+      const items = await engine.query("select id from items where type IN ('Assembly', 'Item Group')", [], true);
+      const retries = params.retries ?? 5;
+      for (const item of items) {
+        const result = await sosApi(`https://api.sosinventory.com/api/v2/item/${item.id}/bom`, "GET", params.sosAuthorization, retries);
+        if (!result.data) continue;
+        if (result.data.lines.length === 0) continue;
+        const lines = result.data.lines;
+        for (const line of lines) {
+          const id = Number(line.lineId);
+          delete line.lineId;
+          const record = {
+            id,
+            itemId: item.id,
+            ...line
+          };
+          await insertRow(engine, table, record);
+        }
+      }
+      return { ok: true };
+    }
+    async function downloadSOS(params) {
+      if (!params.database.engine) {
+        return {
+          ok: false,
+          message: "Missing required parameter: params.database.engine",
+          error: new Error("params.database.engine is required"),
+          extra: null
+        };
+      }
+      if (!params.sosAuthorization) {
+        return {
+          ok: false,
+          message: "Missing required parameter: params.sosAuthorization",
+          error: new Error("params.sosAuthorization is required"),
+          extra: null
+        };
+      }
+      const start = Date.now();
+      const engine = await openDb(params.database);
+      const referenceTables = tables.filter((table) => table.reference === true);
+      for (const table of referenceTables) {
+        const result = await downloadTable(params, engine, table);
+        if (!result.ok) {
+          if (engine.close) await engine.close();
+          return {
+            ok: false,
+            message: `downloadTable failed for ${table.name}`,
+            error: result.error || new Error("Unknown downloadTable error"),
+            extra: { table: table.name, result }
+          };
+        }
+      }
+      const primaryTables = tables.filter((table) => table.primary === true);
+      for (const table of primaryTables) {
+        const result = await downloadTable(params, engine, table);
+        if (!result.ok) {
+          if (engine.close) await engine.close();
+          return {
+            ok: false,
+            message: `downloadTable failed for ${table.name}`,
+            error: result.error || new Error("Unknown downloadTable error"),
+            extra: { table: table.name, result }
+          };
+        }
+      }
+      for (const table of supportTables) {
+        if (table.name === "itemBoms") {
+          const bomResult = await handleItemBoms(params, engine, table);
+          if (!bomResult.ok) {
+            if (engine.close) await engine.close();
+            return {
+              ok: false,
+              message: "handleItemBoms failed",
+              error: bomResult.error || new Error("Unknown item BOM handler error"),
+              extra: { table: table.name, result: bomResult }
+            };
+          }
+        } else {
+          const refResult = await handleSupportTable(engine, table);
+          if (!refResult.ok) {
+            if (engine.close) await engine.close();
+            return {
+              ok: false,
+              message: "handleSupportTable failed",
+              error: refResult.error || new Error("Unknown support table handler error"),
+              extra: { table: table.name, result: refResult }
+            };
+          }
+        }
+      }
+      if (engine.close) await engine.close();
+      const runtime = formatRuntime(Date.now() - start);
+      return {
+        ok: true,
+        message: "downloadSOS completed successfully",
+        error: null,
+        extra: { runtime }
+      };
+    }
+    module.exports = downloadSOS;
+  }
+});
+export default require_index();

package/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "sos-inventory-db",
+  "version": "1.0.0",
+  "description": "A schema‑driven relational database model for SOS Inventory, including full API contract definitions and ingestion support for SQLite, PostgreSQL, MariaDB, MySQL, and SQL Server.",
+  "type": "commonjs",
+
+  "main": "./dist/index.cjs",
+  "exports": {
+    "import": "./dist/index.mjs",
+    "require": "./dist/index.cjs"
+  },
+
+  "files": [
+    "dist/",
+    "README.md",
+    "LICENSE"
+  ],
+
+  "scripts": {
+    "build": "node build.js",
+    "test": "node test/sos-download-test.js"
+  },
+
+  "keywords": [
+    "sos-inventory",
+    "sos inventory",
+    "inventory",
+    "api",
+    "etl",
+    "data-ingestion",
+    "database",
+    "sqlite",
+    "postgresql",
+    "mysql",
+    "mariadb",
+    "sqlserver",
+    "schema",
+    "sync",
+    "mirror",
+    "downloader"
+  ],
+  "author": "Chris Rowley",
+  "license": "MIT",
+  "optionalDependencies": {
+    "better-sqlite3": "^12.8.0",
+    "mariadb": "^3.5.2",
+    "mssql": "^12.2.1",
+    "mysql2": "^3.20.0",
+    "pg": "^8.20.0"
+  },
+  "devDependencies": {
+    "dotenv": "^17.3.1",
+    "esbuild": "^0.27.7",
+    "eslint": "^10.1.0",
+    "globals": "^17.4.0"
+  }
+}