@wmfs/pg-delta-file 1.69.0 → 1.73.0

package/.circleci/config.yml

@@ -10,9 +10,10 @@ jobs:
           TZ: "Europe/London"
           PG_CONNECTION_STRING: postgres://postgres:postgres@localhost:5432/circle_test
           TIMEOUT: 15000
-      - image: circleci/postgres:9.6-alpine-postgis-ram
+      - image: cimg/postgres:13.4-postgis
         environment:
           POSTGRES_USER: postgres
+          POSTGRES_HOST_AUTH_METHOD: trust
           POSTGRES_DB: circle_test
           TZ: "Europe/London"
     working_directory: ~/repo

package/CHANGELOG.md

@@ -1,3 +1,86 @@
+# [1.73.0](https://github.com/wmfs/pg-delta-file/compare/v1.72.0...v1.73.0) (2022-01-25)
+
+
+### 🛠 Builds
+
+* **deps:** update dependency luxon to v2 ([4cfdea8](https://github.com/wmfs/pg-delta-file/commit/4cfdea8bea8c4d4a75af19329b435e71fb9f69f6))
+
+# [1.72.0](https://github.com/wmfs/pg-delta-file/compare/v1.71.0...v1.72.0) (2022-01-25)
+
+
+### 🛠 Builds
+
+* **deps-dev:** Bump [@wmfs](https://github.com/wmfs)/hl-pg-client from 1.27.0 to 1.28.0 ([3ed1bc9](https://github.com/wmfs/pg-delta-file/commit/3ed1bc9008af08a84ace32b80e1ad0ae8cd31ad4))
+* **deps-dev:** Bump codecov from 3.8.2 to 3.8.3 ([0cfeecb](https://github.com/wmfs/pg-delta-file/commit/0cfeecbd164f12626a439b5c734b31f48ff5c9b9))
+* **deps-dev:** Bump mocha from 8.4.0 to 9.0.0 ([2415420](https://github.com/wmfs/pg-delta-file/commit/2415420bfcde85ac620c248d2adc9ce2714befc8))
+* **deps-dev:** Bump mocha from 9.0.0 to 9.0.1 ([0af15f7](https://github.com/wmfs/pg-delta-file/commit/0af15f7172c48ff12b3cad1a11042626364fc566))
+* **deps-dev:** Bump mocha from 9.0.1 to 9.0.2 ([d675f32](https://github.com/wmfs/pg-delta-file/commit/d675f325161711d7da5a57dc9633c30e8449ee0a))
+* **deps-dev:** Bump mocha from 9.0.2 to 9.0.3 ([0df6f3c](https://github.com/wmfs/pg-delta-file/commit/0df6f3c31528b46b302dd1f2ac783728ca74ba29))
+* **deps-dev:** Bump semantic-release from 17.4.3 to 17.4.4 ([ad32508](https://github.com/wmfs/pg-delta-file/commit/ad32508be8374f9ac78ef6116e95ce1fa2490b09))
+* **deps-dev:** update dependency [@semantic-release](https://github.com/semantic-release)/changelog to v6 ([b9ad2c6](https://github.com/wmfs/pg-delta-file/commit/b9ad2c6690c602bea6211772b7614a3220ea896d))
+* **deps-dev:** update dependency [@semantic-release](https://github.com/semantic-release)/git to v10 ([12e5b80](https://github.com/wmfs/pg-delta-file/commit/12e5b80f17e2fb93fede1db7d050def6f335cc83))
+* **deps-dev:** update dependency [@semantic-release](https://github.com/semantic-release)/git to v9.0.1 ([5055fff](https://github.com/wmfs/pg-delta-file/commit/5055fffb9eef7a70d3e8c2b9ad76ffc995379ef9))
+* **deps-dev:** update dependency mocha to v9.1.0 ([db5ee87](https://github.com/wmfs/pg-delta-file/commit/db5ee87472f5ccb2d7b535e8636887cdcc79b7cb))
+* **deps-dev:** update dependency mocha to v9.1.1 ([2bcceb7](https://github.com/wmfs/pg-delta-file/commit/2bcceb75124e012cbf5e867078d161509ee4fa92))
+* **deps-dev:** update dependency mocha to v9.1.2 ([9dc47b0](https://github.com/wmfs/pg-delta-file/commit/9dc47b02a3f726a6d4c17d285434c9ff88fe93a2))
+* **deps-dev:** update dependency mocha to v9.1.3 ([b2bbe70](https://github.com/wmfs/pg-delta-file/commit/b2bbe7045435c8770340cb4f2a5024a913c6e349))
+* **deps-dev:** update dependency mocha to v9.1.4 ([561e50a](https://github.com/wmfs/pg-delta-file/commit/561e50abb55bd0d035014001b44fc46e414e8b09))
+* **deps-dev:** update dependency mocha to v9.2.0 ([502bb69](https://github.com/wmfs/pg-delta-file/commit/502bb69c987286039c6ae1efb4857de0dd71e540))
+* **deps-dev:** update dependency semantic-release to v17.4.5 ([fc7da16](https://github.com/wmfs/pg-delta-file/commit/fc7da1662a66b9be0fbcca303966c4dc0a7d9c40))
+* **deps-dev:** update dependency semantic-release to v17.4.6 ([1a663d9](https://github.com/wmfs/pg-delta-file/commit/1a663d9473ebb1ff1fa0228c68d059edd16e178e))
+* **deps-dev:** update dependency semantic-release to v17.4.7 ([7d1abda](https://github.com/wmfs/pg-delta-file/commit/7d1abda27ee60a87f45016a73fe85fd217116135))
+* **deps-dev:** update dependency semantic-release to v18 ([e1c04f3](https://github.com/wmfs/pg-delta-file/commit/e1c04f3ac238ee177a1fb84a84072c6a455ef7e7))
+* **deps-dev:** update dependency semantic-release to v18.0.1 ([32c5a14](https://github.com/wmfs/pg-delta-file/commit/32c5a148dabf97df2e785802211f264fe2d849aa))
+* **deps-dev:** update dependency semantic-release to v19 ([754d79a](https://github.com/wmfs/pg-delta-file/commit/754d79aa13e12e23a2ac02dabd7469cce7a4a10a))
+* **deps-dev:** update dependency standard to v16.0.4 ([817fbcf](https://github.com/wmfs/pg-delta-file/commit/817fbcf362fcd5295e78dc2bcaf374a2438826f5))
+* **deps-dev:** update semantic-release monorepo ([b07495e](https://github.com/wmfs/pg-delta-file/commit/b07495e1dfbed391e2444e98fd344a26371146f0))
+* **deps:** update dependency luxon to v1.28.0 ([0fedca1](https://github.com/wmfs/pg-delta-file/commit/0fedca13c1681881cb6d52d455feaa3f34114f45))
+
+
+### ⚙️ Continuous Integrations
+
+* **circle:** CircleCI update next gen PostgreSQL image [sc-11442] ([4e377f0](https://github.com/wmfs/pg-delta-file/commit/4e377f0048a1c546893f9b81c253b3385ce71e58))
+
+
+### ♻️ Chores
+
+* add renovate config [ch6600] ([5aefeae](https://github.com/wmfs/pg-delta-file/commit/5aefeae5b0c91a3d96b8435321536523b76e7eb4))
+
+# [1.71.0](https://github.com/wmfs/pg-delta-file/compare/v1.70.0...v1.71.0) (2021-05-26)
+
+
+### 🛠 Builds
+
+* **deps:** Bump luxon from 1.26.0 to 1.27.0 ([5c8cdc2](https://github.com/wmfs/pg-delta-file/commit/5c8cdc26799ab8700c0881b4db18e86ff3f14b06))
+* **deps-dev:** Bump codecov from 3.8.1 to 3.8.2 ([5d82a11](https://github.com/wmfs/pg-delta-file/commit/5d82a11c62497a666acfb0babad1a9760f247e83))
+* **deps-dev:** Bump mocha from 8.3.2 to 8.4.0 ([663bcda](https://github.com/wmfs/pg-delta-file/commit/663bcda96317aed2886c59345438022e1170f5a6))
+* **deps-dev:** Bump semantic-release from 17.4.2 to 17.4.3 ([c05f321](https://github.com/wmfs/pg-delta-file/commit/c05f321793f24c8cec861da30978d3be2c272df3))
+
+# [1.70.0](https://github.com/wmfs/pg-delta-file/compare/v1.69.0...v1.70.0) (2021-03-31)
+
+
+### ✨ Features
+
+* 🎸 New option - deletesFunction(namespace,model,sinceDate,oStream) ([0b942bf](https://github.com/wmfs/pg-delta-file/commit/0b942bf89516e5b4bed44b7ec380d112528f32fb))
+
+
+### 🛠 Builds
+
+* **deps-dev:** Bump chai from 4.3.0 to 4.3.1 ([5677d33](https://github.com/wmfs/pg-delta-file/commit/5677d33161632a35cff62c914e48221fa5f91efc))
+* **deps-dev:** Bump chai from 4.3.1 to 4.3.2 ([83260cb](https://github.com/wmfs/pg-delta-file/commit/83260cbac7f007ee59d9134bb1fac3f92dbcea4d))
+* **deps-dev:** Bump chai from 4.3.2 to 4.3.3 ([48e129c](https://github.com/wmfs/pg-delta-file/commit/48e129cdd67a67047e97cf57e13a9807c31f4e8a))
+* **deps-dev:** Bump chai from 4.3.3 to 4.3.4 ([e6726a6](https://github.com/wmfs/pg-delta-file/commit/e6726a631bf43614c746629b544c2338db917bce))
+* **deps-dev:** Bump mocha from 8.3.0 to 8.3.1 ([bda1d78](https://github.com/wmfs/pg-delta-file/commit/bda1d78598bf242b2370b24240555cabc6fc1d4e))
+* **deps-dev:** Bump mocha from 8.3.1 to 8.3.2 ([a97b476](https://github.com/wmfs/pg-delta-file/commit/a97b476d940af866d6b1d5f02228a911540a70a2))
+* **deps-dev:** Bump semantic-release from 17.3.9 to 17.4.0 ([e64ffa0](https://github.com/wmfs/pg-delta-file/commit/e64ffa0fd5e072decd147a8d5dbbd2af172f82c6))
+* **deps-dev:** Bump semantic-release from 17.4.0 to 17.4.1 ([d0f84b5](https://github.com/wmfs/pg-delta-file/commit/d0f84b554bcf23cf633ac15d6a70895312f193e9))
+* **deps-dev:** Bump semantic-release from 17.4.1 to 17.4.2 ([0e602be](https://github.com/wmfs/pg-delta-file/commit/0e602be52f4f4737b4770a78f7879f1d90e4a893))
+
+
+### ⚙️ Continuous Integrations
+
+* **circle:** update postgres config to 13.2 ([fef7a09](https://github.com/wmfs/pg-delta-file/commit/fef7a09df279122f45786c66aed0fe2296ac735c))
+
 # [1.69.0](https://github.com/wmfs/pg-delta-file/compare/v1.68.0...v1.69.0) (2021-02-24)
 
 

package/lib/Transformer.js

@@ -3,10 +3,10 @@ const csvEncoder = require('./simple-csv-encoder')
 const DateTime = require('luxon').DateTime
 
 class Transformer extends Transform {
-  constructor (info, model, options) {
+  constructor (info, model, options, processingDeletes) {
     super({ objectMode: true })
     this.info = info
-    this.modelInfo = { totalCount: 0 }
+    this.modelInfo = this.info[model] || { totalCount: 0 }
     this.info[model] = this.modelInfo
 
     const transformers = options.transformFunction ? options.transformFunction : identityTransform
@@ -29,6 +29,8 @@ class Transformer extends Transform {
 
     this.progressCallback = options.progressCallback
 
+    const actionTransformer = !processingDeletes ? insertOrUpdateAction(options) : deleteAction(options)
+
     this.transformers = options.csvExtracts[model].map(csvColumnSource => {
       switch (csvColumnSource[0]) {
         case '$': {
@@ -37,21 +39,7 @@ class Transformer extends Transform {
             case 'ROW_NUM':
               return (row, info) => info.totalCount
             case 'ACTION':
-              return row => {
-                // TODO: handle deleted action
-                const createdCol = options.createdColumnName || '_created'
-                const modifiedCol = options.modifiedColumnName || '_modified'
-                const created = new Date(row[createdCol])
-                const modified = new Date(row[modifiedCol])
-                const since = new Date(options.since)
-
-                if (modified >= since && created >= since) {
-                  return options.actionAliases.insert
-                }
-                if (modified >= since && created <= since) {
-                  return options.actionAliases.update
-                }
-              }
+              return actionTransformer
             case 'TIMESTAMP':
               return () => DateTime.local().toLocaleString(DateTime.TIME_24_WITH_SECONDS)
             case 'DATESTAMP':
@@ -122,4 +110,25 @@ function identityTransform (row, callback) {
   callback(null, row)
 }
 
+function insertOrUpdateAction (options) {
+  return row => {
+    const createdCol = options.createdColumnName || '_created'
+    const modifiedCol = options.modifiedColumnName || '_modified'
+    const created = new Date(row[createdCol])
+    const modified = new Date(row[modifiedCol])
+    const since = new Date(options.since)
+
+    if (modified >= since && created >= since) {
+      return options.actionAliases.insert
+    }
+    if (modified >= since && created <= since) {
+      return options.actionAliases.update
+    }
+  }
+}
+
+function deleteAction (options) {
+  return () => options.actionAliases.delete
+}
+
 module.exports = Transformer

package/lib/generate-delta.js

@@ -20,6 +20,8 @@ module.exports = async function generateDelta (options) {
       }
 
       await extractModel(info, model, options, deltaFileWriteStream)
+
+      await processDeletes(info, model, options, deltaFileWriteStream)
     }
   } finally {
     progressCallback(info, true)
@@ -73,3 +75,12 @@ function tableName (namespace, model) {
 
   return (separator === -1) ? `${snakeCase(namespace)}.${model}` : model
 } // tableName
+
+function processDeletes (info, model, options, outStream) {
+  if (!options.deletesFunction) { return }
+  const csvTransformer = new Transformer(info, model, options, true)
+  csvTransformer.pipe(outStream, { end: false })
+
+  const since = new Date(options.since)
+  return options.deletesFunction(options.namespace, model, since, csvTransformer)
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wmfs/pg-delta-file",
-  "version": "1.69.0",
+  "version": "1.73.0",
   "description": "Outputs change-only-update CSV files (or “delta” files) that contain all the necessary actions required to re-synchronize rows in a cloned table.",
   "author": "West Midlands Fire Service",
   "homepage": "https://github.com/wmfs/pg-delta-file#readme",
@@ -22,24 +22,24 @@
   "main": "./lib/index.js",
   "dependencies": {
     "lodash": "4.17.21",
-    "luxon": "1.26.0",
+    "luxon": "2.3.0",
     "make-dir": "3.1.0",
     "pg": "8.5.1",
     "pg-query-stream": "4.0.0"
   },
   "devDependencies": {
-    "chai": "4.3.0",
-    "codecov": "3.8.1",
+    "chai": "4.3.4",
+    "codecov": "3.8.3",
     "conventional-changelog-metahub": "4.0.1",
     "cz-conventional-changelog": "3.3.0",
     "dirty-chai": "2.0.1",
-    "mocha": "8.3.0",
+    "mocha": "9.2.0",
     "nyc": "15.1.0",
-    "semantic-release": "17.3.9",
-    "standard": "16.0.3",
-    "@semantic-release/changelog": "5.0.1",
-    "@semantic-release/git": "9.0.0",
-    "@wmfs/hl-pg-client": "1.27.0"
+    "semantic-release": "19.0.2",
+    "standard": "16.0.4",
+    "@semantic-release/changelog": "6.0.1",
+    "@semantic-release/git": "10.0.1",
+    "@wmfs/hl-pg-client": "1.28.0"
   },
   "scripts": {
     "lint": "standard",

package/renovate.json

@@ -0,0 +1,26 @@
+{
+  "extends": [
+    "config:base"
+  ],
+  "dependencyDashboard": true,
+  "enabledManagers": ["npm"],
+  "packageRules": [
+    {
+      "matchDepTypes": ["devDependencies"],
+      "automerge": true,
+      "semanticCommitType": "build",
+      "semanticCommitScope": "deps-dev",
+      "labels": ["devDependencies"]
+    },
+    {
+      "matchDepTypes": ["dependencies"],
+      "semanticCommitType": "build",
+      "semanticCommitScope": "deps",
+      "labels": ["dependencies"],
+      "reviewers": ["team:tymly-reviewers"]
+    }
+  ],
+    "vulnerabilityAlerts": {
+    "labels": ["security"]
+  }
+}

package/test/fixtures/expected/with-deletes.csv

@@ -0,0 +1,6 @@
+73,"i",1,1,"Homer","Simpson",39
+73,"u",2,2,"Marge","Simpson",36
+73,"i",3,5,"Montgomery","Burns",123
+73,"i",4,6,"Ned","Flanders",60
+73,"i",5,8,"Bart","Simpson",10
+73,"d",6,99,"Marvin","Monroe",45

package/test/tests.js

@@ -264,6 +264,45 @@ describe('Run the basic usage example', function () {
           ]
         }
       }
+    },
+    {
+      name: 'delta with delete',
+      file: 'with-deletes.csv',
+      count: 6,
+      info: {
+        totalCount: 6,
+        people: {
+          totalCount: 6
+        }
+      },
+      delta: {
+        namespace: 'springfield',
+        since: '2016-06-03 15:02:38.000000 GMT',
+        actionAliases: {
+          insert: 'i',
+          update: 'u',
+          delete: 'd'
+        },
+        deletesFunction: function (namespace, tableName, sinceDate, outputStream) {
+          outputStream.write({
+            social_security_id: 99,
+            first_name: 'Marvin',
+            last_name: 'Monroe',
+            age: 45
+          })
+        },
+        csvExtracts: {
+          people: [
+            73,
+            '$ACTION',
+            '$ROW_NUM',
+            '@social_security_id',
+            '@first_name',
+            '@last_name',
+            '@age'
+          ]
+        }
+      }
     }
   ]