@cap-js/postgres 1.0.0 → 1.1.0

package/CHANGELOG.md

@@ -4,7 +4,36 @@
 - The format is based on [Keep a Changelog](http://keepachangelog.com/).
 - This project adheres to [Semantic Versioning](http://semver.org/).
 
+## Version 1.1.0 - 2023-08-01
+
+### Added
+
+- Connectivity to Azure PostgreSQL.
+
+### Fixed
+
+- Order by collation waterfall:
+  1. ICU
+  2. best-effort mapping (`xx` -> `xx_YY`, where `xx_YY` is the first match)
+  3. without collation
+- More stable configuration of `schema_evolution = 'auto'`.
+- Log `hostname` preferrably during deployment.
+- Allow overriding of pool configuration.
+
+### Changed
+
+- Session context variables are set as lower case instead of upper case.
+
+## Version 1.0.1 - 2023-07-03
+
+### Added
+
+- `pg` profile as mentioned in documentation
+
+### Changed
+
+- Updated minimum required version of `@cap-js/db-service` 
 
 ## Version 1.0.0 - 2023-06-23
 
-- Initial Release
+- Initial Release

package/README.md

@@ -1,5 +1,153 @@
 # CDS database service for Postgres
 
 Welcome to the new Postgres database service for [SAP Cloud Application Programming Model](https://cap.cloud.sap) Node.js, based on new, streamlined database architecture and [*pg* driver](https://www.npmjs.com/package/pg) .
+Find full documentation at https://cap.cloud.sap/docs/guides/databases-postgres.
 
-Find documentation at https://cap.cloud.sap/docs/guides/databases-postgres.
+## migration guide from `cds-pg` to `@cap-js/postgres`
+
+`@cap-js/postgres` works as a drop-in replacement for `cds-pg`.  
+However, some preliminary checks and cleanups help:
+
+- for using the BTP Postgres Hyperscaler as database, 
+  - know that the credentials are picked up automatically by from the enviornment (`VCAP_SERVICES.postgres`)
+  - the service binding label is `postgresql-db`
+  - `cds-dbm` is replaced by a hand-crafted "db-deployer" app → see below
+- your local `package.json`: you can safely remove the entry `cds.requires.postgres` previously mandatory for `cds-pg`
+- recommendation: set the env var `DEBUG=sql` during local development to see DB-level output from PostgreSQL
+
+### schema migration
+
+`@cap-js/postgres` brings the same schema evolution capabilities to PostgreSQL known from HANA and SQLite.  
+Enabling schema migration in an existing `cds-pg`-based project consists of generating and deploying a "csn-snapshot" of your database structure.
+
+#### local development
+
+First, set a basis for the evolution
+`$> cds deploy --model-only`  
+→ this will create the table `cds_model` laying the foundation for the schema migration
+
+Subsequent deployments can then re-use the standard deploy mechanism via `$> cds deploy`
+
+#### On BTP, Cloud Foundry environment
+
+The above "csn-snapshots" can be implemented via the `mtar`-based approach. At the same time, the same `mtar` can be used for subsequent PostgreSQL deployments (with schema evolution).
+
+Two major steps in addition to enabling the schema evolution are included in this `mtar`.
+
+1. create local folder `deployer` (any name works)
+2. in `deployer`, create a `package.json` containing
+
+  ```json
+  ...
+  "//npm run migrate": "only one-time!",
+  "migrate": "cds deploy --model-only",
+  "//npm run deploy": "subsequent deployments",
+  "deploy": "cds deploy"
+  ...
+  ```
+
+3. add a section to your `/mta.yaml` denoting the `deployer` directory as a standalone application that runs one-time
+
+```yaml
+- name: pg-db-deployer
+    type: custom
+    path: deployer
+    parameters:
+      buildpacks: nodejs_buildpack
+      no-route: true
+      no-start: true
+      disk-quota: 2GB
+      memory: 512MB
+      tasks:
+      - name: migrate
+        command: npm run migrate
+      # # for subsequent deployments
+      # - name: deploy
+      #  command: npm run deploy
+        disk-quota: 2GB
+        memory: 512MB
+    build-parameters:
+      before-all:
+        custom: 
+        - npm i
+        # generate the "csn-snapshot" - only necessary for one-time migration,
+        # can be commented out on subsequent deployments
+        - cds compile '*' -2 json > deployer/schema.csn
+      ignore: ["node_modules/"]
+    requires:
+      - name: pg-database
+
+resources:
+  - name: pg-database
+    parameters:
+      path: ./pg-options.json
+      service: postgresql-db
+      service-plan: trial # change to yours!
+      skip-service-updates:
+        parameters: true
+      service-tags:
+        - plain
+    type: org.cloudfoundry.managed-service
+```
+
+## migration points to consider
+
+### mixed-case identifiers
+
+even though column names that are not double-quoted are folded to lowercase in PostgreSQL (`yourName` -> `yourname`, `"yourName"` -> `yourName`),  
+you can use the mixed case definitions from your `.cds` files to reference them.  
+
+example: `brewery_id` on DB level -> `brewery_ID` on CDS level
+
+formerly w/ `cds-pg` you had to follow the DB level: `SELECT.from(Beers).columns('brewery_id').groupBy('brewery_id')`  
+now, re-use the CDS definitions: `SELECT.from(Beers).columns('brewery_ID').groupBy('brewery_ID')`
+
+So please adjust your `CQL` statements accordingly.
+
+### timezones (potential _**BREAKING CHANGE**_)
+
+any date- + time-type will get stored in [`UTC`](https://en.wikipedia.org/wiki/Coordinated_Universal_Time) **without any timezone identifier in the actual data field**.  
+CAP's inbound- and outbound adapters take care of converting incoming and outgoing data from/to the desired time zones.  
+So when a `dateime` comes in being in [an ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) compatible format  
+  `2009-01-01T15:00:00+01:00` (15:00:00 on January 1 2009 in Vienna (CEST))  
+will get stored as  
+  `2009-01-01T13:00:00` (13:00:00 on January 1 2009 in UTC).
+
+Please be aware of that concept and rely on the client to parse UTC in your desired timezone (format).
+
+### `cds.DatabaseService` consumption
+
+`InsertResult` now does only return the affected rows and their `ID`s.
+
+```js
+const entries = [
+  { name: 'Beer1', /* ... */ },
+  { name: 'Beer2', /* ... */ },
+  { name: 'Beer3', /* ... */ }
+]
+const insertResult = await cds.run(INSERT.into(Beers).entries(entries))
+expect(insertResult.affectedRows).to.equal(3)
+const beers = [...insertResult] //> this calls the [Symbol.iterator] method of the insert result
+// beers:
+// [ 
+//   { ID: "f81d7ee5-922b-48a1-a12a-a899b8594c99" }, 
+//   { ID: "ddda7f8e-e26b-430f-a80c-ac2c7df29510" },
+//   { ID: "7228c40f-0046-4f53-8a2b-3d55ad825f59" }
+// ]
+```
+
+In `cds-pg`, we additionally surfaced the entire inserted dataset.
+
+```js
+// continuing after the insert of the above example:
+// const insertResult = await cds.run(INSERT.into(Beers).entries(entries))
+
+// this works NO MORE - see above
+const beers = insertResult.results
+expect(beers.length).toStrictEqual(3)
+expect(beers[0].ID).toMatch(uuidRegex)
+expect(beers[0].createdAt.toISOString()).toMatch(timestampRegex)
+expect(beers[0].modifiedAt.toISOString()).toMatch(timestampRegex)
+```
+
+So please adjust your runtime coding accordingly.

package/lib/PostgresService.js

@@ -17,11 +17,11 @@ class PostgresService extends SQLService {
   get factory() {
     return {
       options: {
-        ...this.options.pool,
         min: 0,
         testOnBorrow: true,
         acquireTimeoutMillis: 1000,
         destroyTimeoutMillis: 1000,
+        ...this.options.pool,
       },
       create: async () => {
         const cr = this.options.credentials || {}
@@ -36,10 +36,14 @@ class PostgresService extends SQLService {
           database: cr.dbname || cr.database,
           schema: cr.schema,
           sslRequired: cr.sslrootcert && (cr.sslrootcert ?? true),
-          ssl: cr.sslrootcert && {
-            rejectUnauthorized: false,
-            ca: cr.sslrootcert,
-          },
+          // from pg driver docs:
+          // passed directly to node.TLSSocket, supports all tls.connect options
+          ssl:
+            cr.ssl /* enable pg module setting to connect to Azure postgres */ ||
+            (cr.sslrootcert && {
+              rejectUnauthorized: false,
+              ca: cr.sslrootcert,
+            }),
         }
         const dbc = new Client(credentials)
         await dbc.connect()
@@ -52,18 +56,18 @@ class PostgresService extends SQLService {
 
   url4() {
     // TODO: Maybe log which database and which user? Be more robust against missing properties?
-    let { host, port } = this.options?.credentials || this.options || {}
-    return host + ':' + (port || 5432)
+    let { host, hostname, port } = this.options?.credentials || this.options || {}
+    return (hostname || host) + ':' + (port || 5432)
   }
 
   async set(variables) {
     // REVISIT: remove when all environment variables are aligned
     // RESTRICTIONS: 'Custom parameter names must be two or more simple identifiers separated by dots.'
     const nameMap = {
-      '$user.id': 'CAP.APPLICATIONUSER',
-      '$user.locale': 'CAP.LOCALE',
-      '$valid.from': 'CAP.VALID_FROM',
-      '$valid.to': 'CAP.VALID_TO',
+      '$user.id': 'cap.applicationuser',
+      '$user.locale': 'cap.locale',
+      '$valid.from': 'cap.valid_from',
+      '$valid.to': 'cap.valid_to',
     }
 
     const env = {}
@@ -79,22 +83,55 @@ class PostgresService extends SQLService {
         ? [this.exec(`SET search_path TO "${this.options?.credentials?.schema}";`)]
         : []),
 
-      ...(!this._initalCollateCheck
-        ? [
-            (await this.prepare(`SELECT collname FROM pg_collation WHERE collname = 'en_US' OR collname ='en-x-icu';`))
-              .all([])
-              .then(resp => {
-                this._initalCollateCheck = true
-                if (resp.find(row => row.collname === 'en_US')) return
-                if (resp.find(row => row.collname === 'en-x-icu'))
-                  this.class.CQN2SQL.prototype.orderBy = this.class.CQN2SQL.prototype.orderByICU
-                // REVISIT throw error when there is no collated libary found
-              }),
-          ]
-        : []),
+      ...(!this._initalCollateCheck ? [this._checkCollation()] : []),
     ])
   }
 
+  async _checkCollation() {
+    this._initalCollateCheck = true
+
+    const icuPrep = await this.prepare(`SELECT collname FROM pg_collation WHERE collname = 'en-x-icu';`)
+    const icuResp = await icuPrep.all([])
+
+    if (icuResp.length > 0) {
+      this.class.CQN2SQL.prototype.orderBy = this.class.CQN2SQL.prototype.orderByICU
+      return
+    }
+
+    /**
+     * Selects the first two characters of the collation name as key
+     * Select the smallest collation name as value (could also be max)
+     * Filter the collations by the provider c (libc)
+     * Filters the collation names by /.._../ Where '>' points at the '_' that is an actual '_'
+     * The group by is done by the key column to make sure that only one collation per key is returned
+     */
+    const cSQL = `
+SELECT 
+  SUBSTRING(collname, 1, 2) AS K,
+  MIN(collname) AS V 
+FROM 
+  pg_collation 
+WHERE 
+  collprovider = 'c' AND 
+  collname LIKE '__>___' ESCAPE '>' 
+GROUP BY k
+`
+
+    const cPrep = await this.prepare(cSQL)
+    const cResp = await cPrep.all([])
+    if (cResp.length > 0) {
+      const collationMap = (this.class.CQN2SQL.prototype.collationMap = cResp.reduce((ret, row) => {
+        ret[row.k] = row.v
+        return ret
+      }, {}))
+      collationMap.default = collationMap.en || collationMap[Object.keys(collationMap)[0]]
+      this.class.CQN2SQL.prototype.orderBy = this.class.CQN2SQL.prototype.orderByLIBC
+      return
+    }
+
+    // REVISIT: print a warning when no collation is found
+  }
+
   prepare(sql) {
     const query = {
       text: sql,
@@ -178,26 +215,29 @@ class PostgresService extends SQLService {
   }
 
   static CQN2SQL = class CQN2Postgres extends SQLService.CQN2SQL {
-    orderBy(orderBy, localized) {
+    _orderBy(orderBy, localized, locale) {
       return orderBy.map(
         localized
           ? c =>
               this.expr(c) +
-              (c.element?.[this.class._localized] ? ` COLLATE "${this.context.locale}"` : '') +
+              (c.element?.[this.class._localized] ? ` COLLATE "${locale}"` : '') +
               (c.sort === 'desc' || c.sort === -1 ? ' DESC' : ' ASC')
           : c => this.expr(c) + (c.sort === 'desc' || c.sort === -1 ? ' DESC' : ' ASC'),
       )
     }
 
+    orderBy(orderBy) {
+      return this._orderBy(orderBy)
+    }
+
     orderByICU(orderBy, localized) {
-      return orderBy.map(
-        localized
-          ? c =>
-              this.expr(c) +
-              (c.element?.[this.class._localized] ? ` COLLATE "${this.context.locale.replace('_', '-')}-x-icu"` : '') +
-              (c.sort === 'desc' || c.sort === -1 ? ' DESC' : ' ASC')
-          : c => this.expr(c) + (c.sort === 'desc' || c.sort === -1 ? ' DESC' : ' ASC'),
-      )
+      const locale = `${this.context.locale.replace('_', '-')}-x-icu`
+      return this._orderBy(orderBy, localized, locale)
+    }
+
+    orderByLIBC(orderBy, localized) {
+      const locale = this.collationMap[this.context.locale] || this.collationMap.default
+      return this._orderBy(orderBy, localized && locale, locale)
     }
 
     from(from) {

package/lib/func.js

@@ -1,6 +1,5 @@
 const StandardFunctions = {
   countdistinct: x => `count(distinct ${x || '*'})`,
-  average: x => `avg(${x})`,
   contains: (...args) => `(coalesce(strpos(${args}),0) > 0)`,
   indexof: (x, y) => `strpos(${x},${y}) - 1`, // sqlite instr is 1 indexed
   startswith: (x, y) => `strpos(${x},${y}) = 1`, // sqlite instr is 1 indexed

package/package.json

@@ -1,8 +1,15 @@
 {
   "name": "@cap-js/postgres",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "CDS database service for Postgres",
-  "homepage": "https://cap.cloud.sap/",
+  "homepage": "https://github.com/cap-js/cds-dbs/tree/main/postgres#cds-database-service-for-postgres",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/cap-js/cds-dbs"
+  },
+  "bugs": {
+    "url": "https://github.com/cap-js/cds-dbs/issues"
+  },
   "keywords": [
     "CAP",
     "CDS",
@@ -24,7 +31,7 @@
     "test": "npm run setup && jest --silent"
   },
   "dependencies": {
-    "@cap-js/db-service": "^1",
+    "@cap-js/db-service": "^1.1.0",
     "pg": "^8"
   },
   "peerDependencies": {
@@ -36,6 +43,16 @@
         "sql": {
           "[production]": {
             "kind": "postgres"
+          },
+          "[pg!]": {
+            "kind": "postgres",
+            "credentials": {
+              "host": "localhost",
+              "port": 5432,
+              "user": "postgres",
+              "password": "postgres",
+              "database": "postgres"
+            }
           }
         },
         "postgres": {
@@ -43,7 +60,8 @@
           "dialect": "postgres",
           "vcap": {
             "label": "postgresql-db"
-          }
+          },
+          "schema_evolution": "auto"
         }
       },
       "db": "sql"