pogi 2.11.0

package/docs/executingSqlFile.md

@@ -0,0 +1,44 @@
+## Executing sql files
+
+### Run an sql file for a single schema
+
+```js
+
+    let pgdb = await PgDb.connect(..);
+    await pgdb.execute(__dirname + '/init.sql');
+    
+```
+
+
+### Run an sql file for multiple schemas:
+
+```js
+imports ..
+
+let pgdb = await PgDb.connect(..);
+try {
+    for (let schemaName of ['test1', 'test2']) {
+        await pgdb.execute(__dirname + '/db_upgrade/all.sql', 
+                           (cmd)=>cmd.replace(/__SCHEMA__/g, '"' + schemaName + '"'));
+    }
+} catch (e) {
+    //log
+}
+```
+
+for example the sql file is (note: `__SCHEMA__` will be replaced with the `schemaName` see above)
+```sql
+
+DO $$
+    BEGIN
+        BEGIN
+            ALTER TABLE __SCHEMA__.webapp add column lang char(2) default 'JS';
+        EXCEPTION
+            WHEN duplicate_column THEN RAISE NOTICE 'column <column_name> already exists in <table_name>.';
+        END;
+    END;
+$$;
+
+UPDATE __SCHEMA__.webapp set lang='TS' where lang='JS';
+
+```

package/docs/faq.md

@@ -0,0 +1,15 @@
+##Does it support prepared statements?
+At the moment not, we can add it, but according to 
+[this](https://github.com/brianc/node-postgres/wiki/Parameterized-queries-and-Prepared-Statements)
+they do not add big values.
+
+##What is pogi?
+**pgdb** name was taken, so we renamed it to **pogi**. Pogi is the nickname of a delicious pastry. Yam!
+Search for 'pogacsa'.
+
+##Is there any company behind it?
+It was originally developed at [www.labcup.net](http://www.labcup.net/), where we also use it. 
+
+##My IDE don't show pogi's typing definitions
+Typescript should get it from npm, but maybe this help:
+`typings install pogi=github:holdfenytolvaj/pogi/lib/index.d.ts --save`

package/docs/functions.md

@@ -0,0 +1,19 @@
+#Using stored procedures
+When loading PgDb, stored procedures/functions will be loaded as well to the `fn` namespace for schemas. Like this function
+```SQL
+CREATE OR REPLACE FUNCTION increment(i INT)
+RETURNS INT AS $$
+BEGIN
+  RETURN i + 1;
+END;
+$$ LANGUAGE plpgsql;
+```
+could be used as easy as this
+```js
+    var num = pgdb.fn.increment(4);
+    
+    //or if is in a specific schema:
+     
+    num = pgdb[schema].fn.increment(4); 
+```
+It will detect return values, and if it is a single value or an array of single values it will resolve result row(s) for you

package/docs/generatingInterfaceForTables.md

@@ -0,0 +1,35 @@
+#Generating interface for tables
+You can easily generate interface for tables to facilitate code, just run
+```bash
+export PGUSER='test'
+export PGPASSWORD='test'
+export PGDATABASE='test'
+#using PGUSER, PGPASSWORD and PGDATABASE env variables
+node --harmony pgdb/lib/bin/generateInterface > testDbInterface.ts
+```
+It will generate something like:
+```js 
+import {PgDb, PgSchema, PgTable} from "pogi";
+
+export interface PgDbType extends PgDb {
+    'pgdb_test': PgSchema_pgdb_test;
+    'schemas': {
+        'pgdb_test': PgSchema_pgdb_test;
+    }
+}
+
+export interface PgSchema_pgdb_test extends PgSchema {
+    'users': PgTable;
+    tables: {
+        'users': PgTable;
+    }
+}
+```
+
+So you can use it as 
+```js
+let pgdb = <PgDbType>await PgDb.connect({connectionString: "postgres://"});
+let users = await pgdb.pgdb_test.users.findAll();
+```
+
+If you want to help, you can add, the table definition generation as well, also to merge if schemas or tables have the same type.

package/docs/index.md

@@ -0,0 +1,48 @@
+![Image](https://c3.staticflickr.com/6/5680/29544431994_954237c121_b.jpg) 
+
+## pogi 
+**pogi** is an easy to use PostgreSQL handler for javascript, built on top of [pg.js](https://github.com/brianc/node-postgres) 
+(and inherited few things from [MassiveJS](https://github.com/robconery/massive-js)). Supporting connection pooling, transaction, 
+typescript, async-await, assignable logger, stream, executable sql files and what not, top of that with a lot of sensible default.
+
+It is not a full-featured ORM, it is rather aligned with KISS. 
+Therefore no initial definitions are needed. It rather issues some initial queries at startup
+to look up the schemas, tablenames and some special column types.
+Aim to makes a seamless integration with js objects and removes boiler plate code but keeps 
+the power of custom queries.
+
+```js
+import {PgDb} from "pogi";
+
+(async()=> {
+    let pgdb = await PgDb.connect({connectionString: "postgres://"});
+    
+    let table = pgdb['test']['users']; //or pgdb.test.users if you generate the interface
+    
+    let c1 = await pgdb.query(`SELECT COUNT(*) as c FROM ${table} WHERE active=:active`, {active: true});
+    let c2 = await table.count({active: true});
+    c1[0].c == c2 //true
+    
+    let user = {name: 'admin'}
+    await table.insert(user);
+    
+    await table.update({id: 1}, user);
+    
+    let res = await table.find({id: [1,2,3]});
+    ...
+
+    pgdb.close(); //optional 
+    
+})().catch(console.error)
+```
+
+Typescript should get typing definition from npm package, but if doesn't you can add with typings:
+```sh 
+typings install pogi=github:holdfenytolvaj/pogi/lib/index.d.ts --save
+```
+
+## Why we need +1?
+Since wanted to keep things simple (and use Postgre full power as much as possible), ORMs were out of consideration. pg.js on the other 
+hand was too basic, still required a lot of boiler plate code. MassiveJs looked promising, but
+there again too much restriction applied (no pool, no logger, no typescript, no transaction, no mixing of 
+relational and jsonb columns (not safely at least), etc) and adding these were not possible without redesign.

package/docs/logger.md

@@ -0,0 +1,40 @@
+## Logging
+
+The logger need to implement the following interface:
+
+``` js 
+export interface PgDbLogger {
+    log: Function;
+    error: Function;
+}
+```
+
+where log most of the time is called with 3 params:
+1. sql query
+2. parameter array if any
+3. connection id (from the pool)
+
+It is possible to specify separate loggers per schemas (and tables or queries). E.g.
+```js
+let myDbLogger = console;
+let mySchemaLogger = {log:()=>{}, error:console.error};
+let myTableLogger = {log:(sql, params, connectionId) => (connectionId) ? console.log('[',connectionId,']', sql,' < ',params) : console.log(sql), 
+                     error:console.error};
+let myQueryLogger = console;
+                     
+pgdb.setLogger(myDbLogger); //default console
+pgdb.myschema.setLogger(mySchemaLogger);
+pgdb.myschema.users.setLogger(myTableLogger); 
+
+pgdb.run('SELECT password FROM myschema.users');  //logged by myDbLogger
+pgdb.myschema.run('SELECT password FROM myschema.users'); //logged by mySchemaLogger
+pgdb.myschema.machines.findAll(); //logged by mySchemaLogger
+
+pgdb.myschema.users.findAll(); //logged by myTableLogger
+
+pgdb.myschema.users.findAll({logger:myQueryLogger}); //logged by myQueryLogger
+```
+The first logger found will be the one to be used.
+
+
+

package/docs/mappingDatabaseTypes.md

@@ -0,0 +1,89 @@
+##Type conversion
+
+By default all field returned as a string from postgre. Pg.js convert some of it, e.g. dates
+ to js types, pgdb will convert a bit more e.g. dates arrays, number arrays, bigInts,   
+ 
+### The simplest way 
+
+```ts
+    var numWithValidation = val => {
+        let v = +val;
+        if (v > Number.MAX_SAFE_INTEGER || v < Number.MIN_SAFE_INTEGER) {
+            throw Error("Number can't be represented in javascript precisely: " + val);
+        }
+        return v;
+    };
+
+    await pgdb.setTypeParser('int8', numWithValidation, 'myschema'); 
+    //leaving out 'myschema' would apply to all schemas
+   
+```
+
+### Looks simple, but...
+pg.js doesn't handle well the exception during type conversion. 
+If exception is thrown the node process will exit. So you can add your converter
+to pgdb layer if exception is possible. Note the 'PgDb' in the function name.
+
+```ts
+    await pgdb.setPgDbTypeParser('int8', numWithValidation); 
+```
+
+### complex types and complex type arrays
+Complex type looks promising instead of link tables, but unfortunately 
+they do not have to foreign key check at the moment in PostgreSQL(9.6). 
+Also right now we didn't put much effort to this feature, we rather use jsonb columns instead. 
+But it would be relative easy to add support for them to save and read as js objects (and might be support the operators).
+
+####Example complex type:
+```sql
+    CREATE TYPE "permissionType" AS ENUM ('read', 'write', 'admin');
+    CREATE TYPE "permissionForResourceType" AS (
+        "permission"    "permissionType",
+        "resource"      "text"
+    );
+```
+
+####Add parsing to list:
+```js
+function parseComplexType(str) {
+    //cut of '(', ')'
+    str = str.substring(1, str.length-1);
+    let e = /"((?:[^"]|"")*)"(?:,|$)|([^,]*)(?:,|$)/g;
+    let valList = [];
+    let parsingResult;
+    let valStr;
+    let hasNextValue;
+    /**
+     * parsingResult.index<str.length check for finish is not reliable
+     * as if the last value is null it goes undetected, e.g. (,,)
+     */
+    do {
+        parsingResult = e.exec(str);
+        valStr = (parsingResult[0]=='' || parsingResult[0]==',' || parsingResult[2]=='null') ? null : parsingResult[1] || parsingResult[2] ;
+        if (parsingResult[0]=='"",' || parsingResult[0]=='""') {
+            valStr = '';
+        }
+        valList.push(valStr ? valStr.replace(/""/g,'"') : valStr);
+        hasNextValue = parsingResult[0].substring(parsingResult[0].length-1,parsingResult[0].length)==',';
+    } while (hasNextValue);
+    return valList;
+}
+
+await pgdb.setTypeParser('permissionForResourceType', parseComplexType, 'myschema'); 
+
+```
+
+####Add parsing for array:
+```js
+function parseComplexTypeArray(str) {
+    let list = JSON.parse('[' + str.substring(1, str.length - 1) + ']');
+
+    let result = [];
+    for (let elementStr of list) {
+        result.push(parseComplexType(elementStr));
+    }
+    return result;
+}
+
+await pgdb.setTypeParser('_permissionForResourceType', parseComplexTypeArray, 'myschema');
+```

package/docs/notification.md

@@ -0,0 +1,19 @@
+## Notification
+as simple as:
+```js
+    let result = '';
+    await db.listen('channel', (data) => { result += data.payload; });
+    await db.listen('channel', () => { result += ',nextCallback'; });
+
+    await db.run(`NOTIFY channel, 'data'`);
+    //same as: await db.notify('channel', 'data');
+    //result will be: 'data,nextCallback'
+
+    await db.unlisten('channel'); 
+    //dedicated listener connection now released.
+```
+
+See [Postgresql documentation](https://www.postgresql.org/docs/current/sql-notify.html)
+
+## Some comments
+Notification listeners uses a dedicated connection. If e.g. the postgresql server restarts, some notifications might not be received, but the connection and the listeners will be re-created.

package/docs/pitfalls.md

@@ -0,0 +1,73 @@
+
+## Known pitfalls
+>Nothing is without pitfalls, but for most libraries it's well hidden...
+
+### Named parameter + `column in ()` expression
+
+```
+let dates = ['2000-01-01','2000-02-02'];
+//don't work:
+await pgdb.query(`SELECT * FROM ${dbTable} b WHERE DATE(b.from) IN :dates`, {dates});
+
+//work:
+await pgdb.query(`SELECT * FROM ${dbTable} b WHERE ARRAY[DATE(b.from)] && :dates`, {dates});
+```
+
+
+### postgre data types vs javascript types - general
+pg.js is a powerful library but left many decision to the user, e.g. converting types. 
+By default it doesn't convert arrays or integers, dates are also tricky. We've added some basic 
+types, but rarely used types (e.g. date-ranges, custom complex types) still need to be converted.
+
+### postgre data types vs javascript types - number
+Javacript does not handle integers precisely out of range [Number.MIN_SAFE_INTEGER, Number.MAX_SAFE_INTEGER],
+and this is why it is not converted by default by pg.js, but its rarely reached,
+thus it is safe to convert by default, and just throw an exception when it is above that number (9007199254740991); 
+
+### postgre data types vs javascript types - date
+e.g. when you write your own queries, do not do this:
+```js
+let date = new Date().toISOString();
+await pgdb.query(`SELECT * FROM users where created>'${date}'`);
+```
+this equals to (or something similar depending on your timezone):
+```sql
+select * from users where created>'1999-12-31T18:00:00.000Z';
+```
+it will have some side effect as postgre will compare them as a string, so you need to enforce the conversion:
+```js
+await pgdb.query(`SELECT * FROM users where created>'${date}'::timestamptz`);
+//or 
+await userTable.find({'created >':date});
+await userTable.find({'created >': new Date()});//no need for toISOString()
+```
+this equals to:
+```sql
+select * from users where created>'2000-01-01 00:00:00';
+```
+
+### Table or column type change / truncate
+If a column type changes or truncates occur (truncate needs a double check, it might be possible), 
+the postgres data type's oid numbers might changes, after
+these queries the pgdb needs to rerun the initial column type queries. 
+This can be done as easy as 
+```js
+await pgdb.reload();
+```
+
+### Results' object constructor
+Result objects have a special constructor that are not callable outside pg.js. 
+It's rarely an issue, except e.g. if you use xlsx npm package, where after cloning 
+the object, it calls the object's constructor it can cause some issues
+
+### Connectivity(?)
+See [pg-ka-fix](https://github.com/numminorihsf/pg-ka-fix). Haven't met with the issue,
+but need to investigate, leave it here as a possible issue. 
+
+### Field name collision
+Result fields do not keep their table alias references, so the following query will result
+in name collision:
+```js
+    await pgdb.query(`select u1.id, u2.id from ${table} u1 left join ${table} u2 ON true `);
+```
+not a big issue, just needs aliases, but keep in mind. In case this happens an exception will be thrown.

package/docs/streams.md

@@ -0,0 +1,68 @@
+#Example
+
+##Use stream instead of e.g.:
+```ts
+    let total = await table.count();
+    for (let offset = 0; offset < total; offset += 100)) {
+        let list = await table.findAll({offset, limit:100});
+        for (let row of list) {
+            ...
+            //unless you have a stable ordering and use it in the query above
+            //you might get some rows multiple times here...
+            //use streams instead
+        }
+    }
+```
+
+##Using queryAsStream
+```ts
+
+let stream = await pgdb.queryAsStream(`SELECT * FROM generate_series(0, 1001) num`);
+
+stream.on('data', (c: any)=> {
+    //do or not do stuff
+});
+
+await new Promise((resolve, reject)=> {
+    stream.on('end', resolve);
+    stream.on('error', reject);
+});
+
+```
+
+##Using find
+
+find/findAll/findWhere all can be used to return a stream, passing it to the options.
+
+```ts
+
+let stream;
+stream = await pgdb.users.find({'id >':1}, {stream:true});
+stream = await pgdb.users.findAll({stream:true});
+stream = await pgdb.users.findWhere('true', null, {stream:true});
+
+
+```
+
+##Using async in stream
+```ts
+let stream = await pgdb.users.find({'id >':1}, {stream:true});
+stream.on('data', (user: User)=> {
+    stream.pause();
+    
+    (async function() {
+        ...
+        await mightDoSth(user);
+        ...        
+        stream.resume();
+    })().catch(e=>{
+        console.error(e);
+        stream.emit( "error", e);
+    });
+});
+
+await new Promise((resolve, reject)=> {
+    stream.on('end', resolve);
+    stream.on('error', reject);
+});
+```