@profullstack/libsql-pg 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Profullstack, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,128 @@
+# @profullstack/libsql-pg
+
+A drop-in replacement for `@libsql/client` that talks to Postgres, plus the two
+tools that move a Turso/libSQL database across: a schema converter and a copier.
+
+Porting an app off Turso becomes three steps:
+
+```sh
+npx libsql-pg convert-schema db/schema.sql -o migrations-pg/0001_schema.sql   # 1. DDL
+psql "$DATABASE_URL" -f migrations-pg/0001_schema.sql
+npx libsql-pg copy --from "$TURSO_DATABASE_URL" --token "$TURSO_AUTH_TOKEN" \
+    --to "$DATABASE_URL" --verify                                          # 2. rows
+# 3. in the app: import { createClient } from '@profullstack/libsql-pg'
+```
+
+The pattern comes from rssamplifier.com's port (2026-09-25), where a libSQL
+surface over `pg` let the app keep every query and change one import.
+
+## Client
+
+```js
+import { createClient } from '@profullstack/libsql-pg';
+
+const db = createClient({ url: process.env.DATABASE_URL }); // postgres://...
+const rs = await db.execute({ sql: 'select * from feeds where id = ?', args: [id] });
+rs.rows[0].title; rs.rows[0][1]; rs.rowsAffected; rs.lastInsertRowid;
+await db.batch([{ sql: 'insert into a (x) values (?)', args: [1] }, 'delete from b'], 'write');
+const tx = await db.transaction('write'); await tx.execute(...); await tx.commit();
+```
+
+- The `@libsql/client` surface: `execute`, `batch`, `transaction`, `executeMultiple`,
+  `close`, `sync` (no-op), `protocol` = `'postgres'`. Positional `?` and named
+  `:name` / `@name` / `$name` arguments.
+- `ResultSet` shape matches libsql: `columns`, `columnTypes`, `rows` (objects with
+  column keys AND numeric indexes, plus `length`), `rowsAffected`, and
+  `lastInsertRowid` as a BigInt for an INSERT into a table with an identity or
+  serial primary key (a `RETURNING` of that key is appended when the statement
+  has none; the key is looked up once per table and cached).
+- Errors keep the messages SQLite code matches on: `UNIQUE constraint failed:
+  t.col`, `FOREIGN KEY constraint failed`, `NOT NULL constraint failed: t.col`;
+  the Postgres `code` is preserved.
+- `dialect: 'sqlite'` (the default) rewrites each statement before binding, with
+  a cache keyed by SQL text. `dialect: 'postgres'` sends SQL as-is.
+- `url` must be `postgres://` or `postgresql://`; `libsql://` and `file:` throw.
+- Options: `pool: { max }`, `dialect`.
+
+## What the rewriter changes for you
+
+| SQLite | Postgres |
+| --- | --- |
+| `INSERT OR IGNORE INTO t ...` | `INSERT INTO t ... ON CONFLICT DO NOTHING` |
+| `INSERT OR REPLACE INTO t (cols) ...`, `REPLACE INTO` | `... ON CONFLICT (pk or unique cols) DO UPDATE SET col = EXCLUDED.col` (keys looked up in `pg_index`) |
+| `datetime('now')`, `datetime('now','localtime')` | `now()` |
+| `date('now')` | `current_date` |
+| `strftime('%s','now')`, `unixepoch()` | `extract(epoch from now())::bigint` |
+| `strftime(fmt, x)` (common formats) | `to_char(x at time zone 'utc', ...)` |
+| `json_extract(col, '$.a.b[0]')` | `(col #>> '{a,b,0}')` |
+| `lower(hex(randomblob(16)))`, `hex(randomblob(n))` | `encode(gen_random_bytes(n), 'hex')` |
+| `group_concat(x, sep)` | `string_agg(x::text, sep)` |
+| `ifnull(a, b)` | `coalesce(a, b)` |
+| `CAST(x AS INTEGER)` | `CAST(x AS BIGINT)` (SQLite's integer is 64-bit) |
+| `` `backticked` `` identifiers | `"quoted"` |
+| `PRAGMA ...` | no-op, empty result |
+| `CREATE TABLE`/`ALTER TABLE` inline | passed through the schema converter |
+| `... MATCH ...` against an FTS5 table | throws, naming the table (see FTS below) |
+
+`unsupportedIdioms(sql)` lists what a statement still needs by hand: `printf`,
+`format`, `typeof`, `last_insert_rowid()`, `changes()`, `total_changes()`,
+`COLLATE NOCASE` (use `citext` or `lower()`), `GLOB` (use `LIKE` or `~`),
+`IS NOT <value>` (use `IS DISTINCT FROM`), `random()` (double in [0,1) in
+Postgres, 64-bit integer in SQLite), and bare `rowid` (Postgres tables have no
+implicit rowid; give the table an identity column named `rowid` or use the key).
+Also by hand: `json_each`/`json_tree` (use `jsonb_array_elements`), `?NNN`
+numbered parameters, integer booleans in comparisons (`= 1` against a boolean
+column), and `LIMIT -1`.
+
+## `convert-schema`
+
+```sh
+libsql-pg convert-schema schema.sqlite.sql [-o out.sql] [--json jsonb] [--search-column name] [--ts-config english]
+```
+
+`INTEGER PRIMARY KEY [AUTOINCREMENT]` becomes `bigint generated by default as
+identity primary key`; `INTEGER` -> `bigint`, `REAL` -> `double precision`,
+`BLOB` -> `bytea`, `DATETIME`/`TIMESTAMP` -> `timestamptz`, `BOOLEAN` ->
+`boolean`, `TEXT` stays; `DEFAULT (datetime('now'))`/`CURRENT_TIMESTAMP` ->
+`default now()`, `DEFAULT (strftime('%s','now'))` -> epoch default; `WITHOUT
+ROWID` dropped; indexes kept. `CREATE VIRTUAL TABLE x USING fts5(...)` becomes a
+generated `tsvector` column plus a GIN index on the content table when the
+converter can find it (and a commented TODO otherwise); triggers are emitted as
+commented TODOs. Fixtures in `test/fixtures/`.
+
+## `copy` and `verify`
+
+```sh
+libsql-pg copy --from libsql://... --token ... --to postgres://... \
+   [--tables a,b] [--exclude x,y] [--truncate] [--upsert] [--batch N] [--workers N] [--verify] [--dry-run]
+libsql-pg verify --from ... --to ...
+```
+
+Discovers tables from `sqlite_master`, orders them so foreign-key parents load
+first, streams rows in batches (600 s read timeout with retries; min/max rowid
+are two lookups, never one full scan), coerces SQLite integers into `boolean`
+and `timestamptz` columns using the target's `information_schema`, resets
+identity sequences with `setval(max)`, and `--verify` compares `count(*)` per
+table. `--truncate` is `TRUNCATE ONLY` (a parent with children errors: that is
+the point, use `--upsert`). `--upsert` refreshes in place by primary key and
+mirrors deletes, so a second run before cutover closes the gap without a
+rebuild.
+
+## Traps (all met in the rssamplifier port)
+
+- `TRUNCATE ... CASCADE` empties the child tables too; the copier never uses it.
+- An identity column must not appear in the `DO UPDATE SET` list.
+- A fresh Postgres has no statistics: run `ANALYZE` after the load or the first
+  plans are terrible.
+- Feed-style queries (`where id in (select ...) order by created_at limit n`)
+  want `with picked as materialized ... join lateral (...)`; the planner
+  otherwise scans the whole child table.
+- Postgres rejects `sslmode=no-verify` in `pg` older than 8.x and Bun.SQL
+  forwards unknown URL parameters to the server; pass `ssl` options explicitly.
+
+## Development
+
+Node >= 20, ESM, `pg` is the only runtime dependency (`@libsql/client` is an
+optional peer for the copier's source side). `node --test` runs the unit tests
+without a database; the integration tests skip unless `TEST_DATABASE_URL` is set
+(CI runs them against `postgres:17-alpine`). MIT.

package/bin/libsql-pg.js

@@ -0,0 +1,11 @@
+#!/usr/bin/env node
+import { main } from '../src/cli.js';
+
+main(process.argv.slice(2))
+  .then((code) => {
+    process.exitCode = code;
+  })
+  .catch((err) => {
+    console.error(err?.stack ?? String(err));
+    process.exitCode = 1;
+  });

package/index.js

@@ -0,0 +1,8 @@
+export { createClient, connectionSettings } from './src/client.js';
+export { createRewriter, rewriteSql, rewriteStatement, rewriteFunctions, insertTarget, jsonPathToArray, unsupportedIdioms } from './src/rewrite.js';
+export { convertSchema, convertStatement, convertDdl, mapType } from './src/schema.js';
+export { positional, named, bind, prepare } from './src/bind.js';
+export { translateError, ftsError } from './src/errors.js';
+export { toResultSet, makeRow, emptyResultSet } from './src/result.js';
+export { copyDatabase, verifyCopy, orderTables, referencedTables, userTables, coerce, insertSql } from './src/copy.js';
+export { splitStatements, codeMask } from './src/sqlparse.js';

package/package.json

@@ -0,0 +1,73 @@
+{
+  "name": "@profullstack/libsql-pg",
+  "version": "0.1.0",
+  "type": "module",
+  "description": "Drop-in replacement for @libsql/client that talks to Postgres, plus the tools to move a Turso/libSQL database into it: the same execute/batch/transaction surface, SQLite idioms rewritten on the way through, a schema converter and a row copier.",
+  "keywords": [
+    "libsql",
+    "turso",
+    "sqlite",
+    "postgres",
+    "postgresql",
+    "migration",
+    "port",
+    "adapter",
+    "drop-in",
+    "pg",
+    "schema",
+    "copy"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/profullstack/libsql-pg.git"
+  },
+  "homepage": "https://github.com/profullstack/libsql-pg#readme",
+  "bugs": {
+    "url": "https://github.com/profullstack/libsql-pg/issues"
+  },
+  "license": "MIT",
+  "author": "Profullstack, Inc.",
+  "exports": {
+    ".": {
+      "types": "./index.d.ts",
+      "default": "./index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "types": "./index.d.ts",
+  "bin": {
+    "libsql-pg": "./bin/libsql-pg.js"
+  },
+  "files": [
+    "src",
+    "bin",
+    "index.js",
+    "index.d.ts",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node --test \"test/*.test.js\"",
+    "test:unit": "node --test test/rewrite.test.js test/bind.test.js test/result.test.js test/errors.test.js test/schema.test.js test/copy-plan.test.js"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "pg": "^8.13.0"
+  },
+  "peerDependencies": {
+    "@libsql/client": ">=0.5.0"
+  },
+  "peerDependenciesMeta": {
+    "@libsql/client": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@libsql/client": "^0.15.0"
+  }
+}

package/src/bind.js

@@ -0,0 +1,141 @@
+import { codeMask } from './sqlparse.js';
+
+/**
+ * Placeholder translation and value binding.
+ *
+ * libSQL accepts `?`, `?NNN`, `:name`, `@name` and `$name`; Postgres wants
+ * `$1..$n`. Positional and named forms are handled here, outside literals and
+ * comments, and the args are lined up with the numbers that come out.
+ */
+
+/**
+ * Turn `?` and `?NNN` placeholders into `$1..$n`.
+ *
+ * `?` counts up; `?3` becomes `$3` (libSQL numbers from 1, as Postgres does).
+ * A `?` that is a jsonb operator (`?`, `?|`, `?&`) cannot be told apart from
+ * a placeholder; use `jsonb_exists()` in Postgres-native SQL instead.
+ *
+ * @param {string} sql
+ * @returns {string}
+ */
+export function positional(sql) {
+  const mask = codeMask(sql);
+  let out = '';
+  let n = 0;
+  let i = 0;
+  while (i < sql.length) {
+    if (mask[i] === '?') {
+      const num = /^\d+/.exec(mask.slice(i + 1, i + 8));
+      if (num) {
+        out += `$${num[0]}`;
+        i += 1 + num[0].length;
+      } else {
+        n += 1;
+        out += `$${n}`;
+        i += 1;
+      }
+    } else {
+      out += sql[i];
+      i += 1;
+    }
+  }
+  return out;
+}
+
+/**
+ * Turn `:name`, `@name` and `$name` placeholders into `$1..$n` and line the
+ * named values up with them. A name used twice gets one number. A `::` cast
+ * and a `$1` that is already numeric are left alone.
+ *
+ * @param {string} sql
+ * @param {Record<string, unknown>} named keys with or without their prefix
+ * @returns {{ sql: string, values: unknown[] }}
+ */
+export function named(sql, named) {
+  const lookup = new Map();
+  for (const [k, v] of Object.entries(named)) lookup.set(k.replace(/^[:@$]/, ''), v);
+  const mask = codeMask(sql);
+  const order = [];
+  const numbers = new Map();
+  let out = '';
+  let i = 0;
+  while (i < sql.length) {
+    const c = mask[i];
+    if ((c === ':' || c === '@' || c === '$') && /[A-Za-z_]/.test(mask[i + 1] ?? '')) {
+      const prev = i > 0 ? mask[i - 1] : '';
+      if (c === ':' && prev === ':') {
+        out += c;
+        i++;
+        continue;
+      }
+      // `::text` casts: the second colon is followed by a type name.
+      if (c === ':' && mask[i + 1] === ':') {
+        out += c;
+        i++;
+        continue;
+      }
+      const m = /^[A-Za-z_][A-Za-z0-9_]*/.exec(mask.slice(i + 1, i + 128));
+      const name = m[0];
+      // A `:` right after an identifier character is not a placeholder
+      // (`a:b` never appears in SQL, but a cast `x::y` is handled above).
+      if (c === ':' && /[A-Za-z0-9_)]/.test(prev)) {
+        out += c;
+        i++;
+        continue;
+      }
+      if (!lookup.has(name)) {
+        throw new Error(`named argument "${name}" was not supplied (have: ${[...lookup.keys()].join(', ') || 'none'})`);
+      }
+      if (!numbers.has(name)) {
+        numbers.set(name, order.length + 1);
+        order.push(lookup.get(name));
+      }
+      out += `$${numbers.get(name)}`;
+      i += 1 + name.length;
+    } else {
+      out += sql[i];
+      i++;
+    }
+  }
+  return { sql: out, values: order };
+}
+
+/**
+ * libSQL binds JS values loosely; pg is stricter. Undefined is null (the
+ * local libSQL client does the same). Booleans go as 1/0: Postgres reads
+ * `'1'` into a boolean column and into a bigint flag column kept from SQLite
+ * alike, where `'true'` only fits the first (`nativeBooleans: true` sends
+ * true/false). BigInts go as decimal strings. Dates become ISO strings. Typed
+ * arrays become Buffers (bytea).
+ *
+ * @param {unknown[]} args
+ * @param {{ nativeBooleans?: boolean }} [opts]
+ */
+export function bind(args, opts = {}) {
+  return args.map((v) => {
+    if (v === undefined) return null;
+    if (typeof v === 'boolean') return opts.nativeBooleans ? v : v ? 1 : 0;
+    if (typeof v === 'bigint') return v.toString();
+    if (v instanceof Date) return v.toISOString();
+    if (v instanceof ArrayBuffer) return Buffer.from(v);
+    if (ArrayBuffer.isView(v) && !(v instanceof Buffer)) return Buffer.from(v.buffer, v.byteOffset, v.byteLength);
+    return v;
+  });
+}
+
+/**
+ * Take a libSQL statement in any of its three shapes and produce the pg
+ * `{ text, values }` pair.
+ *
+ * @param {string} sql already rewritten SQL
+ * @param {unknown[] | Record<string, unknown> | undefined} args
+ * @param {{ nativeBooleans?: boolean }} [opts]
+ * @returns {{ text: string, values: unknown[] }}
+ */
+export function prepare(sql, args, opts = {}) {
+  if (args && !Array.isArray(args) && typeof args === 'object') {
+    const r = named(sql, args);
+    return { text: r.sql, values: bind(r.values, opts) };
+  }
+  return { text: positional(sql), values: bind(args ?? [], opts) };
+}

package/src/cli.js

@@ -0,0 +1,123 @@
+import { readFile, writeFile } from 'node:fs/promises';
+import { createRequire } from 'node:module';
+
+import { commaList, copyDatabase, verifyCopy } from './copy.js';
+import { convertSchema } from './schema.js';
+
+const require = createRequire(import.meta.url);
+const { version } = require('../package.json');
+
+const USAGE = `libsql-pg ${version}
+
+  libsql-pg convert-schema <sqlite-schema.sql> [-o out.sql] [--json jsonb] [--search-column name] [--ts-config english]
+      Convert SQLite DDL to Postgres DDL (stdout unless -o).
+
+  libsql-pg copy --from <libsql://...|file:...> [--token ...] --to <postgres://...>
+      [--tables a,b] [--exclude x,y] [--truncate] [--upsert] [--batch N] [--workers N] [--verify] [--dry-run]
+      Copy every user table from the source into Postgres. Postgres must
+      already hold the schema. Default: skip a table that already has rows.
+        --truncate   TRUNCATE ONLY each table first (a parent with children errors; use --upsert)
+        --upsert     refresh in place by primary key, mirroring deletes
+        --verify     after loading (or alone), compare count(*) per table; exit 1 on a difference
+      Env fallbacks: TURSO_DATABASE_URL / LIBSQL_URL, TURSO_AUTH_TOKEN, DATABASE_URL.
+
+  libsql-pg verify --from ... --to ...     Only the count comparison.
+`;
+
+/** Flags that never take a value. */
+const BOOLEAN_FLAGS = new Set(['truncate', 'upsert', 'verify', 'dry-run', 'help', 'version']);
+
+/** @param {string[]} argv */
+export function parseArgs(argv) {
+  const flags = {};
+  const positional = [];
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a.startsWith('--')) {
+      const [k, inline] = a.slice(2).split('=', 2);
+      if (inline !== undefined) flags[k] = inline;
+      else if (BOOLEAN_FLAGS.has(k)) flags[k] = true;
+      else if (argv[i + 1] !== undefined && !argv[i + 1].startsWith('-')) flags[k] = argv[++i];
+      else flags[k] = true;
+    } else if (a === '-o') flags.out = argv[++i];
+    else if (a === '-h') flags.help = true;
+    else positional.push(a);
+  }
+  return { flags, positional };
+}
+
+/**
+ * @param {string[]} argv
+ * @param {{ stdout?: (s: string) => void, stderr?: (s: string) => void }} [io]
+ * @returns {Promise<number>} exit code
+ */
+export async function main(argv, io = {}) {
+  const out = io.stdout ?? ((s) => process.stdout.write(s));
+  const err = io.stderr ?? ((s) => process.stderr.write(s));
+  const { flags, positional } = parseArgs(argv);
+  const command = positional[0];
+  if (flags.version) {
+    out(`${version}\n`);
+    return 0;
+  }
+  if (!command || flags.help) {
+    out(USAGE);
+    return command ? 0 : 1;
+  }
+
+  if (command === 'convert-schema') {
+    const file = positional[1];
+    if (!file) {
+      err('convert-schema: a SQLite schema file is required\n');
+      return 1;
+    }
+    const sql = await readFile(file, 'utf8');
+    const converted = convertSchema(sql, {
+      json: flags.json === 'jsonb' ? 'jsonb' : 'text',
+      searchColumn: typeof flags['search-column'] === 'string' ? flags['search-column'] : undefined,
+      textSearchConfig: typeof flags['ts-config'] === 'string' ? flags['ts-config'] : undefined,
+    });
+    if (typeof flags.out === 'string') {
+      await writeFile(flags.out, converted);
+      err(`wrote ${flags.out} (${converted.split('\n').filter((l) => /TODO/.test(l)).length} TODO line(s))\n`);
+    } else out(converted);
+    return 0;
+  }
+
+  if (command === 'copy' || command === 'verify') {
+    const env = process.env;
+    const from = typeof flags.from === 'string' ? flags.from : env.TURSO_DATABASE_URL ?? env.LIBSQL_URL;
+    const token = typeof flags.token === 'string' ? flags.token : env.TURSO_AUTH_TOKEN;
+    const to = typeof flags.to === 'string' ? flags.to : env.DATABASE_URL;
+    if (!from || !to) {
+      err(`${command}: --from and --to are required (or TURSO_DATABASE_URL and DATABASE_URL)\n`);
+      return 1;
+    }
+    const common = {
+      from,
+      token,
+      to,
+      tables: commaList(typeof flags.tables === 'string' ? flags.tables : undefined),
+      exclude: commaList(typeof flags.exclude === 'string' ? flags.exclude : undefined),
+      log: (line) => err(`${new Date().toISOString().slice(11, 19)} ${line}\n`),
+    };
+    if (command === 'copy') {
+      const reports = await copyDatabase({
+        ...common,
+        truncate: flags.truncate === true,
+        upsert: flags.upsert === true,
+        batch: typeof flags.batch === 'string' ? Number(flags.batch) : undefined,
+        workers: typeof flags.workers === 'string' ? Number(flags.workers) : undefined,
+        dryRun: flags['dry-run'] === true,
+      });
+      out(`\n${'table'.padEnd(32)} ${'mode'.padEnd(9)} ${'rows'.padStart(12)} ${'secs'.padStart(6)}\n`);
+      for (const r of reports) out(`${r.table.padEnd(32)} ${r.mode.padEnd(9)} ${String(r.rows).padStart(12)} ${String(r.seconds).padStart(6)}${r.skipped ? `  ${r.skipped}` : ''}\n`);
+      if (flags['dry-run'] === true || flags.verify !== true) return 0;
+    }
+    const v = await verifyCopy({ ...common, log: (line) => out(`${line}\n`) });
+    return v.ok ? 0 : 1;
+  }
+
+  err(`unknown command "${command}"\n\n${USAGE}`);
+  return 1;
+}