fhir-persistence 0.1.0 → 0.4.0

package/CHANGELOG.md

@@ -5,17 +5,118 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2025-03-15
+
+### Fixed
+
+#### PostgreSQL DDL Compatibility (Phase D)
+
+- **`migration-runner.ts`** — Tracking table `_migrations` now uses `TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP` on PostgreSQL instead of SQLite-only `datetime('now')`
+- **`package-registry-repo.ts`** — `_packages` and `_schema_version` tables use dialect-aware timestamp defaults; `INSERT OR REPLACE` replaced with `INSERT ... ON CONFLICT ... DO UPDATE` on PostgreSQL
+- **`reindex-scheduler.ts`** — `_reindex_jobs` table uses `SERIAL PRIMARY KEY` on PostgreSQL instead of `AUTOINCREMENT`; `datetime('now')` replaced with `CURRENT_TIMESTAMP`
+- **`valueset-repo.ts`** — `terminology_valuesets` table uses dialect-aware timestamp defaults; `INSERT OR REPLACE` replaced with `INSERT ... ON CONFLICT ... DO UPDATE` on PostgreSQL
+- **`lookup-table-writer.ts`** — 4 global lookup tables (`HumanName`, `Address`, `ContactPoint`, `Identifier`) use `SERIAL PRIMARY KEY` on PostgreSQL instead of `AUTOINCREMENT`
+
+### Changed
+
+- **`ig-persistence-manager.ts`** — Now passes `dialect` to `PackageRegistryRepo`, `MigrationRunnerV2`, and `ReindexScheduler` constructors
+- **`indexing-pipeline.ts`** — `IndexingPipelineOptions` accepts `dialect` parameter, passed to `LookupTableWriter`
+- **`fhir-system.ts`** — Passes `dialect` through to `IndexingPipeline` via `FhirPersistence` options
+- All new `dialect` parameters default to `'sqlite'` — fully backward-compatible
+
+### Test Coverage
+
+- **1014 total tests** (1006 passing, 8 skipped) across 56 test files — no regressions
+
+## [0.3.0] - 2025-03-15
+
+### Added
+
+#### Dual-Backend Validation
+
+- Comprehensive dual-backend test suite (`dual-backend-validation.test.ts`) — 41 tests covering SQLite and PostgreSQL
+  - **Schema DDL correctness** — generate and execute DDL on both backends, verify tables/columns/indexes
+  - **IG lifecycle validation** — `compareSchemas` → `generateMigration` → apply migration on both backends
+  - **CRUD correctness** — `FhirStore` create/read/update/delete/history/vread on both SQLite and PostgreSQL
+  - Transaction atomicity verification on PostgreSQL
+  - Optimistic locking (`ifMatch`) verification on both backends
+  - History auto-increment `versionSeq` verification on PostgreSQL
+
+#### PostgreSQL Integration Tests
+
+- 23 PostgreSQL integration tests (`postgres-adapter.integration.test.ts`) covering CRUD, transactions, DDL generation, migrations, search SQL generation, concurrency, streaming, and NULL handling
+
+### Changed
+
+- `buildTableSet` helper now accepts parameterized `resourceType` for test isolation (unique constraint/index names per test run)
+- Schema DDL comparison tests verify CREATE TABLE count parity rather than exact statement count (SQLite generates extra AUTOINCREMENT index)
+
+### Fixed
+
+- PostgreSQL test isolation — unique resource type names per run prevent constraint/index name collisions across test runs
+
+### Test Coverage
+
+- **1014 total tests** (1006 passing, 8 skipped) across 56 test files
+- Full CRUD verified on both SQLite (in-memory) and PostgreSQL (localhost:5433)
+
+## [0.2.0] - 2025-03-15
+
+### Added
+
+#### PostgreSQL Support
+
+- `PostgresAdapter` — full `StorageAdapter` implementation for PostgreSQL via `pg` Pool
+  - Automatic `?` → `$1, $2, ...` placeholder rewriting
+  - Transaction support via pool client + BEGIN/COMMIT/ROLLBACK
+  - `queryStream` via cursor-like row iteration
+  - Serialization failure retry (40001) with exponential backoff
+  - `close()` guard preventing use-after-close
+- `PostgresDialect` — `SqlDialect` implementation for PostgreSQL
+  - Native `TEXT[]` array operators (`&&`, `@>`) instead of `json_each()`
+  - `GENERATED ALWAYS AS IDENTITY` for history sequence columns
+  - PostgreSQL-native type mappings (TIMESTAMPTZ, TEXT[], BOOLEAN, etc.)
+- 23 PostgreSQL integration tests covering CRUD, transactions, DDL, migrations, search SQL, concurrency, streaming, and NULL handling
+
+#### SQL Dialect Abstraction
+
+- `SqlDialect` interface — abstracts SQL syntax differences between SQLite and PostgreSQL
+- `SQLiteDialect` — `SqlDialect` implementation for SQLite (json_each, AUTOINCREMENT)
+- Dialect-aware WHERE builders: `arrayContainsV2`, `arrayNotContainsV2`, `arrayContainsLikeV2`
+- `buildWhereFragmentV2`, `buildWhereClauseV2` now accept optional `dialect` parameter
+- `buildSearchSQLv2`, `buildCountSQLv2`, `buildTwoPhaseSearchSQLv2` accept optional `dialect`
+- `executeSearchV2` accepts `dialect` via options
+- Compartment filters are dialect-aware (json_each for SQLite, ARRAY operators for PostgreSQL)
+
+### Changed
+
+- `TransactionContext` methods (`execute`, `query`, `queryOne`) are now `async` (Promise-based)
+- All transaction callers updated to `await` transaction operations
+- `BetterSqlite3Adapter` transaction wraps sync operations in async interface
+
+### Removed
+
+- `sql.js` WASM adapter (`SQLiteAdapter`) — replaced by `BetterSqlite3Adapter`
+- `sql.js` dependency removed from `package.json`
+
+### Dependencies
+
+- `pg` ^8.0.0 added as optional peer dependency
+- `pg` ^8.20.0, `@types/pg` ^8.18.0 added as dev dependencies
+
 ## [0.1.0] - 2025-03-13
 
 ### Added
 
 #### Storage Layer
+
 - `StorageAdapter` interface — unified async database abstraction
 - `SQLiteAdapter` — sql.js (WASM) implementation for cross-platform use
 - `BetterSqlite3Adapter` — native better-sqlite3 implementation for production Node.js
 - `SQLiteDialect` / PostgreSQL dialect support for DDL generation
 
 #### CRUD & Persistence
+
 - `FhirStore` — basic CRUD with soft delete and version tracking
 - `FhirPersistence` — end-to-end facade with automatic search indexing
 - `ConditionalService` — conditionalCreate / conditionalUpdate / conditionalDelete
@@ -24,6 +125,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - History tracking via dedicated `_History` tables
 
 #### Search
+
 - `SearchParameterRegistry` — index FHIR SearchParameter bundles
 - `parseSearchRequest` — parse FHIR search URL query parameters
 - `buildSearchSQLv2` — generate SQL with `?` placeholders (SQLite + PG compatible)
@@ -34,6 +136,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `SearchBundleBuilder` — FHIR Bundle response construction with pagination
 
 #### Indexing
+
 - `IndexingPipeline` — automatic search column population on CRUD
 - `RuntimeProvider` bridge — FHIRPath-driven extraction via `fhir-runtime`
 - `buildSearchColumns` — fallback property-path extraction
@@ -42,6 +145,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Column strategy, token-column strategy, lookup-table strategy
 
 #### Schema & Migration
+
 - `StructureDefinitionRegistry` — register and resolve FHIR StructureDefinitions
 - `buildResourceTableSet` — StructureDefinition + SearchParameter → table schema
 - DDL generators for Main, History, References tables + indexes
@@ -53,25 +157,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `ReindexScheduler` — schedule reindex jobs for SP expression changes
 
 #### Terminology
+
 - `TerminologyCodeRepo` — code system concept storage
 - `ValueSetRepo` — value set expansion storage
 
 #### Platform IG
+
 - Built-in platform resource types: User, Bot, Project, Agent, ClientApplication
 - `initializePlatformIG` — auto-register platform search parameters
 
 #### Provider Bridges (for fhir-engine)
+
 - `FhirDefinitionBridge` — wraps `fhir-definition` `DefinitionRegistry` → `DefinitionProvider`
 - `FhirRuntimeProvider` — wraps `fhir-runtime` `FhirRuntimeInstance` → `RuntimeProvider`
 - `FhirSystem` — end-to-end startup orchestrator
 
 #### Production
+
 - `ResourceCacheV2` — in-memory resource cache with TTL
 - `SearchLogger` — search query logging and diagnostics
 - `reindexResourceTypeV2` / `reindexAllV2` — CLI reindex utilities
 
 ### Dependencies
+
 - `fhir-definition` ^0.5.0
 - `fhir-runtime` ^0.8.1
 - `better-sqlite3` ^12.6.2
-- `sql.js` ^1.14.1

package/README.md

@@ -5,10 +5,13 @@ Embedded FHIR R4 persistence layer — CRUD, search, indexing, and schema migrat
 [![npm version](https://img.shields.io/npm/v/fhir-persistence)](https://www.npmjs.com/package/fhir-persistence)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
+> **v0.4.0** — Full PostgreSQL DDL compatibility for all system and lookup tables
+
 ## Features
 
 - **StorageAdapter abstraction** — unified async interface for SQLite and PostgreSQL
-- **Two SQLite adapters** — `BetterSqlite3Adapter` (native, production) + `SQLiteAdapter` (sql.js WASM, cross-platform)
+- **Dual-backend support** — `BetterSqlite3Adapter` (native SQLite) + `PostgresAdapter` (pg)
+- **SqlDialect abstraction** — dialect-aware SQL generation for array operators, DDL, upsert
 - **3-table-per-resource pattern** — Main + History + References tables per FHIR resource type
 - **Automatic search indexing** — column, token-column, and lookup-table strategies
 - **FHIRPath-driven extraction** — optional `RuntimeProvider` bridge for `fhir-runtime` powered indexing
@@ -34,21 +37,24 @@ npm install fhir-persistence
 
 ```bash
 npm install fhir-definition fhir-runtime
+
+# For PostgreSQL support (optional):
+npm install pg
 ```
 
 ## Quick Start
 
-### Standalone (low-level)
+### SQLite (standalone)
 
 ```typescript
 import {
-  SQLiteAdapter,
+  BetterSqlite3Adapter,
   FhirPersistence,
   SearchParameterRegistry,
 } from "fhir-persistence";
 
 // 1. Create storage adapter
-const adapter = new SQLiteAdapter(":memory:");
+const adapter = new BetterSqlite3Adapter({ path: "./fhir.db" });
 
 // 2. Create search parameter registry
 const spRegistry = new SearchParameterRegistry();
@@ -78,6 +84,29 @@ const result = await persistence.searchResources({
 });
 ```
 
+### PostgreSQL
+
+```typescript
+import { PostgresAdapter, FhirStore } from "fhir-persistence";
+import { Pool } from "pg";
+
+const pool = new Pool({
+  host: "localhost",
+  port: 5432,
+  database: "fhir_db",
+  user: "postgres",
+  password: "secret",
+});
+const adapter = new PostgresAdapter(pool);
+const store = new FhirStore(adapter);
+
+// Same CRUD API as SQLite — no code changes needed
+const patient = await store.createResource("Patient", {
+  resourceType: "Patient",
+  name: [{ family: "Smith" }],
+});
+```
+
 ### With FhirSystem (recommended for fhir-engine)
 
 ```typescript
@@ -96,7 +125,7 @@ const adapter = new BetterSqlite3Adapter({ path: './fhir.db' });
 const definitionBridge = new FhirDefinitionBridge(registry);
 
 // 4. Bootstrap via FhirSystem
-const system = new FhirSystem(adapter, { dialect: 'sqlite' });
+const system = new FhirSystem(adapter, { dialect: 'sqlite' });  // or 'postgres'
 const { persistence, sdRegistry, spRegistry, igResult } =
   await system.initialize(definitionBridge);
 
@@ -107,9 +136,13 @@ const patient = await persistence.createResource('Patient', { resourceType: 'Pat
 ## Architecture
 
 ```
-StorageAdapter (SQLite / better-sqlite3 / PostgreSQL)
+StorageAdapter (BetterSqlite3Adapter / PostgresAdapter)
+  │
+  ├── SqlDialect (SQLiteDialect / PostgresDialect)
+  │     └── Dialect-aware: DDL, array operators, upsert, timestamps
+  │
   └── FhirPersistence (end-to-end facade)
-        ├── FhirStore (basic CRUD + soft delete + versioning)
+        ├── FhirStore (basic CRUD + soft delete + optimistic locking + versioning)
         ├── IndexingPipeline
         │     ├── RuntimeProvider (FHIRPath extraction, optional)
         │     ├── buildSearchColumns (fallback row indexer)
@@ -119,7 +152,7 @@ StorageAdapter (SQLite / better-sqlite3 / PostgreSQL)
         ├── BundleProcessorV2 (transaction / batch)
         ├── SearchParameterRegistry
         └── Search Engine
-              ├── WhereBuilder v2 (chain search, ? placeholders)
+              ├── WhereBuilder v2 (chain search, dialect-aware)
               ├── SearchPlanner (filter reorder, two-phase recommendation)
               ├── SearchSQLBuilder v2 (single-phase + two-phase)
               ├── SearchExecutor (_include / _revinclude / _include:iterate)
@@ -131,8 +164,7 @@ StorageAdapter (SQLite / better-sqlite3 / PostgreSQL)
 | Adapter                | Backend                 | Use Case                          |
 | ---------------------- | ----------------------- | --------------------------------- |
 | `BetterSqlite3Adapter` | better-sqlite3 (native) | Production Node.js, CLI, Electron |
-| `SQLiteAdapter`        | sql.js (WASM)           | Browser, cross-platform, testing  |
-| `PostgresAdapter`      | pg                      | Production server                 |
+| `PostgresAdapter`      | pg (connection pool)    | Production server                 |
 
 ## Search
 
@@ -155,7 +187,7 @@ const request = parseSearchRequest(
   registry,
 );
 
-// Single-phase SQL
+// Single-phase SQL (dialect-aware)
 const sql = buildSearchSQLv2(request, registry);
 
 // Two-phase SQL for large tables
@@ -184,7 +216,8 @@ The IG persistence manager automatically handles schema evolution:
 ```typescript
 import { IGPersistenceManager } from "fhir-persistence";
 
-const igManager = new IGPersistenceManager(adapter, "sqlite");
+// Dialect: 'sqlite' or 'postgres'
+const igManager = new IGPersistenceManager(adapter, "postgres");
 const result = await igManager.initialize({
   name: "hl7.fhir.r4.core",
   version: "4.0.1",
@@ -202,7 +235,7 @@ const result = await igManager.initialize({
 import { createFhirEngine } from "fhir-engine";
 
 const engine = await createFhirEngine({
-  database: { type: "sqlite", url: "./fhir.db" },
+  database: { type: "sqlite", url: "./fhir.db" }, // or { type: 'postgres', ... }
   packages: { path: "./fhir-packages" },
 });
 
@@ -218,7 +251,7 @@ const engine = await createFhirEngine({
 | `fhir-definition` | StructureDefinition, SearchParameter, ValueSet, CodeSystem definitions |
 | `fhir-runtime`    | FHIRPath evaluation, validation, search value extraction               |
 | `better-sqlite3`  | Native SQLite bindings (production)                                    |
-| `sql.js`          | WebAssembly SQLite (cross-platform)                                    |
+| `pg`              | PostgreSQL client (optional peer dependency)                           |
 
 ## License