dzql 0.1.6 → 0.2.0

package/README.md

@@ -1,11 +1,12 @@
 # DZQL
 
-PostgreSQL-powered framework with automatic CRUD operations and real-time WebSocket synchronization.
+PostgreSQL-powered framework with automatic CRUD operations, live query subscriptions, and real-time WebSocket synchronization.
 
 ## Documentation
 
 - **[Getting Started Guide](docs/GETTING_STARTED.md)** - Complete tutorial with working todo app
 - **[API Reference](docs/REFERENCE.md)** - Complete API documentation
+- **[Live Query Subscriptions](../../docs/SUBSCRIPTIONS_QUICK_START.md)** - Real-time denormalized documents (NEW in v0.2.0)
 - **[Compiler Documentation](docs/compiler/)** - Entity compilation guide and coding standards
 - **[Claude Guide](docs/CLAUDE.md)** - Development guide for AI assistants
 - **[Venues Example](../venues/)** - Full working application
@@ -29,6 +30,12 @@ await ws.connect();
 // All 5 operations work automatically
 const user = await ws.api.save.users({ name: 'Alice' });
 const results = await ws.api.search.users({ filters: { name: 'alice' } });
+
+// NEW in v0.2.0: Live query subscriptions
+const { data, unsubscribe } = await ws.api.subscribe_venue_detail(
+  { venue_id: 123 },
+  (updated) => console.log('Venue changed!', updated)
+);
 ```
 
 ## DZQL Compiler

package/docs/REFERENCE.md

@@ -12,6 +12,7 @@ Complete API documentation for DZQL framework. For tutorials, see [GETTING_START
 - [Custom Functions](#custom-functions)
 - [Authentication](#authentication)
 - [Real-time Events](#real-time-events)
+- [Live Query Subscriptions](#live-query-subscriptions)
 - [Temporal Relationships](#temporal-relationships)
 - [Error Messages](#error-messages)
 
@@ -864,6 +865,144 @@ ws.onBroadcast((method, params) => {
 
 ---
 
+## Live Query Subscriptions
+
+Subscribe to denormalized documents and receive automatic updates when underlying data changes. Subscriptions use a PostgreSQL-first architecture where all change detection happens in the database.
+
+For complete documentation, see **[Live Query Subscriptions Guide](../../../docs/LIVE_QUERY_SUBSCRIPTIONS.md)** and **[Quick Start](../../../docs/SUBSCRIPTIONS_QUICK_START.md)**.
+
+### Quick Example
+
+```javascript
+// Subscribe to venue with all related data
+const { data, unsubscribe } = await ws.api.subscribe_venue_detail(
+  { venue_id: 123 },
+  (updatedVenue) => {
+    // Called automatically when venue, org, or sites change
+    console.log('Updated:', updatedVenue);
+    // updatedVenue = { id: 123, name: '...', org: {...}, sites: [...] }
+  }
+);
+
+// Initial data available immediately
+console.log('Initial:', data);
+
+// Later: cleanup
+await unsubscribe();
+```
+
+### Creating a Subscribable
+
+Define subscribables in SQL:
+
+```sql
+SELECT dzql.register_subscribable(
+  'venue_detail',                                               -- Name
+  '{"subscribe": ["@org_id->acts_for[org_id=$]{active}.user_id"]}'::jsonb,  -- Permissions
+  '{"venue_id": "int"}'::jsonb,                                -- Parameters
+  'venues',                                                     -- Root table
+  '{
+    "org": "organisations",
+    "sites": {"entity": "sites", "filter": "venue_id=$venue_id"}
+  }'::jsonb                                                     -- Relations
+);
+```
+
+### Compile and Deploy
+
+```bash
+# Compile subscribable to PostgreSQL functions
+node packages/dzql/compile-subscribable.js venue.sql | psql $DATABASE_URL
+```
+
+This generates three functions:
+- `venue_detail_can_subscribe(user_id, params)` - Permission check
+- `get_venue_detail(params, user_id)` - Query builder
+- `venue_detail_affected_documents(table, op, old, new)` - Change detector
+
+### Subscription Lifecycle
+
+1. **Subscribe**: Client calls `ws.api.subscribe_<name>(params, callback)`
+2. **Permission Check**: `<name>_can_subscribe()` validates access
+3. **Initial Query**: `get_<name>()` returns denormalized document
+4. **Register**: Server stores subscription in-memory
+5. **Database Change**: Any relevant table modification
+6. **Detect**: `<name>_affected_documents()` identifies affected subscriptions
+7. **Re-query**: `get_<name>()` fetches fresh data
+8. **Update**: Callback invoked with new data
+
+### Unsubscribe
+
+```javascript
+// Method 1: Use returned unsubscribe function
+const { unsubscribe } = await ws.api.subscribe_venue_detail(...);
+await unsubscribe();
+
+// Method 2: Direct unsubscribe call
+await ws.api.unsubscribe_venue_detail({ venue_id: 123 });
+```
+
+### Architecture Benefits
+
+- **PostgreSQL-First**: All logic executes in database, not application code
+- **Zero Configuration**: Pattern matching on method names - no server changes needed
+- **Type Safe**: Compiled functions validated at deploy time
+- **Efficient**: In-memory registry, PostgreSQL does matching
+- **Secure**: Permission paths enforced at database level
+- **Scalable**: Stateless server, can add instances freely
+
+### Common Patterns
+
+**Single Table:**
+```sql
+SELECT dzql.register_subscribable(
+  'user_settings',
+  '{"subscribe": ["@user_id"]}'::jsonb,
+  '{"user_id": "int"}'::jsonb,
+  'user_settings',
+  '{}'::jsonb
+);
+```
+
+**With Relations:**
+```sql
+SELECT dzql.register_subscribable(
+  'booking_detail',
+  '{"subscribe": ["@user_id"]}'::jsonb,
+  '{"booking_id": "int"}'::jsonb,
+  'bookings',
+  '{
+    "venue": "venues",
+    "customer": "users",
+    "items": {"entity": "booking_items", "filter": "booking_id=$booking_id"}
+  }'::jsonb
+);
+```
+
+**Multiple Permission Paths (OR logic):**
+```sql
+SELECT dzql.register_subscribable(
+  'venue_admin',
+  '{
+    "subscribe": [
+      "@owner_id",
+      "@org_id->acts_for[org_id=$]{active}.user_id"
+    ]
+  }'::jsonb,
+  '{"venue_id": "int"}'::jsonb,
+  'venues',
+  '{"sites": {"entity": "sites", "filter": "venue_id=$venue_id"}}'::jsonb
+);
+```
+
+### See Also
+
+- **[Live Query Subscriptions Guide](../../../docs/LIVE_QUERY_SUBSCRIPTIONS.md)** - Complete reference
+- **[Quick Start Guide](../../../docs/SUBSCRIPTIONS_QUICK_START.md)** - 5-minute tutorial
+- **[Permission Paths](#permission--notification-paths)** - Path DSL syntax
+
+---
+
 ## Temporal Relationships
 
 Handle time-based relationships with `valid_from`/`valid_to` fields.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dzql",
-  "version": "0.1.6",
+  "version": "0.2.0",
   "description": "PostgreSQL-powered framework with zero boilerplate CRUD operations and real-time WebSocket synchronization",
   "type": "module",
   "main": "src/server/index.js",
@@ -22,7 +22,7 @@
   ],
   "scripts": {
     "test": "bun test",
-    "prepublishOnly": "echo '✅ Publishing DZQL v0.1.5...'"
+    "prepublishOnly": "echo '✅ Publishing DZQL v0.2.0...'"
   },
   "dependencies": {
     "jose": "^6.1.0",

package/src/client/ws.js

@@ -59,12 +59,11 @@ class WebSocketManager {
     this.pendingRequests = new Map();
     this.broadcastCallbacks = new Set();
     this.sidRequestHandlers = new Set();
+    this.subscriptions = new Map(); // subscription_id -> { callback, unsubscribe }
     this.reconnectAttempts = 0;
     this.maxReconnectAttempts = options.maxReconnectAttempts ?? 5;
     this.isShuttingDown = false;
 
-    // Ad
-
     // DZQL nested proxy API - matches server-side db.api pattern
     // Proxy handles both DZQL operations and custom functions
     const dzqlOps = {
@@ -137,6 +136,18 @@ class WebSocketManager {
         if (prop in target) {
           return target[prop];
         }
+        // Handle subscribe_* methods specially
+        if (prop.startsWith('subscribe_')) {
+          return (params = {}, callback) => {
+            return this.subscribe(prop, params, callback);
+          };
+        }
+        // Handle unsubscribe_* methods
+        if (prop.startsWith('unsubscribe_')) {
+          return (params = {}) => {
+            return this.unsubscribe(prop, params);
+          };
+        }
         // All other properties are treated as custom function calls
         return (params = {}) => {
           return this.call(prop, params);
@@ -314,6 +325,16 @@ class WebSocketManager {
         resolve(message.result);
       }
     } else {
+      // Handle subscription updates
+      if (message.method === "subscription:update") {
+        const { subscription_id, data } = message.params;
+        const sub = this.subscriptions.get(subscription_id);
+        if (sub && sub.callback) {
+          sub.callback(data);
+        }
+        return;
+      }
+
       // Handle broadcasts and SID requests
 
       // Check if this is a SID request from server
@@ -376,6 +397,70 @@ class WebSocketManager {
     });
   }
 
+  /**
+   * Subscribe to a live query
+   *
+   * @param {string} method - Method name (subscribe_<subscribable>)
+   * @param {object} params - Subscription parameters
+   * @param {function} callback - Callback function for updates
+   * @returns {Promise<{data, subscription_id, unsubscribe}>} Initial data and unsubscribe function
+   *
+   * @example
+   * const { data, unsubscribe } = await ws.api.subscribe_venue_detail(
+   *   { venue_id: 1 },
+   *   (updated) => console.log('Updated:', updated)
+   * );
+   *
+   * // Use initial data
+   * console.log('Initial:', data);
+   *
+   * // Later: unsubscribe
+   * unsubscribe();
+   */
+  async subscribe(method, params = {}, callback) {
+    if (!callback || typeof callback !== 'function') {
+      throw new Error('Subscribe requires a callback function');
+    }
+
+    // Call server to register subscription
+    const result = await this.call(method, params);
+    const { subscription_id, data } = result;
+
+    // Create unsubscribe function
+    const unsubscribeFn = async () => {
+      const unsubMethod = method.replace('subscribe_', 'unsubscribe_');
+      await this.call(unsubMethod, params);
+      this.subscriptions.delete(subscription_id);
+    };
+
+    // Store callback for updates
+    this.subscriptions.set(subscription_id, {
+      callback,
+      unsubscribe: unsubscribeFn
+    });
+
+    // Return initial data and unsubscribe function
+    return {
+      data,
+      subscription_id,
+      unsubscribe: unsubscribeFn
+    };
+  }
+
+  /**
+   * Unsubscribe from a live query
+   *
+   * @param {string} method - Method name (unsubscribe_<subscribable>)
+   * @param {object} params - Subscription parameters
+   * @returns {Promise<{success: boolean}>}
+   *
+   * @example
+   * await ws.api.unsubscribe_venue_detail({ venue_id: 1 });
+   */
+  async unsubscribe(method, params = {}) {
+    return await this.call(method, params);
+  }
+
   /**
    * Register callback for real-time broadcast events
    *

package/src/compiler/codegen/subscribable-codegen.js

@@ -0,0 +1,396 @@
+/**
+ * Subscribable Code Generator
+ * Generates PostgreSQL functions for live query subscriptions
+ *
+ * For each subscribable, generates:
+ * 1. get_<name>(params, user_id) - Query function that builds the document
+ * 2. <name>_affected_documents(table, op, old, new) - Determines which subscription instances are affected
+ * 3. <name>_can_subscribe(user_id, params) - Access control check
+ */
+
+import { PathParser } from '../parser/path-parser.js';
+
+export class SubscribableCodegen {
+  constructor(subscribable) {
+    this.name = subscribable.name;
+    this.permissionPaths = subscribable.permissionPaths || {};
+    this.paramSchema = subscribable.paramSchema || {};
+    this.rootEntity = subscribable.rootEntity;
+    this.relations = subscribable.relations || {};
+    this.parser = new PathParser();
+  }
+
+  /**
+   * Generate all functions for this subscribable
+   * @returns {string} SQL for all subscribable functions
+   */
+  generate() {
+    const sections = [];
+
+    // Header comment
+    sections.push(this._generateHeader());
+
+    // 1. Access control function
+    sections.push(this._generateAccessControlFunction());
+
+    // 2. Query function (builds the document)
+    sections.push(this._generateQueryFunction());
+
+    // 3. Affected documents function (determines which subscriptions to update)
+    sections.push(this._generateAffectedDocumentsFunction());
+
+    return sections.join('\n\n');
+  }
+
+  /**
+   * Generate header comment
+   * @private
+   */
+  _generateHeader() {
+    return `-- ============================================================================
+-- Subscribable: ${this.name}
+-- Root Entity: ${this.rootEntity}
+-- Generated: ${new Date().toISOString()}
+-- ============================================================================`;
+  }
+
+  /**
+   * Generate access control function
+   * @private
+   */
+  _generateAccessControlFunction() {
+    let subscribePaths = this.permissionPaths.subscribe || [];
+
+    // Ensure it's an array
+    if (!Array.isArray(subscribePaths)) {
+      subscribePaths = [subscribePaths];
+    }
+
+    // If no paths, it's public
+    if (subscribePaths.length === 0) {
+      return `CREATE OR REPLACE FUNCTION ${this.name}_can_subscribe(
+  p_user_id INT,
+  p_params JSONB
+) RETURNS BOOLEAN AS $$
+BEGIN
+  RETURN TRUE; -- Public access
+END;
+$$ LANGUAGE plpgsql STABLE SECURITY DEFINER;`;
+    }
+
+    // Generate permission check logic
+    const checks = subscribePaths.map(path => {
+      const ast = this.parser.parse(path);
+      return this._generatePathCheck(ast, 'p_params', 'p_user_id');
+    });
+
+    const checkSQL = checks.join(' OR\n    ');
+
+    return `CREATE OR REPLACE FUNCTION ${this.name}_can_subscribe(
+  p_user_id INT,
+  p_params JSONB
+) RETURNS BOOLEAN AS $$
+BEGIN
+  RETURN (
+    ${checkSQL}
+  );
+END;
+$$ LANGUAGE plpgsql STABLE SECURITY DEFINER;`;
+  }
+
+  /**
+   * Generate path check SQL from AST
+   * @private
+   */
+  _generatePathCheck(ast, recordVar, userIdVar) {
+    // Handle direct field reference: @owner_id
+    if (ast.type === 'field_ref') {
+      return `(${recordVar}->>'${ast.field}')::int = ${userIdVar}`;
+    }
+
+    // Handle traversal with steps: @org_id->acts_for[org_id=$]{active}.user_id
+    if (ast.type === 'traversal' && ast.steps) {
+      const fieldRef = ast.steps[0];  // First step is the field reference
+      const tableRef = ast.steps[1];   // Second step is the table reference
+
+      if (!fieldRef || !tableRef || tableRef.type !== 'table_ref') {
+        return 'FALSE';
+      }
+
+      const startField = fieldRef.field;
+      const targetTable = tableRef.table;
+      const targetField = tableRef.targetField;
+
+      const startValue = `(${recordVar}->>'${startField}')::int`;
+
+      // Build WHERE clause
+      const whereClauses = [];
+
+      // Add filter conditions from the table_ref
+      if (tableRef.filter && tableRef.filter.length > 0) {
+        for (const filterCondition of tableRef.filter) {
+          const field = filterCondition.field;
+          if (filterCondition.value.type === 'param') {
+            // Parameter reference: org_id=$
+            whereClauses.push(`${targetTable}.${field} = ${startValue}`);
+          } else {
+            // Literal value
+            whereClauses.push(`${targetTable}.${field} = '${filterCondition.value}'`);
+          }
+        }
+      }
+
+      // Add temporal marker if present
+      if (tableRef.temporal) {
+        whereClauses.push(`${targetTable}.valid_to IS NULL`);
+      }
+
+      return `EXISTS (
+      SELECT 1 FROM ${targetTable}
+      WHERE ${whereClauses.join('\n        AND ')}
+        AND ${targetTable}.${targetField} = ${userIdVar}
+    )`;
+    }
+
+    return 'FALSE';
+  }
+
+  /**
+   * Generate filter SQL
+   * @private
+   */
+  _generateFilterSQL(filter, tableAlias) {
+    const conditions = [];
+    for (const [key, value] of Object.entries(filter)) {
+      if (value === '$') {
+        // Placeholder - will be replaced with actual value
+        conditions.push(`${tableAlias}.${key} = ${tableAlias}.${key}`);
+      } else {
+        conditions.push(`${tableAlias}.${key} = '${value}'`);
+      }
+    }
+    return conditions.join(' AND ');
+  }
+
+  /**
+   * Generate query function that builds the document
+   * @private
+   */
+  _generateQueryFunction() {
+    const params = Object.keys(this.paramSchema);
+    const paramDeclarations = params.map(p => `  v_${p} ${this.paramSchema[p]};`).join('\n');
+    const paramExtractions = params.map(p =>
+      `  v_${p} := (p_params->>'${p}')::${this.paramSchema[p]};`
+    ).join('\n');
+
+    // Build root WHERE clause based on params
+    const rootFilter = this._generateRootFilter();
+
+    // Build relation subqueries
+    const relationSelects = this._generateRelationSelects();
+
+    return `CREATE OR REPLACE FUNCTION get_${this.name}(
+  p_params JSONB,
+  p_user_id INT
+) RETURNS JSONB AS $$
+DECLARE
+${paramDeclarations}
+  v_result JSONB;
+BEGIN
+  -- Extract parameters
+${paramExtractions}
+
+  -- Check access control
+  IF NOT ${this.name}_can_subscribe(p_user_id, p_params) THEN
+    RAISE EXCEPTION 'Permission denied';
+  END IF;
+
+  -- Build document with root and all relations
+  SELECT jsonb_build_object(
+    '${this.rootEntity}', row_to_json(root.*)${relationSelects}
+  )
+  INTO v_result
+  FROM ${this.rootEntity} root
+  WHERE ${rootFilter};
+
+  RETURN v_result;
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;`;
+  }
+
+  /**
+   * Generate root filter based on params
+   * @private
+   */
+  _generateRootFilter() {
+    const params = Object.keys(this.paramSchema);
+
+    // Assume first param is the root entity ID
+    // TODO: Make this more flexible based on param naming conventions
+    if (params.length > 0) {
+      const firstParam = params[0];
+      // Convention: venue_id -> id, org_id -> id, etc.
+      return `root.id = v_${firstParam}`;
+    }
+
+    return 'TRUE';
+  }
+
+  /**
+   * Generate relation subqueries
+   * @private
+   */
+  _generateRelationSelects() {
+    if (Object.keys(this.relations).length === 0) {
+      return '';
+    }
+
+    const selects = Object.entries(this.relations).map(([relName, relConfig]) => {
+      const relEntity = typeof relConfig === 'string' ? relConfig : relConfig.entity;
+      const relFilter = typeof relConfig === 'object' ? relConfig.filter : null;
+      const relIncludes = typeof relConfig === 'object' ? relConfig.include : null;
+
+      // Build filter condition
+      let filterSQL = this._generateRelationFilter(relFilter, relEntity);
+
+      // Build nested includes if any
+      let nestedSelect = 'row_to_json(rel.*)';
+      if (relIncludes) {
+        const nestedFields = Object.entries(relIncludes).map(([nestedName, nestedEntity]) => {
+          return `'${nestedName}', (
+          SELECT jsonb_agg(row_to_json(nested.*))
+          FROM ${nestedEntity} nested
+          WHERE nested.${relEntity}_id = rel.id
+        )`;
+        }).join(',\n        ');
+
+        nestedSelect = `jsonb_build_object(
+        '${relEntity}', row_to_json(rel.*),
+        ${nestedFields}
+      )`;
+      }
+
+      return `,
+    '${relName}', (
+      SELECT jsonb_agg(${nestedSelect})
+      FROM ${relEntity} rel
+      WHERE ${filterSQL}
+    )`;
+    }).join('');
+
+    return selects;
+  }
+
+  /**
+   * Generate filter for relation subquery
+   * @private
+   */
+  _generateRelationFilter(filter, relEntity) {
+    if (!filter) {
+      // Default: foreign key to root
+      return `rel.${this.rootEntity}_id = root.id`;
+    }
+
+    // Parse filter expression like "venue_id=$venue_id"
+    // Replace $param with v_param variable
+    return filter.replace(/\$(\w+)/g, 'v_$1');
+  }
+
+  /**
+   * Generate affected documents function
+   * @private
+   */
+  _generateAffectedDocumentsFunction() {
+    const cases = [];
+
+    // Case 1: Root entity changed
+    cases.push(this._generateRootAffectedCase());
+
+    // Case 2: Related entities changed
+    for (const [relName, relConfig] of Object.entries(this.relations)) {
+      cases.push(this._generateRelationAffectedCase(relName, relConfig));
+    }
+
+    const casesSQL = cases.join('\n\n    ');
+
+    return `CREATE OR REPLACE FUNCTION ${this.name}_affected_documents(
+  p_table_name TEXT,
+  p_op TEXT,
+  p_old JSONB,
+  p_new JSONB
+) RETURNS JSONB[] AS $$
+DECLARE
+  v_affected JSONB[];
+BEGIN
+  CASE p_table_name
+    ${casesSQL}
+
+    ELSE
+      v_affected := ARRAY[]::JSONB[];
+  END CASE;
+
+  RETURN v_affected;
+END;
+$$ LANGUAGE plpgsql IMMUTABLE;`;
+  }
+
+  /**
+   * Generate case for root entity changes
+   * @private
+   */
+  _generateRootAffectedCase() {
+    const params = Object.keys(this.paramSchema);
+    const firstParam = params[0] || 'id';
+
+    return `-- Root entity (${this.rootEntity}) changed
+    WHEN '${this.rootEntity}' THEN
+      v_affected := ARRAY[
+        jsonb_build_object('${firstParam}', COALESCE((p_new->>'id')::int, (p_old->>'id')::int))
+      ];`;
+  }
+
+  /**
+   * Generate case for related entity changes
+   * @private
+   */
+  _generateRelationAffectedCase(relName, relConfig) {
+    const relEntity = typeof relConfig === 'string' ? relConfig : relConfig.entity;
+    const relFK = typeof relConfig === 'object' && relConfig.foreignKey
+      ? relConfig.foreignKey
+      : `${this.rootEntity}_id`;
+
+    const params = Object.keys(this.paramSchema);
+    const firstParam = params[0] || 'id';
+
+    // Check if this is a nested relation (has parent FK)
+    const nestedIncludes = typeof relConfig === 'object' ? relConfig.include : null;
+
+    if (nestedIncludes) {
+      // Nested relation: need to traverse up to root
+      return `-- Nested relation (${relEntity}) changed
+    WHEN '${relEntity}' THEN
+      -- Find parent and then root
+      SELECT ARRAY_AGG(jsonb_build_object('${firstParam}', parent.${this.rootEntity}_id))
+      INTO v_affected
+      FROM ${relEntity} rel
+      JOIN ${Object.keys(nestedIncludes)[0]} parent ON parent.id = rel.${Object.keys(nestedIncludes)[0]}_id
+      WHERE rel.id = COALESCE((p_new->>'id')::int, (p_old->>'id')::int);`;
+    }
+
+    return `-- Related entity (${relEntity}) changed
+    WHEN '${relEntity}' THEN
+      v_affected := ARRAY[
+        jsonb_build_object('${firstParam}', COALESCE((p_new->>'${relFK}')::int, (p_old->>'${relFK}')::int))
+      ];`;
+  }
+}
+
+/**
+ * Generate subscribable functions from config
+ * @param {Object} subscribable - Subscribable configuration
+ * @returns {string} Generated SQL
+ */
+export function generateSubscribable(subscribable) {
+  const codegen = new SubscribableCodegen(subscribable);
+  return codegen.generate();
+}