dzql 0.5.33 → 0.6.1

package/docs/README.md

@@ -1,54 +1,327 @@
-# DZQL Documentation
+# DZQL: The Compile-Only Realtime Database Framework
 
-Complete documentation for the DZQL PostgreSQL-powered framework.
+DZQL ("Database Zero Query Language") is a PostgreSQL-native framework for building realtime, reactive applications without the runtime overhead or complexity of traditional ORMs or BaaS solutions.
 
-## 📚 Getting Started
+## Quick Start
 
-New to DZQL? Start here:
+The fastest way to get started is with `bun create`:
 
-- **[Tutorial](getting-started/tutorial.md)** - Complete step-by-step guide with a working todo app
-- **[Interpreter vs Compiler](guides/interpreter-vs-compiler.md)** - Understand the two execution modes
-- **[Subscriptions Quick Start](getting-started/subscriptions-quick-start.md)** - Get real-time subscriptions working in 5 minutes
+```bash
+bun create dzql my-app
+cd my-app
+bun install
+bun run db:rebuild
+bun run dev
+```
 
-## 📖 Guides
+This creates a full-stack app with Vue/Vite frontend, DZQL server, and PostgreSQL database.
 
-Feature-specific guides and how-tos:
+## The Problem
 
-- **[Live Query Subscriptions](guides/subscriptions.md)** - Real-time denormalized documents
-- **[Many-to-Many Relationships](guides/many-to-many.md)** - Junction table management
-- **[Composite Primary Keys](guides/composite-primary-keys.md)** - Tables with compound keys
-- **[Field Defaults](guides/field-defaults.md)** - Auto-populate fields on create
-- **[Custom Functions](guides/custom-functions.md)** - Extend with PostgreSQL or Bun functions
-- **[Client Stores](guides/client-stores.md)** - Pinia store patterns for Vue.js
+Building realtime apps is hard. You typically have to:
+1.  **Sync State:** Manually keep your frontend Pinia/Redux store in sync with your backend database.
+2.  **Manage Permissions:** Re-implement row-level security in your API layer (and hope it matches your DB).
+3.  **Handle Atomicity:** Ensure that complex operations (e.g., "Create Order + Reserve Inventory + Notify User") happen in a single transaction.
+4.  **Optimistic Updates:** Write complex client-side logic to "guess" the server's response, often leading to data divergence.
 
-## 📘 Reference
+## The DZQL Solution
 
-Complete API documentation:
+DZQL takes a radically different approach: **Compilation**.
 
-- **[API Reference](reference/api.md)** - The 5 operations, entities, permissions, graph rules
-- **[Client API](reference/client.md)** - WebSocket client and connection management
-- **[Compiler](compiler/)** - Entity compilation, code generation, and coding standards
+Instead of a heavy runtime framework, you define your **Domain Schema** (Entities, Relationships, Permissions) in a simple TypeScript configuration. DZQL compiles this definition into:
 
-### Compiler Documentation
+1.  **Optimized SQL:** Specialized PostgreSQL functions (`save_order`, `get_product`) with *inlined* permission checks and *atomic* graph operations.
+2.  **Type-Safe Client SDK:** A generated TypeScript client that knows your exact API surface.
+3.  **Smart Pinia Stores:** Generated Vue stores that automatically handle realtime synchronization using atomic "Patch Events" from the database.
 
-- [Quickstart](compiler/QUICKSTART.md) - Get started with the DZQL compiler
-- [Advanced Filters](compiler/ADVANCED_FILTERS.md) - Complex search operators
-- [Coding Standards](compiler/CODING_STANDARDS.md) - Best practices for DZQL code
-- [Comparison](compiler/COMPARISON.md) - Runtime vs compiled side-by-side
+### Key Features
 
-## 🤖 For AI Assistants
+*   **Zero Runtime Interpretation:** No slow ORM query builders. Everything is compiled to native PL/pgSQL.
+*   **Security by Construction:** The runtime is a "dumb" gateway that routes requests by OID allowlist. It *cannot* execute arbitrary SQL.
+*   **Atomic Everything:** Complex graph operations (cascading creates/deletes) happen in a single database transaction.
+*   **Realtime by Default:** Every database write emits an atomic event batch. The client SDK automatically patches your local state. No "refetching" required.
+*   **JavaScript/TypeScript Native:** Define your schema in code you understand, get full type safety end-to-end.
 
-- **[Claude Guide](for-ai/claude-guide.md)** - Complete guide for AI-assisted DZQL development
+## Manual Setup
 
-## 🔗 Quick Links
+If you prefer to set up manually instead of using `bun create dzql`:
 
-- [npm Package](https://www.npmjs.com/package/dzql)
-- [GitHub Repository](https://github.com/blueshed/dzql)
-- [Issue Tracker](https://github.com/blueshed/dzql/issues)
+### 1. Define your Domain (`domain.ts`)
 
-## Need Help?
+```typescript
+export const entities = {
+  posts: {
+    schema: { id: 'serial PRIMARY KEY', title: 'text', author_id: 'int' },
+    permissions: { create: ['@author_id == @user_id'] } // Inlined SQL security
+  }
+};
 
-- 📖 Check the guides above
-- 🐛 [Report an issue](https://github.com/blueshed/dzql/issues)
-- 💬 [Start a discussion](https://github.com/blueshed/dzql/discussions)
-- 🤖 Ask your AI assistant (they have access to this documentation!)
+export const subscribables = {
+  post_feed: {
+    root: { entity: 'posts' },
+    scopeTables: ['posts']
+  }
+};
+```
+
+### 2. Compile
+
+```bash
+bunx dzql domain.ts -o generated
+```
+
+### 3. Use in Client
+
+```typescript
+import { usePostFeedStore } from '@generated/client/stores';
+
+const feed = usePostFeedStore();
+// Automatically fetches data AND subscribes to realtime updates
+// bind() is async - awaits until first data arrives
+const { data, loading } = await feed.bind({ user_id: 1 }); 
+// data.value is now populated
+```
+
+## Architecture
+
+*   **Compiler:** CLI tool that analyzes your domain and generates artifacts.
+*   **Runtime:** A lightweight Bun/Node server that handles Auth (JWT) and WebSocket connection pooling.
+*   **Client:** A robust WebSocket SDK that manages reconnection and dispatches atomic patches to stores.
+*   **Namespace:** Direct database access for CLI tools like `invokej`.
+
+## Package Exports
+
+```typescript
+import { ... } from 'dzql';           // Runtime server
+import { ... } from 'dzql/client';    // WebSocket client SDK  
+import { ... } from 'dzql/compiler';  // CLI compiler
+import { DzqlNamespace } from 'dzql/namespace';  // CLI/invokej integration
+```
+
+## Client Connection & Authentication
+
+When a client connects to the WebSocket server, it immediately receives a `connection:ready` message containing the authenticated user profile (or `null` if not authenticated).
+
+### Connection Flow
+
+1. Client connects with optional `?token=...` in URL
+2. Server validates token (if present) and fetches user profile
+3. Server sends: `{"method": "connection:ready", "params": {"user": {...} | null}}`
+4. Client knows auth state immediately
+
+### Client API
+
+```typescript
+import { WebSocketManager } from 'dzql/client';
+
+const ws = new WebSocketManager();
+await ws.connect('ws://localhost:3000/ws');
+
+// Check connection state
+console.log(ws.ready);  // true after connection:ready received
+console.log(ws.user);   // user profile object or null
+
+// Register callback for ready state
+ws.onReady((user) => {
+  if (user) {
+    console.log('Authenticated as:', user.email);
+  } else {
+    console.log('Anonymous connection');
+  }
+});
+
+// Authentication methods
+await ws.login({ email: '...', password: '...' });  // Stores token in localStorage
+await ws.logout();  // Clears token and user state
+```
+
+### Vue/Pinia Usage Pattern
+
+```vue
+<template>
+  <div v-if="!ws.ready">Loading...</div>
+  <LoginModal v-else-if="!ws.user" />
+  <RouterView v-else />
+</template>
+```
+
+## Generated Pinia Subscribable Stores
+
+TZQL generates Pinia stores for each subscribable that handle:
+- Initial data fetch via WebSocket subscription
+- Automatic realtime patching when related data changes
+- Deduplication of subscriptions by parameter key
+
+### Store Structure
+
+Each generated store exports:
+
+```typescript
+const store = useVenueDetailStore();
+
+// Main API
+store.bind(params)    // Async - subscribes and returns { data, loading, ready }
+store.unbind(params)  // Unsubscribes and removes document from store
+store.documents       // Ref containing all bound documents keyed by JSON.stringify(params)
+```
+
+### Basic Usage
+
+```typescript
+import { useVenueDetailStore } from '@/generated/client/stores';
+
+const store = useVenueDetailStore();
+
+// bind() is async - returns when first data arrives
+const { data, loading, ready } = await store.bind({ venue_id: 1 });
+
+// data is reactive and contains the document
+console.log(data); // { id: 1, name: 'My Venue', sites: [...], org: {...} }
+
+// loading is false after first data
+console.log(loading); // false
+
+// Subsequent calls with same params return cached subscription
+const same = await store.bind({ venue_id: 1 }); // Returns immediately, no new subscription
+```
+
+### Vue Component Patterns
+
+**Pattern 1: Top-level await (recommended with `<Suspense>`)**
+
+```vue
+<script setup>
+import { useVenueDetailStore } from '@/generated/client/stores';
+
+const props = defineProps(['venueId']);
+const store = useVenueDetailStore();
+
+// Await at top level - component suspends until data arrives
+const { data } = await store.bind({ venue_id: props.venueId });
+</script>
+
+<template>
+  <!-- data is guaranteed to be populated -->
+  <h1>{{ data.name }}</h1>
+  <p>Organization: {{ data.org.name }}</p>
+  <ul>
+    <li v-for="site in data.sites" :key="site.id">
+      {{ site.name }} ({{ site.allocations.length }} allocations)
+    </li>
+  </ul>
+</template>
+```
+
+**Pattern 2: Reactive binding with loading state**
+
+```vue
+<script setup>
+import { useVenueDetailStore } from '@/generated/client/stores';
+import { ref, onMounted, watch } from 'vue';
+
+const props = defineProps(['venueId']);
+const store = useVenueDetailStore();
+const docState = ref({ data: null, loading: true });
+
+onMounted(async () => {
+  docState.value = await store.bind({ venue_id: props.venueId });
+});
+
+// Re-bind when venueId changes
+watch(() => props.venueId, async (newId) => {
+  docState.value = await store.bind({ venue_id: newId });
+});
+</script>
+
+<template>
+  <div v-if="docState.loading">Loading...</div>
+  <div v-else>
+    <h1>{{ docState.data.name }}</h1>
+  </div>
+</template>
+```
+
+**Pattern 3: Multiple subscriptions**
+
+```vue
+<script setup>
+import { useVenueDetailStore } from '@/generated/client/stores';
+
+const store = useVenueDetailStore();
+
+// Bind multiple venues - each gets its own cached subscription
+const venues = await Promise.all([
+  store.bind({ venue_id: 1 }),
+  store.bind({ venue_id: 2 }),
+  store.bind({ venue_id: 3 }),
+]);
+</script>
+```
+
+### Accessing All Documents
+
+The store's `documents` ref contains all bound subscriptions:
+
+```typescript
+const store = useVenueDetailStore();
+
+await store.bind({ venue_id: 1 });
+await store.bind({ venue_id: 2 });
+
+// Access all documents
+console.log(store.documents);
+// {
+//   '{"venue_id":1}': { data: {...}, loading: false, ready: Promise },
+//   '{"venue_id":2}': { data: {...}, loading: false, ready: Promise }
+// }
+```
+
+### How Realtime Works
+
+The WebSocket is a pure transport layer - it does not manage or cache data. Stores own their data.
+
+When data changes in the database:
+1. PostgreSQL triggers emit atomic events to the `dzql_v2.events` table
+2. The runtime calls `*_affected_keys()` to find which subscriptions are affected
+3. Events are broadcast to subscribed clients via WebSocket
+4. The **store's** `subscription_event` function receives the event
+5. The store's `applyPatch` function updates its local document in-place
+6. Vue reactivity updates the UI automatically
+
+**No refetching required** - changes are applied incrementally via patching.
+
+### Patch Events
+
+The store receives events with this structure:
+
+```typescript
+{
+  table: 'sites',           // Which table changed
+  op: 'insert' | 'update' | 'delete',
+  pk: { id: 123 },          // Primary key of affected row
+  data: { ... }             // Full row data
+}
+```
+
+The generated `applyPatch` function routes events to the correct location in the document graph based on the subscribable's `includes` structure.
+
+## Why "Compile-Only"?
+
+By moving complexity to build-time, we get:
+*   **Performance:** The database does the heavy lifting.
+*   **Correctness:** If it compiles, your permissions and relationships are valid.
+*   **Simplicity:** The runtime is tiny and easy to audit.
+
+## Entity Options
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `schema` | object | Column definitions with PostgreSQL types |
+| `primaryKey` | string[] | Composite primary key fields (default: `['id']`) |
+| `label` | string | Field used for display/autocomplete |
+| `softDelete` | boolean | Use `deleted_at` instead of hard delete |
+| `managed` | boolean | Set to `false` to skip CRUD generation (for junction tables) |
+| `permissions` | object | Row-level security rules |
+| `graphRules` | object | Side effects on create/update/delete |
+| `manyToMany` | object | M2M relationship definitions |
+| `fieldDefaults` | object | Default values for fields on INSERT |

package/docs/feature-requests/applyPatch-bug-report.md

@@ -0,0 +1,85 @@
+# tzql Bug Report
+
+## Generated Store applyPatch Doesn't Match Data Structure
+
+**Severity:** High (Breaking)
+
+**Status:** ✅ FIXED
+
+**Description:**
+The generated Pinia store's `applyPatch` function assumes a flat data structure, but the subscribable SQL was returning nested wrapper objects.
+
+**SQL was returning:**
+```json
+{
+  "venues": {...},
+  "sites": [
+    { "sites": { "id": 1, "name": "..." }, "allocations": [...] },
+    { "sites": { "id": 2, "name": "..." }, "allocations": [...] }
+  ]
+}
+```
+
+**Fix:**
+Updated `subscribable_sql.ts` to use JSONB concatenation (`||`) to merge entity fields with nested includes into a flat structure:
+
+```sql
+SELECT jsonb_agg(
+  row_to_json(rel.*) || jsonb_build_object(
+    'allocations', COALESCE((
+      SELECT jsonb_agg(row_to_json(nested.*))
+      FROM allocations nested
+      WHERE nested.site_id = rel.id
+    ), '[]'::jsonb))
+)
+```
+
+**SQL now returns:**
+```json
+{
+  "venues": {...},
+  "sites": [
+    { "id": 1, "name": "...", "allocations": [...] },
+    { "id": 2, "name": "...", "allocations": [...] }
+  ]
+}
+```
+
+This flat structure matches what `applyPatch` and `handleArrayPatch` expect, so realtime updates now work correctly for nested includes.
+
+---
+
+## Bug 2: json/jsonb Type Mismatch in Subscribable SQL
+
+**Severity:** Critical (Blocking)
+
+**Status:** ✅ FIXED
+
+**Description:**
+The generated subscribable SQL used `row_to_json()` which returns `json` type, but concatenated it with `jsonb_build_object()` which returns `jsonb`. PostgreSQL cannot concatenate these types.
+
+**Generated SQL (before):**
+```sql
+row_to_json(rel.*) || jsonb_build_object(...)
+```
+
+**Error:**
+```
+operator does not exist: json || jsonb
+```
+
+**Fix:**
+Changed to use `to_jsonb()` instead of `row_to_json()` when concatenating with nested includes:
+
+```sql
+to_jsonb(rel.*) || jsonb_build_object(...)
+```
+
+---
+
+## Environment
+
+- tzql version: local development (linked)
+- Database: PostgreSQL 17 (Docker)
+- Client: Vue 3 + Pinia + TypeScript
+- Runtime: Bun

package/docs/feature-requests/connection-ready-profile.md

@@ -0,0 +1,57 @@
+# Feature Request: Send User Profile on WebSocket Connect
+
+**Status: IMPLEMENTED**
+
+## Summary
+
+When a client connects to the tzql WebSocket server, the server should immediately send a connection:ready message containing the authenticated user profile (or null if not authenticated).
+
+## Current Behavior
+
+1. Client connects with optional ?token=... in URL
+2. Server opens connection but sends nothing
+3. Client must call auth RPC to authenticate, then separately fetch profile
+4. Client has no immediate knowledge of auth state
+
+## Proposed Behavior
+
+1. Client connects with optional ?token=... in URL
+2. Server validates token (if present) and fetches user profile
+3. Server sends first message:
+
+{"method": "connection:ready", "params": {"user": {"id": 1, "name": "...", "email": "..."} | null}}
+
+4. Client knows auth state immediately, can render accordingly
+
+## Why This Matters
+
+- Single source of truth: WebSocket connection determines auth state, not localStorage
+- No race conditions: UI waits for connection:ready before rendering
+- Simpler client code: No need for separate auth check after connect
+- Better UX: App shows loading state until connection ready, then immediately correct view
+
+## Client Usage Pattern
+
+Template:
+  div v-if="!ready" - loading spinner
+  LoginModal v-else-if="!user"
+  RouterView v-else
+
+## Suggested Server Changes
+
+In src/runtime/ws.ts, modify handleOpen to:
+1. Parse token from URL query params
+2. If valid, verify token and fetch user profile
+3. Send connection:ready message with user (or null)
+
+## Suggested Client Changes
+
+In src/client/ws.ts:
+1. Add user and ready properties to class
+2. Handle connection:ready message in handleMessage
+3. Add onReady callback method
+
+## Migration
+
+- Existing clients that dont handle connection:ready will ignore it (no breaking change)
+- New clients can opt-in to the pattern

package/docs/feature-requests/hidden-bug-report.md

@@ -0,0 +1,111 @@
+# tzql Bug Report
+
+Bugs discovered while building the Venues application with tzql.
+
+---
+
+## Bug 1: Hidden Fields Exposed in Subscribables
+
+**Severity:** High (Security)
+
+**Status:** ✅ FIXED
+
+**Description:**
+Fields marked as `hidden: true` in the domain schema are still exposed when using subscribables. The generated SQL uses `row_to_json(root.*)` which includes all columns regardless of the `hidden` property.
+
+**Fix:**
+Added `buildVisibleRowJson` helper in `subscribable_sql.ts` that generates `jsonb_build_object()` with only visible fields instead of `row_to_json(root.*)`. Also updated all other SQL generators (`sql.ts`) to exclude hidden fields from `get_*`, `search_*`, `save_*`, and `delete_*` functions.
+
+---
+
+## Bug 2: Generated Pinia Stores Don't Await Async Subscribe
+
+**Severity:** Medium
+
+**Status:** ✅ FIXED
+
+**Description:**
+The generated Pinia stores call the async `subscribe` method but don't await it. This means `store.data` is undefined immediately after calling `store.subscribe()` even though the subscription has been initiated.
+
+**Fix:**
+Updated `subscribable_store.ts` to make the `bind` function async. It now:
+1. Creates a `ready` Promise that resolves when first data arrives
+2. Awaits the `ready` Promise before returning
+3. Stores the `ready` Promise so repeat calls can also await it
+
+**Generated Code (after fix):**
+```typescript
+async function bind(params) {
+  const key = JSON.stringify(params);
+  if (documents.value[key]) {
+    const existing = documents.value[key];
+    if (existing.loading.value) {
+      await existing.ready;
+    }
+    return existing;
+  }
+
+  const docState = ref(null);
+  const loading = ref(true);
+  let resolveReady;
+  const ready = new Promise((resolve) => { resolveReady = resolve; });
+
+  ws.api.subscribe_venue_detail(params, (initialData) => {
+    docState.value = initialData;
+    loading.value = false;
+    resolveReady();
+  });
+
+  documents.value[key] = { data: docState, loading, ready };
+  await ready;
+  return documents.value[key];
+}
+```
+
+**Usage:**
+```typescript
+const store = useMyProfileStore();
+const { data, loading } = await store.bind({ user_id: 1 });
+// data.value is now populated
+```
+
+---
+
+## Bug 3: Subscribable Permission Check Uses Param Name Instead of Column Name
+
+**Severity:** High (Breaking)
+
+**Status:** ✅ FIXED
+
+**Description:**
+When generating SQL for subscribable permission checks, the compiler uses the parameter name instead of the actual database column name. This causes SQL errors when the param name differs from the column name.
+
+**Fix:**
+Updated `compileSubscribePermission` in `subscribable_sql.ts` to map param names to the root entity's `id` column when they match the rootKey. For example, `@org_id` now correctly resolves to `v_root.id` instead of `v_root.org_id`.
+
+---
+
+## Summary
+
+| Bug | Severity | Status | Notes |
+|-----|----------|--------|-------|
+| Hidden fields exposed | High | ✅ Fixed | Uses `jsonb_build_object` to exclude hidden fields |
+| Stores don't await | Medium | ✅ Fixed | `bind()` is now async and awaits first data |
+| Permission param/column mismatch | High | ✅ Fixed | Maps param name to `id` column |
+
+---
+
+## Environment
+
+- tzql version: (linked local development version)
+- Database: PostgreSQL 17 (via Docker)
+- Client: Vue 3 + Pinia + TypeScript
+- Runtime: Bun
+
+---
+
+## Related Files
+
+Detailed bug documents created in tzql docs:
+- `/packages/tzql/docs/feature-requests/hidden-fields-in-subscribables.md`
+- `/packages/tzql/docs/feature-requests/subscribable-param-key-bug.md`

package/docs/feature-requests/hidden-fields-subscribables.md

@@ -0,0 +1,34 @@
+# Bug: Hidden fields exposed in subscribables
+
+**Status: IMPLEMENTED**
+
+## Issue
+
+Entity hidden fields are not excluded from subscribable queries.
+
+## Example
+
+domain.js:
+users: {
+  schema: { ... },
+  hidden: ["password_hash"],  // Should be excluded from all queries
+}
+
+But get_my_profile returns:
+{
+  "users": {
+    "password_hash": "$2a$06$..."  // EXPOSED!
+  }
+}
+
+## Root Cause
+
+The subscribable SQL generator uses row_to_json(root.*) which returns all columns.
+
+## Fix
+
+When generating subscribable SQL, exclude hidden fields by using explicit column list.
+
+## Impact
+
+Security vulnerability - sensitive data like password hashes exposed to clients.

package/docs/feature-requests/subscribable-param-key-bug.md

@@ -0,0 +1,38 @@
+# Bug: Subscribable permission check uses param name instead of column name
+
+## Issue
+
+When generating subscribable SQL, the permission check incorrectly uses the param name instead of the actual column name on the root entity.
+
+## Example
+
+Domain:
+```javascript
+org_dashboard: {
+  params: { org_id: "int" },
+  root: { entity: "organisations", key: "org_id" },
+  canSubscribe: ["@org_id->acts_for[org_id=$]{active}.user_id"]
+}
+```
+
+Generated SQL:
+```sql
+-- WRONG: v_root.org_id doesn't exist on organisations table
+WHERE acts_for.org_id = v_root.org_id
+```
+
+Should be:
+```sql
+-- CORRECT: organisations.id is the actual column
+WHERE acts_for.org_id = v_root.id
+```
+
+## Error
+
+```
+ERROR: record "v_root" has no field "org_id"
+```
+
+## Fix
+
+The compiler should resolve `key: "org_id"` to mean "param org_id maps to the root entity's primary key (id)", not "access v_root.org_id".