dzql 0.2.1 → 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -4
- package/docs/README.md +71 -0
- package/docs/compiler/README.md +84 -0
- package/docs/{CLAUDE.md → for-ai/claude-guide.md} +2 -2
- package/docs/{SUBSCRIPTIONS_QUICK_START.md → getting-started/subscriptions-quick-start.md} +3 -3
- package/docs/{GETTING_STARTED.md → getting-started/tutorial.md} +1 -1
- package/docs/{LIVE_QUERY_SUBSCRIPTIONS.md → guides/subscriptions.md} +4 -4
- package/docs/{REFERENCE.md → reference/api.md} +8 -8
- package/docs/{CLIENT-QUICK-START.md → reference/client.md} +2 -2
- package/package.json +2 -2
- package/src/client/ws.js +3 -1
- package/src/database/migrations/008a_meta.sql +7 -0
- package/src/server/db.js +7 -0
- package/docs/LIVE_QUERY_SUBSCRIPTIONS_STRATEGY.md +0 -488
- /package/docs/{CLIENT-STORES.md → guides/client-stores.md} +0 -0
package/README.md
CHANGED
|
@@ -4,11 +4,12 @@ PostgreSQL-powered framework with automatic CRUD operations, live query subscrip
|
|
|
4
4
|
|
|
5
5
|
## Documentation
|
|
6
6
|
|
|
7
|
-
- **[
|
|
8
|
-
- **[
|
|
9
|
-
- **[
|
|
7
|
+
- **[Documentation Hub](docs/)** - Complete documentation index
|
|
8
|
+
- **[Getting Started Tutorial](docs/getting-started/tutorial.md)** - Complete tutorial with working todo app
|
|
9
|
+
- **[API Reference](docs/reference/api.md)** - Complete API documentation
|
|
10
|
+
- **[Live Query Subscriptions](docs/getting-started/subscriptions-quick-start.md)** - Real-time denormalized documents (NEW in v0.2.0)
|
|
10
11
|
- **[Compiler Documentation](docs/compiler/)** - Entity compilation guide and coding standards
|
|
11
|
-
- **[Claude Guide](docs/
|
|
12
|
+
- **[Claude Guide](docs/for-ai/claude-guide.md)** - Development guide for AI assistants
|
|
12
13
|
- **[Venues Example](../venues/)** - Full working application
|
|
13
14
|
|
|
14
15
|
## Quick Install
|
package/docs/README.md
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# DZQL Documentation
|
|
2
|
+
|
|
3
|
+
Complete documentation for the DZQL PostgreSQL-powered framework.
|
|
4
|
+
|
|
5
|
+
## 📚 Getting Started
|
|
6
|
+
|
|
7
|
+
New to DZQL? Start here:
|
|
8
|
+
|
|
9
|
+
- **[Tutorial](getting-started/tutorial.md)** - Complete step-by-step guide with a working todo app
|
|
10
|
+
- **[Subscriptions Quick Start](getting-started/subscriptions-quick-start.md)** - Get real-time subscriptions working in 5 minutes
|
|
11
|
+
|
|
12
|
+
## 📖 Guides
|
|
13
|
+
|
|
14
|
+
Feature-specific guides and how-tos:
|
|
15
|
+
|
|
16
|
+
- **[Live Query Subscriptions](guides/subscriptions.md)** - Real-time denormalized documents
|
|
17
|
+
- **[Client Stores](guides/client-stores.md)** - Pinia store patterns for Vue.js
|
|
18
|
+
|
|
19
|
+
## 📘 Reference
|
|
20
|
+
|
|
21
|
+
Complete API documentation:
|
|
22
|
+
|
|
23
|
+
- **[API Reference](reference/api.md)** - The 5 operations, entities, permissions, graph rules
|
|
24
|
+
- **[Client API](reference/client.md)** - WebSocket client and connection management
|
|
25
|
+
- **[Compiler](compiler/)** - Entity compilation, code generation, and coding standards
|
|
26
|
+
|
|
27
|
+
### Compiler Documentation
|
|
28
|
+
|
|
29
|
+
- [Quickstart](compiler/QUICKSTART.md) - Get started with the DZQL compiler
|
|
30
|
+
- [Advanced Filters](compiler/ADVANCED_FILTERS.md) - Complex search operators
|
|
31
|
+
- [Coding Standards](compiler/CODING_STANDARDS.md) - Best practices for DZQL code
|
|
32
|
+
- [Comparison](compiler/COMPARISON.md) - DZQL vs other approaches
|
|
33
|
+
|
|
34
|
+
## 🤖 For AI Assistants
|
|
35
|
+
|
|
36
|
+
- **[Claude Guide](for-ai/claude-guide.md)** - Complete guide for AI-assisted DZQL development
|
|
37
|
+
|
|
38
|
+
## 🔗 Quick Links
|
|
39
|
+
|
|
40
|
+
- [npm Package](https://www.npmjs.com/package/dzql)
|
|
41
|
+
- [GitHub Repository](https://github.com/blueshed/dzql)
|
|
42
|
+
- [Issue Tracker](https://github.com/blueshed/dzql/issues)
|
|
43
|
+
- [Changelog](../../../CHANGELOG.md)
|
|
44
|
+
- [Contributing](../../../CONTRIBUTING.md)
|
|
45
|
+
|
|
46
|
+
## 🏗️ Architecture
|
|
47
|
+
|
|
48
|
+
Looking for architecture and design docs? See the [repository docs](../../../docs/):
|
|
49
|
+
|
|
50
|
+
- [Permissions System](../../../docs/architecture/PERMISSIONS.md)
|
|
51
|
+
- [Project Roadmap](../../../docs/architecture/ROADMAP.md)
|
|
52
|
+
- [Subscription Architecture](../../../docs/architecture/SUBSCRIPTIONS_STRATEGY.md)
|
|
53
|
+
|
|
54
|
+
## 🧪 Development
|
|
55
|
+
|
|
56
|
+
Contributing to DZQL? See development documentation:
|
|
57
|
+
|
|
58
|
+
- [TDD Workflow](../../../docs/development/TDD_WORKFLOW.md)
|
|
59
|
+
- [WebSocket Testing](../../../docs/development/WEBSOCKET_TESTING.md)
|
|
60
|
+
- [Claude Web Setup](../../../docs/development/CLAUDE-WEB.md)
|
|
61
|
+
|
|
62
|
+
## 📦 Package Contents
|
|
63
|
+
|
|
64
|
+
This documentation is published with the npm package. For repository-wide documentation (contributors, development workflow, architecture), see [`/docs/`](../../../docs/) in the repository root.
|
|
65
|
+
|
|
66
|
+
## Need Help?
|
|
67
|
+
|
|
68
|
+
- 📖 Check the guides above
|
|
69
|
+
- 🐛 [Report an issue](https://github.com/blueshed/dzql/issues)
|
|
70
|
+
- 💬 [Start a discussion](https://github.com/blueshed/dzql/discussions)
|
|
71
|
+
- 🤖 Ask your AI assistant (they have access to this documentation!)
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# DZQL Compiler Documentation
|
|
2
|
+
|
|
3
|
+
The DZQL Compiler transforms declarative entity definitions into optimized PostgreSQL stored procedures.
|
|
4
|
+
|
|
5
|
+
## Quick Start
|
|
6
|
+
|
|
7
|
+
- **[Quickstart Guide](QUICKSTART.md)** - Get started with the compiler in 5 minutes
|
|
8
|
+
|
|
9
|
+
## Guides
|
|
10
|
+
|
|
11
|
+
- **[Advanced Filters](ADVANCED_FILTERS.md)** - Complex search operators and patterns
|
|
12
|
+
- **[Coding Standards](CODING_STANDARDS.md)** - Best practices for DZQL code
|
|
13
|
+
|
|
14
|
+
## Reference
|
|
15
|
+
|
|
16
|
+
- **[Comparison](COMPARISON.md)** - How DZQL compares to other approaches
|
|
17
|
+
- **[Session Summary](SESSION_SUMMARY.md)** - Development session documentation
|
|
18
|
+
- **[Summary](SUMMARY.md)** - Compiler overview and architecture
|
|
19
|
+
- **[Overnight Build](OVERNIGHT_BUILD.md)** - Batch compilation process
|
|
20
|
+
|
|
21
|
+
## Using the Compiler
|
|
22
|
+
|
|
23
|
+
### Via CLI
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
dzql compile database/domain.sql -o compiled/
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### Programmatically
|
|
30
|
+
|
|
31
|
+
```javascript
|
|
32
|
+
import { DZQLCompiler } from 'dzql/compiler';
|
|
33
|
+
|
|
34
|
+
const compiler = new DZQLCompiler();
|
|
35
|
+
const result = compiler.compileFromSQL(sqlContent);
|
|
36
|
+
|
|
37
|
+
console.log(result.sql); // Generated PostgreSQL
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Registering Entities
|
|
41
|
+
|
|
42
|
+
```sql
|
|
43
|
+
SELECT dzql.register_entity(
|
|
44
|
+
'todos', -- Table name
|
|
45
|
+
'title', -- Label field
|
|
46
|
+
array['title', 'description'], -- Searchable fields
|
|
47
|
+
'{}'::jsonb, -- FK includes
|
|
48
|
+
false, -- Soft delete
|
|
49
|
+
'{}'::jsonb, -- Graph rules
|
|
50
|
+
jsonb_build_object( -- Notification paths
|
|
51
|
+
'owner', array['@user_id']
|
|
52
|
+
),
|
|
53
|
+
jsonb_build_object( -- Permission paths
|
|
54
|
+
'view', array['@user_id'],
|
|
55
|
+
'create', array['@user_id'],
|
|
56
|
+
'update', array['@user_id'],
|
|
57
|
+
'delete', array['@user_id']
|
|
58
|
+
),
|
|
59
|
+
'{}'::jsonb -- Temporal config
|
|
60
|
+
);
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
This generates 5 PostgreSQL functions:
|
|
64
|
+
- `get_todos(params, user_id)` - Retrieve single record
|
|
65
|
+
- `save_todos(params, user_id)` - Create or update
|
|
66
|
+
- `delete_todos(params, user_id)` - Delete record
|
|
67
|
+
- `lookup_todos(params, user_id)` - Autocomplete
|
|
68
|
+
- `search_todos(params, user_id)` - Search with filters
|
|
69
|
+
|
|
70
|
+
## Architecture
|
|
71
|
+
|
|
72
|
+
The compiler uses a three-phase approach:
|
|
73
|
+
|
|
74
|
+
1. **Parse** - Extract entity definitions from SQL
|
|
75
|
+
2. **Generate** - Create optimized PostgreSQL functions
|
|
76
|
+
3. **Deploy** - Execute generated SQL
|
|
77
|
+
|
|
78
|
+
All business logic runs in PostgreSQL, not application code.
|
|
79
|
+
|
|
80
|
+
## See Also
|
|
81
|
+
|
|
82
|
+
- [Main Documentation](../) - Full DZQL documentation
|
|
83
|
+
- [API Reference](../reference/api.md) - The 5 operations
|
|
84
|
+
- [For AI](../for-ai/claude-guide.md) - AI-assisted development
|
|
@@ -1163,7 +1163,7 @@ array['name', 'address', 'city', 'description', 'notes', 'tags', 'metadata']
|
|
|
1163
1163
|
|
|
1164
1164
|
## Additional Resources
|
|
1165
1165
|
|
|
1166
|
-
- **API Reference**: See [
|
|
1167
|
-
- **Tutorial**: See [
|
|
1166
|
+
- **API Reference**: See [API Reference](../reference/api.md) for complete API documentation
|
|
1167
|
+
- **Tutorial**: See [Getting Started Tutorial](../getting-started/tutorial.md) for hands-on guide
|
|
1168
1168
|
- **Examples**: See `packages/venues/` for complete working application
|
|
1169
1169
|
- **Tests**: See `packages/venues/tests/` for comprehensive test patterns
|
|
@@ -61,9 +61,9 @@ All change detection happens in PostgreSQL - zero configuration needed on the se
|
|
|
61
61
|
|
|
62
62
|
## Next Steps
|
|
63
63
|
|
|
64
|
-
- [Full Documentation](
|
|
65
|
-
- [Permission Paths Guide](
|
|
66
|
-
- [
|
|
64
|
+
- [Full Documentation](../guides/subscriptions.md)
|
|
65
|
+
- [Permission Paths Guide](../../../../docs/architecture/PERMISSIONS.md)
|
|
66
|
+
- [API Reference](../reference/api.md)
|
|
67
67
|
|
|
68
68
|
## Common Patterns
|
|
69
69
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
DZQL is a PostgreSQL framework that gives you **atomic real-time updates** via WebSocket. Every database change broadcasts instantly to all connected clients. Zero boilerplate.
|
|
4
4
|
|
|
5
|
-
> **See also:** [
|
|
5
|
+
> **See also:** [API Reference](../reference/api.md) for complete API documentation | [Claude Guide](../for-ai/claude-guide.md) for AI development guide
|
|
6
6
|
|
|
7
7
|
## The Core Pattern
|
|
8
8
|
|
|
@@ -529,7 +529,7 @@ ws.api.subscribe_venue_detail(
|
|
|
529
529
|
|
|
530
530
|
## See Also
|
|
531
531
|
|
|
532
|
-
- [Vision Document](
|
|
533
|
-
- [
|
|
534
|
-
- [
|
|
535
|
-
- [Compiler
|
|
532
|
+
- [Vision Document](../../../../vision.md) - Architecture overview and patterns
|
|
533
|
+
- [Permission Paths](../../../../docs/architecture/PERMISSIONS.md) - Permission path DSL syntax
|
|
534
|
+
- [Subscription Architecture](../../../../docs/architecture/SUBSCRIPTIONS_STRATEGY.md) - Design decisions
|
|
535
|
+
- [Compiler Documentation](../compiler/) - Code generation and compilation guide
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# DZQL API Reference
|
|
2
2
|
|
|
3
|
-
Complete API documentation for DZQL framework. For tutorials, see [
|
|
3
|
+
Complete API documentation for DZQL framework. For tutorials, see [Getting Started Tutorial](../getting-started/tutorial.md). For AI development guide, see [Claude Guide](../for-ai/claude-guide.md).
|
|
4
4
|
|
|
5
5
|
## Table of Contents
|
|
6
6
|
|
|
@@ -869,7 +869,7 @@ ws.onBroadcast((method, params) => {
|
|
|
869
869
|
|
|
870
870
|
Subscribe to denormalized documents and receive automatic updates when underlying data changes. Subscriptions use a PostgreSQL-first architecture where all change detection happens in the database.
|
|
871
871
|
|
|
872
|
-
For complete documentation, see **[Live Query Subscriptions Guide](
|
|
872
|
+
For complete documentation, see **[Live Query Subscriptions Guide](../guides/subscriptions.md)** and **[Quick Start](../getting-started/subscriptions-quick-start.md)**.
|
|
873
873
|
|
|
874
874
|
### Quick Example
|
|
875
875
|
|
|
@@ -997,8 +997,8 @@ SELECT dzql.register_subscribable(
|
|
|
997
997
|
|
|
998
998
|
### See Also
|
|
999
999
|
|
|
1000
|
-
- **[Live Query Subscriptions Guide](
|
|
1001
|
-
- **[Quick Start Guide](
|
|
1000
|
+
- **[Live Query Subscriptions Guide](../guides/subscriptions.md)** - Complete reference
|
|
1001
|
+
- **[Quick Start Guide](../getting-started/subscriptions-quick-start.md)** - 5-minute tutorial
|
|
1002
1002
|
- **[Permission Paths](#permission--notification-paths)** - Path DSL syntax
|
|
1003
1003
|
|
|
1004
1004
|
---
|
|
@@ -1093,7 +1093,7 @@ const result = await db.api.myCustomFunction({param: 'value'}, userId);
|
|
|
1093
1093
|
|
|
1094
1094
|
## See Also
|
|
1095
1095
|
|
|
1096
|
-
- [
|
|
1097
|
-
- [
|
|
1098
|
-
- [README
|
|
1099
|
-
- [Venues Example](
|
|
1096
|
+
- [Getting Started Tutorial](../getting-started/tutorial.md) - Hands-on tutorial
|
|
1097
|
+
- [Claude Guide](../for-ai/claude-guide.md) - AI development guide
|
|
1098
|
+
- [Project README](../../../../README.md) - Project overview
|
|
1099
|
+
- [Venues Example](../../../venues/) - Complete working application
|
|
@@ -154,7 +154,7 @@ You now have:
|
|
|
154
154
|
|
|
155
155
|
## Next Steps
|
|
156
156
|
|
|
157
|
-
- Read [
|
|
157
|
+
- Read [Client Stores Guide](../guides/client-stores.md) for complete API reference
|
|
158
158
|
- Customize the App.vue template
|
|
159
159
|
- Add your own components
|
|
160
160
|
- Style with Tailwind/DaisyUI
|
|
@@ -179,5 +179,5 @@ Check `packages/client` for a complete working example.
|
|
|
179
179
|
## Help
|
|
180
180
|
|
|
181
181
|
For more help, see:
|
|
182
|
-
- [
|
|
182
|
+
- [Client Stores Guide](../guides/client-stores.md) - Complete documentation
|
|
183
183
|
- [GitHub Issues](https://github.com/blueshed/dzql/issues)
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dzql",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.3",
|
|
4
4
|
"description": "PostgreSQL-powered framework with zero boilerplate CRUD operations and real-time WebSocket synchronization",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/server/index.js",
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
],
|
|
23
23
|
"scripts": {
|
|
24
24
|
"test": "bun test",
|
|
25
|
-
"prepublishOnly": "echo '✅ Publishing DZQL v0.2.
|
|
25
|
+
"prepublishOnly": "echo '✅ Publishing DZQL v0.2.3...'"
|
|
26
26
|
},
|
|
27
27
|
"dependencies": {
|
|
28
28
|
"jose": "^6.1.0",
|
package/src/client/ws.js
CHANGED
|
@@ -52,6 +52,7 @@ class WebSocketManager {
|
|
|
52
52
|
*
|
|
53
53
|
* @param {Object} [options={}] - Configuration options
|
|
54
54
|
* @param {number} [options.maxReconnectAttempts=5] - Maximum reconnection attempts before giving up
|
|
55
|
+
* @param {string} [options.tokenName='dzql_token'] - Name of the localStorage key for JWT token
|
|
55
56
|
*/
|
|
56
57
|
constructor(options = {}) {
|
|
57
58
|
this.ws = null;
|
|
@@ -62,6 +63,7 @@ class WebSocketManager {
|
|
|
62
63
|
this.subscriptions = new Map(); // subscription_id -> { callback, unsubscribe }
|
|
63
64
|
this.reconnectAttempts = 0;
|
|
64
65
|
this.maxReconnectAttempts = options.maxReconnectAttempts ?? 5;
|
|
66
|
+
this.tokenName = options.tokenName ?? 'dzql_token';
|
|
65
67
|
this.isShuttingDown = false;
|
|
66
68
|
|
|
67
69
|
// DZQL nested proxy API - matches server-side db.api pattern
|
|
@@ -265,7 +267,7 @@ class WebSocketManager {
|
|
|
265
267
|
|
|
266
268
|
// Add JWT token as query parameter if available
|
|
267
269
|
if (typeof localStorage !== 'undefined'){
|
|
268
|
-
const storedToken = localStorage.getItem(
|
|
270
|
+
const storedToken = localStorage.getItem(this.tokenName);
|
|
269
271
|
if (storedToken) {
|
|
270
272
|
wsUrl += `?token=${encodeURIComponent(storedToken)}`;
|
|
271
273
|
}
|
|
@@ -30,6 +30,13 @@ begin
|
|
|
30
30
|
'temporal_fields', e.temporal_fields,
|
|
31
31
|
'notification_paths', e.notification_paths,
|
|
32
32
|
'permission_paths', e.permission_paths,
|
|
33
|
+
'primary_key', (
|
|
34
|
+
-- Get primary key columns
|
|
35
|
+
select jsonb_agg(a.attname order by a.attnum)
|
|
36
|
+
from pg_index i
|
|
37
|
+
join pg_attribute a on a.attrelid = i.indrelid and a.attnum = any(i.indkey)
|
|
38
|
+
where i.indrelid = e.table_name::regclass and i.indisprimary
|
|
39
|
+
),
|
|
33
40
|
'schema', (
|
|
34
41
|
-- Get column schema from information_schema
|
|
35
42
|
select jsonb_agg(
|
package/src/server/db.js
CHANGED
|
@@ -10,11 +10,17 @@ const DB_MAX_CONNECTIONS = parseInt(process.env.DB_MAX_CONNECTIONS || "10", 10);
|
|
|
10
10
|
const DB_IDLE_TIMEOUT = parseInt(process.env.DB_IDLE_TIMEOUT || "20", 10);
|
|
11
11
|
const DB_CONNECT_TIMEOUT = parseInt(process.env.DB_CONNECT_TIMEOUT || "10", 10);
|
|
12
12
|
|
|
13
|
+
// SSL configuration for Heroku and other hosted databases
|
|
14
|
+
const sslConfig = process.env.DATABASE_SSL === 'true'
|
|
15
|
+
? { rejectUnauthorized: false } // Heroku uses self-signed certs
|
|
16
|
+
: undefined;
|
|
17
|
+
|
|
13
18
|
// Main PostgreSQL connection for queries
|
|
14
19
|
export const sql = postgres(DATABASE_URL, {
|
|
15
20
|
max: DB_MAX_CONNECTIONS,
|
|
16
21
|
idle_timeout: DB_IDLE_TIMEOUT,
|
|
17
22
|
connect_timeout: DB_CONNECT_TIMEOUT,
|
|
23
|
+
ssl: sslConfig,
|
|
18
24
|
// Suppress NOTICE messages in test environment
|
|
19
25
|
onnotice: process.env.NODE_ENV === 'test' ? () => {} : undefined,
|
|
20
26
|
});
|
|
@@ -24,6 +30,7 @@ export const listen_sql = postgres(DATABASE_URL, {
|
|
|
24
30
|
max: 1,
|
|
25
31
|
idle_timeout: 0,
|
|
26
32
|
connect_timeout: DB_CONNECT_TIMEOUT,
|
|
33
|
+
ssl: sslConfig,
|
|
27
34
|
// Suppress NOTICE messages in test environment
|
|
28
35
|
onnotice: process.env.NODE_ENV === 'test' ? () => {} : undefined,
|
|
29
36
|
});
|
|
@@ -1,488 +0,0 @@
|
|
|
1
|
-
# Live Query Subscriptions Strategy
|
|
2
|
-
|
|
3
|
-
**Date:** 2025-11-16
|
|
4
|
-
**Status:** Phase 1 Complete (Compiler), Phase 2-4 Pending
|
|
5
|
-
|
|
6
|
-
---
|
|
7
|
-
|
|
8
|
-
## Overview
|
|
9
|
-
|
|
10
|
-
Live Query Subscriptions implement **Pattern 1** from `vision.md` - allowing clients to subscribe to denormalized documents and receive automatic updates when any related data changes.
|
|
11
|
-
|
|
12
|
-
### Architecture Principles
|
|
13
|
-
|
|
14
|
-
1. **PostgreSQL-First**: Database determines which subscriptions are affected
|
|
15
|
-
2. **Compiler-Driven**: All logic compiled to PostgreSQL functions (zero runtime interpretation)
|
|
16
|
-
3. **In-Memory Subscriptions**: Server holds active subscriptions in memory for performance
|
|
17
|
-
4. **Naming Convention**: `subscribe_<name>` / `unsubscribe_<name>` for pattern matching
|
|
18
|
-
|
|
19
|
-
---
|
|
20
|
-
|
|
21
|
-
## Core Concept: Subscribables
|
|
22
|
-
|
|
23
|
-
**Subscribables** are separate from entities - they define denormalized documents that:
|
|
24
|
-
- Combine data from multiple entities (root + relations)
|
|
25
|
-
- Have their own access control (permission paths)
|
|
26
|
-
- Define subscription parameters (subscription key)
|
|
27
|
-
- Compile to PostgreSQL functions that determine affected documents
|
|
28
|
-
|
|
29
|
-
### Example Subscribable
|
|
30
|
-
|
|
31
|
-
```sql
|
|
32
|
-
SELECT dzql.register_subscribable(
|
|
33
|
-
'venue_detail', -- Subscribable name
|
|
34
|
-
|
|
35
|
-
-- Permission: who can subscribe?
|
|
36
|
-
jsonb_build_object(
|
|
37
|
-
'subscribe', ARRAY['@org_id->acts_for[org_id=$]{active}.user_id']
|
|
38
|
-
),
|
|
39
|
-
|
|
40
|
-
-- Parameters: subscription key
|
|
41
|
-
jsonb_build_object(
|
|
42
|
-
'venue_id', 'int'
|
|
43
|
-
),
|
|
44
|
-
|
|
45
|
-
-- Root entity
|
|
46
|
-
'venues',
|
|
47
|
-
|
|
48
|
-
-- Relations to include
|
|
49
|
-
jsonb_build_object(
|
|
50
|
-
'org', 'organisations',
|
|
51
|
-
'sites', jsonb_build_object(
|
|
52
|
-
'entity', 'sites',
|
|
53
|
-
'filter', 'venue_id=$venue_id'
|
|
54
|
-
),
|
|
55
|
-
'packages', jsonb_build_object(
|
|
56
|
-
'entity', 'packages',
|
|
57
|
-
'filter', 'venue_id=$venue_id',
|
|
58
|
-
'include', jsonb_build_object(
|
|
59
|
-
'allocations', 'allocations'
|
|
60
|
-
)
|
|
61
|
-
)
|
|
62
|
-
)
|
|
63
|
-
);
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
---
|
|
67
|
-
|
|
68
|
-
## Generated PostgreSQL Functions
|
|
69
|
-
|
|
70
|
-
For each subscribable, the compiler generates 3 functions:
|
|
71
|
-
|
|
72
|
-
###1. Access Control: `<name>_can_subscribe(user_id, params)`
|
|
73
|
-
|
|
74
|
-
```sql
|
|
75
|
-
CREATE OR REPLACE FUNCTION venue_detail_can_subscribe(
|
|
76
|
-
p_user_id INT,
|
|
77
|
-
p_params JSONB
|
|
78
|
-
) RETURNS BOOLEAN AS $$
|
|
79
|
-
BEGIN
|
|
80
|
-
-- Check permission path: @org_id->acts_for[org_id=$]{active}.user_id
|
|
81
|
-
RETURN EXISTS (
|
|
82
|
-
SELECT 1
|
|
83
|
-
FROM venues v
|
|
84
|
-
JOIN acts_for af ON af.org_id = v.org_id
|
|
85
|
-
WHERE v.id = (p_params->>'venue_id')::int
|
|
86
|
-
AND af.user_id = p_user_id
|
|
87
|
-
AND af.valid_to IS NULL
|
|
88
|
-
);
|
|
89
|
-
END;
|
|
90
|
-
$$ LANGUAGE plpgsql STABLE SECURITY DEFINER;
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
### 2. Query Function: `get_<name>(params, user_id)`
|
|
94
|
-
|
|
95
|
-
```sql
|
|
96
|
-
CREATE OR REPLACE FUNCTION get_venue_detail(
|
|
97
|
-
p_params JSONB,
|
|
98
|
-
p_user_id INT
|
|
99
|
-
) RETURNS JSONB AS $$
|
|
100
|
-
DECLARE
|
|
101
|
-
v_venue_id int;
|
|
102
|
-
v_result JSONB;
|
|
103
|
-
BEGIN
|
|
104
|
-
v_venue_id := (p_params->>'venue_id')::int;
|
|
105
|
-
|
|
106
|
-
-- Check access control
|
|
107
|
-
IF NOT venue_detail_can_subscribe(p_user_id, p_params) THEN
|
|
108
|
-
RAISE EXCEPTION 'Permission denied';
|
|
109
|
-
END IF;
|
|
110
|
-
|
|
111
|
-
-- Build document with root and all relations
|
|
112
|
-
SELECT jsonb_build_object(
|
|
113
|
-
'venues', row_to_json(root.*),
|
|
114
|
-
'org', (SELECT row_to_json(o.*) FROM organisations o WHERE o.id = root.org_id),
|
|
115
|
-
'sites', (SELECT jsonb_agg(s.*) FROM sites s WHERE s.venue_id = root.id),
|
|
116
|
-
'packages', (
|
|
117
|
-
SELECT jsonb_agg(
|
|
118
|
-
jsonb_build_object(
|
|
119
|
-
'package', row_to_json(p.*),
|
|
120
|
-
'allocations', (SELECT jsonb_agg(a.*) FROM allocations a WHERE a.package_id = p.id)
|
|
121
|
-
)
|
|
122
|
-
)
|
|
123
|
-
FROM packages p WHERE p.venue_id = root.id
|
|
124
|
-
)
|
|
125
|
-
)
|
|
126
|
-
INTO v_result
|
|
127
|
-
FROM venues root
|
|
128
|
-
WHERE root.id = v_venue_id;
|
|
129
|
-
|
|
130
|
-
RETURN v_result;
|
|
131
|
-
END;
|
|
132
|
-
$$ LANGUAGE plpgsql SECURITY DEFINER;
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
### 3. Affected Documents: `<name>_affected_documents(table, op, old, new)`
|
|
136
|
-
|
|
137
|
-
```sql
|
|
138
|
-
CREATE OR REPLACE FUNCTION venue_detail_affected_documents(
|
|
139
|
-
p_table_name TEXT,
|
|
140
|
-
p_op TEXT,
|
|
141
|
-
p_old JSONB,
|
|
142
|
-
p_new JSONB
|
|
143
|
-
) RETURNS JSONB[] AS $$
|
|
144
|
-
DECLARE
|
|
145
|
-
v_affected JSONB[];
|
|
146
|
-
BEGIN
|
|
147
|
-
CASE p_table_name
|
|
148
|
-
-- Venue changed: affects subscription for that venue
|
|
149
|
-
WHEN 'venues' THEN
|
|
150
|
-
v_affected := ARRAY[
|
|
151
|
-
jsonb_build_object('venue_id', COALESCE(p_new->>'id', p_old->>'id')::int)
|
|
152
|
-
];
|
|
153
|
-
|
|
154
|
-
-- Organisation changed: affects all venues in that org
|
|
155
|
-
WHEN 'organisations' THEN
|
|
156
|
-
SELECT ARRAY_AGG(jsonb_build_object('venue_id', v.id))
|
|
157
|
-
INTO v_affected
|
|
158
|
-
FROM venues v
|
|
159
|
-
WHERE v.org_id = COALESCE((p_new->>'id')::int, (p_old->>'id')::int);
|
|
160
|
-
|
|
161
|
-
-- Site changed: affects parent venue
|
|
162
|
-
WHEN 'sites' THEN
|
|
163
|
-
v_affected := ARRAY[
|
|
164
|
-
jsonb_build_object('venue_id', COALESCE(p_new->>'venue_id', p_old->>'venue_id')::int)
|
|
165
|
-
];
|
|
166
|
-
|
|
167
|
-
-- Package changed: affects parent venue
|
|
168
|
-
WHEN 'packages' THEN
|
|
169
|
-
v_affected := ARRAY[
|
|
170
|
-
jsonb_build_object('venue_id', COALESCE(p_new->>'venue_id', p_old->>'venue_id')::int)
|
|
171
|
-
];
|
|
172
|
-
|
|
173
|
-
-- Allocation changed: affects venue via package
|
|
174
|
-
WHEN 'allocations' THEN
|
|
175
|
-
SELECT ARRAY_AGG(jsonb_build_object('venue_id', p.venue_id))
|
|
176
|
-
INTO v_affected
|
|
177
|
-
FROM packages p
|
|
178
|
-
WHERE p.id = COALESCE((p_new->>'package_id')::int, (p_old->>'package_id')::int);
|
|
179
|
-
|
|
180
|
-
ELSE
|
|
181
|
-
v_affected := ARRAY[]::JSONB[];
|
|
182
|
-
END CASE;
|
|
183
|
-
|
|
184
|
-
RETURN v_affected;
|
|
185
|
-
END;
|
|
186
|
-
$$ LANGUAGE plpgsql IMMUTABLE;
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
---
|
|
190
|
-
|
|
191
|
-
## Server Implementation (In-Memory)
|
|
192
|
-
|
|
193
|
-
### Subscription Registry
|
|
194
|
-
|
|
195
|
-
```javascript
|
|
196
|
-
// In-memory storage
|
|
197
|
-
const subscriptions = new Map();
|
|
198
|
-
// subscription_id -> { subscribable, user_id, connection_id, params }
|
|
199
|
-
|
|
200
|
-
const connectionSubscriptions = new Map();
|
|
201
|
-
// connection_id -> Set<subscription_id>
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
### RPC Handlers
|
|
205
|
-
|
|
206
|
-
```javascript
|
|
207
|
-
// Pattern matching on method names
|
|
208
|
-
if (method.startsWith('subscribe_')) {
|
|
209
|
-
const subscribableName = method.replace('subscribe_', '');
|
|
210
|
-
const subscriptionId = crypto.randomUUID();
|
|
211
|
-
|
|
212
|
-
// Execute initial query (checks permissions)
|
|
213
|
-
const data = await db.query(
|
|
214
|
-
`SELECT get_${subscribableName}($1, $2) as data`,
|
|
215
|
-
[params, userId]
|
|
216
|
-
);
|
|
217
|
-
|
|
218
|
-
// Store in memory
|
|
219
|
-
subscriptions.set(subscriptionId, {
|
|
220
|
-
subscribable: subscribableName,
|
|
221
|
-
user_id: userId,
|
|
222
|
-
connection_id: connectionId,
|
|
223
|
-
params
|
|
224
|
-
});
|
|
225
|
-
|
|
226
|
-
return {
|
|
227
|
-
subscription_id: subscriptionId,
|
|
228
|
-
data: data.rows[0].data
|
|
229
|
-
};
|
|
230
|
-
}
|
|
231
|
-
|
|
232
|
-
if (method.startsWith('unsubscribe_')) {
|
|
233
|
-
// Remove from in-memory registry
|
|
234
|
-
// Find and delete by params + connection
|
|
235
|
-
}
|
|
236
|
-
```
|
|
237
|
-
|
|
238
|
-
### Event Listener
|
|
239
|
-
|
|
240
|
-
```javascript
|
|
241
|
-
setupListeners(async (event) => {
|
|
242
|
-
const { table, op, before, after } = event;
|
|
243
|
-
|
|
244
|
-
// EXISTING: Pattern 2 - Need to Know notifications
|
|
245
|
-
broadcast(...);
|
|
246
|
-
|
|
247
|
-
// NEW: Pattern 1 - Live Query subscriptions
|
|
248
|
-
|
|
249
|
-
// Group subscriptions by subscribable name
|
|
250
|
-
const subsByName = new Map();
|
|
251
|
-
for (const [subId, sub] of subscriptions.entries()) {
|
|
252
|
-
if (!subsByName.has(sub.subscribable)) {
|
|
253
|
-
subsByName.set(sub.subscribable, []);
|
|
254
|
-
}
|
|
255
|
-
subsByName.get(sub.subscribable).push({ subId, ...sub });
|
|
256
|
-
}
|
|
257
|
-
|
|
258
|
-
// For each subscribable, ask PostgreSQL which instances are affected
|
|
259
|
-
for (const [subscribableName, subs] of subsByName.entries()) {
|
|
260
|
-
const result = await db.query(
|
|
261
|
-
`SELECT ${subscribableName}_affected_documents($1, $2, $3, $4) as affected`,
|
|
262
|
-
[table, op, before, after]
|
|
263
|
-
);
|
|
264
|
-
|
|
265
|
-
const affectedParamSets = result.rows[0]?.affected || [];
|
|
266
|
-
|
|
267
|
-
// Match affected params to active subscriptions (in-memory)
|
|
268
|
-
for (const affectedParams of affectedParamSets) {
|
|
269
|
-
for (const sub of subs) {
|
|
270
|
-
if (paramsMatch(sub.params, affectedParams)) {
|
|
271
|
-
// Re-execute query
|
|
272
|
-
const updated = await db.query(
|
|
273
|
-
`SELECT get_${subscribableName}($1, $2) as data`,
|
|
274
|
-
[sub.params, sub.user_id]
|
|
275
|
-
);
|
|
276
|
-
|
|
277
|
-
// Broadcast to connection
|
|
278
|
-
broadcastToConnection(sub.connection_id, {
|
|
279
|
-
method: 'subscription:update',
|
|
280
|
-
params: {
|
|
281
|
-
subscription_id: sub.subId,
|
|
282
|
-
data: updated.rows[0].data
|
|
283
|
-
}
|
|
284
|
-
});
|
|
285
|
-
}
|
|
286
|
-
}
|
|
287
|
-
}
|
|
288
|
-
}
|
|
289
|
-
});
|
|
290
|
-
```
|
|
291
|
-
|
|
292
|
-
---
|
|
293
|
-
|
|
294
|
-
## Client API
|
|
295
|
-
|
|
296
|
-
```javascript
|
|
297
|
-
// Subscribe
|
|
298
|
-
const { data, unsubscribe } = await ws.api.subscribe_venue_detail(
|
|
299
|
-
{ venue_id: 1 },
|
|
300
|
-
(updated) => {
|
|
301
|
-
console.log('Venue updated:', updated);
|
|
302
|
-
// Update UI
|
|
303
|
-
}
|
|
304
|
-
);
|
|
305
|
-
|
|
306
|
-
// Initial data available immediately
|
|
307
|
-
console.log('Initial:', data);
|
|
308
|
-
|
|
309
|
-
// Unsubscribe
|
|
310
|
-
unsubscribe();
|
|
311
|
-
|
|
312
|
-
// Or call directly
|
|
313
|
-
await ws.api.unsubscribe_venue_detail({ venue_id: 1 });
|
|
314
|
-
```
|
|
315
|
-
|
|
316
|
-
---
|
|
317
|
-
|
|
318
|
-
## Implementation Status
|
|
319
|
-
|
|
320
|
-
### ✅ Phase 1: Compiler (COMPLETE)
|
|
321
|
-
|
|
322
|
-
**Files Created:**
|
|
323
|
-
- `/packages/dzql/src/compiler/codegen/subscribable-codegen.js` - Code generation
|
|
324
|
-
- `/packages/dzql/src/compiler/parser/subscribable-parser.js` - SQL parsing
|
|
325
|
-
- `/packages/dzql/src/compiler/compiler.js` - Extended with subscribable support
|
|
326
|
-
|
|
327
|
-
**Exports:**
|
|
328
|
-
- `compileSubscribable(subscribable)` - Compile single subscribable
|
|
329
|
-
- `compileAllSubscribables(subscribables[])` - Compile multiple
|
|
330
|
-
- `compileSubscribablesFromSQL(sqlContent)` - Parse and compile from SQL file
|
|
331
|
-
|
|
332
|
-
**Generated Functions:**
|
|
333
|
-
- `<name>_can_subscribe(user_id, params)` - Access control
|
|
334
|
-
- `get_<name>(params, user_id)` - Query function
|
|
335
|
-
- `<name>_affected_documents(table, op, old, new)` - Affected params
|
|
336
|
-
|
|
337
|
-
**Known Issue:**
|
|
338
|
-
- Parser needs improvement for nested `jsonb_build_object()` calls
|
|
339
|
-
- Test compilation failing on parameter splitting
|
|
340
|
-
|
|
341
|
-
---
|
|
342
|
-
|
|
343
|
-
### 🔨 Phase 2: Database Schema (TODO)
|
|
344
|
-
|
|
345
|
-
**Tasks:**
|
|
346
|
-
1. Create `dzql.subscribables` table (metadata only)
|
|
347
|
-
2. Create `register_subscribable()` SQL function
|
|
348
|
-
3. Migration file: `011_subscriptions.sql`
|
|
349
|
-
|
|
350
|
-
**Schema:**
|
|
351
|
-
```sql
|
|
352
|
-
CREATE TABLE IF NOT EXISTS dzql.subscribables (
|
|
353
|
-
name TEXT PRIMARY KEY,
|
|
354
|
-
permission_paths jsonb NOT NULL,
|
|
355
|
-
param_schema jsonb NOT NULL,
|
|
356
|
-
root_entity text NOT NULL,
|
|
357
|
-
relations jsonb NOT NULL,
|
|
358
|
-
created_at timestamptz DEFAULT NOW()
|
|
359
|
-
);
|
|
360
|
-
```
|
|
361
|
-
|
|
362
|
-
---
|
|
363
|
-
|
|
364
|
-
### 🔨 Phase 3: Server Integration (TODO)
|
|
365
|
-
|
|
366
|
-
**Files to Modify:**
|
|
367
|
-
1. `/packages/dzql/src/server/ws.js`
|
|
368
|
-
- Add in-memory subscription registry
|
|
369
|
-
- Add `subscribe_*` / `unsubscribe_*` handlers
|
|
370
|
-
- Add `broadcastToConnection()` function
|
|
371
|
-
|
|
372
|
-
2. `/packages/dzql/src/server/index.js`
|
|
373
|
-
- Extend event listener to check affected subscriptions
|
|
374
|
-
- Re-execute queries and broadcast updates
|
|
375
|
-
|
|
376
|
-
**Estimated Effort:** 2-3 days
|
|
377
|
-
|
|
378
|
-
---
|
|
379
|
-
|
|
380
|
-
### 🔨 Phase 4: Client Integration (TODO)
|
|
381
|
-
|
|
382
|
-
**Files to Modify:**
|
|
383
|
-
1. `/packages/dzql/src/client/ws.js`
|
|
384
|
-
- Add `subscribe_*` method handling
|
|
385
|
-
- Handle `subscription:update` messages
|
|
386
|
-
- Return `{ data, unsubscribe }` pattern
|
|
387
|
-
|
|
388
|
-
**Estimated Effort:** 1-2 days
|
|
389
|
-
|
|
390
|
-
---
|
|
391
|
-
|
|
392
|
-
## Testing Strategy
|
|
393
|
-
|
|
394
|
-
### Unit Tests (Compiler)
|
|
395
|
-
```javascript
|
|
396
|
-
test('generates subscribable functions', () => {
|
|
397
|
-
const result = compileSubscribable({
|
|
398
|
-
name: 'venue_detail',
|
|
399
|
-
permissionPaths: { subscribe: ['@org_id->acts_for...'] },
|
|
400
|
-
paramSchema: { venue_id: 'int' },
|
|
401
|
-
rootEntity: 'venues',
|
|
402
|
-
relations: { org: 'organisations', sites: 'sites' }
|
|
403
|
-
});
|
|
404
|
-
|
|
405
|
-
expect(result.sql).toContain('venue_detail_can_subscribe');
|
|
406
|
-
expect(result.sql).toContain('get_venue_detail');
|
|
407
|
-
expect(result.sql).toContain('venue_detail_affected_documents');
|
|
408
|
-
});
|
|
409
|
-
```
|
|
410
|
-
|
|
411
|
-
### Integration Tests (Database)
|
|
412
|
-
```sql
|
|
413
|
-
-- Test affected documents function
|
|
414
|
-
SELECT venue_detail_affected_documents(
|
|
415
|
-
'venues', 'update',
|
|
416
|
-
'{"id": 1, "name": "Old"}'::jsonb,
|
|
417
|
-
'{"id": 1, "name": "New"}'::jsonb
|
|
418
|
-
);
|
|
419
|
-
-- Should return: [{"venue_id": 1}]
|
|
420
|
-
|
|
421
|
-
-- Test query function
|
|
422
|
-
SELECT get_venue_detail('{"venue_id": 1}'::jsonb, 5);
|
|
423
|
-
-- Should return denormalized document
|
|
424
|
-
```
|
|
425
|
-
|
|
426
|
-
### E2E Tests
|
|
427
|
-
```javascript
|
|
428
|
-
test('subscription receives updates', async () => {
|
|
429
|
-
const updates = [];
|
|
430
|
-
|
|
431
|
-
const { data, unsubscribe } = await ws.api.subscribe_venue_detail(
|
|
432
|
-
{ venue_id: 1 },
|
|
433
|
-
(updated) => updates.push(updated)
|
|
434
|
-
);
|
|
435
|
-
|
|
436
|
-
// Trigger change
|
|
437
|
-
await ws.api.save.venues({ id: 1, name: 'Updated' });
|
|
438
|
-
|
|
439
|
-
await waitFor(() => updates.length > 0);
|
|
440
|
-
expect(updates[0].venues.name).toBe('Updated');
|
|
441
|
-
|
|
442
|
-
unsubscribe();
|
|
443
|
-
});
|
|
444
|
-
```
|
|
445
|
-
|
|
446
|
-
---
|
|
447
|
-
|
|
448
|
-
## Next Steps
|
|
449
|
-
|
|
450
|
-
1. **Fix Parser** - Handle nested `jsonb_build_object()` correctly
|
|
451
|
-
2. **Test Compilation** - Verify generated SQL is correct
|
|
452
|
-
3. **Create Migration** - `011_subscriptions.sql` with schema
|
|
453
|
-
4. **Implement Server Handlers** - In-memory subscriptions + event processing
|
|
454
|
-
5. **Implement Client Support** - `subscribe_*` methods
|
|
455
|
-
6. **Integration Testing** - End-to-end subscription flow
|
|
456
|
-
7. **Documentation** - API docs and examples
|
|
457
|
-
|
|
458
|
-
---
|
|
459
|
-
|
|
460
|
-
## Estimated Timeline
|
|
461
|
-
|
|
462
|
-
| Phase | Tasks | Effort |
|
|
463
|
-
|-------|-------|--------|
|
|
464
|
-
| Phase 1: Compiler | ✅ Complete | 4 hours |
|
|
465
|
-
| Phase 2: Database | Schema + migration | 1 day |
|
|
466
|
-
| Phase 3: Server | Handlers + event processing | 2-3 days |
|
|
467
|
-
| Phase 4: Client | Client API | 1-2 days |
|
|
468
|
-
| Testing | Unit + Integration + E2E | 2-3 days |
|
|
469
|
-
| **Total** | | **7-10 days** |
|
|
470
|
-
|
|
471
|
-
---
|
|
472
|
-
|
|
473
|
-
## Success Criteria
|
|
474
|
-
|
|
475
|
-
- ✅ Compiler generates 3 PostgreSQL functions per subscribable
|
|
476
|
-
- ⏳ PostgreSQL determines affected subscription instances (not server)
|
|
477
|
-
- ⏳ Server holds subscriptions in-memory (fast lookup)
|
|
478
|
-
- ⏳ Naming convention: `subscribe_<name>` / `unsubscribe_<name>`
|
|
479
|
-
- ⏳ Client receives automatic updates on data changes
|
|
480
|
-
- ⏳ < 100ms latency from DB change to client update
|
|
481
|
-
- ⏳ Supports 1000+ concurrent subscriptions per server
|
|
482
|
-
- ⏳ Zero runtime interpretation (all logic compiled)
|
|
483
|
-
|
|
484
|
-
---
|
|
485
|
-
|
|
486
|
-
**Implementation by:** Claude Sonnet 4.5
|
|
487
|
-
**Project:** DZQL Live Query Subscriptions
|
|
488
|
-
**Version:** 0.1.4+subscriptions
|
|
File without changes
|