dzql 0.1.3 → 0.1.5

package/docs/compiler/QUICKSTART.md

@@ -0,0 +1,326 @@
+# DZQL Compiler - Quick Start Guide
+
+Get up and running with the DZQL Compiler in 5 minutes.
+
+## Installation
+
+```bash
+cd /home/user/dzql/packages/dzql-compiler
+bun install
+```
+
+## Basic Usage
+
+### 1. Compile Your First Entity
+
+```bash
+# Compile the venues domain (9 entities)
+bun src/cli/index.js /home/user/dzql/packages/venues/database/init_db/009_venues_domain.sql -o compiled/
+
+# Output will be in compiled/ directory
+```
+
+### 2. Examine the Output
+
+```bash
+# View generated SQL for venues
+cat compiled/venues.sql
+
+# Check checksums
+cat compiled/checksums.json
+```
+
+### 3. Run the Tests
+
+```bash
+# All tests should pass
+bun test
+```
+
+## What You Get
+
+For each entity, the compiler generates:
+
+### Permission Functions
+```sql
+can_create_venues(p_user_id INT, p_record JSONB) → BOOLEAN
+can_update_venues(p_user_id INT, p_record JSONB) → BOOLEAN
+can_delete_venues(p_user_id INT, p_record JSONB) → BOOLEAN
+can_view_venues(p_user_id INT, p_record JSONB) → BOOLEAN
+```
+
+### CRUD Operations
+```sql
+get_venues(p_user_id INT, p_id INT) → JSONB
+save_venues(p_user_id INT, p_data JSONB) → JSONB
+delete_venues(p_user_id INT, p_id INT) → JSONB
+lookup_venues(p_user_id INT, p_filter TEXT) → JSONB
+search_venues(p_user_id INT, p_filters JSONB, p_search TEXT, ...) → JSONB
+```
+
+> **Note:** `p_user_id` is always the first parameter in all functions. See [CODING_STANDARDS.md](./CODING_STANDARDS.md).
+
+## Example: Compile a Simple Entity
+
+Create `examples/todos.sql`:
+
+```sql
+select dzql.register_entity(
+  'todos',
+  'title',
+  array['title', 'description'],
+  '{}',  -- no FK includes
+  false, -- no soft delete
+  '{}',  -- no temporal fields
+  '{}',  -- no notifications
+  jsonb_build_object(
+    'view', array[]::text[],      -- public
+    'create', array[]::text[],    -- public
+    'update', array['@owner_id'], -- owner only
+    'delete', array['@owner_id']  -- owner only
+  )
+);
+```
+
+Compile it:
+
+```bash
+bun src/cli/index.js examples/todos.sql -o compiled/
+```
+
+Result: `compiled/todos.sql` with 5 operations + 4 permission checks.
+
+## Using Compiled Functions
+
+Once deployed to PostgreSQL:
+
+```sql
+-- Get a todo (p_user_id first, then p_id)
+SELECT get_todos(42, 1);  -- user_id=42, id=1
+
+-- Create a todo (p_user_id first, then p_data)
+SELECT save_todos(42, '{"title": "Learn DZQL", "owner_id": 42}'::jsonb);
+
+-- Search todos (p_user_id first)
+SELECT search_todos(
+  42,             -- user_id
+  '{}',           -- filters
+  'DZQL',         -- search text
+  '{"field": "title", "order": "asc"}', -- sort
+  1,              -- page
+  25              -- limit
+);
+
+-- Delete a todo (p_user_id first, then p_id)
+SELECT delete_todos(42, 1);
+```
+
+## Development Workflow
+
+### 1. Define Entity
+
+Edit `entities/my_entity.sql`:
+
+```sql
+select dzql.register_entity(
+  'my_entity',
+  'name',
+  array['name'],
+  '{}',
+  false
+);
+```
+
+### 2. Compile
+
+```bash
+bun src/cli/index.js entities/my_entity.sql -o compiled/
+```
+
+### 3. Deploy
+
+```bash
+psql -U dzql -d dzql < compiled/my_entity.sql
+```
+
+### 4. Use
+
+```sql
+SELECT save_my_entity(1, '{"name": "Test"}'::jsonb);  -- user_id first, then data
+```
+
+## Programmatic API
+
+```javascript
+import { DZQLCompiler } from './src/compiler.js';
+
+const compiler = new DZQLCompiler();
+
+// Compile from object
+const result = compiler.compile({
+  tableName: 'posts',
+  labelField: 'title',
+  searchableFields: ['title', 'body'],
+  permissionPaths: {
+    view: [],
+    update: ['@author_id']
+  }
+});
+
+console.log(result.sql);
+console.log(result.checksum);
+
+// Compile from SQL
+import { readFileSync } from 'fs';
+
+const sql = readFileSync('entities/posts.sql', 'utf-8');
+const results = compiler.compileFromSQL(sql);
+
+for (const entity of results.results) {
+  console.log(`Compiled ${entity.tableName}: ${entity.checksum}`);
+}
+```
+
+## Verifying Compilation
+
+### Check Checksums
+
+```javascript
+import { readFileSync } from 'fs';
+
+const checksums = JSON.parse(readFileSync('compiled/checksums.json', 'utf-8'));
+
+console.log(checksums.venues);
+// {
+//   checksum: "9c116484...",
+//   generatedAt: "2025-11-16T01:38:54.321Z",
+//   compilationTime: 12
+// }
+```
+
+### Test Functions
+
+```sql
+-- Test permission check (p_user_id first)
+SELECT can_update_venues(42, '{"org_id": 1}'::jsonb);
+
+-- Test GET with FK expansion (p_user_id first, then p_id)
+SELECT get_venues(42, 1);
+-- Returns: { id: 1, name: "...", org: { id: 1, name: "..." }, sites: [...] }
+
+-- Test SEARCH (p_user_id first)
+SELECT search_venues(42, '{}', 'garden', null, 1, 10);
+-- Returns: { data: [...], total: 5, page: 1, limit: 10 }
+```
+
+## Common Patterns
+
+### Public Read, Owner Write
+
+```sql
+select dzql.register_entity(
+  'blog_posts',
+  'title',
+  array['title', 'body'],
+  '{}',
+  false,
+  '{}',
+  '{}',
+  jsonb_build_object(
+    'view', array[]::text[],      -- anyone can read
+    'create', array[]::text[],    -- anyone can create
+    'update', array['@author_id'], -- only author can update
+    'delete', array['@author_id']  -- only author can delete
+  )
+);
+```
+
+### Organization-Scoped
+
+```sql
+select dzql.register_entity(
+  'projects',
+  'name',
+  array['name', 'description'],
+  '{"org": "organisations"}',
+  false,
+  '{}',
+  '{}',
+  jsonb_build_object(
+    'view', array['@org_id->members[org_id=$]{active}.user_id'],
+    'update', array['@org_id->members[org_id=$,role=admin]{active}.user_id'],
+    'delete', array['@org_id->members[org_id=$,role=admin]{active}.user_id'],
+    'create', array['@org_id->members[org_id=$]{active}.user_id']
+  )
+);
+```
+
+### Temporal Data
+
+```sql
+select dzql.register_entity(
+  'memberships',
+  'user_id',
+  array['user_id', 'org_id'],
+  '{"user": "users", "org": "organisations"}',
+  false,
+  '{"valid_from": "valid_from", "valid_to": "valid_to"}', -- temporal!
+  '{}',
+  '{}'
+);
+```
+
+## Debugging
+
+### View Generated SQL
+
+```bash
+# Pretty print generated SQL
+cat compiled/venues.sql | less
+
+# Search for specific function
+grep -A 20 "CREATE OR REPLACE FUNCTION get_venues" compiled/venues.sql
+```
+
+### Check Compilation Errors
+
+```bash
+# Compile with verbose output
+DZQL_COMPILER_VERBOSE=true bun src/cli/index.js entities/my_entity.sql -o compiled/
+```
+
+### Test in PostgreSQL
+
+```sql
+-- Enable query logging
+SET log_statement = 'all';
+
+-- Test function (p_user_id first)
+SELECT get_venues(42, 1);
+
+-- View execution plan
+EXPLAIN ANALYZE SELECT get_venues(42, 1);
+```
+
+## Next Steps
+
+1. **Compile existing entities** - Try compiling your current DZQL entities
+2. **Review generated SQL** - Understand what the compiler produces
+3. **Test in staging** - Deploy to a test database and verify behavior
+4. **Benchmark performance** - Compare runtime vs compiled versions
+5. **Report issues** - Let us know what works and what doesn't!
+
+## Need Help?
+
+- **README.md** - Complete documentation
+- **SUMMARY.md** - What was built and how
+- **tests/compiler.test.js** - Usage examples
+- **examples/compiled/** - Real compiled output
+
+## Resources
+
+- Vision document: `/home/user/dzql/vision.md`
+- Current DZQL: `/home/user/dzql/packages/dzql/`
+- Example apps: `/home/user/dzql/packages/venues/`
+
+Happy compiling! 🚀

package/docs/compiler/SESSION_SUMMARY.md

@@ -0,0 +1,266 @@
+# Compiler Development Session Summary
+
+## Latest Session: Coding Standards Compliance (2025-11-16)
+
+This session enforced DZQL coding standards across all generated functions to ensure consistency, security, and proper WebSocket API protection.
+
+### ✅ Coding Standards Fixes
+
+**What Changed**: Fixed the compiler to generate code compliant with DZQL conventions:
+
+1. **Parameter Ordering** - `p_user_id INT` now FIRST in all functions
+   - ✅ `get_*(p_user_id INT, p_id INT, ...)`
+   - ✅ `save_*(p_user_id INT, p_data JSONB)`
+   - ✅ `delete_*(p_user_id INT, p_id INT)`
+   - ✅ `lookup_*(p_user_id INT, p_filter TEXT, ...)`
+   - ✅ `search_*(p_user_id INT, p_filters JSONB, ...)`
+
+2. **Helper Function Prefixes** - Internal functions now use `_` prefix
+   - ✅ `_graph_*_on_create(p_user_id INT, p_record JSONB)`
+   - ✅ `_resolve_notification_paths_*(p_user_id INT, p_record JSONB)`
+   - Prevents direct websocket client access to internal functions
+
+3. **Standard Permission Functions** - All 4 always generated
+   - ✅ `can_view_*`, `can_create_*`, `can_update_*`, `can_delete_*`
+   - Even for public access (returns `true`)
+
+4. **Clean Function Names** - Fixed malformed names with embedded comments
+   - ❌ Before: `can_-- Anyone can create...'update_organisations`
+   - ✅ After: `can_update_organisations`
+
+**Files Modified**:
+- `src/codegen/permission-codegen.js` - Clean operation names, generate all 4 functions
+- `src/codegen/operation-codegen.js` - Fix parameter ordering, update helper calls
+- `src/codegen/notification-codegen.js` - Add underscore prefix, p_user_id first
+- `src/codegen/graph-rules-codegen.js` - Add underscore prefix, p_user_id first
+- `tests/sql-validation.test.js` - Update expectations to match standards
+
+**Documentation Added**:
+- `CODING_STANDARDS.md` - Complete coding standards reference
+- Updated `README.md` with correct function signatures
+- Updated `QUICKSTART.md` with correct examples
+
+**Test Results**: ✅ All 55 tests passing
+
+---
+
+## Previous Session: Graph Rules & Advanced Filters
+
+This session continued work on the DZQL Compiler from a previous session where the core compilation infrastructure was built.
+
+## Completed Features
+
+### 1. ✅ Graph Rules Compilation
+
+**What**: Transforms declarative graph rules into executable PostgreSQL functions
+
+**Implementation**:
+- Created `GraphRulesCodegen` class
+- Supports all action types: `create`, `update`, `delete`, `validate`, `execute`
+- Resolves special variables:
+  - `@user_id` → `p_user_id`
+  - `@today` → `CURRENT_DATE`
+  - `@now` → `NOW()`
+  - `@field` → `(p_record->>'field')`
+- Generates trigger-based functions: `graph_{table}_{on_create|on_update|on_delete}`
+- Integrated into SAVE and DELETE operations
+
+**Example Generated Code** (now with correct coding standards):
+```sql
+CREATE OR REPLACE FUNCTION _graph_organisations_on_create(
+  p_user_id INT,
+  p_record JSONB
+) RETURNS VOID AS $$
+BEGIN
+  -- Creator becomes owner
+  INSERT INTO acts_for (user_id, org_id, valid_from)
+  VALUES (p_user_id, (p_record->>'id'), CURRENT_DATE);
+END;
+$$ LANGUAGE plpgsql SECURITY DEFINER;
+```
+
+> Note: Helper functions now use `_` prefix and have `p_user_id` first
+
+**Files Changed**:
+- `src/codegen/graph-rules-codegen.js` (new)
+- `src/compiler.js` (integrated)
+- `src/codegen/operation-codegen.js` (calls graph functions)
+
+### 2. ✅ Advanced SEARCH Filter Operators
+
+**What**: Comprehensive JSONB-based filtering with multiple operators
+
+**Operators Implemented**:
+- **Comparison**: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`
+- **Array Membership**: `in`
+- **Pattern Matching**: `like`, `ilike`
+
+**Features**:
+- Dynamic WHERE clause builder
+- Type-safe value handling
+- SQL injection protection via `format()`
+- Combines with text search and temporal filters
+- Multiple operators per field
+- Multiple fields per query
+
+**Example Usage**:
+```json
+{
+  "age": {"gte": 18, "lte": 65},
+  "status": {"in": ["active", "pending"]},
+  "email": {"ilike": "%@company.com"}
+}
+```
+
+**Generated SQL**:
+```sql
+WHERE TRUE
+  AND age >= '18'
+  AND age <= '65'
+  AND status = ANY(ARRAY['active', 'pending']::TEXT[])
+  AND email ILIKE '%@company.com'
+```
+
+**Files Changed**:
+- `src/codegen/operation-codegen.js` (enhanced SEARCH function)
+- `docs/ADVANCED_FILTERS.md` (comprehensive documentation)
+
+### 3. ✅ Comprehensive Test Suite
+
+**SQL Validation Tests** (42 tests):
+- Function signature validation
+- Permission logic verification
+- CRUD operation structure
+- Graph rules generation
+- Filter operator inclusion
+- SQL syntax validation
+- Metadata verification
+
+**Integration Tests** (prepared):
+- Full workflow testing
+- Real database operations
+- Permission enforcement
+- Advanced filter testing
+- Graph rules execution
+
+**Test Organization**:
+```bash
+bun test                  # Unit + validation (55 tests)
+bun test:integration      # Integration tests (requires PostgreSQL)
+bun test:all             # Everything
+```
+
+**Files Added**:
+- `tests/sql-validation.test.js` (42 tests)
+- `tests/integration.test.skip.js` (prepared for PostgreSQL)
+- `tests/integration-README.md` (setup instructions)
+
+## Test Results
+
+✅ **55 tests passing** (13 compiler + 42 validation)
+- All core compiler functionality verified
+- All generated SQL validated
+- No regressions introduced
+
+## Commits Made
+
+1. **feat: Complete graph rules compilation** (95d4257)
+   - Full graph rules compilation implementation
+   - Integration with SAVE/DELETE operations
+   - All action types supported
+
+2. **feat: Add advanced SEARCH filter operators** (dc8b0fe)
+   - 9 filter operators implemented
+   - Dynamic WHERE clause building
+   - Comprehensive documentation
+
+3. **test: Add comprehensive test suite for compiled SQL** (1d14209)
+   - 42 SQL validation tests
+   - Integration test framework
+   - Test documentation
+
+## Code Statistics
+
+**Lines Added**: ~1,500 lines across:
+- Graph rules code generation
+- Advanced filter implementation
+- Test suites
+- Documentation
+
+**Files Created**: 6
+- `src/codegen/graph-rules-codegen.js`
+- `docs/ADVANCED_FILTERS.md`
+- `tests/sql-validation.test.js`
+- `tests/integration.test.skip.js`
+- `tests/integration-README.md`
+- `docs/SESSION_SUMMARY.md`
+
+**Files Modified**: 4
+- `src/compiler.js`
+- `src/codegen/operation-codegen.js`
+- `package.json`
+- (various generated files)
+
+## Current Compiler Capabilities
+
+The DZQL Compiler now supports:
+
+### ✅ Entity Definition Parsing
+- Complete DZQL entity syntax
+- Nested JSONB structures
+- Array parameters with depth tracking
+
+### ✅ Permission Compilation
+- Direct field checks
+- Graph traversal with filters
+- Temporal filtering (`{active}`)
+- All CRUD operations (view, create, update, delete)
+
+### ✅ Notification Path Compilation
+- User resolution via graph traversal
+- Temporal filtering
+- Multiple notification paths
+
+### ✅ Graph Rules Compilation
+- on_create, on_update, on_delete triggers
+- All action types
+- Special variable resolution
+- Automatic integration with operations
+
+### ✅ CRUD Operations
+- GET with FK expansion
+- SAVE with upsert logic
+- DELETE (soft or hard)
+- LOOKUP with filtering
+- SEARCH with advanced filters
+
+### ✅ Advanced Features
+- 9 filter operators in SEARCH
+- Dynamic WHERE clause building
+- Reproducible builds (SHA-256 checksums)
+- Git-trackable output
+- Comprehensive error handling
+
+## What's Next?
+
+Potential future enhancements:
+- [ ] Performance benchmarking vs runtime DZQL
+- [ ] Migration generator (schema changes)
+- [ ] Index recommendation based on paths
+- [ ] Query plan analysis
+- [ ] Incremental compilation
+- [ ] Type inference for better error messages
+- [ ] GraphQL/REST API generation from entities
+
+## Branch Status
+
+Branch: `claude/new-features-planning-015GH92pzBxTkA1uWjnjJqbZ`
+
+All changes committed and pushed. Ready for review or PR creation.
+
+---
+
+**Session Duration**: Continued from previous session
+**Total Tests**: 55 passing
+**Code Coverage**: All major compilation paths tested
+**Documentation**: Comprehensive docs for all features