@lix-js/sdk 0.5.0-preview.0 → 0.5.0-preview.1

package/src/entity-views/README.md

@@ -0,0 +1,214 @@
+# Entity Views
+
+## Overview
+
+Entity views are SQL views that provide a typed, structured interface over the underlying `state` table. They transform the generic state storage into domain-specific entity tables, making it easy to work with entities using standard SQL operations while maintaining version control and change tracking.
+
+## Architecture
+
+Entity views are "merely a view over the state table" - they don't store any data themselves but provide a convenient interface:
+
+```
+┌─────────────────────────────────────────────────┐
+│             User SQL Queries                    │
+│   (SELECT, INSERT, UPDATE, DELETE on entities)  │
+└─────────────────────┬───────────────────────────┘
+                      │
+                      v
+┌─────────────────────────────────────────────────┐
+│              Entity Views                       │
+│                                                 │
+│  • Extract JSON properties as columns           │
+│  • Add operational metadata (lixcol_*)          │
+│  • Provide CRUD operations via triggers         │
+└─────────────────────┬───────────────────────────┘
+                      │
+                      v
+┌─────────────────────────────────────────────────┐
+│          State Virtual Table                    │
+│                                                 │
+│  • Generic entity storage                       │
+│  • Version control                              │
+│  • Change tracking                              │
+└─────────────────────────────────────────────────┘
+```
+
+## Example: Views as Filters
+
+Entity views are essentially filters over the generic state table. Here's how it works:
+
+### What's in the state_all table
+
+```sql
+state_all:
+┌─────────────┬─────────────────┬──────────┬───────────────────────────────────┐
+│ entity_id   │ schema_key      │ file_id  │ snapshot_content                  │
+├─────────────┼─────────────────┼──────────┼───────────────────────────────────┤
+│ greeting    │ lix:key_value   │ file1    │ {"key":"greeting","value":"hello"}│
+│ farewell    │ lix:key_value   │ file1    │ {"key":"farewell","value":"bye"}  │
+│ user123     │ lix:account     │ lix      │ {"id":"user123","name":"Alice"}   │
+│ config1     │ lix:key_value   │ file2    │ {"key":"config1","value":"true"}  │
+└─────────────┴─────────────────┴──────────┴───────────────────────────────────┘
+```
+
+### What the key_value view returns
+
+```sql
+SELECT * FROM key_value;
+
+┌──────────┬───────┬─────────────────┬──────────────────┐
+│ key      │ value │ lixcol_file_id  │ lixcol_change_id │
+├──────────┼───────┼─────────────────┼──────────────────┤
+│ greeting │ hello │ file1           │ change-123       │
+│ farewell │ bye   │ file1           │ change-124       │
+│ config1  │ true  │ file2           │ change-125       │
+└──────────┴───────┴─────────────────┴──────────────────┘
+```
+
+The view:
+
+- **Filters** rows where `schema_key = 'lix:key_value'`
+- **Extracts** JSON properties (`key`, `value`) as columns
+- **Excludes** entities with different schemas (like `lix:account`)
+- **Adds** operational columns prefixed with `lixcol_`
+
+## Three Types of Entity Views
+
+For each entity schema, three views are created:
+
+### 1. Primary View (e.g., `key_value`)
+
+- **Purpose**: Work with entities in the active version only
+- **Based on**: `state` view (filtered to current version)
+- **Features**:
+  - Hides `lixcol_version_id` to prevent cross-version accidents
+  - Full CRUD support via INSTEAD OF triggers
+  - Most commonly used for application logic
+
+### 2. All-Versions View (e.g., `key_value_all`)
+
+- **Purpose**: Work with entities across all versions
+- **Based on**: `state_all` virtual table
+- **Features**:
+  - Exposes `lixcol_version_id` for version-specific queries
+  - Full CRUD support with explicit version control
+  - Used for cross-version operations and migrations
+
+### 3. History View (e.g., `key_value_history`)
+
+- **Purpose**: Read historical states and track changes
+- **Based on**: `state_history` table
+- **Features**:
+  - Read-only (no CRUD triggers)
+  - Includes depth information for blame functionality
+  - Used for audit trails and debugging
+
+## How Entity Views Work
+
+### JSON Property Extraction
+
+Entity properties are stored as JSON in the state table's `snapshot_content` column. Views extract these as proper columns:
+
+```sql
+-- State table stores generic JSON
+snapshot_content: {"value": "hello", "description": "greeting"}
+
+-- Entity view exposes as columns
+SELECT value, description FROM key_value
+-- Returns: value='hello', description='greeting'
+```
+
+### Operational Columns (lixcol\_\*)
+
+Entity views add metadata columns prefixed with `lixcol_`:
+
+- `lixcol_file_id` - Links entity to a file
+- `lixcol_created_at` - When entity was created in this version
+- `lixcol_updated_at` - When entity was last updated in this version
+- `lixcol_change_id` - Links to the change that created/modified the entity
+- `lixcol_untracked` - Bypasses change tracking (for UI state)
+- `lixcol_inherited_from_version_id` - Tracks entity lineage across branches
+- `lixcol_version_id` - Version ID (only in `_all` views)
+
+### INSTEAD OF Triggers
+
+Entity views use SQLite's INSTEAD OF triggers to intercept CRUD operations and redirect them to the state table:
+
+```sql
+-- User writes to entity view
+INSERT INTO key_value (key, value) VALUES ('greeting', 'hello')
+
+-- Trigger converts to state table operation
+INSERT INTO state_all (
+  entity_id: 'greeting',
+  schema_key: 'lix:key_value',
+  snapshot_content: '{"key": "greeting", "value": "hello"}'
+  -- ... other metadata
+)
+```
+
+## Type Safety
+
+The TypeScript type system ensures compile-time safety:
+
+```typescript
+// Schema definition
+type KeyValue = {
+	key: string;
+	value: string;
+	description?: string;
+};
+
+// Generated types for queries
+type KeyValueRow = State<KeyValue>; // For SELECT from key_value
+type NewKeyValueRow = NewState<KeyValue>; // For INSERT into key_value
+```
+
+## Usage Examples
+
+### Working with Active Version
+
+```sql
+-- Insert a new entity (goes through state virtual table)
+INSERT INTO key_value (key, value) VALUES ('greeting', 'hello');
+
+-- Update an entity
+UPDATE key_value SET value = 'hi' WHERE key = 'greeting';
+
+-- Delete an entity
+DELETE FROM key_value WHERE key = 'greeting';
+
+-- Query entities
+SELECT * FROM key_value WHERE value LIKE 'h%';
+```
+
+### Working Across Versions
+
+```sql
+-- Query specific version
+SELECT * FROM key_value_all WHERE lixcol_version_id = 'version-123';
+
+-- Insert into specific version
+INSERT INTO key_value_all (key, value, lixcol_version_id)
+VALUES ('greeting', 'hello', 'version-123');
+```
+
+### Viewing History
+
+```sql
+-- See all historical states of an entity
+SELECT * FROM key_value_history WHERE key = 'greeting' ORDER BY depth;
+
+-- Find who changed an entity
+SELECT * FROM key_value_history
+WHERE key = 'greeting' AND lixcol_change_id IS NOT NULL;
+```
+
+## Benefits
+
+1. **Natural SQL Interface** - Work with entities using familiar SQL
+2. **Type Safety** - Full TypeScript types for all operations
+3. **Version Control** - Built-in branching and merging support
+4. **Change Tracking** - Automatic audit trail for all modifications
+5. **Performance** - Leverages SQLite's query optimizer
+6. **Flexibility** - JSON storage allows schema evolution

package/src/entity-views/entity-state-all.test.ts

@@ -442,4 +442,206 @@ describe("createEntityAllViewIfNotExists", () => {
 		expect(onlyTracked).toHaveLength(1);
 		expect(onlyTracked[0]?.id).toBe("tracked_entity");
 	});
+
+	test("should expose lixcol_commit_id for history queries", async () => {
+		const lix = await openLix({});
+
+		// Store the test schema
+		await lix.db
+			.insertInto("stored_schema")
+			.values({ key: "test_entity", value: testSchema as any })
+			.execute();
+
+		createEntityStateAllView({
+			lix,
+			schema: testSchema,
+			overrideName: "test_view_all",
+			pluginKey: "test_plugin",
+			hardcodedFileId: "test_file",
+		});
+
+		// Get the active version ID
+		const activeVersion = await lix.db
+			.selectFrom("active_version")
+			.select("version_id")
+			.executeTakeFirst();
+
+		// Insert test data through the view
+		await lix.db
+			.insertInto("test_view_all" as any)
+			.values({
+				id: "test_id",
+				name: "test_name",
+				value: 42,
+				lixcol_version_id: activeVersion?.version_id,
+			})
+			.execute();
+
+		// Query the view and verify lixcol_commit_id is exposed
+		const result = await lix.db
+			.selectFrom("test_view_all" as any)
+			.selectAll()
+			.execute();
+
+		expect(result).toHaveLength(1);
+		const entity = result[0];
+
+		// Verify that lixcol_commit_id is exposed
+		expect(entity).toHaveProperty("lixcol_commit_id");
+		expect(entity?.lixcol_commit_id).toBeDefined();
+		expect(typeof entity?.lixcol_commit_id).toBe("string");
+
+		// Verify we can query by lixcol_commit_id
+		const queryByChangeSetId = await lix.db
+			.selectFrom("test_view_all" as any)
+			.where("lixcol_commit_id", "=", entity?.lixcol_commit_id)
+			.where("lixcol_version_id", "=", activeVersion?.version_id)
+			.selectAll()
+			.execute();
+
+		expect(queryByChangeSetId).toHaveLength(1);
+		expect(queryByChangeSetId[0]?.id).toBe("test_id");
+
+		// Also verify it's exposed alongside other lixcol columns
+		expect(entity).toHaveProperty("lixcol_version_id");
+		expect(entity).toHaveProperty("lixcol_inherited_from_version_id");
+		expect(entity).toHaveProperty("lixcol_created_at");
+		expect(entity).toHaveProperty("lixcol_updated_at");
+		expect(entity).toHaveProperty("lixcol_file_id");
+		expect(entity).toHaveProperty("lixcol_change_id");
+		expect(entity).toHaveProperty("lixcol_untracked");
+	});
+
+	test("should expose lixcol_entity_id, lixcol_schema_key, lixcol_file_id, and lixcol_plugin_key in _all view", async () => {
+		const lix = await openLix({});
+
+		createEntityStateAllView({
+			lix,
+			schema: testSchema,
+			overrideName: "test_view_all",
+			pluginKey: "test_plugin",
+			hardcodedFileId: "test_file",
+		});
+
+		// Get active version
+		const activeVersion = await lix.db
+			.selectFrom("active_version")
+			.select("version_id")
+			.executeTakeFirstOrThrow();
+
+		// Insert test data
+		await lix.db
+			.insertInto("test_view_all" as any)
+			.values({
+				id: "test_id",
+				name: "test_name",
+				value: 42,
+				lixcol_version_id: activeVersion.version_id,
+			})
+			.execute();
+
+		// Query the view to verify all entity identification columns are exposed
+		const result = await lix.db
+			.selectFrom("test_view_all" as any)
+			.select([
+				"id",
+				"name",
+				"lixcol_entity_id",
+				"lixcol_schema_key",
+				"lixcol_file_id",
+				"lixcol_plugin_key",
+				"lixcol_version_id", // _all views also have version_id
+			])
+			.executeTakeFirst();
+
+		expect(result).toBeDefined();
+
+		// Verify all required lixcol_ columns are present and have correct values
+		expect(result?.lixcol_entity_id).toBe("test_id");
+		expect(result?.lixcol_schema_key).toBe("test_entity");
+		expect(result?.lixcol_file_id).toBe("test_file");
+		expect(result?.lixcol_plugin_key).toBe("test_plugin");
+		expect(result?.lixcol_version_id).toBeDefined();
+	});
+
+	test("DELETE with version_id only affects the specified version", async () => {
+		const lix = await openLix({});
+
+		// Create the entity view
+		createEntityStateAllView({
+			lix,
+			schema: testSchema,
+			overrideName: "test_view_all",
+			pluginKey: "test_plugin",
+			hardcodedFileId: "test_file",
+		});
+
+		// Create parent and child versions
+		const { createVersion } = await import("../version/create-version.js");
+		await createVersion({ lix, id: "parent-version" });
+		await createVersion({
+			lix,
+			id: "child-version",
+			inherits_from_version_id: "parent-version",
+		});
+
+		// Insert entity into parent version
+		await lix.db
+			.insertInto("test_view_all" as any)
+			.values({
+				id: "shared-entity",
+				name: "parent-value",
+				value: 100,
+				lixcol_version_id: "parent-version",
+			})
+			.execute();
+
+		// Verify entity exists in parent version's state_all
+		const parentStateBefore = await lix.db
+			.selectFrom("state_all")
+			.where("entity_id", "=", "shared-entity")
+			.where("version_id", "=", "parent-version")
+			.selectAll()
+			.execute();
+
+		expect(parentStateBefore).toHaveLength(1);
+		expect(parentStateBefore[0]!.snapshot_content).toEqual({
+			id: "shared-entity",
+			name: "parent-value",
+			value: 100,
+		});
+
+		// Attempt to delete the entity from child version
+		// This should not affect any data since the entity doesn't exist in child version
+		await lix.db
+			.deleteFrom("test_view_all" as any)
+			.where("id", "=", "shared-entity")
+			.where("lixcol_version_id", "=", "child-version")
+			.execute();
+
+		// Verify parent version still has the entity
+		const parentStateAfter = await lix.db
+			.selectFrom("state_all")
+			.where("entity_id", "=", "shared-entity")
+			.where("version_id", "=", "parent-version")
+			.selectAll()
+			.execute();
+
+		expect(parentStateAfter).toHaveLength(1);
+		expect(parentStateAfter[0]!.snapshot_content).toEqual({
+			id: "shared-entity",
+			name: "parent-value",
+			value: 100,
+		});
+
+		// Verify child version has no entry
+		const childState = await lix.db
+			.selectFrom("state_all")
+			.where("entity_id", "=", "shared-entity")
+			.where("version_id", "=", "child-version")
+			.selectAll()
+			.execute();
+
+		expect(childState).toHaveLength(0);
+	});
 });

package/src/entity-views/entity-state-all.ts

@@ -28,6 +28,20 @@ import type { ValidationRule, ValidationCallbacks } from "./entity-state.js";
  * ```
  */
 export type StateEntityAllView = {
+	/**
+	 * The unique identifier for this entity within its schema and file.
+	 *
+	 * This is the primary identifier used to reference this specific entity.
+	 */
+	lixcol_entity_id: Generated<string>;
+
+	/**
+	 * The schema key that defines the structure and type of this entity.
+	 *
+	 * This references the schema definition that validates and types this entity.
+	 */
+	lixcol_schema_key: Generated<string>;
+
 	/**
 	 * File identifier where this entity is stored.
 	 *
@@ -36,6 +50,13 @@ export type StateEntityAllView = {
 	 */
 	lixcol_file_id: Generated<string>;
 
+	/**
+	 * The plugin key that manages this entity type.
+	 *
+	 * This identifies which plugin is responsible for handling this entity.
+	 */
+	lixcol_plugin_key: Generated<string>;
+
 	/**
 	 * Version identifier for this specific state of the entity.
 	 *
@@ -95,6 +116,15 @@ export type StateEntityAllView = {
 	 * in the state resolution order: untracked > tracked > inherited.
 	 */
 	lixcol_untracked: Generated<boolean>;
+
+	/**
+	 * Commit identifier that contains this entity's last change.
+	 *
+	 * This references the commit.id that contains the last change to this entity.
+	 * Useful for understanding which commit a particular entity state belongs to,
+	 * enabling history queries and version comparison.
+	 */
+	lixcol_commit_id: Generated<string>;
 };
 
 /**
@@ -112,6 +142,20 @@ export type StateEntityAllView = {
  * ```
  */
 export type EntityStateAllColumns = {
+	/**
+	 * The unique identifier for this entity within its schema and file.
+	 *
+	 * This is the primary identifier used to reference this specific entity.
+	 */
+	lixcol_entity_id: LixGenerated<string>;
+
+	/**
+	 * The schema key that defines the structure and type of this entity.
+	 *
+	 * This references the schema definition that validates and types this entity.
+	 */
+	lixcol_schema_key: LixGenerated<string>;
+
 	/**
 	 * File identifier where this entity is stored.
 	 *
@@ -120,6 +164,13 @@ export type EntityStateAllColumns = {
 	 */
 	lixcol_file_id: LixGenerated<string>;
 
+	/**
+	 * The plugin key that manages this entity type.
+	 *
+	 * This identifies which plugin is responsible for handling this entity.
+	 */
+	lixcol_plugin_key: LixGenerated<string>;
+
 	/**
 	 * Version identifier for this specific state of the entity.
 	 *
@@ -179,6 +230,15 @@ export type EntityStateAllColumns = {
 	 * in the state resolution order: untracked > tracked > inherited.
 	 */
 	lixcol_untracked: LixGenerated<boolean>;
+
+	/**
+	 * Commit identifier that contains this entity's last change.
+	 *
+	 * This references the commit.id that contains the last change to this entity.
+	 * Useful for understanding which commit a particular entity state belongs to,
+	 * enabling history queries and version comparison.
+	 */
+	lixcol_commit_id: LixGenerated<string>;
 };
 
 /**
@@ -236,10 +296,13 @@ export function createEntityStateAllView(args: {
 	validation?: ValidationCallbacks;
 }): void {
 	const view_name = args.overrideName ?? args.schema["x-lix-key"] + "_all";
+	// Quote the view name to handle SQL reserved keywords
+	const quoted_view_name = `"${view_name}"`;
 
 	createSingleEntityAllView({
 		...args,
 		viewName: view_name,
+		quotedViewName: quoted_view_name,
 		stateTable: "state_all",
 	});
 }
@@ -248,6 +311,7 @@ function createSingleEntityAllView(args: {
 	lix: Pick<Lix, "sqlite">;
 	schema: LixSchemaDefinition;
 	viewName: string;
+	quotedViewName?: string;
 	stateTable: "state_all";
 	/** Plugin identifier for the entity */
 	pluginKey: string;
@@ -270,6 +334,7 @@ function createSingleEntityAllView(args: {
 	}
 
 	const view_name = args.viewName;
+	const quoted_view_name = args.quotedViewName || `"${view_name}"`;
 	const schema_key = args.schema["x-lix-key"];
 	const properties = Object.keys((args.schema as any).properties);
 	const primaryKeys = args.schema["x-lix-primary-key"];
@@ -280,11 +345,11 @@ function createSingleEntityAllView(args: {
 	const entityIdNew =
 		primaryKeys.length === 1
 			? `NEW.${primaryKeys[0]}`
-			: `(${primaryKeys.map((key) => `NEW.${key}`).join(" || '::' || ")})`;
+			: `(${primaryKeys.map((key) => `NEW.${key}`).join(" || '~' || ")})`;
 	const entityIdOld =
 		primaryKeys.length === 1
 			? `OLD.${primaryKeys[0]}`
-			: `(${primaryKeys.map((key) => `OLD.${key}`).join(" || '::' || ")})`;
+			: `(${primaryKeys.map((key) => `OLD.${key}`).join(" || '~' || ")})`;
 
 	// Create UDFs for default values
 	if (args.defaultValues) {
@@ -366,13 +431,17 @@ function createSingleEntityAllView(args: {
 
 	// Operational columns for _all view (includes version_id)
 	const operationalColumns = [
+		"entity_id AS lixcol_entity_id",
+		"schema_key AS lixcol_schema_key",
+		"file_id AS lixcol_file_id",
+		"plugin_key AS lixcol_plugin_key",
 		"version_id AS lixcol_version_id",
 		"inherited_from_version_id AS lixcol_inherited_from_version_id",
 		"created_at AS lixcol_created_at",
 		"updated_at AS lixcol_updated_at",
-		"file_id AS lixcol_file_id",
 		"change_id AS lixcol_change_id",
 		"untracked AS lixcol_untracked",
+		"commit_id AS lixcol_commit_id",
 	];
 
 	// Handle version_id for _all view
@@ -413,7 +482,7 @@ function createSingleEntityAllView(args: {
 
 	// Generated SQL query - set breakpoint here to inspect the generated SQL during debugging
 	const sqlQuery = `
-    CREATE VIEW IF NOT EXISTS ${view_name} AS
+    CREATE VIEW IF NOT EXISTS ${quoted_view_name} AS
       SELECT
         ${Object.keys((args.schema as any).properties)
 					.map(
@@ -425,7 +494,7 @@ function createSingleEntityAllView(args: {
       WHERE schema_key = '${schema_key}';
 
       CREATE TRIGGER IF NOT EXISTS ${view_name}_insert
-      INSTEAD OF INSERT ON ${view_name}
+      INSTEAD OF INSERT ON ${quoted_view_name}
       BEGIN      
         ${insertValidationSQL}
         INSERT INTO state_all (
@@ -448,13 +517,13 @@ function createSingleEntityAllView(args: {
           json_object(${properties.map((prop) => `'${prop}', with_default_values.${prop}`).join(", ")}),
           '${args.schema["x-lix-version"]}',
           ${versionIdReference.replace(/NEW\./g, "with_default_values.")},
-          with_default_values.lixcol_untracked
+          COALESCE(with_default_values.lixcol_untracked, 0)
         FROM (
           SELECT
             ${defaultsSubquery},
             ${versionIdInDefaults}
             NEW.lixcol_file_id AS lixcol_file_id,
-            NEW.lixcol_untracked AS lixcol_untracked
+            COALESCE(NEW.lixcol_untracked, 0) AS lixcol_untracked
         ) AS with_default_values`
 						: `
         VALUES (
@@ -465,13 +534,13 @@ function createSingleEntityAllView(args: {
           json_object(${properties.map((prop) => `'${prop}', NEW.${prop}`).join(", ")}),
           '${args.schema["x-lix-version"]}',
           ${versionIdReference},
-          NEW.lixcol_untracked
+          COALESCE(NEW.lixcol_untracked, 0)
         )`
 				};
       END;
 
       CREATE TRIGGER IF NOT EXISTS ${view_name}_update
-      INSTEAD OF UPDATE ON ${view_name}
+      INSTEAD OF UPDATE ON ${quoted_view_name}
       BEGIN
         ${updateValidationSQL}
         UPDATE state_all
@@ -491,14 +560,14 @@ function createSingleEntityAllView(args: {
       END;
 
       CREATE TRIGGER IF NOT EXISTS ${view_name}_delete
-      INSTEAD OF DELETE ON ${view_name}
+      INSTEAD OF DELETE ON ${quoted_view_name}
       BEGIN
         ${deleteValidationSQL}
         DELETE FROM state_all
         WHERE entity_id = ${entityIdOld}
         AND schema_key = '${schema_key}'
         AND file_id = ${args.hardcodedFileId ? `'${args.hardcodedFileId}'` : "OLD.lixcol_file_id"}
-        ${args.hardcodedVersionId ? `AND version_id = '${args.hardcodedVersionId}'` : ""};
+        AND version_id = ${args.hardcodedVersionId ? `'${args.hardcodedVersionId}'` : "OLD.lixcol_version_id"};
       END;
     `;