@harperfast/template-react-studio 1.6.3 → 1.6.4

package/.agents/skills/harper-best-practices/rules/real-time-apps.md

@@ -1,43 +1,102 @@
 ---
 name: real-time-apps
 description: How to build real-time features in Harper using WebSockets and Pub/Sub.
+metadata:
+  mode: generate
+  sources:
+    - reference/v5/rest/websockets.md
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: a8afd4d3a52f77ba
 ---
 
-# Real-time Applications
+# Real-Time Apps with WebSockets and Pub/Sub
 
-Instructions for the agent to follow when building real-time applications in Harper.
+Instructions for the agent to follow when building real-time features in Harper using WebSockets and Pub/Sub.
 
 ## When to Use
 
-Use this skill when you need to stream live updates to clients, implement chat features, or provide real-time data synchronization between the database and a frontend.
+Apply this rule when implementing any feature that requires real-time bidirectional communication, live data streaming, or push-based updates in a Harper application. This includes chat, live dashboards, sensor feeds, and any scenario where clients must receive resource changes as they happen.
 
 ## How It Works
 
-1. **Check Automatic WebSockets**: If you only need to stream table changes, use [Automatic APIs](automatic-apis.md) which provide a WebSocket endpoint for every `@export`ed table.
-2. **Implement `connect` in a Resource**: For custom bi-directional logic, implement the `connect` method.
-3. **Use Pub/Sub**: Use `tables.TableName.subscribe(query)` to listen for specific data changes and stream them to the client.
-4. **Handle SSE**: Ensure your `connect` method gracefully handles cases where `incomingMessages` is null (Server-Sent Events).
-5. **Connect from Client**: Use standard WebSockets (`new WebSocket('wss://...')`) to connect to your resource endpoint. Ensure you use the appropriate scheme (`ws://` for HTTP, `wss://` for HTTPS).
+1. **Enable WebSocket support**: WebSocket support is enabled automatically when the `rest` plugin is enabled. To explicitly disable it, set the following in your config:
 
-## Examples
+   ```yaml
+   rest:
+     webSocket: false
+   ```
 
-### Bi-directional WebSocket Resource
+2. **Connect a client to a resource**: A WebSocket connection to a resource URL automatically subscribes to that resource. When the record changes or a message is published to it, the connection receives the update.
 
-```typescript
-import { Resource, tables } from 'harper';
+   ```javascript
+   let ws = new WebSocket('wss://server/my-resource/341');
+   ws.onmessage = (event) => {
+   	let data = JSON.parse(event.data);
+   };
+   ```
 
-export class MySocket extends Resource {
-	async *connect(target, incomingMessages) {
-		// Subscribe to table changes
-		const subscription = await tables.MyTable.subscribe(target);
-		if (!incomingMessages) {
-			return subscription; // SSE mode
-		}
+   `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it.
+
+3. **Implement a custom `connect()` handler**: Override the `connect(incomingMessages)` method on a resource class to control WebSocket behavior. The method must return an async iterable (or generator) that produces messages to send to the client. See [automatic-apis.md](automatic-apis.md) for more on defining resource classes.
+
+4. **Use the default `connect()` for event-style access**: Call `super.connect()` to get a streaming iterable that provides:
+   - A `send(message)` method for pushing outgoing messages
+   - A `close` event for cleanup on disconnect
+
+5. **Handle message ordering in distributed environments**: Harper delivers messages to local subscribers immediately without inter-node coordination delay.
+
+   | Message Type                                             | Behavior                                                                |
+   | -------------------------------------------------------- | ----------------------------------------------------------------------- |
+   | Non-retained (no `retain` flag)                          | Every message delivered in order received; suitable for chat            |
+   | Retained (published with `retain`, or PUT/updated in DB) | Only the latest-timestamp message is kept; suitable for sensor readings |
 
-		// Handle incoming client messages
+6. **Use MQTT over WebSockets** when needed by setting the sub-protocol header:
+   ```
+   Sec-WebSocket-Protocol: mqtt
+   ```
+
+## Examples
+
+**Simple echo server** — override `connect(incomingMessages)` to yield each incoming message back to the client:
+
+```javascript
+export class Echo extends Resource {
+	async *connect(incomingMessages) {
 		for await (let message of incomingMessages) {
-			yield { received: message };
+			yield message; // echo each message back
 		}
 	}
 }
 ```
+
+**Custom connect with timer and event-style access** — use `super.connect()` to get the outgoing stream, push periodic messages, echo incoming messages, and clean up on disconnect:
+
+```javascript
+export class Example extends Resource {
+	connect(incomingMessages) {
+		let outgoingMessages = super.connect();
+
+		let timer = setInterval(() => {
+			outgoingMessages.send({ greeting: 'hi again!' });
+		}, 1000);
+
+		incomingMessages.on('data', (message) => {
+			outgoingMessages.send(message); // echo incoming messages
+		});
+
+		outgoingMessages.on('close', () => {
+			clearInterval(timer);
+		});
+
+		return outgoingMessages;
+	}
+}
+```
+
+## Notes
+
+- WebSocket connections target a resource URL path. By default, connecting to a resource subscribes to changes for that resource.
+- The `connect(incomingMessages)` method **must** return an async iterable or generator; returning a plain value will not work.
+- `super.connect()` returns a streaming iterable with `send(message)` and a `close` event — use this when you need to push messages outside of the incoming message loop.
+- For one-way real-time streaming without bidirectional communication, consider Server-Sent Events instead.
+- For full pub/sub capabilities, Harper also supports MQTT; set `Sec-WebSocket-Protocol: mqtt` to use MQTT over WebSockets.

package/.agents/skills/harper-best-practices/rules/schema-design-tooling.md

@@ -1,6 +1,8 @@
 ---
 name: schema-design-tooling
 description: Best practices for Harper schema design, including core directives and GraphQL tooling configuration.
+metadata:
+  mode: synthesized
 ---
 
 # Schema Design & Tooling

package/.agents/skills/harper-best-practices/rules/serving-web-content.md

@@ -1,6 +1,8 @@
 ---
 name: serving-web-content
 description: How to serve static files and integrated Vite/React applications in Harper.
+metadata:
+  mode: synthesized
 ---
 
 # Serving Web Content

package/.agents/skills/harper-best-practices/rules/typescript-type-stripping.md

@@ -1,6 +1,8 @@
 ---
 name: typescript-type-stripping
 description: How to run TypeScript files directly in Harper without a build step.
+metadata:
+  mode: synthesized
 ---
 
 # TypeScript Type Stripping

package/.agents/skills/harper-best-practices/rules/using-blob-datatype.md

@@ -1,6 +1,8 @@
 ---
 name: using-blob-datatype
 description: How to use the Blob data type for efficient binary storage in Harper.
+metadata:
+  mode: synthesized
 ---
 
 # Using Blob Datatype

package/.agents/skills/harper-best-practices/rules/vector-indexing.md

@@ -1,40 +1,56 @@
 ---
 name: vector-indexing
 description: How to enable and query vector indexes for similarity search in Harper.
+metadata:
+  mode: generate
+  sources:
+    - reference/v5/database/schema.md#Vector Indexing
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: 3732961c671aac00
 ---
 
 # Vector Indexing
 
-Instructions for the agent to follow when implementing vector search in Harper.
+Instructions for the agent to follow when enabling and querying vector indexes for similarity search in Harper using the HNSW algorithm.
 
 ## When to Use
 
-Use this skill when you need to perform similarity searches on high-dimensional data, such as AI embeddings for semantic search, recommendations, or image retrieval.
+Apply this rule when adding a vector index to a Harper table schema or writing similarity search queries against high-dimensional vector fields. Use it whenever you need approximate nearest-neighbor search, distance-threshold filtering, or distance-scored results.
 
 ## How It Works
 
-1. **Enable Vector Indexing**: In your GraphQL schema, add `@indexed(type: "HNSW")` to a numeric array field:
+1. **Declare the vector index on a `[Float]` field**: Add `@indexed(type: "HNSW")` to any `[Float]` attribute in a `@table` type. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for general schema setup.
+
    ```graphql
-   type Product @table {
-   	id: ID @primaryKey
+   type Document @table {
+   	id: Long @primaryKey
    	textEmbeddings: [Float] @indexed(type: "HNSW")
    }
    ```
-2. **Configure Index Options (Optional)**: Fine-tune the index with parameters like `distance` (`cosine` or `euclidean`), `M`, and `efConstruction`.
-3. **Query with Vector Search**: Use `tables.Table.search()` with a `sort` object containing the `target` vector:
+
+2. **Query by nearest neighbors using `sort`**: Call `Document.search()` with a `sort` object containing `attribute` (the indexed field name) and `target` (the query vector). Include `limit` to cap results.
+
    ```javascript
-   const results = await tables.Product.search({
-     select: ['name', '$distance'],
-     sort: {
-       attribute: 'textEmbeddings',
-       target: [0.1, 0.2, ...], // query vector
-     },
-     limit: 5,
+   let results = Document.search({
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
    });
    ```
-4. **Filter by Distance**: Use `conditions` with a `target` vector and a `comparator` (e.g., `lt`) to return results within a similarity threshold:
+
+3. **Combine with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
+
    ```javascript
-   const results = await tables.Product.search({
+   let results = Document.search({
+   	conditions: [{ attribute: 'price', comparator: 'lt', value: 50 }],
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
+   });
+   ```
+
+4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. Omit `sort`.
+
+   ```javascript
+   let results = Document.search({
    	conditions: {
    		attribute: 'textEmbeddings',
    		comparator: 'lt',
@@ -43,117 +59,66 @@ Use this skill when you need to perform similarity searches on high-dimensional
    	},
    });
    ```
-5. **Generate Embeddings**: Use external services (OpenAI, Ollama) to generate the numeric vectors before storing or searching them in Harper.
-
-```typescript
-import OpenAI from 'openai';
-import ollama from 'ollama';
-
-const { Product } = tables;
-const openai = new OpenAI();
-// the name of the OpenAI embedding model
-const OPENAI_EMBEDDING_MODEL = 'text-embedding-3-small';
-
-// the name of the Ollama embedding model
-const OLLAMA_EMBEDDING_MODEL = 'llama3';
-
-const SIMILARITY_THRESHOLD = 0.5;
-
-export class ProductSearch extends Resource {
-	// based on env variable we choose the appropriate embedding generator
-	generateEmbedding =
-		process.env.EMBEDDING_GENERATOR === 'ollama'
-			? this._generateOllamaEmbedding
-			: this._generateOpenAIEmbedding;
-
-	/**
-	 * Executes a search query using a generated text embedding and returns the matching products.
-	 *
-	 * @param {Object} data - The input data for the request.
-	 * @param {string} data.prompt - The prompt to generate the text embedding from.
-	 * @return {Promise<Array>} Returns a promise that resolves to an array of products matching the conditions,
-	 * including fields: name, description, price, and $distance.
-	 */
-	async post(data) {
-		const embedding = await this.generateEmbedding(data.prompt);
-
-		return await Product.search({
-			select: ['name', 'description', 'price', '$distance'],
-			conditions: {
-				attribute: 'textEmbeddings',
-				comparator: 'lt',
-				value: SIMILARITY_THRESHOLD,
-				target: embedding[0],
-			},
-			limit: 5,
-		});
-	}
-
-	/**
-	 * Generates an embedding using the Ollama API.
-	 *
-	 * @param {string} promptData - The input data for which the embedding is to be generated.
-	 * @return {Promise<number[][]>} A promise that resolves to the generated embedding as an array of numbers.
-	 */
-	async _generateOllamaEmbedding(promptData) {
-		const embedding = await ollama.embed({
-			model: OLLAMA_EMBEDDING_MODEL,
-			input: promptData,
-		});
-		return embedding?.embeddings;
-	}
-
-	/**
-	 * Generates OpenAI embeddings based on the given prompt data.
-	 *
-	 * @param {string} promptData - The input data used for generating the embedding.
-	 * @return {Promise<number[][]>} A promise that resolves to an array of embeddings, where each embedding is an array of floats.
-	 */
-	async _generateOpenAIEmbedding(promptData) {
-		const embedding = await openai.embeddings.create({
-			model: OPENAI_EMBEDDING_MODEL,
-			input: promptData,
-			encoding_format: 'float',
-		});
-
-		let embeddings = [];
-		embedding.data.forEach((embeddingData) => {
-			embeddings.push(embeddingData.embedding);
-		});
-
-		return embeddings;
-	}
-}
-```
 
-## Examples
+5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Works with both `sort`-based and `conditions`-based queries.
 
-Sample request to the `ProductSearch` resource which prompts to find "shorts for the gym":
+   ```javascript
+   let results = Document.search({
+   	select: ['name', '$distance'],
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
+   });
+   ```
 
-```bash
-curl -X POST "http://localhost:9926/ProductSearch/" \
--H "Accept: application/json" \
--H "Content-Type: application/json" \
--H "Authorization: Basic <YOUR_AUTH>" \
--d '{"prompt": "shorts for the gym"}'
-```
+6. **Tune HNSW parameters**: Pass additional parameters to `@indexed(type: "HNSW", ...)` to control index quality and performance.
 
----
+   | Parameter              | Default           | Description                                                                                         |
+   | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------- |
+   | `distance`             | `"cosine"`        | Distance function: `"euclidean"` or `"cosine"` (negative cosine similarity)                         |
+   | `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance    |
+   | `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data |
+   | `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)          |
+   | `mL`                   | computed from `M` | Normalization factor for level generation                                                           |
+   | `efSearchConstruction` | `50`              | Max nodes explored during search                                                                    |
 
-## When to Use Vector Indexing
+## Examples
 
-Vector indexing is ideal when:
+**Schema with custom HNSW parameters:**
 
-- Storing embedding vectors from ML models
-- Performing semantic or similarity-based search
-- Working with high-dimensional numeric data
-- Exact-match indexes are insufficient
+```graphql
+type Document @table {
+	id: Long @primaryKey
+	textEmbeddings: [Float]
+		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efSearchConstruction: 100)
+}
+```
 
----
+**Nearest-neighbor search with distance score:**
+
+```javascript
+let results = Document.search({
+	select: ['name', '$distance'],
+	sort: { attribute: 'textEmbeddings', target: searchVector },
+	limit: 5,
+});
+```
+
+**Distance-threshold filter (no ranking):**
+
+```javascript
+let results = Document.search({
+	conditions: {
+		attribute: 'textEmbeddings',
+		comparator: 'lt',
+		value: 0.1,
+		target: searchVector,
+	},
+});
+```
 
-## Summary
+## Notes
 
-- Vector indexing enables fast similarity search on numeric arrays
-- Defined using `@indexed(type: "HNSW")`
-- Queried using a target vector in search sorting
-- Tunable for performance and accuracy
+- The default `distance` function is `cosine`. Pass `distance: "euclidean"` to switch.
+- `efConstruction` controls index build quality; raising it improves recall at the cost of build time.
+- `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
+- Use the threshold (`conditions` + `target`) form when you want to bound result quality by a similarity cutoff rather than ranking by similarity.

package/.agents/skills/harper-best-practices/rules.manifest.yaml

@@ -0,0 +1,258 @@
+# Rules manifest for the harper-best-practices skill.
+#
+# Declarative source of truth for rule taxonomy, sources, and generation mode.
+# Owned by humans; the generator reads this file and writes derived artifacts
+# (rule bodies, AGENTS.md). See docs/plans/docs-driven-skills.md for the
+# full schema definition and Manifest ↔ frontmatter reconciliation semantics.
+#
+# Phase 2 flipped vector-indexing to mode: generate.
+# Phase 3 flips 7 obvious 1:1 rules to mode: generate (automatic-apis,
+# querying-rest-apis, real-time-apps, checking-authentication, logging,
+# deploying-to-harper-fabric, caching). All others remain synthesized.
+
+rules:
+  # ===========================================================================
+  # Schema & Data Design (priority 1 — HIGH)
+  # ===========================================================================
+
+  - rule: adding-tables-with-schemas
+    description: Guidelines for adding tables to a Harper database using GraphQL schemas.
+    category: schema
+    priority: 1
+    order: 1
+    mode: synthesized
+
+  - rule: schema-design-tooling
+    description: Best practices for Harper schema design, including core directives and GraphQL tooling configuration.
+    category: schema
+    priority: 1
+    order: 2
+    mode: synthesized
+
+  - rule: defining-relationships
+    description: How to define and use relationships between tables in Harper using GraphQL.
+    category: schema
+    priority: 1
+    order: 3
+    mode: synthesized
+
+  - rule: vector-indexing
+    description: How to enable and query vector indexes for similarity search in Harper.
+    category: schema
+    priority: 1
+    order: 4
+    mode: generate
+    sources:
+      - path: reference/v5/database/schema.md
+        section: 'Vector Indexing'
+        role: primary
+    must_cover:
+      - '@indexed(type: "HNSW")'
+      - 'sort'
+      - 'target'
+      - 'cosine'
+      - 'efConstruction'
+    cross_links:
+      - adding-tables-with-schemas
+
+  - rule: using-blob-datatype
+    description: How to use the Blob data type for efficient binary storage in Harper.
+    category: schema
+    priority: 1
+    order: 5
+    mode: synthesized
+
+  - rule: handling-binary-data
+    description: How to store and serve binary data like images or audio in Harper.
+    category: schema
+    priority: 1
+    order: 6
+    mode: synthesized
+
+  # ===========================================================================
+  # API & Communication (priority 2 — HIGH)
+  # ===========================================================================
+
+  - rule: automatic-apis
+    description: How to use Harper's automatically generated REST and WebSocket APIs.
+    category: api
+    priority: 2
+    order: 1
+    mode: generate
+    sources:
+      - path: reference/v5/rest/overview.md
+        role: primary
+      - path: reference/v5/rest/websockets.md
+        role: supplemental
+    must_cover:
+      - 'rest: true'
+      - '@export'
+      - 'WebSocket'
+    cross_links:
+      - querying-rest-apis
+      - real-time-apps
+
+  - rule: querying-rest-apis
+    description: How to use query parameters to filter, sort, and paginate Harper REST APIs.
+    category: api
+    priority: 2
+    order: 2
+    mode: generate
+    sources:
+      - path: reference/v5/rest/querying.md
+        role: primary
+    must_cover:
+      - '=gt='
+      - '=lt='
+      - 'select('
+      - 'sort('
+      - 'limit('
+    cross_links:
+      - automatic-apis
+
+  - rule: real-time-apps
+    description: How to build real-time features in Harper using WebSockets and Pub/Sub.
+    category: api
+    priority: 2
+    order: 3
+    mode: generate
+    sources:
+      - path: reference/v5/rest/websockets.md
+        role: primary
+    must_cover:
+      - 'WebSocket'
+      - 'connect('
+    cross_links:
+      - automatic-apis
+
+  - rule: checking-authentication
+    description: How to handle user authentication and sessions in Harper Resources.
+    category: api
+    priority: 2
+    order: 4
+    mode: generate
+    sources:
+      - path: reference/v5/resources/resource-api.md
+        section: '`getCurrentUser(): User | undefined`'
+        role: primary
+      - path: reference/v5/resources/resource-api.md
+        section: 'Session and Login from a Resource'
+        role: primary
+      - path: reference/v5/security/jwt-authentication.md
+        role: supplemental
+    must_cover:
+      - 'getCurrentUser()'
+      - 'getContext()'
+      - 'context.login'
+      - 'enableSessions'
+    cross_links:
+      - custom-resources
+
+  # ===========================================================================
+  # Logic & Extension (priority 3 — MEDIUM)
+  # ===========================================================================
+
+  - rule: custom-resources
+    description: How to define custom REST endpoints with JavaScript or TypeScript in Harper.
+    category: logic
+    priority: 3
+    order: 1
+    mode: synthesized
+
+  - rule: extending-tables
+    description: How to add custom logic to automatically generated table resources in Harper.
+    category: logic
+    priority: 3
+    order: 2
+    mode: synthesized
+
+  - rule: programmatic-table-requests
+    description: How to interact with Harper tables programmatically using the `tables` object.
+    category: logic
+    priority: 3
+    order: 3
+    mode: synthesized
+
+  - rule: typescript-type-stripping
+    description: How to run TypeScript files directly in Harper without a build step.
+    category: logic
+    priority: 3
+    order: 4
+    mode: synthesized
+
+  - rule: caching
+    description: How to implement integrated data caching in Harper from external sources.
+    category: logic
+    priority: 3
+    order: 5
+    mode: generate
+    sources:
+      - path: learn/developers/caching-with-harper.md
+        role: primary
+    must_cover:
+      - 'expiration'
+      - 'sourcedFrom'
+      - '@table'
+    cross_links:
+      - custom-resources
+      - automatic-apis
+
+  # ===========================================================================
+  # Infrastructure & Ops (priority 4 — MEDIUM)
+  # ===========================================================================
+
+  - rule: deploying-to-harper-fabric
+    description: How to deploy a Harper application to the Harper Fabric cloud.
+    category: ops
+    priority: 4
+    order: 1
+    mode: generate
+    sources:
+      - path: reference/v5/components/applications.md
+        section: 'Remote Management'
+        role: primary
+      - path: fabric/cluster-creation-management.md
+        section: 'Connecting the Harper CLI to a Cluster'
+        role: supplemental
+    must_cover:
+      - 'harper deploy'
+      - 'harper login'
+    cross_links:
+      - creating-a-fabric-account-and-cluster
+
+  - rule: creating-a-fabric-account-and-cluster
+    description: How to create a Harper Fabric account, organization, and cluster.
+    category: ops
+    priority: 4
+    order: 2
+    mode: synthesized
+
+  - rule: creating-harper-apps
+    description: How to initialize a new Harper application using the CLI.
+    category: ops
+    priority: 4
+    order: 3
+    mode: synthesized
+
+  - rule: serving-web-content
+    description: How to serve static files and integrated Vite/React applications in Harper.
+    category: ops
+    priority: 4
+    order: 4
+    mode: synthesized
+
+  - rule: logging
+    description: Best practices for logging in Harper, including console capture, the granular logger interface, and programmatic log retrieval.
+    category: ops
+    priority: 4
+    order: 5
+    mode: generate
+    sources:
+      - path: reference/v5/logging/overview.md
+        role: primary
+      - path: reference/v5/logging/api.md
+        role: primary
+    must_cover:
+      - 'console.log'
+      - 'logger'
+      - 'withTag('

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@harperfast/template-react-studio",
-	"version": "1.6.3",
+	"version": "1.6.4",
 	"type": "module",
 	"repository": "github:HarperFast/create-harper",
 	"scripts": {},