@harperfast/template-react-studio 1.6.2 → 1.6.4

package/.agents/skills/harper-best-practices/rules/real-time-apps.md

@@ -1,43 +1,102 @@
 ---
 name: real-time-apps
 description: How to build real-time features in Harper using WebSockets and Pub/Sub.
+metadata:
+  mode: generate
+  sources:
+    - reference/v5/rest/websockets.md
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: a8afd4d3a52f77ba
 ---
 
-# Real-time Applications
+# Real-Time Apps with WebSockets and Pub/Sub
 
-Instructions for the agent to follow when building real-time applications in Harper.
+Instructions for the agent to follow when building real-time features in Harper using WebSockets and Pub/Sub.
 
 ## When to Use
 
-Use this skill when you need to stream live updates to clients, implement chat features, or provide real-time data synchronization between the database and a frontend.
+Apply this rule when implementing any feature that requires real-time bidirectional communication, live data streaming, or push-based updates in a Harper application. This includes chat, live dashboards, sensor feeds, and any scenario where clients must receive resource changes as they happen.
 
 ## How It Works
 
-1. **Check Automatic WebSockets**: If you only need to stream table changes, use [Automatic APIs](automatic-apis.md) which provide a WebSocket endpoint for every `@export`ed table.
-2. **Implement `connect` in a Resource**: For custom bi-directional logic, implement the `connect` method.
-3. **Use Pub/Sub**: Use `tables.TableName.subscribe(query)` to listen for specific data changes and stream them to the client.
-4. **Handle SSE**: Ensure your `connect` method gracefully handles cases where `incomingMessages` is null (Server-Sent Events).
-5. **Connect from Client**: Use standard WebSockets (`new WebSocket('wss://...')`) to connect to your resource endpoint. Ensure you use the appropriate scheme (`ws://` for HTTP, `wss://` for HTTPS).
+1. **Enable WebSocket support**: WebSocket support is enabled automatically when the `rest` plugin is enabled. To explicitly disable it, set the following in your config:
 
-## Examples
+   ```yaml
+   rest:
+     webSocket: false
+   ```
 
-### Bi-directional WebSocket Resource
+2. **Connect a client to a resource**: A WebSocket connection to a resource URL automatically subscribes to that resource. When the record changes or a message is published to it, the connection receives the update.
 
-```typescript
-import { Resource, tables } from 'harperdb';
+   ```javascript
+   let ws = new WebSocket('wss://server/my-resource/341');
+   ws.onmessage = (event) => {
+   	let data = JSON.parse(event.data);
+   };
+   ```
 
-export class MySocket extends Resource {
-	async *connect(target, incomingMessages) {
-		// Subscribe to table changes
-		const subscription = await tables.MyTable.subscribe(target);
-		if (!incomingMessages) {
-			return subscription; // SSE mode
-		}
+   `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it.
+
+3. **Implement a custom `connect()` handler**: Override the `connect(incomingMessages)` method on a resource class to control WebSocket behavior. The method must return an async iterable (or generator) that produces messages to send to the client. See [automatic-apis.md](automatic-apis.md) for more on defining resource classes.
+
+4. **Use the default `connect()` for event-style access**: Call `super.connect()` to get a streaming iterable that provides:
+   - A `send(message)` method for pushing outgoing messages
+   - A `close` event for cleanup on disconnect
+
+5. **Handle message ordering in distributed environments**: Harper delivers messages to local subscribers immediately without inter-node coordination delay.
+
+   | Message Type                                             | Behavior                                                                |
+   | -------------------------------------------------------- | ----------------------------------------------------------------------- |
+   | Non-retained (no `retain` flag)                          | Every message delivered in order received; suitable for chat            |
+   | Retained (published with `retain`, or PUT/updated in DB) | Only the latest-timestamp message is kept; suitable for sensor readings |
 
-		// Handle incoming client messages
+6. **Use MQTT over WebSockets** when needed by setting the sub-protocol header:
+   ```
+   Sec-WebSocket-Protocol: mqtt
+   ```
+
+## Examples
+
+**Simple echo server** — override `connect(incomingMessages)` to yield each incoming message back to the client:
+
+```javascript
+export class Echo extends Resource {
+	async *connect(incomingMessages) {
 		for await (let message of incomingMessages) {
-			yield { received: message };
+			yield message; // echo each message back
 		}
 	}
 }
 ```
+
+**Custom connect with timer and event-style access** — use `super.connect()` to get the outgoing stream, push periodic messages, echo incoming messages, and clean up on disconnect:
+
+```javascript
+export class Example extends Resource {
+	connect(incomingMessages) {
+		let outgoingMessages = super.connect();
+
+		let timer = setInterval(() => {
+			outgoingMessages.send({ greeting: 'hi again!' });
+		}, 1000);
+
+		incomingMessages.on('data', (message) => {
+			outgoingMessages.send(message); // echo incoming messages
+		});
+
+		outgoingMessages.on('close', () => {
+			clearInterval(timer);
+		});
+
+		return outgoingMessages;
+	}
+}
+```
+
+## Notes
+
+- WebSocket connections target a resource URL path. By default, connecting to a resource subscribes to changes for that resource.
+- The `connect(incomingMessages)` method **must** return an async iterable or generator; returning a plain value will not work.
+- `super.connect()` returns a streaming iterable with `send(message)` and a `close` event — use this when you need to push messages outside of the incoming message loop.
+- For one-way real-time streaming without bidirectional communication, consider Server-Sent Events instead.
+- For full pub/sub capabilities, Harper also supports MQTT; set `Sec-WebSocket-Protocol: mqtt` to use MQTT over WebSockets.

package/.agents/skills/harper-best-practices/rules/schema-design-tooling.md

@@ -1,6 +1,8 @@
 ---
 name: schema-design-tooling
 description: Best practices for Harper schema design, including core directives and GraphQL tooling configuration.
+metadata:
+  mode: synthesized
 ---
 
 # Schema Design & Tooling
@@ -9,7 +11,7 @@ Harper uses GraphQL schemas to define database tables, relationships, and APIs.
 
 ## Core Harper Directives
 
-Harper extends GraphQL with custom directives that define database behavior. These are typically defined in `node_modules/harperdb/schema.graphql`. If you don't have access to that file, here is a reference of the most important ones:
+Harper extends GraphQL with custom directives that define database behavior. These are typically defined in `node_modules/harper/schema.graphql`. If you don't have access to that file, here is a reference of the most important ones:
 
 ### Table Definition
 
@@ -43,7 +45,7 @@ Create a file named `graphql.config.yml` in your project root with the following
 
 ```yaml
 schema:
-  - 'node_modules/harperdb/schema.graphql'
+  - 'node_modules/harper/schema.graphql'
   - 'schema.graphql'
   - 'schemas/*.graphql'
 ```

package/.agents/skills/harper-best-practices/rules/serving-web-content.md

@@ -1,6 +1,8 @@
 ---
 name: serving-web-content
 description: How to serve static files and integrated Vite/React applications in Harper.
+metadata:
+  mode: synthesized
 ---
 
 # Serving Web Content
@@ -59,7 +61,6 @@ Use this skill when you need to serve a frontend (HTML, CSS, JS, or a React app)
 4. **Deploy for Production**: For Vite apps, use a build script to generate static files into a `web/` folder and deploy them using the static handler pattern. For example, these scripts in a package.json can perform the necessary steps:
    ```json
    "build": "vite build",
-   "deploy": "rm -Rf deploy && npm run build && mkdir deploy && mv web deploy/ && cp -R deploy-template/* deploy/ && cp -R schemas resources deploy/ && dotenv -- npm run deploy:component && rm -Rf deploy",
-   "deploy:component": "(cd deploy && harper deploy_component . project=web restart=rolling replicated=true)"
+   "deploy": "rm -Rf deploy && npm run build && mkdir deploy && mv web deploy/ && cp -R deploy-template/* deploy/ && cp -R schemas resources deploy/ && (cd deploy && harper deploy_component . project=web restart=rolling replicated=true) && rm -Rf deploy",
    ```
    Then in production, the "Static Plugin" option will performantly and securely serve your assets. `npm create harper@latest` scaffolds all of this for you.

package/.agents/skills/harper-best-practices/rules/typescript-type-stripping.md

@@ -1,6 +1,8 @@
 ---
 name: typescript-type-stripping
 description: How to run TypeScript files directly in Harper without a build step.
+metadata:
+  mode: synthesized
 ---
 
 # TypeScript Type Stripping
@@ -17,7 +19,7 @@ Use this skill when you want to write Harper Resources in TypeScript and have th
 2. **Name Files with `.ts`**: Create your resource files in the `resources/` directory with a `.ts` extension.
 3. **Use TypeScript Syntax**: Write your resource classes using standard TypeScript (interfaces, types, etc.).
    ```typescript
-   import { Resource } from 'harperdb';
+   import { Resource } from 'harper';
    export class MyResource extends Resource {
    	async get(): Promise<{ message: string }> {
    		return { message: 'Running TS directly!' };

package/.agents/skills/harper-best-practices/rules/using-blob-datatype.md

@@ -1,6 +1,8 @@
 ---
 name: using-blob-datatype
 description: How to use the Blob data type for efficient binary storage in Harper.
+metadata:
+  mode: synthesized
 ---
 
 # Using Blob Datatype
@@ -22,7 +24,7 @@ Use this skill when you need to store unstructured or large binary data (media,
    ```
 2. **Create and Store Blobs**: Use `createBlob()` from Harper's globals to wrap Buffers or Streams:
    ```javascript
-   import { tables } from 'harperdb';
+   import { tables } from 'harper';
    const blob = createBlob(largeBuffer);
    await tables.MyTable.put('my-id', { data: blob });
    ```

package/.agents/skills/harper-best-practices/rules/vector-indexing.md

@@ -1,40 +1,56 @@
 ---
 name: vector-indexing
 description: How to enable and query vector indexes for similarity search in Harper.
+metadata:
+  mode: generate
+  sources:
+    - reference/v5/database/schema.md#Vector Indexing
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: 3732961c671aac00
 ---
 
 # Vector Indexing
 
-Instructions for the agent to follow when implementing vector search in Harper.
+Instructions for the agent to follow when enabling and querying vector indexes for similarity search in Harper using the HNSW algorithm.
 
 ## When to Use
 
-Use this skill when you need to perform similarity searches on high-dimensional data, such as AI embeddings for semantic search, recommendations, or image retrieval.
+Apply this rule when adding a vector index to a Harper table schema or writing similarity search queries against high-dimensional vector fields. Use it whenever you need approximate nearest-neighbor search, distance-threshold filtering, or distance-scored results.
 
 ## How It Works
 
-1. **Enable Vector Indexing**: In your GraphQL schema, add `@indexed(type: "HNSW")` to a numeric array field:
+1. **Declare the vector index on a `[Float]` field**: Add `@indexed(type: "HNSW")` to any `[Float]` attribute in a `@table` type. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for general schema setup.
+
    ```graphql
-   type Product @table {
-   	id: ID @primaryKey
+   type Document @table {
+   	id: Long @primaryKey
    	textEmbeddings: [Float] @indexed(type: "HNSW")
    }
    ```
-2. **Configure Index Options (Optional)**: Fine-tune the index with parameters like `distance` (`cosine` or `euclidean`), `M`, and `efConstruction`.
-3. **Query with Vector Search**: Use `tables.Table.search()` with a `sort` object containing the `target` vector:
+
+2. **Query by nearest neighbors using `sort`**: Call `Document.search()` with a `sort` object containing `attribute` (the indexed field name) and `target` (the query vector). Include `limit` to cap results.
+
    ```javascript
-   const results = await tables.Product.search({
-     select: ['name', '$distance'],
-     sort: {
-       attribute: 'textEmbeddings',
-       target: [0.1, 0.2, ...], // query vector
-     },
-     limit: 5,
+   let results = Document.search({
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
    });
    ```
-4. **Filter by Distance**: Use `conditions` with a `target` vector and a `comparator` (e.g., `lt`) to return results within a similarity threshold:
+
+3. **Combine with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
+
    ```javascript
-   const results = await tables.Product.search({
+   let results = Document.search({
+   	conditions: [{ attribute: 'price', comparator: 'lt', value: 50 }],
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
+   });
+   ```
+
+4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. Omit `sort`.
+
+   ```javascript
+   let results = Document.search({
    	conditions: {
    		attribute: 'textEmbeddings',
    		comparator: 'lt',
@@ -43,117 +59,66 @@ Use this skill when you need to perform similarity searches on high-dimensional
    	},
    });
    ```
-5. **Generate Embeddings**: Use external services (OpenAI, Ollama) to generate the numeric vectors before storing or searching them in Harper.
-
-```typescript
-import OpenAI from 'openai';
-import ollama from 'ollama';
-
-const { Product } = tables;
-const openai = new OpenAI();
-// the name of the OpenAI embedding model
-const OPENAI_EMBEDDING_MODEL = 'text-embedding-3-small';
-
-// the name of the Ollama embedding model
-const OLLAMA_EMBEDDING_MODEL = 'llama3';
-
-const SIMILARITY_THRESHOLD = 0.5;
-
-export class ProductSearch extends Resource {
-	// based on env variable we choose the appropriate embedding generator
-	generateEmbedding =
-		process.env.EMBEDDING_GENERATOR === 'ollama'
-			? this._generateOllamaEmbedding
-			: this._generateOpenAIEmbedding;
-
-	/**
-	 * Executes a search query using a generated text embedding and returns the matching products.
-	 *
-	 * @param {Object} data - The input data for the request.
-	 * @param {string} data.prompt - The prompt to generate the text embedding from.
-	 * @return {Promise<Array>} Returns a promise that resolves to an array of products matching the conditions,
-	 * including fields: name, description, price, and $distance.
-	 */
-	async post(data) {
-		const embedding = await this.generateEmbedding(data.prompt);
-
-		return await Product.search({
-			select: ['name', 'description', 'price', '$distance'],
-			conditions: {
-				attribute: 'textEmbeddings',
-				comparator: 'lt',
-				value: SIMILARITY_THRESHOLD,
-				target: embedding[0],
-			},
-			limit: 5,
-		});
-	}
-
-	/**
-	 * Generates an embedding using the Ollama API.
-	 *
-	 * @param {string} promptData - The input data for which the embedding is to be generated.
-	 * @return {Promise<number[][]>} A promise that resolves to the generated embedding as an array of numbers.
-	 */
-	async _generateOllamaEmbedding(promptData) {
-		const embedding = await ollama.embed({
-			model: OLLAMA_EMBEDDING_MODEL,
-			input: promptData,
-		});
-		return embedding?.embeddings;
-	}
-
-	/**
-	 * Generates OpenAI embeddings based on the given prompt data.
-	 *
-	 * @param {string} promptData - The input data used for generating the embedding.
-	 * @return {Promise<number[][]>} A promise that resolves to an array of embeddings, where each embedding is an array of floats.
-	 */
-	async _generateOpenAIEmbedding(promptData) {
-		const embedding = await openai.embeddings.create({
-			model: OPENAI_EMBEDDING_MODEL,
-			input: promptData,
-			encoding_format: 'float',
-		});
-
-		let embeddings = [];
-		embedding.data.forEach((embeddingData) => {
-			embeddings.push(embeddingData.embedding);
-		});
-
-		return embeddings;
-	}
-}
-```
 
-## Examples
+5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Works with both `sort`-based and `conditions`-based queries.
 
-Sample request to the `ProductSearch` resource which prompts to find "shorts for the gym":
+   ```javascript
+   let results = Document.search({
+   	select: ['name', '$distance'],
+   	sort: { attribute: 'textEmbeddings', target: searchVector },
+   	limit: 5,
+   });
+   ```
 
-```bash
-curl -X POST "http://localhost:9926/ProductSearch/" \
--H "Accept: application/json" \
--H "Content-Type: application/json" \
--H "Authorization: Basic <YOUR_AUTH>" \
--d '{"prompt": "shorts for the gym"}'
-```
+6. **Tune HNSW parameters**: Pass additional parameters to `@indexed(type: "HNSW", ...)` to control index quality and performance.
 
----
+   | Parameter              | Default           | Description                                                                                         |
+   | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------- |
+   | `distance`             | `"cosine"`        | Distance function: `"euclidean"` or `"cosine"` (negative cosine similarity)                         |
+   | `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance    |
+   | `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data |
+   | `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)          |
+   | `mL`                   | computed from `M` | Normalization factor for level generation                                                           |
+   | `efSearchConstruction` | `50`              | Max nodes explored during search                                                                    |
 
-## When to Use Vector Indexing
+## Examples
 
-Vector indexing is ideal when:
+**Schema with custom HNSW parameters:**
 
-- Storing embedding vectors from ML models
-- Performing semantic or similarity-based search
-- Working with high-dimensional numeric data
-- Exact-match indexes are insufficient
+```graphql
+type Document @table {
+	id: Long @primaryKey
+	textEmbeddings: [Float]
+		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efSearchConstruction: 100)
+}
+```
 
----
+**Nearest-neighbor search with distance score:**
+
+```javascript
+let results = Document.search({
+	select: ['name', '$distance'],
+	sort: { attribute: 'textEmbeddings', target: searchVector },
+	limit: 5,
+});
+```
+
+**Distance-threshold filter (no ranking):**
+
+```javascript
+let results = Document.search({
+	conditions: {
+		attribute: 'textEmbeddings',
+		comparator: 'lt',
+		value: 0.1,
+		target: searchVector,
+	},
+});
+```
 
-## Summary
+## Notes
 
-- Vector indexing enables fast similarity search on numeric arrays
-- Defined using `@indexed(type: "HNSW")`
-- Queried using a target vector in search sorting
-- Tunable for performance and accuracy
+- The default `distance` function is `cosine`. Pass `distance: "euclidean"` to switch.
+- `efConstruction` controls index build quality; raising it improves recall at the cost of build time.
+- `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
+- Use the threshold (`conditions` + `target`) form when you want to bound result quality by a similarity cutoff rather than ranking by similarity.