@harperfast/template-vue-ts-studio 1.9.1 → 1.10.0

package/.agents/skills/harper-best-practices/rules/using-blob-datatype.md

@@ -2,37 +2,117 @@
 name: using-blob-datatype
 description: How to use the Blob data type for efficient binary storage in Harper.
 metadata:
-  mode: synthesized
+  mode: generate
+  sources:
+    - reference/v5/database/schema.md#Blob Type
+    - reference/v5/database/api.md#Streaming
+    - reference/v5/database/api.md#`BlobOptions`
+    - reference/v5/database/api.md#Blob Coercion
+  sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
+  inputHash: 71a3e738ebf87fa4
 ---
 
-# Using Blob Datatype
+# Using the Blob Data Type
 
-Instructions for the agent to follow when working with the Blob data type in Harper.
+Instructions for the agent to follow when storing and retrieving large binary content using Harper's `Blob` data type.
 
 ## When to Use
 
-Use this skill when you need to store unstructured or large binary data (media, documents) that is too large for standard JSON fields. Blobs provide efficient storage and integrated streaming support.
+Apply this rule when a schema field needs to store large binary content such as images, video, audio, or large HTML — typically content larger than 20KB. Use `Blob` instead of `Bytes` when you need streaming support or want to avoid loading the entire value into memory. See [handling-binary-data.md](handling-binary-data.md) for broader binary data guidance.
 
 ## How It Works
 
-1. **Define Blob Fields**: In your GraphQL schema, use the `Blob` type:
+1. **Declare a `Blob` field in your schema**: Add a field typed as `Blob` to a `@table` type.
+
    ```graphql
    type MyTable @table {
-   	id: ID @primaryKey
+   	id: Any! @primaryKey
    	data: Blob
    }
    ```
-2. **Create and Store Blobs**: Use `createBlob()` from Harper's globals to wrap Buffers or Streams:
+
+2. **Create a blob with `createBlob()`**: Pass a buffer, string, or stream as the first argument. Pass a `BlobOptions` object as the second argument to configure behavior.
+
+   ```javascript
+   let blob = createBlob(largeBuffer);
+   await MyTable.put({ id: 'my-record', data: blob });
+   ```
+
+3. **Read blob data using standard Web API methods**: The `Blob` type implements the Web API `Blob` interface. Use `.bytes()`, `.text()`, `.arrayBuffer()`, `.stream()`, or `.slice()` to access content.
+
+   ```javascript
+   let record = await MyTable.get('my-record');
+   let buffer = await record.data.bytes(); // ArrayBuffer
+   let text = await record.data.text(); // string
+   let stream = record.data.stream(); // ReadableStream
+   ```
+
+4. **Use `saveBeforeCommit` for ACID-compliant writes**: By default, blobs are not ACID-compliant — a record can reference a blob before it is fully written. Set `saveBeforeCommit: true` to wait for the full write before the transaction commits.
+
    ```javascript
-   import { tables } from 'harper';
-   const blob = createBlob(largeBuffer);
-   await tables.MyTable.put('my-id', { data: blob });
+   let blob = createBlob(stream, { saveBeforeCommit: true });
+   await MyTable.put({ id: 'my-record', data: blob });
+   // put() resolves only after blob is fully written and record is committed
    ```
-3. **Use Streaming (Optional)**: For very large files, pass a stream to `createBlob()` to avoid loading the entire file into memory.
-4. **Read Blob Data**: Retrieve the record and use `.bytes()` or streaming interfaces on the blob field:
+
+5. **Register an error handler when returning a blob via REST**: Interrupted streams must be handled explicitly to avoid stale records.
+
    ```javascript
-   const record = await tables.MyTable.get('my-id');
-   const buffer = await record.data.bytes();
+   export class MyEndpoint extends MyTable {
+   	static async get(target) {
+   		const record = super.get(target);
+   		let blob = record.data;
+   		blob.on('error', () => {
+   			MyTable.invalidate(target);
+   		});
+   		return { status: 200, headers: {}, body: blob };
+   	}
+   }
    ```
-5. **Ensure Write Completion**: Use `saveBeforeCommit: true` in `createBlob` options if you need the blob fully written before the record is committed.
-6. **Handle Errors**: Attach error listeners to the blob object to handle streaming failures.
+
+6. **Rely on automatic coercion where applicable**: When a field is typed as `Blob` in the schema, any string or buffer assigned via `put`, `patch`, or `publish` is automatically coerced to a `Blob`. Manual `createBlob()` calls are not required for plain JSON HTTP bodies or MQTT messages in most cases.
+
+### `BlobOptions` Reference
+
+| Option             | Type      | Default     | Description                                                                                                              |
+| ------------------ | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `type`             | `string`  | `undefined` | MIME type to associate with the blob (e.g., `image/jpeg`). Readable via `blob.type` and used when serving HTTP.          |
+| `size`             | `number`  | `undefined` | Size of the data in bytes, if known ahead of time. Otherwise inferred from a buffer or determined as a stream completes. |
+| `saveBeforeCommit` | `boolean` | `false`     | Wait for the blob to be fully written before committing the transaction.                                                 |
+| `compress`         | `boolean` | `false`     | Compress the stored data with deflate.                                                                                   |
+| `flush`            | `boolean` | `false`     | Flush the file to disk after writing, before the `createBlob` promise chain resolves.                                    |
+
+## Examples
+
+**Store an image with a MIME type:**
+
+```javascript
+let blob = createBlob(imageBuffer, { type: 'image/jpeg' });
+await Photo.put({ id, data: blob });
+```
+
+**Stream large media with low latency:**
+
+```javascript
+let blob = createBlob(incomingStream);
+// blob exists, but data is still streaming to storage
+await MyTable.put({ id: 'my-record', data: blob });
+
+let record = await MyTable.get('my-record');
+// blob data is accessible as it arrives
+let outgoingStream = record.data.stream();
+```
+
+**Guaranteed write before commit:**
+
+```javascript
+let blob = createBlob(stream, { saveBeforeCommit: true });
+await MyTable.put({ id: 'my-record', data: blob });
+```
+
+## Notes
+
+- `Blob` stores data separately from the record; `Bytes` does not. Prefer `Blob` for content larger than 20KB.
+- All standard Web API `Blob` methods are available: `.bytes()`, `.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`.
+- Blobs are **not** ACID-compliant by default when created from a stream. Use `saveBeforeCommit: true` to enforce transactional consistency.
+- Always attach an `error` handler on blobs returned as HTTP response bodies to handle interrupted streams.

package/.agents/skills/harper-best-practices/rules/vector-indexing.md

@@ -5,21 +5,21 @@ metadata:
   mode: generate
   sources:
     - reference/v5/database/schema.md#Vector Indexing
-  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
-  inputHash: 3732961c671aac00
+  sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
+  inputHash: d90b1b74597d08a6
 ---
 
 # Vector Indexing
 
-Instructions for the agent to follow when enabling and querying vector indexes for similarity search in Harper using the HNSW algorithm.
+Instructions for the agent to enable HNSW vector indexes on table fields and query them for similarity search in Harper.
 
 ## When to Use
 
-Apply this rule when adding a vector index to a Harper table schema or writing similarity search queries against high-dimensional vector fields. Use it whenever you need approximate nearest-neighbor search, distance-threshold filtering, or distance-scored results.
+Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or tuning index construction and search parameters. Use it alongside [adding-tables-with-schemas.md](adding-tables-with-schemas.md) when defining the schema that hosts the vector field.
 
 ## How It Works
 
-1. **Declare the vector index on a `[Float]` field**: Add `@indexed(type: "HNSW")` to any `[Float]` attribute in a `@table` type. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for general schema setup.
+1. **Declare the vector index on a field**: Add `@indexed(type: "HNSW")` to a `[Float]` field inside a `@table` type. This creates an HNSW (Hierarchical Navigable Small World) index for approximate nearest-neighbor search.
 
    ```graphql
    type Document @table {
@@ -28,7 +28,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
    }
    ```
 
-2. **Query by nearest neighbors using `sort`**: Call `Document.search()` with a `sort` object containing `attribute` (the indexed field name) and `target` (the query vector). Include `limit` to cap results.
+2. **Query by nearest neighbors using `sort`**: Call `.search()` with a `sort` descriptor that specifies the indexed `attribute` and a `target` vector. Use `limit` to cap results.
 
    ```javascript
    let results = Document.search({
@@ -47,7 +47,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
    });
    ```
 
-4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. Omit `sort`.
+4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. This bounds result quality rather than ranking by similarity.
 
    ```javascript
    let results = Document.search({
@@ -60,7 +60,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
    });
    ```
 
-5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Works with both `sort`-based and `conditions`-based queries.
+5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Available in both `sort`-based and threshold-based queries.
 
    ```javascript
    let results = Document.search({
@@ -70,43 +70,59 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
    });
    ```
 
-6. **Tune HNSW parameters**: Pass additional parameters to `@indexed(type: "HNSW", ...)` to control index quality and performance.
+6. **Tune per-query search options**: Pass `distance` and `ef` directly on the `sort` descriptor to override index defaults for a single query.
 
-   | Parameter              | Default           | Description                                                                                         |
-   | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------- |
-   | `distance`             | `"cosine"`        | Distance function: `"euclidean"` or `"cosine"` (negative cosine similarity)                         |
-   | `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance    |
-   | `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data |
-   | `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)          |
-   | `mL`                   | computed from `M` | Normalization factor for level generation                                                           |
-   | `efSearchConstruction` | `50`              | Max nodes explored during search                                                                    |
+   ```javascript
+   let results = Document.search({
+   	sort: { attribute: 'textEmbeddings', target: searchVector, distance: 'dotProduct', ef: 200 },
+   	limit: 5,
+   });
+   ```
+
+   - `distance` — overrides the distance function for this query: `"cosine"`, `"euclidean"`, or `"dotProduct"`.
+   - `ef` — overrides the search exploration budget. Higher values improve recall at the cost of latency.
+
+7. **Configure HNSW index parameters**: Pass parameters directly in the `@indexed` directive. Structural parameters (`distance`, `M`, `efConstruction`, `quantization`) trigger an index rebuild when changed; `efConstructionSearch` does not.
+
+   ```graphql
+   type Document @table {
+   	id: Long @primaryKey
+   	textEmbeddings: [Float]
+   		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
+   }
+   ```
+
+8. **Enable vector quantization**: Use `quantization: "int8"` to store vectors as 8-bit integers, reducing index size and memory usage. Harper re-ranks nearest-neighbor `sort` results against full-precision vectors automatically.
+
+   ```graphql
+   type Document @table {
+   	id: Long @primaryKey
+   	textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
+   }
+   ```
 
 ## Examples
 
-**Schema with custom HNSW parameters:**
+Full schema with custom HNSW parameters and a nearest-neighbor query with distance output:
 
 ```graphql
 type Document @table {
 	id: Long @primaryKey
 	textEmbeddings: [Float]
-		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efSearchConstruction: 100)
+		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
 }
 ```
 
-**Nearest-neighbor search with distance score:**
-
 ```javascript
+// Nearest-neighbor search with distance scores
 let results = Document.search({
 	select: ['name', '$distance'],
 	sort: { attribute: 'textEmbeddings', target: searchVector },
 	limit: 5,
 });
-```
 
-**Distance-threshold filter (no ranking):**
-
-```javascript
-let results = Document.search({
+// Distance-threshold query (no ranking)
+let closeMatches = Document.search({
 	conditions: {
 		attribute: 'textEmbeddings',
 		comparator: 'lt',
@@ -118,7 +134,21 @@ let results = Document.search({
 
 ## Notes
 
-- The default `distance` function is `cosine`. Pass `distance: "euclidean"` to switch.
-- `efConstruction` controls index build quality; raising it improves recall at the cost of build time.
+### HNSW Parameters
+
+| Parameter              | Default           | Description                                                                                            |
+| ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
+| `distance`             | `"cosine"`        | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"`                                        |
+| `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance       |
+| `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data    |
+| `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)             |
+| `mL`                   | computed from `M` | Normalization factor for level generation                                                              |
+| `efConstructionSearch` | auto-scaled       | Max nodes explored during search. When unset, auto-scales with index size; setting it fixes the budget |
+| `quantization`         | —                 | `"int8"` stores vectors quantized to int8                                                              |
+
+- The `distance` option on a per-query `sort` descriptor accepts `"cosine"`, `"euclidean"`, or `"dotProduct"`.
+- When no `ef` is passed and `efConstructionSearch` (or `efConstruction`) is not explicitly set on the index, the search budget auto-scales with index size.
+- `efConstruction` seeds the initial value of `efConstructionSearch`; setting either one fixes the search budget.
+- The correct parameter name is `efConstructionSearch` (not `efSearchConstruction`).
 - `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
-- Use the threshold (`conditions` + `target`) form when you want to bound result quality by a similarity cutoff rather than ranking by similarity.
+- For `quantization: "int8"`, distance-threshold (`lt`/`le`) queries filter on approximate distance; `sort` queries re-rank against full-precision vectors.

package/.agents/skills/harper-best-practices/rules.manifest.yaml

@@ -2,7 +2,7 @@
 #
 # Declarative source of truth for rule taxonomy, sources, and generation mode.
 # Owned by humans; the generator reads this file and writes derived artifacts
-# (rule bodies, AGENTS.md). See docs/plans/docs-driven-skills.md for the
+# (rule bodies, AGENTS.md). See docs/plans-archive/docs-driven-skills.md for the
 # full schema definition and Manifest ↔ frontmatter reconciliation semantics.
 #
 # Phase 2 flipped vector-indexing to mode: generate.
@@ -53,7 +53,20 @@ rules:
     category: schema
     priority: 1
     order: 3
-    mode: synthesized
+    mode: generate
+    sources:
+      - path: reference/v5/database/schema.md
+        section: 'Relationships'
+        role: primary
+      - path: reference/v5/rest/querying.md
+        section: 'Relationships and Joins'
+        role: supplemental
+    must_cover:
+      - '@relationship'
+      - 'from'
+      - 'to'
+      - 'one-to-many'
+      - 'many-to-one'
 
   - rule: vector-indexing
     description: How to enable and query vector indexes for similarity search in Harper.
@@ -79,14 +92,51 @@ rules:
     category: schema
     priority: 1
     order: 5
-    mode: synthesized
+    mode: generate
+    sources:
+      - path: reference/v5/database/schema.md
+        section: 'Blob Type'
+        role: primary
+      - path: reference/v5/database/api.md
+        section: 'Streaming'
+        role: supplemental
+      - path: reference/v5/database/api.md
+        section: '`BlobOptions`'
+        role: supplemental
+      - path: reference/v5/database/api.md
+        section: 'Blob Coercion'
+        role: supplemental
+    must_cover:
+      - 'Blob'
+      - 'createBlob'
+      - 'saveBeforeCommit'
+      - '.bytes()'
+    cross_links:
+      - handling-binary-data
 
   - rule: handling-binary-data
     description: How to store and serve binary data like images or audio in Harper.
     category: schema
     priority: 1
     order: 6
-    mode: synthesized
+    mode: generate
+    sources:
+      - path: reference/v5/database/api.md
+        section: 'Accepting Binary in JSON Requests'
+        role: primary
+      - path: reference/v5/database/api.md
+        section: 'Serving Binary from a Resource'
+        role: primary
+      - path: reference/v5/rest/content-types.md
+        section: 'Storing Arbitrary Content Types'
+        role: supplemental
+    must_cover:
+      - 'createBlob'
+      - 'Content-Type'
+      - 'application/octet-stream'
+      - 'base64'
+    cross_links:
+      - using-blob-datatype
 
   # ===========================================================================
   # API & Communication (priority 2 — HIGH)
@@ -176,14 +226,41 @@ rules:
     category: logic
     priority: 3
     order: 1
-    mode: synthesized
+    mode: generate
+    sources:
+      - path: reference/v5/resources/overview.md
+        section: 'Custom External Data Source'
+        role: primary
+      - path: reference/v5/resources/overview.md
+        section: 'Exporting Resources as Endpoints'
+        role: primary
+      - path: reference/v5/components/javascript-environment.md
+        section: 'Module Loading'
+        role: supplemental
+    must_cover:
+      - 'extends Resource'
+      - 'export class'
+      - 'server.resources.set('
+      - 'static'
 
   - rule: extending-tables
     description: How to add custom logic to automatically generated table resources in Harper.
     category: logic
     priority: 3
     order: 2
-    mode: synthesized
+    mode: generate
+    sources:
+      - path: reference/v5/resources/overview.md
+        section: 'Extending a Table'
+        role: primary
+      - path: reference/v5/resources/resource-api.md
+        section: 'Throwing Errors'
+        role: supplemental
+    must_cover:
+      - 'extends tables.'
+      - 'super.'
+      - '@export'
+      - 'statusCode'
 
   - rule: programmatic-table-requests
     description: How to interact with Harper tables programmatically using the `tables` object.