@harperfast/template-react-ts-studio 1.11.2 → 1.11.4

package/agent/skills/harper-best-practices/rules/v5-upgrade.md

@@ -7,101 +7,137 @@ metadata:
   mode: generate
   sources:
     - release-notes/v5-lincoln/v5-migration.md
-  sourceCommit: de2aaf1c759a7ff5b93e862ba704153e2a392fcb
-  inputHash: 80314689f3a7f42e
+  sourceCommit: 3749d0c54be457a2a65d9a63c738a5dc88989ecd
+  inputHash: 5ee6407370219136
 ---
 
-# Upgrading a Harper Application to v5
+# v5 Upgrade: Breaking Changes and Migration Guide
 
-Instructions for the agent to follow when migrating an existing Harper application to version 5, addressing the breaking changes and adopting the recommended patterns.
+Instructions for the agent to apply when migrating a Harper application to v5, covering all breaking changes and required code updates.
 
 ## When to Use
 
-Apply this rule when upgrading a Harper application from v4.x (or earlier) to v5, or when code written against older APIs behaves differently after a v5 upgrade — for example a record that can no longer be mutated, a query that returns stale data, or a `spawn` call that now throws. Harper v5 introduces breaking changes; applications built on documented APIs need the updates below, while code relying on undocumented behavior or timing may need broader review.
+Apply this rule when upgrading an existing Harper application to v5, when encountering runtime errors related to renamed packages, changed APIs, or security restrictions introduced in v5, or when reviewing application code for v5 compatibility before deployment.
 
 ## How It Works
 
-1. **Adopt the `harper` package name**: HarperDB is now Harper, and the package was renamed. Install the open source edition with `npm i -g harper` and the pro edition with `npm i -g @harperfast/harper-pro`. Update application imports from `harperdb` to `harper`:
+1. **Update the package import from `harperdb` to `harper`**: All application code must import from `harper` instead of `harperdb`.
 
    ```javascript
    import { tables } from 'harper';
    ```
 
-2. **Opt in to install scripts when required**: Harper now installs packages with `--ignore-scripts` to guard against accidental script execution, a common security risk. If an application genuinely needs install scripts to run (for example to build native binaries), pass the `allowInstallScripts` option when deploying.
+2. **Enable `allowInstallScripts` if packages require install scripts**: Harper v5 uses `--ignore-scripts` by default when installing packages. If your application requires installation scripts (e.g., to install additional binaries), set the `allowInstallScripts` option when deploying.
 
-3. **Update `Table.get` return-value handling**: `Table.get` now returns a plain, frozen record object rather than an instance of the table class, so instance methods are no longer available on the result. The commonly used `wasLoadedFromSource()` method is gone; cache-vs-origin information now lives on the request `target` as `target.loadedFromSource`. Pass a `RequestTarget` to `Table.get` and read the flag from it. Because the record is frozen, create a copy instead of mutating it in place.
+3. **Update `Table.get` usage — return value is now a frozen record object**: `Table.get` now returns a plain record object, not a table class instance. The record is frozen; you cannot mutate it directly.
+   - Replace `wasLoadedFromSource()` with `target.loadedFromSource`:
 
-4. **Account for automatic transaction context**: With RocksDB, transactions are fully supported by the storage engine, and Harper v5 uses asynchronous context tracking to carry the current transaction across calls automatically. A nested `Table.get` now reads within the current transaction's snapshot, so it will not observe newly written data until you commit. Access the context through `getContext()` and either commit the current transaction (`getContext().transaction.commit()`) or run the read inside a fresh `transaction(...)` to see the latest data. This matters most for code executing outside the context of a Harper request.
+     ```javascript
+     // Old — remove this pattern:
+     const record = await Table.get(id);
+     if (record.wasLoadedFromSource()) { ... }
 
-5. **Register spawnable commands**: Spawning processes is now tightly controlled. `spawn`, `exec`, and `execFile` may only run executables listed in the `applications.allowedSpawnCommands` configuration, and each call must pass a `name` in its `options` so that only a single named process is started across Harper's multiple threads. Use a distinct `name` when you deliberately need a separate process.
+     // New — use loadedFromSource on the target:
+     const target = new RequestTarget();
+     target.id = id;
+     const record = await Table.get(target);
+     if (target.loadedFromSource) {
+       // record was loaded from origin (not cache)
+     }
+     ```
 
-6. **Return Response-like objects to set headers**: If a REST method returns an object with a `headers` property, Harper uses it as the response headers.
+   - Replace in-place mutation with object spread, since records are frozen:
 
-7. **Replace `blob.save()`**: The `blob.save()` method has been removed. Pass the `saveBeforeCommit` flag in the options to the `Blob` constructor instead.
+     ```javascript
+     // Old — throws in v5:
+     const record = await Table.get(id);
+     record.property = 'changed';
 
-8. **Configure the VM module loader**: v5 loads application modules through Node.js's VM module API, giving each application its own module cache and an application-scoped `harper` module. All of this is controlled by the `applications` section of `harperdb-config.yaml`:
-   - `moduleLoader` — `vm-current-context` (default), `vm`, `native`, or `compartment`. The default shares intrinsics with Harper for best compatibility; choose `vm` only if you need per-application intrinsics, and `native` to disable the VM loader entirely (restores pre-v5 loading, but per-app `logger`/`config` context is unavailable).
-   - `lockdown` — `freeze-after-load` (default), `freeze`, `ses`, or `none`. Freezing intrinsics prevents prototype-pollution attacks; set `lockdown: none` only as a temporary workaround if a dependency mutates built-in prototypes.
-   - `allowedDirectory` — `app` (default) restricts module loading to the application's own directory tree; set `any` if the app must load files from outside it in production.
-   - `allowedBuiltinModules` — an optional allowlist restricting which Node.js built-ins the application may import (all are allowed if omitted).
-   - `dependencyLoader` — `auto` (default), `app`, or `native`, controlling how npm dependencies are loaded through or around the VM loader.
+     // New — copy instead of mutate:
+     let record = await Table.get(id);
+     record = { ...record, property: 'changed' };
+     ```
 
-## Examples
+   - `getUpdatedTime` and `getExpiresAt` methods remain available on the record object.
 
-**Updating `Table.get` and its cache check:**
+4. **Update transaction and context handling**: Harper v5 uses asynchronous context tracking. `Table.get` and other table calls now automatically inherit the current transaction context. Code that previously omitted context to bypass a transaction will no longer work as expected. Use `getContext` (imported from `harper`) to access and commit the current transaction explicitly when you need to see updated data.
 
-```javascript
-// Old (v4.x)
-const record = await Table.get(id);
-if (record.wasLoadedFromSource()) {
-	// loaded from origin, not cache
-}
-
-// New (v5)
-const target = new RequestTarget(); // passed in when overriding `get`
-target.id = id;
-const record = await Table.get(target);
-if (target.loadedFromSource) {
-	// loaded from origin, not cache
-}
-```
+   ```javascript
+   import { setTimeout as delay } from 'node:timers/promises';
+   import { getContext, transaction } from 'harper';
+
+   class MyResource {
+   	static async get(target) {
+   		// Commit the current transaction to read latest data:
+   		await getContext().transaction.commit();
+   		// Optionally wrap each poll in a new transaction:
+   		while ((await transaction(() => Table.get(target))).status !== 'ready') {
+   			await delay(100);
+   		}
+   		return Table.get(target);
+   	}
+   }
+   ```
 
-**Committing the transaction to read fresh data:**
+5. **Register allowed spawn commands via `allowedSpawnCommands`**: Any use of `spawn`, `exec`, or `execFile` from `node:child_process` must reference executables listed in `applications.allowedSpawnCommands` in `harperdb-config.yaml`. Provide a `name` property in the options argument to ensure only a single named process is started across threads.
 
-```javascript
-import { setTimeout as delay } from 'node:timers/promises';
-import { getContext, transaction } from 'harper';
-
-class MyResource {
-	static async get(target) {
-		// The current transaction is a consistent snapshot; commit it to see updates.
-		await getContext().transaction.commit();
-		while ((await transaction(() => Table.get(target))).status !== 'ready') {
-			delay(100);
-		}
-		return Table.get(target);
-	}
-}
-```
+6. **Replace `blob.save()` with `saveBeforeCommit`**: The `blob.save()` method has been removed. Use the `saveBeforeCommit` flag in the options passed to the `Blob` constructor instead.
 
-**Module loader and security configuration in `harperdb-config.yaml`:**
+7. **Configure the `moduleLoader` and `lockdown` settings**: Harper v5 loads application modules through Node.js's VM module API. Control this behavior in `harperdb-config.yaml` under the `applications` key.
+
+   | Setting            | Default              | Options                                             |
+   | ------------------ | -------------------- | --------------------------------------------------- |
+   | `moduleLoader`     | `vm-current-context` | `vm-current-context`, `vm`, `native`, `compartment` |
+   | `lockdown`         | `freeze-after-load`  | `freeze-after-load`, `freeze`, `ses`, `none`        |
+   | `dependencyLoader` | `auto`               | `auto`, `app`, `native`                             |
+   | `allowedDirectory` | `app`                | `app`, `any`                                        |
+   - Use `moduleLoader: native` to disable the VM loader entirely and restore pre-v5 behavior (application-specific context such as tagged logging and per-app `config` will not be available).
+   - Use `lockdown: none` as a temporary workaround if a dependency modifies intrinsic prototypes at runtime and throws a `TypeError`.
+
+## Examples
+
+### Full `harperdb-config.yaml` `applications` block
 
 ```yaml
 applications:
-  lockdown: freeze-after-load # default
-  moduleLoader: vm-current-context # vm-current-context (default) | vm | native | compartment
-  dependencyLoader: auto # auto (default) | app | native
-  allowedDirectory: app # app (default) | any
+  lockdown: freeze-after-load
+  moduleLoader: vm-current-context
+  dependencyLoader: auto
+  allowedDirectory: app
   allowedSpawnCommands:
     - npm
     - node
-  # allowedBuiltinModules: [] # if omitted, all Node.js built-ins are allowed
 ```
 
-## Recommended Changes
+### Restricting allowed built-in modules
+
+```yaml
+applications:
+  allowedBuiltinModules:
+    - fs
+    - path
+    - http
+```
+
+### Disabling the VM loader for package compatibility
+
+```yaml
+applications:
+  moduleLoader: native
+```
+
+### Accessing context and committing a transaction
+
+```javascript
+import { getContext } from 'harper';
+
+await getContext().transaction.commit();
+```
 
-Beyond the required fixes above, Harper v5 encourages these patterns for new and migrated code:
+## Notes
 
-- Implement endpoints with `static` methods on Resources/Tables, reading request information from the request `target` argument (or from `getContext()`).
-- Rely on automatic context propagation rather than threading context through every call manually; access it via `getContext()` exported from `harper`.
-- Access Harper functions and APIs through the `harper` package rather than through global variables.
+- The `lockdown: freeze-after-load` default freezes `Object`, `Array`, `Promise`, `Map`, `Set`, and other intrinsics after all application code loads. Any code or dependency that mutates intrinsic prototypes after startup will throw a `TypeError`.
+- Under `lockdown: ses`, the constrained `fetch` (https-only) is applied only in `vm` mode. In `vm-current-context` and `native` modes, application code uses the standard global `fetch`.
+- In production, `allowedDirectory: app` prevents loading modules from outside the application's own directory tree. Set `allowedDirectory: any` only if your application legitimately requires it.
+- `dependencyLoader: native` is a narrower alternative to `moduleLoader: native` — it uses native loading only for npm packages while keeping the VM loader for first-party application source files.
+- Access Harper functions and APIs through the `harper` package rather than through global variables. Use `getContext` from `harper` to access request context without passing it explicitly through every call.

package/agent/skills/harper-best-practices/rules/vector-indexing.md

@@ -5,8 +5,8 @@ metadata:
   mode: generate
   sources:
     - reference/v5/database/schema.md#Vector Indexing
-  sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
-  inputHash: d90b1b74597d08a6
+  sourceCommit: 3749d0c54be457a2a65d9a63c738a5dc88989ecd
+  inputHash: 225b9bea420d9adc
 ---
 
 # Vector Indexing
@@ -15,11 +15,11 @@ Instructions for the agent to enable HNSW vector indexes on table fields and que
 
 ## When to Use
 
-Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or tuning index construction and search parameters. Use it alongside [adding-tables-with-schemas.md](adding-tables-with-schemas.md) when defining the schema that hosts the vector field.
+Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or combining vector search with record-level access control. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for how to define the surrounding table schema.
 
 ## How It Works
 
-1. **Declare the vector index on a field**: Add `@indexed(type: "HNSW")` to a `[Float]` field inside a `@table` type. This creates an HNSW (Hierarchical Navigable Small World) index for approximate nearest-neighbor search.
+1. **Declare the vector index** on a `[Float]` field using `@indexed(type: "HNSW")`:
 
    ```graphql
    type Document @table {
@@ -28,7 +28,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    }
    ```
 
-2. **Query by nearest neighbors using `sort`**: Call `.search()` with a `sort` descriptor that specifies the indexed `attribute` and a `target` vector. Use `limit` to cap results.
+2. **Query nearest neighbors** using the `sort` parameter with `attribute` and `target`:
 
    ```javascript
    let results = Document.search({
@@ -37,7 +37,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-3. **Combine with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
+3. **Combine with filter conditions** to narrow results before or during traversal:
 
    ```javascript
    let results = Document.search({
@@ -47,7 +47,25 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. This bounds result quality rather than ranking by similarity.
+   Conditions are evaluated _during_ graph traversal (predicate-aware search), not after. Very selective conditions are automatically diverted to an exact-scan strategy.
+
+4. **Use a `vectorFilter` function** for predicates not expressible as conditions (JavaScript API only):
+
+   ```javascript
+   let results = Document.search(
+   	{
+   		sort: { attribute: 'textEmbeddings', target: searchVector },
+   		vectorFilter: (record) =>
+   			record.tenantId === context.user.tenantId && record.status === 'published',
+   		limit: 10,
+   	},
+   	context,
+   );
+   ```
+
+   The function receives a frozen candidate record and must return a boolean synchronously. It must be side-effect free and fast — it can run once per candidate visited during traversal (verdicts are memoized per query).
+
+5. **Filter by distance threshold** using `target` on a condition instead of `sort`:
 
    ```javascript
    let results = Document.search({
@@ -60,7 +78,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Available in both `sort`-based and threshold-based queries.
+6. **Include computed distance in results** using the `$distance` field in `select`:
 
    ```javascript
    let results = Document.search({
@@ -70,7 +88,9 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-6. **Tune per-query search options**: Pass `distance` and `ef` directly on the `sort` descriptor to override index defaults for a single query.
+   `$distance` works in both `sort`-based and threshold-based queries.
+
+7. **Override the distance function or exploration budget per query** via options on the `sort` descriptor:
 
    ```javascript
    let results = Document.search({
@@ -79,31 +99,76 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-   - `distance` — overrides the distance function for this query: `"cosine"`, `"euclidean"`, or `"dotProduct"`.
-   - `ef` — overrides the search exploration budget. Higher values improve recall at the cost of latency.
+   - `distance` — `"cosine"`, `"euclidean"`, or `"dotProduct"`.
+   - `ef` — overrides the search exploration budget for this query. Higher values improve recall at the cost of latency.
 
-7. **Configure HNSW index parameters**: Pass parameters directly in the `@indexed` directive. Structural parameters (`distance`, `M`, `efConstruction`, `quantization`) trigger an index rebuild when changed; `efConstructionSearch` does not.
+8. **Tune filtered traversal** when a `vectorFilter` is very selective by raising `ef` or `filterExpansion`:
 
-   ```graphql
-   type Document @table {
-   	id: Long @primaryKey
-   	textEmbeddings: [Float]
-   		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
-   }
+   ```javascript
+   let results = Document.search(
+   	{
+   		sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 },
+   		vectorFilter: (record) => record.category === 'rare',
+   		limit: 10,
+   	},
+   	context,
+   );
    ```
 
-8. **Enable vector quantization**: Use `quantization: "int8"` to store vectors as 8-bit integers, reducing index size and memory usage. Harper re-ranks nearest-neighbor `sort` results against full-precision vectors automatically.
+   Filtered traversal visits at most `ef * filterExpansion` nodes (`filterExpansion` defaults to `24`). If the budget is exhausted before results fill, the search returns what was found rather than erroring.
+
+9. **Configure HNSW index parameters** directly on the directive:
 
    ```graphql
    type Document @table {
    	id: Long @primaryKey
-   	textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
+   	textEmbeddings: [Float]
+   		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
    }
    ```
 
+   | Parameter              | Default           | Description                                                                                      |
+   | ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------ |
+   | `distance`             | `"cosine"`        | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"`                                  |
+   | `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance |
+   | `M`                    | `16`              | Preferred connections per graph layer                                                            |
+   | `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)       |
+   | `mL`                   | computed from `M` | Normalization factor for level generation                                                        |
+   | `efConstructionSearch` | auto-scaled       | Max nodes explored during search. When unset, auto-scales with index size                        |
+   | `quantization`         | —                 | `"int8"` stores vectors quantized to int8                                                        |
+   | `filterExpansion`      | `24`              | Visit-budget multiplier for filtered search                                                      |
+
+   Changing `efConstructionSearch` on an existing index does not trigger a rebuild. Changing structural parameters (`distance`, `M`, `efConstruction`, `quantization`) does rebuild the index.
+
+10. **Enable int8 quantization** to reduce index size and memory usage:
+
+    ```graphql
+    type Document @table {
+    	id: Long @primaryKey
+    	textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
+    }
+    ```
+
+    Graph navigation uses quantized distances. For `sort` queries, Harper re-ranks results against full-precision vectors, restoring exact ordering and exact `$distance` values. Distance-threshold queries filter on the approximate distance.
+
+11. **Implement record-level access control** by overriding `allowRead(user, target, context)` on the table resource. During vector queries the check participates in graph traversal, so a restricted user receives the k nearest records they are allowed to see:
+
+    ```javascript
+    export class Reports extends tables.Reports {
+    	allowRead(user, target, context) {
+    		if (!super.allowRead(user, target, context)) return false;
+    		if (user.role.permission.super_user) return true;
+    		if (this.ownerId == null) return true;
+    		return this.ownerId === user.id;
+    	}
+    }
+    ```
+
+    The check must be synchronous, side-effect free, and fast. `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed).
+
 ## Examples
 
-Full schema with custom HNSW parameters and a nearest-neighbor query with distance output:
+**Full schema with custom HNSW parameters:**
 
 ```graphql
 type Document @table {
@@ -113,16 +178,33 @@ type Document @table {
 }
 ```
 
+**Nearest-neighbor search with distance output:**
+
 ```javascript
-// Nearest-neighbor search with distance scores
 let results = Document.search({
 	select: ['name', '$distance'],
 	sort: { attribute: 'textEmbeddings', target: searchVector },
 	limit: 5,
 });
+```
+
+**Filtered vector search with tuned traversal budget:**
 
-// Distance-threshold query (no ranking)
-let closeMatches = Document.search({
+```javascript
+let results = Document.search(
+	{
+		sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 },
+		vectorFilter: (record) => record.category === 'rare',
+		limit: 10,
+	},
+	context,
+);
+```
+
+**Distance threshold query (no ranking, cutoff only):**
+
+```javascript
+let results = Document.search({
 	conditions: {
 		attribute: 'textEmbeddings',
 		comparator: 'lt',
@@ -134,21 +216,10 @@ let closeMatches = Document.search({
 
 ## Notes
 
-### HNSW Parameters
-
-| Parameter              | Default           | Description                                                                                            |
-| ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
-| `distance`             | `"cosine"`        | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"`                                        |
-| `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance       |
-| `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data    |
-| `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)             |
-| `mL`                   | computed from `M` | Normalization factor for level generation                                                              |
-| `efConstructionSearch` | auto-scaled       | Max nodes explored during search. When unset, auto-scales with index size; setting it fixes the budget |
-| `quantization`         | —                 | `"int8"` stores vectors quantized to int8                                                              |
-
-- The `distance` option on a per-query `sort` descriptor accepts `"cosine"`, `"euclidean"`, or `"dotProduct"`.
-- When no `ef` is passed and `efConstructionSearch` (or `efConstruction`) is not explicitly set on the index, the search budget auto-scales with index size.
-- `efConstruction` seeds the initial value of `efConstructionSearch`; setting either one fixes the search budget.
-- The correct parameter name is `efConstructionSearch` (not `efSearchConstruction`).
-- `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
-- For `quantization: "int8"`, distance-threshold (`lt`/`le`) queries filter on approximate distance; `sort` queries re-rank against full-precision vectors.
+- Use `@indexed(type: "HNSW")` on a `[Float]` field — not on scalar fields.
+- The default `distance` is `"cosine"`. Override it per-index via the directive or per-query via the `sort` descriptor.
+- `efConstruction` seeds the initial value of `efConstructionSearch`. When neither is set, the search budget auto-scales with index size.
+- `vectorFilter` is available from the JavaScript API only; it cannot be expressed in a REST query string.
+- The parameter name Harper reads is `efConstructionSearch` (not `efSearchConstruction`).
+- `$distance` must be listed explicitly in `select` to appear in results.
+- For record-level `allowRead` overrides: collection-scope calls (where `this.ownerId` is `null`) should return `true` to open the connection; per-record filtering happens during query execution and event delivery.

package/agent/skills/harper-best-practices/rules.manifest.yaml

@@ -422,3 +422,12 @@ rules:
       - 'moduleLoader'
       - 'lockdown'
       - 'Table.get'
+
+  - rule: delegating-to-the-built-in-agent
+    description: How to delegate tasks to Harper's built-in agent via the CLI and the agent operations API.
+    category: ops
+    priority: 4
+    order: 8
+    mode: synthesized
+    cross_links:
+      - deploying-to-harper-fabric