@harperfast/template-react-ts-studio 1.11.3 → 1.11.4

package/agent/skills/harper-best-practices/rules/vector-indexing.md

@@ -5,8 +5,8 @@ metadata:
   mode: generate
   sources:
     - reference/v5/database/schema.md#Vector Indexing
-  sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
-  inputHash: d90b1b74597d08a6
+  sourceCommit: 3749d0c54be457a2a65d9a63c738a5dc88989ecd
+  inputHash: 225b9bea420d9adc
 ---
 
 # Vector Indexing
@@ -15,11 +15,11 @@ Instructions for the agent to enable HNSW vector indexes on table fields and que
 
 ## When to Use
 
-Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or tuning index construction and search parameters. Use it alongside [adding-tables-with-schemas.md](adding-tables-with-schemas.md) when defining the schema that hosts the vector field.
+Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or combining vector search with record-level access control. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for how to define the surrounding table schema.
 
 ## How It Works
 
-1. **Declare the vector index on a field**: Add `@indexed(type: "HNSW")` to a `[Float]` field inside a `@table` type. This creates an HNSW (Hierarchical Navigable Small World) index for approximate nearest-neighbor search.
+1. **Declare the vector index** on a `[Float]` field using `@indexed(type: "HNSW")`:
 
    ```graphql
    type Document @table {
@@ -28,7 +28,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    }
    ```
 
-2. **Query by nearest neighbors using `sort`**: Call `.search()` with a `sort` descriptor that specifies the indexed `attribute` and a `target` vector. Use `limit` to cap results.
+2. **Query nearest neighbors** using the `sort` parameter with `attribute` and `target`:
 
    ```javascript
    let results = Document.search({
@@ -37,7 +37,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-3. **Combine with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
+3. **Combine with filter conditions** to narrow results before or during traversal:
 
    ```javascript
    let results = Document.search({
@@ -47,7 +47,25 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. This bounds result quality rather than ranking by similarity.
+   Conditions are evaluated _during_ graph traversal (predicate-aware search), not after. Very selective conditions are automatically diverted to an exact-scan strategy.
+
+4. **Use a `vectorFilter` function** for predicates not expressible as conditions (JavaScript API only):
+
+   ```javascript
+   let results = Document.search(
+   	{
+   		sort: { attribute: 'textEmbeddings', target: searchVector },
+   		vectorFilter: (record) =>
+   			record.tenantId === context.user.tenantId && record.status === 'published',
+   		limit: 10,
+   	},
+   	context,
+   );
+   ```
+
+   The function receives a frozen candidate record and must return a boolean synchronously. It must be side-effect free and fast — it can run once per candidate visited during traversal (verdicts are memoized per query).
+
+5. **Filter by distance threshold** using `target` on a condition instead of `sort`:
 
    ```javascript
    let results = Document.search({
@@ -60,7 +78,7 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Available in both `sort`-based and threshold-based queries.
+6. **Include computed distance in results** using the `$distance` field in `select`:
 
    ```javascript
    let results = Document.search({
@@ -70,7 +88,9 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-6. **Tune per-query search options**: Pass `distance` and `ef` directly on the `sort` descriptor to override index defaults for a single query.
+   `$distance` works in both `sort`-based and threshold-based queries.
+
+7. **Override the distance function or exploration budget per query** via options on the `sort` descriptor:
 
    ```javascript
    let results = Document.search({
@@ -79,31 +99,76 @@ Apply this rule when adding a vector similarity search capability to a Harper ta
    });
    ```
 
-   - `distance` — overrides the distance function for this query: `"cosine"`, `"euclidean"`, or `"dotProduct"`.
-   - `ef` — overrides the search exploration budget. Higher values improve recall at the cost of latency.
+   - `distance` — `"cosine"`, `"euclidean"`, or `"dotProduct"`.
+   - `ef` — overrides the search exploration budget for this query. Higher values improve recall at the cost of latency.
 
-7. **Configure HNSW index parameters**: Pass parameters directly in the `@indexed` directive. Structural parameters (`distance`, `M`, `efConstruction`, `quantization`) trigger an index rebuild when changed; `efConstructionSearch` does not.
+8. **Tune filtered traversal** when a `vectorFilter` is very selective by raising `ef` or `filterExpansion`:
 
-   ```graphql
-   type Document @table {
-   	id: Long @primaryKey
-   	textEmbeddings: [Float]
-   		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
-   }
+   ```javascript
+   let results = Document.search(
+   	{
+   		sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 },
+   		vectorFilter: (record) => record.category === 'rare',
+   		limit: 10,
+   	},
+   	context,
+   );
    ```
 
-8. **Enable vector quantization**: Use `quantization: "int8"` to store vectors as 8-bit integers, reducing index size and memory usage. Harper re-ranks nearest-neighbor `sort` results against full-precision vectors automatically.
+   Filtered traversal visits at most `ef * filterExpansion` nodes (`filterExpansion` defaults to `24`). If the budget is exhausted before results fill, the search returns what was found rather than erroring.
+
+9. **Configure HNSW index parameters** directly on the directive:
 
    ```graphql
    type Document @table {
    	id: Long @primaryKey
-   	textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
+   	textEmbeddings: [Float]
+   		@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
    }
    ```
 
+   | Parameter              | Default           | Description                                                                                      |
+   | ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------ |
+   | `distance`             | `"cosine"`        | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"`                                  |
+   | `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance |
+   | `M`                    | `16`              | Preferred connections per graph layer                                                            |
+   | `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)       |
+   | `mL`                   | computed from `M` | Normalization factor for level generation                                                        |
+   | `efConstructionSearch` | auto-scaled       | Max nodes explored during search. When unset, auto-scales with index size                        |
+   | `quantization`         | —                 | `"int8"` stores vectors quantized to int8                                                        |
+   | `filterExpansion`      | `24`              | Visit-budget multiplier for filtered search                                                      |
+
+   Changing `efConstructionSearch` on an existing index does not trigger a rebuild. Changing structural parameters (`distance`, `M`, `efConstruction`, `quantization`) does rebuild the index.
+
+10. **Enable int8 quantization** to reduce index size and memory usage:
+
+    ```graphql
+    type Document @table {
+    	id: Long @primaryKey
+    	textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
+    }
+    ```
+
+    Graph navigation uses quantized distances. For `sort` queries, Harper re-ranks results against full-precision vectors, restoring exact ordering and exact `$distance` values. Distance-threshold queries filter on the approximate distance.
+
+11. **Implement record-level access control** by overriding `allowRead(user, target, context)` on the table resource. During vector queries the check participates in graph traversal, so a restricted user receives the k nearest records they are allowed to see:
+
+    ```javascript
+    export class Reports extends tables.Reports {
+    	allowRead(user, target, context) {
+    		if (!super.allowRead(user, target, context)) return false;
+    		if (user.role.permission.super_user) return true;
+    		if (this.ownerId == null) return true;
+    		return this.ownerId === user.id;
+    	}
+    }
+    ```
+
+    The check must be synchronous, side-effect free, and fast. `this` is the frozen record during per-record evaluation. A thrown exception denies that record (fail closed).
+
 ## Examples
 
-Full schema with custom HNSW parameters and a nearest-neighbor query with distance output:
+**Full schema with custom HNSW parameters:**
 
 ```graphql
 type Document @table {
@@ -113,16 +178,33 @@ type Document @table {
 }
 ```
 
+**Nearest-neighbor search with distance output:**
+
 ```javascript
-// Nearest-neighbor search with distance scores
 let results = Document.search({
 	select: ['name', '$distance'],
 	sort: { attribute: 'textEmbeddings', target: searchVector },
 	limit: 5,
 });
+```
+
+**Filtered vector search with tuned traversal budget:**
 
-// Distance-threshold query (no ranking)
-let closeMatches = Document.search({
+```javascript
+let results = Document.search(
+	{
+		sort: { attribute: 'textEmbeddings', target: searchVector, ef: 200, filterExpansion: 40 },
+		vectorFilter: (record) => record.category === 'rare',
+		limit: 10,
+	},
+	context,
+);
+```
+
+**Distance threshold query (no ranking, cutoff only):**
+
+```javascript
+let results = Document.search({
 	conditions: {
 		attribute: 'textEmbeddings',
 		comparator: 'lt',
@@ -134,21 +216,10 @@ let closeMatches = Document.search({
 
 ## Notes
 
-### HNSW Parameters
-
-| Parameter              | Default           | Description                                                                                            |
-| ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
-| `distance`             | `"cosine"`        | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"`                                        |
-| `efConstruction`       | `100`             | Max nodes explored during index construction. Higher = better recall, lower = better performance       |
-| `M`                    | `16`              | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data    |
-| `optimizeRouting`      | `0.5`             | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive)             |
-| `mL`                   | computed from `M` | Normalization factor for level generation                                                              |
-| `efConstructionSearch` | auto-scaled       | Max nodes explored during search. When unset, auto-scales with index size; setting it fixes the budget |
-| `quantization`         | —                 | `"int8"` stores vectors quantized to int8                                                              |
-
-- The `distance` option on a per-query `sort` descriptor accepts `"cosine"`, `"euclidean"`, or `"dotProduct"`.
-- When no `ef` is passed and `efConstructionSearch` (or `efConstruction`) is not explicitly set on the index, the search budget auto-scales with index size.
-- `efConstruction` seeds the initial value of `efConstructionSearch`; setting either one fixes the search budget.
-- The correct parameter name is `efConstructionSearch` (not `efSearchConstruction`).
-- `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
-- For `quantization: "int8"`, distance-threshold (`lt`/`le`) queries filter on approximate distance; `sort` queries re-rank against full-precision vectors.
+- Use `@indexed(type: "HNSW")` on a `[Float]` field — not on scalar fields.
+- The default `distance` is `"cosine"`. Override it per-index via the directive or per-query via the `sort` descriptor.
+- `efConstruction` seeds the initial value of `efConstructionSearch`. When neither is set, the search budget auto-scales with index size.
+- `vectorFilter` is available from the JavaScript API only; it cannot be expressed in a REST query string.
+- The parameter name Harper reads is `efConstructionSearch` (not `efSearchConstruction`).
+- `$distance` must be listed explicitly in `select` to appear in results.
+- For record-level `allowRead` overrides: collection-scope calls (where `this.ownerId` is `null`) should return `true` to open the connection; per-record filtering happens during query execution and event delivery.