@harperfast/template-vue-ts-studio 1.9.2 → 1.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/harper-best-practices/AGENTS.md +916 -338
- package/.agents/skills/harper-best-practices/rules/caching.md +68 -62
- package/.agents/skills/harper-best-practices/rules/custom-resources.md +144 -23
- package/.agents/skills/harper-best-practices/rules/defining-relationships.md +152 -22
- package/.agents/skills/harper-best-practices/rules/extending-tables.md +90 -21
- package/.agents/skills/harper-best-practices/rules/handling-binary-data.md +103 -23
- package/.agents/skills/harper-best-practices/rules/programmatic-table-requests.md +143 -91
- package/.agents/skills/harper-best-practices/rules/schema-design-tooling.md +87 -64
- package/.agents/skills/harper-best-practices/rules/serving-web-content.md +28 -0
- package/.agents/skills/harper-best-practices/rules/using-blob-datatype.md +97 -16
- package/.agents/skills/harper-best-practices/rules/vector-indexing.md +59 -29
- package/.agents/skills/harper-best-practices/rules.manifest.yaml +109 -7
- package/agent/skills/harper-best-practices/AGENTS.md +916 -338
- package/agent/skills/harper-best-practices/rules/caching.md +68 -62
- package/agent/skills/harper-best-practices/rules/custom-resources.md +144 -23
- package/agent/skills/harper-best-practices/rules/defining-relationships.md +152 -22
- package/agent/skills/harper-best-practices/rules/extending-tables.md +90 -21
- package/agent/skills/harper-best-practices/rules/handling-binary-data.md +103 -23
- package/agent/skills/harper-best-practices/rules/programmatic-table-requests.md +143 -91
- package/agent/skills/harper-best-practices/rules/schema-design-tooling.md +87 -64
- package/agent/skills/harper-best-practices/rules/serving-web-content.md +28 -0
- package/agent/skills/harper-best-practices/rules/using-blob-datatype.md +97 -16
- package/agent/skills/harper-best-practices/rules/vector-indexing.md +59 -29
- package/agent/skills/harper-best-practices/rules.manifest.yaml +109 -7
- package/package.json +1 -1
- package/skills-lock.json +1 -1
|
@@ -9,11 +9,11 @@ metadata:
|
|
|
9
9
|
- reference/v5/database/schema.md#Overview
|
|
10
10
|
- reference/v5/database/schema.md#Type Directives
|
|
11
11
|
- reference/v5/database/schema.md#Field Directives
|
|
12
|
-
sourceCommit:
|
|
13
|
-
inputHash:
|
|
12
|
+
sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
|
|
13
|
+
inputHash: 15b9c72decc1f05d
|
|
14
14
|
---
|
|
15
15
|
|
|
16
|
-
# Schema Design and Tooling
|
|
16
|
+
# Schema Design and GraphQL Tooling
|
|
17
17
|
|
|
18
18
|
Instructions for the agent to follow when designing Harper schemas, applying core directives, and configuring GraphQL tooling.
|
|
19
19
|
|
|
@@ -23,7 +23,18 @@ Apply this rule when creating or modifying Harper schema files, configuring `gra
|
|
|
23
23
|
|
|
24
24
|
## How It Works
|
|
25
25
|
|
|
26
|
-
1. **Create a
|
|
26
|
+
1. **Create a schema file** using standard GraphQL type definitions with Harper-specific directives. Name it (e.g., `schema.graphql`) and place it in your component directory.
|
|
27
|
+
|
|
28
|
+
2. **Register the schema** in the component's `config.yaml` using the `graphqlSchema` plugin:
|
|
29
|
+
|
|
30
|
+
```yaml
|
|
31
|
+
graphqlSchema:
|
|
32
|
+
files: 'schema.graphql'
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Both plugins and applications can specify schemas.
|
|
36
|
+
|
|
37
|
+
3. **Mark types as tables** with `@table`. The type name becomes the table name by default:
|
|
27
38
|
|
|
28
39
|
```graphql
|
|
29
40
|
type Dog @table {
|
|
@@ -32,23 +43,36 @@ Apply this rule when creating or modifying Harper schema files, configuring `gra
|
|
|
32
43
|
breed: String
|
|
33
44
|
age: Int
|
|
34
45
|
}
|
|
46
|
+
```
|
|
35
47
|
|
|
48
|
+
4. **Designate a primary key** with `@primaryKey` on exactly one field per type. Primary keys must be unique; duplicate-key inserts are rejected. If no primary key is provided on insert, Harper auto-generates one:
|
|
49
|
+
- `String` or `ID` → UUID string
|
|
50
|
+
- `Int`, `Long`, or `Any` → auto-incrementing integer
|
|
51
|
+
|
|
52
|
+
Use `Long` or `Any` for auto-generated numeric keys; `Int` is 32-bit and may be insufficient for large tables.
|
|
53
|
+
|
|
54
|
+
5. **Add secondary indexes** with `@indexed` on any field that will be used for filtering in REST queries, SQL, or NoSQL operations:
|
|
55
|
+
|
|
56
|
+
```graphql
|
|
36
57
|
type Breed @table {
|
|
37
58
|
id: Long @primaryKey
|
|
38
59
|
name: String @indexed
|
|
39
60
|
}
|
|
40
61
|
```
|
|
41
62
|
|
|
42
|
-
|
|
63
|
+
If the field value is an array, each element is individually indexed. Null values are indexed by default.
|
|
43
64
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
65
|
+
6. **Expose a table as an external endpoint** with `@export`. The optional `name` parameter sets the URL path segment:
|
|
66
|
+
|
|
67
|
+
```graphql
|
|
68
|
+
type MyTable @table @export(name: "my-table") {
|
|
69
|
+
id: Long @primaryKey
|
|
70
|
+
}
|
|
47
71
|
```
|
|
48
72
|
|
|
49
|
-
|
|
73
|
+
Without `name`, the type name is used as the path segment.
|
|
50
74
|
|
|
51
|
-
|
|
75
|
+
7. **Configure `@table` arguments** as needed for database placement, expiration, eviction, and replication:
|
|
52
76
|
|
|
53
77
|
| Argument | Type | Default | Description |
|
|
54
78
|
| -------------- | --------- | ----------------------------- | ------------------------------------------------------------- |
|
|
@@ -59,32 +83,13 @@ Apply this rule when creating or modifying Harper schema files, configuring `gra
|
|
|
59
83
|
| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans |
|
|
60
84
|
| `replicate` | `Boolean` | `true` | Enable replication of this table |
|
|
61
85
|
|
|
62
|
-
|
|
63
|
-
- `
|
|
64
|
-
- `
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
5. **Index fields that need fast querying** with `@indexed`. This is required for filtering by that attribute in REST queries, SQL, or NoSQL operations. If the field value is an array, each element is individually indexed.
|
|
69
|
-
|
|
70
|
-
```graphql
|
|
71
|
-
type Product @table {
|
|
72
|
-
id: Long @primaryKey
|
|
73
|
-
category: String @indexed
|
|
74
|
-
price: Float @indexed
|
|
75
|
-
}
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
6. **Expose a table as an external resource endpoint** with `@export`. This makes the table accessible via REST, MQTT, and other interfaces. The optional `name` parameter sets the URL path segment; without it, the type name is used.
|
|
79
|
-
|
|
80
|
-
```graphql
|
|
81
|
-
type MyTable @table @export(name: "my-table") {
|
|
82
|
-
id: Long @primaryKey
|
|
83
|
-
}
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
7. **Restrict extra properties** with `@sealed` when records must not include attributes beyond those declared. By default, Harper allows additional properties.
|
|
86
|
+
8. **Apply additional field directives** where needed:
|
|
87
|
+
- `@createdTime` — auto-assigns Unix epoch milliseconds on record creation
|
|
88
|
+
- `@updatedTime` — auto-assigns Unix epoch milliseconds on each update
|
|
89
|
+
- `@embed(source: "fieldName", model: "modelName")` — computes an embedding vector on write; attribute type must be `[Float]`
|
|
90
|
+
- `@hidden` — suppresses the field from MCP tool descriptors and OpenAPI documents (not an access-control mechanism)
|
|
87
91
|
|
|
92
|
+
9. **Restrict extra properties** with `@sealed` if records must not include properties beyond those declared:
|
|
88
93
|
```graphql
|
|
89
94
|
type StrictRecord @table @sealed {
|
|
90
95
|
id: Long @primaryKey
|
|
@@ -92,70 +97,88 @@ Apply this rule when creating or modifying Harper schema files, configuring `gra
|
|
|
92
97
|
}
|
|
93
98
|
```
|
|
94
99
|
|
|
95
|
-
8. **Configure expiration, eviction, and scan behavior** together when building caching tables. These three arguments control the full record lifecycle:
|
|
96
|
-
- `expiration` — record becomes stale; next request triggers a source fetch
|
|
97
|
-
- `eviction` — additional time after `expiration` before physical removal
|
|
98
|
-
- `scanInterval` — how often Harper scans for records to evict; clock-aligned, not startup-aligned
|
|
99
|
-
|
|
100
100
|
## Examples
|
|
101
101
|
|
|
102
|
-
**
|
|
102
|
+
**Minimal two-table schema:**
|
|
103
103
|
|
|
104
104
|
```graphql
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
105
|
+
type Dog @table {
|
|
106
|
+
id: Long @primaryKey
|
|
107
|
+
name: String
|
|
108
|
+
breed: String
|
|
109
|
+
age: Int
|
|
109
110
|
}
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
**Table in a named database with expiration and an indexed field:**
|
|
113
111
|
|
|
114
|
-
|
|
115
|
-
type Event @table(database: "analytics", expiration: 86400) {
|
|
112
|
+
type Breed @table {
|
|
116
113
|
id: Long @primaryKey
|
|
117
114
|
name: String @indexed
|
|
118
115
|
}
|
|
119
116
|
```
|
|
120
117
|
|
|
121
|
-
**
|
|
118
|
+
**Table with expiration, eviction, and custom scan interval:**
|
|
122
119
|
|
|
123
120
|
```graphql
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
121
|
+
# Expire after 5 minutes, evict after 1 hour, scan every 10 minutes
|
|
122
|
+
type WeatherCache @table(expiration: 300, eviction: 3300, scanInterval: 600) {
|
|
123
|
+
id: ID @primaryKey
|
|
124
|
+
temperature: Float
|
|
127
125
|
}
|
|
128
126
|
```
|
|
129
127
|
|
|
130
|
-
**
|
|
128
|
+
**Exported table with timestamps and a hidden field:**
|
|
131
129
|
|
|
132
130
|
```graphql
|
|
133
|
-
type
|
|
131
|
+
type Customer @table @export(name: "customers") {
|
|
134
132
|
id: Long @primaryKey
|
|
133
|
+
name: String @indexed
|
|
135
134
|
createdAt: Long @createdTime
|
|
136
135
|
updatedAt: Long @updatedTime
|
|
137
|
-
|
|
136
|
+
|
|
137
|
+
"""
|
|
138
|
+
Internal — do not surface to external consumers.
|
|
139
|
+
"""
|
|
140
|
+
creditScore: Int @hidden
|
|
138
141
|
}
|
|
139
142
|
```
|
|
140
143
|
|
|
141
|
-
**
|
|
144
|
+
**Multiple `@table` argument combinations:**
|
|
142
145
|
|
|
143
146
|
```graphql
|
|
147
|
+
# Override table name
|
|
144
148
|
type Product @table(table: "products") {
|
|
145
149
|
id: Long @primaryKey
|
|
146
|
-
name: String
|
|
147
150
|
}
|
|
148
151
|
|
|
152
|
+
# Place in a specific database
|
|
153
|
+
type Order @table(database: "commerce") {
|
|
154
|
+
id: Long @primaryKey
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
# Auto-expire records after 1 hour
|
|
158
|
+
type Session @table(expiration: 3600) {
|
|
159
|
+
id: Long @primaryKey
|
|
160
|
+
userId: String
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
# Disable replication
|
|
149
164
|
type LocalRecord @table(replicate: false) {
|
|
150
165
|
id: Long @primaryKey
|
|
151
166
|
value: String
|
|
152
167
|
}
|
|
168
|
+
|
|
169
|
+
# Combine multiple arguments
|
|
170
|
+
type Event @table(database: "analytics", expiration: 86400) {
|
|
171
|
+
id: Long @primaryKey
|
|
172
|
+
name: String @indexed
|
|
173
|
+
}
|
|
153
174
|
```
|
|
154
175
|
|
|
155
176
|
## Notes
|
|
156
177
|
|
|
157
|
-
-
|
|
158
|
-
-
|
|
159
|
-
- `
|
|
160
|
-
-
|
|
161
|
-
-
|
|
178
|
+
- All tables default to the `"data"` database. When designing plugins or applications, use unique database names to avoid table naming collisions.
|
|
179
|
+
- Schemas are flexible by default — records may include properties beyond those declared. Use `@sealed` to prevent this.
|
|
180
|
+
- `expiration` marks a record stale; `eviction` controls how long after expiration the record is physically removed. Eviction does not remove records from secondary indexes — Harper fetches the full record on demand if an evicted record matches a query.
|
|
181
|
+
- `scanInterval` is clock-aligned to the server's local timezone, not startup-aligned. The eviction schedule is deterministic across restarts.
|
|
182
|
+
- If replication is disabled on a table and later re-enabled, writes made during the disabled period are not replicated retroactively.
|
|
183
|
+
- `@hidden` (on types or fields) is a metadata-visibility directive only. Use `attribute_permissions` on roles to enforce data access control.
|
|
184
|
+
- Disabling replication (`replicate: false`) and re-enabling it later will not catch up on writes made while replication was disabled.
|
|
@@ -68,6 +68,34 @@ static:
|
|
|
68
68
|
- Install dependencies: `npm install --save-dev vite @harperfast/vite @vitejs/plugin-react` (swap in your framework's Vite plugin, e.g. `@vitejs/plugin-vue`).
|
|
69
69
|
- Then `harper dev .` runs the app with HMR and `harper run .` runs the production build. Vite does _not_ need to be executed separately.
|
|
70
70
|
|
|
71
|
+
## Reading Harper Data During SSR
|
|
72
|
+
|
|
73
|
+
The render entry (`src/entry-server.tsx`) runs **inside Harper**, so it can read straight from the database and render the data into the HTML — no client-side fetch/XHR. `tables` is the same live, process-wide registry available everywhere (see [Programmatic Table Requests](programmatic-table-requests.md)); import it and query a table in an async `render`:
|
|
74
|
+
|
|
75
|
+
```tsx
|
|
76
|
+
import { tables } from 'harper';
|
|
77
|
+
|
|
78
|
+
export async function render(url: string): Promise<string> {
|
|
79
|
+
const product = await tables.Product.get(idFromUrl(url));
|
|
80
|
+
return renderToString(
|
|
81
|
+
<StrictMode>
|
|
82
|
+
<App product={product} />
|
|
83
|
+
</StrictMode>,
|
|
84
|
+
);
|
|
85
|
+
}
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Keep `harper` external in `vite.config.ts` so this import resolves to Harper's running runtime instead of being bundled. `node_modules/harper` is symlinked to the running install, and symlinked deps aren't reliably auto-externalized for SSR:
|
|
89
|
+
|
|
90
|
+
```typescript
|
|
91
|
+
export default defineConfig({
|
|
92
|
+
ssr: { external: ['harper'] },
|
|
93
|
+
// ...plugins, resolve, build
|
|
94
|
+
});
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
To hydrate on the client without re-fetching, embed the rendered data in the HTML (e.g. an inline `<script type="application/json">`) and read it back before hydration — so the page needs no XHR at all.
|
|
98
|
+
|
|
71
99
|
## Deploying to Production
|
|
72
100
|
|
|
73
101
|
Because `@harperfast/vite` builds on the node and `static` serves the output, deploy the component as-is — no manual build-and-move step is needed:
|
|
@@ -2,37 +2,118 @@
|
|
|
2
2
|
name: using-blob-datatype
|
|
3
3
|
description: How to use the Blob data type for efficient binary storage in Harper.
|
|
4
4
|
metadata:
|
|
5
|
-
mode:
|
|
5
|
+
mode: generate
|
|
6
|
+
sources:
|
|
7
|
+
- reference/v5/database/schema.md#Blob Type
|
|
8
|
+
- reference/v5/database/api.md#Streaming
|
|
9
|
+
- reference/v5/database/api.md#`BlobOptions`
|
|
10
|
+
- reference/v5/database/api.md#Blob Coercion
|
|
11
|
+
sourceCommit: f37a8c4021e20d5c74c1d339a6b6c8c196b5603e
|
|
12
|
+
inputHash: 92e03eb0b830f335
|
|
6
13
|
---
|
|
7
14
|
|
|
8
|
-
# Using Blob
|
|
15
|
+
# Using the Blob Data Type
|
|
9
16
|
|
|
10
|
-
Instructions for the agent to follow when
|
|
17
|
+
Instructions for the agent to follow when storing and retrieving large binary content using the `Blob` data type in Harper.
|
|
11
18
|
|
|
12
19
|
## When to Use
|
|
13
20
|
|
|
14
|
-
|
|
21
|
+
Apply this rule when a schema field needs to store large binary content such as images, video, audio, or large HTML — typically content larger than 20KB. Use `Blob` instead of `Bytes` when streaming support and out-of-record storage are required. See [handling-binary-data.md](handling-binary-data.md) for broader binary data guidance.
|
|
15
22
|
|
|
16
23
|
## How It Works
|
|
17
24
|
|
|
18
|
-
1. **
|
|
25
|
+
1. **Declare a `Blob` field in your schema**: Add a field typed as `Blob` to your `@table` type.
|
|
26
|
+
|
|
19
27
|
```graphql
|
|
20
28
|
type MyTable @table {
|
|
21
|
-
id:
|
|
29
|
+
id: Any! @primaryKey
|
|
22
30
|
data: Blob
|
|
23
31
|
}
|
|
24
32
|
```
|
|
25
|
-
|
|
33
|
+
|
|
34
|
+
2. **Create and store a blob with `createBlob()`**: Pass a buffer or stream to `createBlob()`, then `put` the record.
|
|
35
|
+
|
|
36
|
+
```javascript
|
|
37
|
+
let blob = createBlob(largeBuffer);
|
|
38
|
+
await MyTable.put({ id: 'my-record', data: blob });
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
3. **Retrieve blob data using standard Web API methods**: The `Blob` type implements the Web API `Blob` interface. Use `.bytes()`, `.text()`, `.arrayBuffer()`, `.stream()`, or `.slice()` as needed.
|
|
42
|
+
|
|
43
|
+
```javascript
|
|
44
|
+
let record = await MyTable.get('my-record');
|
|
45
|
+
let buffer = await record.data.bytes(); // ArrayBuffer
|
|
46
|
+
let text = await record.data.text(); // string
|
|
47
|
+
let stream = record.data.stream(); // ReadableStream
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
4. **Use `saveBeforeCommit` when full write must precede commit**: By default, `Blob` is not ACID-compliant — a record can reference a blob before it is fully written. Set `saveBeforeCommit: true` to block the transaction until the blob is fully saved.
|
|
51
|
+
|
|
26
52
|
```javascript
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
53
|
+
let blob = createBlob(stream, { saveBeforeCommit: true });
|
|
54
|
+
await MyTable.put({ id: 'my-record', data: blob });
|
|
55
|
+
// put() resolves only after blob is fully written and record is committed
|
|
30
56
|
```
|
|
31
|
-
|
|
32
|
-
|
|
57
|
+
|
|
58
|
+
5. **Register an error handler when returning a blob via REST**: Interrupted streams must be handled explicitly.
|
|
59
|
+
|
|
33
60
|
```javascript
|
|
34
|
-
|
|
35
|
-
|
|
61
|
+
export class MyEndpoint extends MyTable {
|
|
62
|
+
static async get(target) {
|
|
63
|
+
const record = super.get(target);
|
|
64
|
+
let blob = record.data;
|
|
65
|
+
blob.on('error', () => {
|
|
66
|
+
MyTable.invalidate(target);
|
|
67
|
+
});
|
|
68
|
+
return { status: 200, headers: {}, body: blob };
|
|
69
|
+
}
|
|
70
|
+
}
|
|
36
71
|
```
|
|
37
|
-
|
|
38
|
-
6. **
|
|
72
|
+
|
|
73
|
+
6. **Rely on automatic coercion where applicable**: When a field is typed as `Blob` in the schema, any string or buffer assigned via `put`, `patch`, or `publish` is automatically coerced to a `Blob` — no manual `createBlob()` call is needed in those cases.
|
|
74
|
+
|
|
75
|
+
### `BlobOptions` reference
|
|
76
|
+
|
|
77
|
+
Pass an options object as the second argument to `createBlob()`.
|
|
78
|
+
|
|
79
|
+
| Option | Type | Default | Description |
|
|
80
|
+
| ------------------ | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
81
|
+
| `type` | `string` | `undefined` | MIME type to associate with the blob (e.g., `image/jpeg`). Readable via `blob.type` and used when serving HTTP. |
|
|
82
|
+
| `size` | `number` | `undefined` | Size of the data in bytes, if known ahead of time. Otherwise inferred from a buffer or determined as a stream completes. |
|
|
83
|
+
| `saveBeforeCommit` | `boolean` | `false` | Wait until the blob is fully written before the transaction commits. |
|
|
84
|
+
| `compress` | `boolean` | `false` | Compress the stored data with deflate. |
|
|
85
|
+
| `flush` | `boolean` | `false` | Flush the file to disk after writing, before the `createBlob` promise chain resolves. |
|
|
86
|
+
|
|
87
|
+
## Examples
|
|
88
|
+
|
|
89
|
+
**Store an image with a MIME type:**
|
|
90
|
+
|
|
91
|
+
```javascript
|
|
92
|
+
let blob = createBlob(imageBuffer, { type: 'image/jpeg' });
|
|
93
|
+
await Photo.put({ id, data: blob });
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
**Stream a blob in as it streams out (low-latency passthrough):**
|
|
97
|
+
|
|
98
|
+
```javascript
|
|
99
|
+
let blob = createBlob(incomingStream);
|
|
100
|
+
// blob exists, but data is still streaming to storage
|
|
101
|
+
await MyTable.put({ id: 'my-record', data: blob });
|
|
102
|
+
|
|
103
|
+
let record = await MyTable.get('my-record');
|
|
104
|
+
// blob data is accessible as it arrives
|
|
105
|
+
let outgoingStream = record.data.stream();
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
**Guarantee full write before commit using `saveBeforeCommit`:**
|
|
109
|
+
|
|
110
|
+
```javascript
|
|
111
|
+
let blob = createBlob(stream, { saveBeforeCommit: true });
|
|
112
|
+
await MyTable.put({ id: 'my-record', data: blob });
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
## Notes
|
|
116
|
+
|
|
117
|
+
- `Blob` stores data separately from the record. If you need the binary data to be a true, ACID-committed part of the record, use a `Bytes` field instead.
|
|
118
|
+
- All standard Web API `Blob` methods — `.text()`, `.arrayBuffer()`, `.stream()`, `.slice()`, and `.bytes()` — are available on retrieved blob fields.
|
|
119
|
+
- Without `saveBeforeCommit: true`, blobs are **not** ACID-compliant by default; a record can reference a blob before it is fully written to storage.
|
|
@@ -5,21 +5,21 @@ metadata:
|
|
|
5
5
|
mode: generate
|
|
6
6
|
sources:
|
|
7
7
|
- reference/v5/database/schema.md#Vector Indexing
|
|
8
|
-
sourceCommit:
|
|
9
|
-
inputHash:
|
|
8
|
+
sourceCommit: 4fe4c9c95e0974eaa77032f6f10e36fbd8ec64ac
|
|
9
|
+
inputHash: d90b1b74597d08a6
|
|
10
10
|
---
|
|
11
11
|
|
|
12
12
|
# Vector Indexing
|
|
13
13
|
|
|
14
|
-
Instructions for the agent to
|
|
14
|
+
Instructions for the agent to enable HNSW vector indexes on table fields and query them for similarity search in Harper.
|
|
15
15
|
|
|
16
16
|
## When to Use
|
|
17
17
|
|
|
18
|
-
Apply this rule when adding a vector
|
|
18
|
+
Apply this rule when adding a vector similarity search capability to a Harper table — for example, storing text embeddings and querying for nearest neighbors, filtering by distance threshold, or tuning index construction and search parameters. Use it alongside [adding-tables-with-schemas.md](adding-tables-with-schemas.md) when defining the schema that hosts the vector field.
|
|
19
19
|
|
|
20
20
|
## How It Works
|
|
21
21
|
|
|
22
|
-
1. **Declare the vector index on a
|
|
22
|
+
1. **Declare the vector index on a field**: Add `@indexed(type: "HNSW")` to a `[Float]` field inside a `@table` type. This creates an HNSW (Hierarchical Navigable Small World) index for approximate nearest-neighbor search.
|
|
23
23
|
|
|
24
24
|
```graphql
|
|
25
25
|
type Document @table {
|
|
@@ -28,7 +28,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
|
|
|
28
28
|
}
|
|
29
29
|
```
|
|
30
30
|
|
|
31
|
-
2. **Query by nearest neighbors using `sort`**: Call
|
|
31
|
+
2. **Query by nearest neighbors using `sort`**: Call `.search()` with a `sort` descriptor that specifies the indexed `attribute` and a `target` vector. Use `limit` to cap results.
|
|
32
32
|
|
|
33
33
|
```javascript
|
|
34
34
|
let results = Document.search({
|
|
@@ -47,7 +47,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
|
|
|
47
47
|
});
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
-
4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`.
|
|
50
|
+
4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. This bounds result quality rather than ranking by similarity.
|
|
51
51
|
|
|
52
52
|
```javascript
|
|
53
53
|
let results = Document.search({
|
|
@@ -60,7 +60,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
|
|
|
60
60
|
});
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
-
5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector.
|
|
63
|
+
5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Available in both `sort`-based and threshold-based queries.
|
|
64
64
|
|
|
65
65
|
```javascript
|
|
66
66
|
let results = Document.search({
|
|
@@ -70,43 +70,59 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
|
|
|
70
70
|
});
|
|
71
71
|
```
|
|
72
72
|
|
|
73
|
-
6. **Tune
|
|
73
|
+
6. **Tune per-query search options**: Pass `distance` and `ef` directly on the `sort` descriptor to override index defaults for a single query.
|
|
74
74
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
75
|
+
```javascript
|
|
76
|
+
let results = Document.search({
|
|
77
|
+
sort: { attribute: 'textEmbeddings', target: searchVector, distance: 'dotProduct', ef: 200 },
|
|
78
|
+
limit: 5,
|
|
79
|
+
});
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
- `distance` — overrides the distance function for this query: `"cosine"`, `"euclidean"`, or `"dotProduct"`.
|
|
83
|
+
- `ef` — overrides the search exploration budget. Higher values improve recall at the cost of latency.
|
|
84
|
+
|
|
85
|
+
7. **Configure HNSW index parameters**: Pass parameters directly in the `@indexed` directive. Structural parameters (`distance`, `M`, `efConstruction`, `quantization`) trigger an index rebuild when changed; `efConstructionSearch` does not.
|
|
86
|
+
|
|
87
|
+
```graphql
|
|
88
|
+
type Document @table {
|
|
89
|
+
id: Long @primaryKey
|
|
90
|
+
textEmbeddings: [Float]
|
|
91
|
+
@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
|
|
92
|
+
}
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
8. **Enable vector quantization**: Use `quantization: "int8"` to store vectors as 8-bit integers, reducing index size and memory usage. Harper re-ranks nearest-neighbor `sort` results against full-precision vectors automatically.
|
|
96
|
+
|
|
97
|
+
```graphql
|
|
98
|
+
type Document @table {
|
|
99
|
+
id: Long @primaryKey
|
|
100
|
+
textEmbeddings: [Float] @indexed(type: "HNSW", quantization: "int8")
|
|
101
|
+
}
|
|
102
|
+
```
|
|
83
103
|
|
|
84
104
|
## Examples
|
|
85
105
|
|
|
86
|
-
|
|
106
|
+
Full schema with custom HNSW parameters and a nearest-neighbor query with distance output:
|
|
87
107
|
|
|
88
108
|
```graphql
|
|
89
109
|
type Document @table {
|
|
90
110
|
id: Long @primaryKey
|
|
91
111
|
textEmbeddings: [Float]
|
|
92
|
-
@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0,
|
|
112
|
+
@indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efConstructionSearch: 100)
|
|
93
113
|
}
|
|
94
114
|
```
|
|
95
115
|
|
|
96
|
-
**Nearest-neighbor search with distance score:**
|
|
97
|
-
|
|
98
116
|
```javascript
|
|
117
|
+
// Nearest-neighbor search with distance scores
|
|
99
118
|
let results = Document.search({
|
|
100
119
|
select: ['name', '$distance'],
|
|
101
120
|
sort: { attribute: 'textEmbeddings', target: searchVector },
|
|
102
121
|
limit: 5,
|
|
103
122
|
});
|
|
104
|
-
```
|
|
105
123
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
```javascript
|
|
109
|
-
let results = Document.search({
|
|
124
|
+
// Distance-threshold query (no ranking)
|
|
125
|
+
let closeMatches = Document.search({
|
|
110
126
|
conditions: {
|
|
111
127
|
attribute: 'textEmbeddings',
|
|
112
128
|
comparator: 'lt',
|
|
@@ -118,7 +134,21 @@ let results = Document.search({
|
|
|
118
134
|
|
|
119
135
|
## Notes
|
|
120
136
|
|
|
121
|
-
|
|
122
|
-
|
|
137
|
+
### HNSW Parameters
|
|
138
|
+
|
|
139
|
+
| Parameter | Default | Description |
|
|
140
|
+
| ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------ |
|
|
141
|
+
| `distance` | `"cosine"` | Distance function: `"cosine"`, `"euclidean"`, or `"dotProduct"` |
|
|
142
|
+
| `efConstruction` | `100` | Max nodes explored during index construction. Higher = better recall, lower = better performance |
|
|
143
|
+
| `M` | `16` | Preferred connections per graph layer. Higher = more space, better recall for high-dimensional data |
|
|
144
|
+
| `optimizeRouting` | `0.5` | Heuristic aggressiveness for omitting redundant connections (0 = off, 1 = most aggressive) |
|
|
145
|
+
| `mL` | computed from `M` | Normalization factor for level generation |
|
|
146
|
+
| `efConstructionSearch` | auto-scaled | Max nodes explored during search. When unset, auto-scales with index size; setting it fixes the budget |
|
|
147
|
+
| `quantization` | — | `"int8"` stores vectors quantized to int8 |
|
|
148
|
+
|
|
149
|
+
- The `distance` option on a per-query `sort` descriptor accepts `"cosine"`, `"euclidean"`, or `"dotProduct"`.
|
|
150
|
+
- When no `ef` is passed and `efConstructionSearch` (or `efConstruction`) is not explicitly set on the index, the search budget auto-scales with index size.
|
|
151
|
+
- `efConstruction` seeds the initial value of `efConstructionSearch`; setting either one fixes the search budget.
|
|
152
|
+
- The correct parameter name is `efConstructionSearch` (not `efSearchConstruction`).
|
|
123
153
|
- `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
|
|
124
|
-
-
|
|
154
|
+
- For `quantization: "int8"`, distance-threshold (`lt`/`le`) queries filter on approximate distance; `sort` queries re-rank against full-precision vectors.
|