create-harper 0.12.4 → 0.13.1

package/package.json

@@ -1,7 +1,7 @@
 {
 	"name": "create-harper",
 	"description": "Scaffold a new Harper project in JavaScript or TypeScript.",
-	"version": "0.12.4",
+	"version": "0.13.1",
 	"type": "module",
 	"author": {
 		"name": "HarperDB",

package/template-react/AGENTS.md

@@ -16,4 +16,6 @@ This repository contains "skills" that guide AI agents in developing Harper appl
 - [Handling Binary Data](skills/handling-binary-data.md): How to store and serve binary data like images or MP3s.
 - [Serving Web Content](skills/serving-web-content): Two ways to serve web content from a Harper application.
 - [Checking Authentication](skills/checking-authentication.md): How to use sessions to verify user identity and roles.
-  [adding-tables-with-schemas.md](skills/adding-tables-with-schemas.md)
+- [Caching](skills/caching.md): How caching is defined and implemented in Harper applications.
+- [Using Blobs](skills/using-blob-datatype.md): How to store and retrieve large data in HarperDB.
+- [Vector Indexing](skills/vector-indexing.md): How to define and use vector indexes for efficient similarity search.

package/template-react/skills/automatic-apis.md

@@ -47,3 +47,7 @@ This is the easiest way to add real-time capabilities to your application. For m
 The `GET /{TableName}/` and `DELETE /{TableName}/` endpoints can be filtered using query parameters. While basic equality filters are straightforward, Harper supports a rich set of operators, sorting, and pagination.
 
 For a comprehensive guide on advanced querying, see the [Querying REST APIs](querying-rest-apis.md) skill.
+
+## Customizing Resources
+
+If the automatic APIs don't behave how you need, then you can look to [customize the resources](./custom-resources.md).

package/template-react/skills/caching.md

@@ -0,0 +1,113 @@
+# Harper Caching
+
+Harper includes integrated support for **caching data from external sources**, enabling high-performance, low-latency cache storage that is fully queryable and interoperable with your applications. With built-in caching capabilities and distributed responsiveness, Harper makes an ideal **data caching server** for both edge and centralized use cases.
+
+---
+
+## What is Harper Caching?
+
+Harper caching lets you store **cached content** in standard tables, enabling you to:
+
+- Expose cached entries as **queryable structured data** (e.g., JSON or CSV)
+- Serve data to clients with **flexible formats and custom querying**
+- Manage cache control with **timestamps and ETags** for downstream caching layers
+- Implement **active or passive caching** patterns depending on your source and invalidation strategy
+
+---
+
+## Configuring a Cache Table
+
+Define a cache table in your `schema.graphql`:
+
+```graphql
+type MyCache @table(expiration: 3600) @export {
+	id: ID @primaryKey
+}
+```
+
+- `expiration` is defined in seconds
+- Expired records are refreshed on access
+- Evicted records are removed after expiration
+
+---
+
+## Connecting an External Source
+
+Create a resource:
+
+```js
+import { Resource } from 'harperdb';
+
+export class ThirdPartyAPI extends Resource {
+	async get() {
+		const id = this.getId();
+		const response = await fetch(`https://api.example.com/items/${id}`);
+		if (!response.ok) {
+			throw new Error('Source fetch failed');
+		}
+		return await response.json();
+	}
+}
+```
+
+Attach it to your table:
+
+```js
+import { tables } from 'harperdb';
+import { ThirdPartyAPI } from './ThirdPartyAPI.js';
+
+const { MyCache } = tables;
+MyCache.sourcedFrom(ThirdPartyAPI);
+```
+
+---
+
+## Cache Behavior
+
+1. Fresh data is returned immediately
+2. Missing or stale data triggers a fetch
+3. Concurrent misses are deduplicated
+
+---
+
+## Active Caching
+
+Use `subscribe()` to proactively update or invalidate cache entries:
+
+```js
+class MyAPI extends Resource {
+	async *subscribe() {
+		// stream updates
+	}
+}
+```
+
+See [Real Time Apps](real-time-apps.md) for more details.
+
+---
+
+## Write-Through Caching
+
+Propagate updates upstream:
+
+```js
+class ThirdPartyAPI extends Resource {
+	async put(data) {
+		await fetch(`https://some-api.com/${this.getId()}`, {
+			method: 'PUT',
+			body: JSON.stringify(data),
+		});
+	}
+}
+```
+
+---
+
+## Summary
+
+Harper Caching allows you to:
+
+- Cache external APIs efficiently
+- Query cached data like native tables
+- Prevent cache stampedes
+- Build real-time or write-through caches

package/template-react/skills/checking-authentication.md

@@ -177,6 +177,100 @@ export function ensureSuperUser(user: User | undefined) {
 - Sessions are cookie-based; the server handles setting and reading the cookie via Harper. If you make cross-origin requests, ensure the appropriate `credentials` mode and CORS settings.
 - If developing locally, double-check the server config still has `authentication.authorizeLocal: false` to avoid accidental superuser bypass.
 
+## Token-based auth (JWT + refresh token) for non-browser clients
+
+Cookie-backed sessions are great for browser flows. For CLI tools, mobile apps, or other non-browser clients, it’s often easier to use **explicit tokens**:
+
+- **JWT (`operation_token`)**: short-lived bearer token used to authorize API requests.
+- **Refresh token (`refresh_token`)**: longer-lived token used to mint a new JWT when it expires.
+
+This project includes two Resource patterns for that flow:
+
+### Issuing tokens: `IssueTokens`
+
+**Description / use case:** Generate `{ refreshToken, jwt }` either:
+
+- with an existing Authorization token (either Basic Auth or a JWT) and you want to issue new tokens, or
+- from an explicit `{ username, password }` payload (useful for direct “login” from a CLI/mobile client).
+
+```
+js
+export class IssueTokens extends Resource {
+static loadAsInstance = false;
+
+	async get(target) {
+		const { refresh_token: refreshToken, operation_token: jwt } =
+			await databases.system.hdb_user.operation(
+				{ operation: 'create_authentication_tokens' },
+				this.getContext(),
+			);
+		return { refreshToken, jwt };
+	}
+
+	async post(target, data) {
+		if (!data.username || !data.password) {
+			throw new Error('username and password are required');
+		}
+
+		const { refresh_token: refreshToken, operation_token: jwt } =
+			await databases.system.hdb_user.operation({
+				operation: 'create_authentication_tokens',
+				username: data.username,
+				password: data.password,
+			});
+		return { refreshToken, jwt };
+	}
+}
+```
+
+**Recommended documentation notes to include:**
+
+- `GET` variant: intended for “I already have an Authorization token, give me new tokens”.
+- `POST` variant: intended for “I have credentials, give me tokens”.
+- Response shape:
+  - `refreshToken`: store securely (long-lived).
+  - `jwt`: attach to requests (short-lived).
+
+### Refreshing a JWT: `RefreshJWT`
+
+**Description / use case:** When the JWT expires, the client uses the refresh token to get a new JWT without re-supplying username/password.
+
+```
+js
+export class RefreshJWT extends Resource {
+static loadAsInstance = false;
+
+	async post(target, data) {
+		if (!data.refreshToken) {
+			throw new Error('refreshToken is required');
+		}
+
+		const { operation_token: jwt } = await databases.system.hdb_user.operation({
+			operation: 'refresh_operation_token',
+			refresh_token: data.refreshToken,
+		});
+		return { jwt };
+	}
+}
+```
+
+**Recommended documentation notes to include:**
+
+- Requires `refreshToken` in the request body.
+- Returns a new `{ jwt }`.
+- If refresh fails (expired/revoked), client must re-authenticate (e.g., call `IssueTokens.post` again).
+
+### Suggested client flow (high-level)
+
+1. **Sign in (token flow)**
+   - POST /IssueTokens/ with a body of `{ "username": "your username", "password": "your password" }` or GET /IssueTokens/ with an existing Authorization token.
+   - Receive `{ jwt, refreshToken }` in the response
+2. **Call protected APIs**
+   - Send the JWT with each request in the Authorization header (as your auth mechanism expects)
+3. **JWT expires**
+   - POST /RefreshJWT/ with a body of `{ "refreshToken": "your refresh token" }`.
+   - Receive `{ jwt }` in the response and continue
+
 ## Quick checklist
 
 - [ ] Public endpoints explicitly `allowRead`/`allowCreate` as needed.
@@ -184,3 +278,4 @@ export function ensureSuperUser(user: User | undefined) {
 - [ ] Protected routes call `ensureSuperUser(this.getCurrentUser())` (or another role check) before doing work.
 - [ ] Sign-out verifies a session and deletes it.
 - [ ] `authentication.authorizeLocal` is `false` and `enableSessions` is `true` in Harper config.
+- [ ] If using tokens: `IssueTokens` issues `{ jwt, refreshToken }`, `RefreshJWT` refreshes `{ jwt }` with a `refreshToken`.

package/template-react/skills/custom-resources.md

@@ -4,6 +4,10 @@ Custom Resources allow you to define your own REST endpoints with custom logic b
 
 Harper supports [TypeScript Type Stripping](typescript-type-stripping.md), allowing you to use TypeScript directly without additional build tools on supported Node.js versions.
 
+## Do you need a custom resource?
+
+Exported tables have [automatic APIs](./automatic-apis.md) as a part of them. These APIs may be enough for what you need. Take a look at [extendings tables](./extending-tables.md) to learn more about resources with a backing table.
+
 ## Defining a Custom Resource
 
 To create a custom resource:

package/template-react/skills/extending-tables.md

@@ -2,6 +2,10 @@
 
 In Harper, when you define a table in GraphQL and export it, you can extend the automatically generated resource class to add custom logic, validation, or hooks to standard CRUD operations.
 
+## Do you need to extend tables?
+
+Exported tables have [automatic APIs](./automatic-apis.md) as a part of them. These APIs may be sufficient for what you need.
+
 ## How to Extend a Table Resource
 
 1. Identify the table you want to extend (e.g., `ExamplePeople`).

package/template-react/skills/using-blob-datatype.md

@@ -0,0 +1,131 @@
+# Blob (Binary Large Objects)
+
+Harper supports **Blobs** — binary large objects for storing unstructured or large binary data — with integrated streaming support and efficient storage. Blobs are ideal for media files, documents, and any data where size or throughput makes standard JSON fields impractical.
+
+---
+
+## What Are Blobs
+
+Blobs extend the native JavaScript `Blob` type and allow you to store **binary or arbitrary data** inside Harper tables. The blob reference is stored in the record, while the blob’s contents are streamed to and from storage.
+
+- Designed for binary data such as images, audio, and documents
+- Supports streaming reads and writes
+- Blob data is stored separately from record attributes
+- Optimized for large payloads
+
+---
+
+## Defining Blob Fields
+
+Declare a blob field using the `Blob` type in your schema:
+
+```graphql
+type MyTable @table {
+	id: ID @primaryKey
+	data: Blob
+}
+```
+
+Any record written to this field will store a reference to the blob’s contents.
+
+---
+
+## Creating and Storing Blobs
+
+### Creating a Blob from a Buffer
+
+```js
+const blob = createBlob(largeBuffer);
+await MyTable.put({ id: 'my-record', data: blob });
+```
+
+- `createBlob()` returns a blob reference
+- Data is streamed to storage asynchronously
+- Records may be committed before the blob finishes writing
+
+---
+
+### Creating a Blob from a Stream
+
+```js
+const blob = createBlob(stream);
+await MyTable.put({ id: 'streamed-record', data: blob });
+```
+
+Streaming allows large data to be written without loading it fully into memory.
+
+---
+
+## Reading Blob Data
+
+Retrieve a record and read its blob contents:
+
+```js
+const record = await MyTable.get('my-record');
+const buffer = await record.data.bytes();
+```
+
+Blob objects also support streaming interfaces for large reads.
+
+---
+
+## Blob Attributes and Events
+
+### Size
+
+The blob size may not be immediately available when streaming:
+
+```js
+if (blob.size === undefined) {
+	blob.on('size', size => {
+		console.log('Blob size:', size);
+	});
+}
+```
+
+---
+
+### saveBeforeCommit
+
+Blobs are not atomic while streaming. To ensure the blob is fully written before committing the record:
+
+```js
+const blob = createBlob(stream, { saveBeforeCommit: true });
+await MyTable.put({ id: 'safe-record', data: blob });
+```
+
+---
+
+## Error Handling
+
+Handle streaming errors by attaching an error listener:
+
+```js
+blob.on('error', () => {
+	MyTable.invalidate('my-record');
+});
+```
+
+This prevents partially written blobs from being used.
+
+---
+
+## Automatic Coercion
+
+When a field is defined as `Blob`, assigning a string or buffer automatically converts it into a blob when using `put`, `patch`, or `publish`.
+
+---
+
+## Related Skill
+
+- [Handling Binary Data with Blobs](handling-binary-data.md) How to store and serve binary data like images or MP3s using the Blob data type.
+
+---
+
+## Summary
+
+- Blobs store large or binary data efficiently
+- Blob fields reference streamed content
+- Supports buffered and streamed writes
+- Optional write-before-commit behavior
+- Integrates seamlessly with Harper tables

package/template-react/skills/vector-indexing.md

@@ -0,0 +1,215 @@
+# Vector Indexing
+
+Harper supports **vector indexing** on array attributes, enabling efficient similarity search over high-dimensional vector data. This is essential for AI-powered features such as semantic search, recommendations, and embeddings-based retrieval.
+
+---
+
+## What Is Vector Indexing
+
+Vector indexing organizes numeric vectors so that Harper can efficiently find records that are closest to a given query vector using a distance metric such as cosine similarity or Euclidean distance.
+
+Unlike traditional indexes that rely on exact matches, vector indexes enable **nearest-neighbor search** across high-dimensional spaces, making them ideal for embeddings and machine learning workloads.
+
+---
+
+## Enabling a Vector Index
+
+Vector indexes are defined using the `@indexed` directive on numeric array attributes.
+
+```graphql
+type Product @table {
+	id: Long @primaryKey
+	name: String
+	description: String
+	textEmbeddings: [Float] @indexed(type: "HNSW")
+	price: Float
+}
+```
+
+- `type: "HNSW"` enables Harper’s vector index using the HNSW algorithm
+- The indexed field must be an array of numeric values
+- Vector indexes are stored and maintained automatically
+
+---
+
+## Querying with a Vector Index
+
+### Search Vectors with sort
+
+Once defined, vector indexes can be used by specifying a `sort` configuration with a target vector. To view the similarity of a result to a given query vector, use the `$distance` attribute in the `select` clause.
+
+```js
+const results = Product.search({
+	select: ['name', 'description', 'price', '$distance'],
+	sort: {
+		attribute: 'textEmbeddings',
+		target: searchVector,
+	},
+	limit: 5,
+});
+```
+
+- `attribute` is the vector index attribute
+- `target` is the vector to compare against
+- `searchVector` is the embedding to compare against
+- Results are ordered by similarity
+- Vector search can be combined with filters and limits
+- The `$distance` attribute in the `select` (optional) returns the distance between the result and the query vector
+
+### Search Vectors limited by distance
+
+Vector indexes results can be limited by distance using the `conditions` clause. In the following example, results are returned that are less than 0.1 similar to the query vector.
+The `conditions` clause can be combined with `sort` and `limit` and the `comparator` can be any of the following: `lt`, `lte`, `gt`, `gte`, `between`.
+
+```js
+const results = Product.search({
+	select: ['name', 'description', 'price', '$distance'],
+	conditions: {
+		attribute: 'textEmbeddings',
+		comparator: 'lt',
+		value: 0.1, // '0.1' is the similarity threshold
+		target: searchVector,
+	},
+});
+```
+
+- `attribute` is the vector index attribute
+- `comparator` is the comparison operator (`lt`, `lte`, `gt`, `gte`, `between` are accepted)
+- `value` is the threshold value
+- `target` is the vector to compare against
+- `searchVector` is the embedding to compare against
+- Vector search can be combined with filters, sort, and limits
+- The `$distance` attribute in the `select` (optional) returns the distance between the result and the query vector
+
+---
+
+## Vector Index Options
+
+Additional tuning options can be provided on the `@indexed` directive:
+
+| Option                 | Description                                 |
+| ---------------------- | ------------------------------------------- |
+| `distance`             | Similarity metric (`cosine` or `euclidean`) |
+| `efConstruction`       | Index build quality vs performance          |
+| `M`                    | Graph connectivity per HNSW layer           |
+| `optimizeRouting`      | Improves routing efficiency                 |
+| `efSearchConstruction` | Search breadth during queries               |
+
+These options allow fine-tuning for performance and recall tradeoffs.
+
+---
+
+## How to Generate and Search Vector Embeddings
+
+Here is a full example that generates embeddings for a set of products and then searches for similar products using vector indexes. The following example shows how to generate embeddings using OpenAI or Ollama.
+
+```js
+import { Ollama } from 'ollama';
+const ollama = new Ollama({ host: 'http://127.0.0.1:11434' });
+// The name of the ollama embedding model
+const OLLAMA_EMBEDDING_MODEL = 'nomic-embed-text';
+
+const { Product } = tables;
+
+import OpenAI from 'openai';
+const openai = new OpenAI();
+// the name of the OpenAI embedding model
+const OPENAI_EMBEDDING_MODEL = 'text-embedding-3-small';
+
+const SIMILARITY_THRESHOLD = 0.5;
+
+export class ProductSearch extends Resource {
+	// based on env variable we choose the appropriate embedding generator
+	generateEmbedding = process.env.EMBEDDING_GENERATOR === 'ollama'
+		? this._generateOllamaEmbedding
+		: this._generateOpenAIEmbedding;
+
+	/**
+	 * Executes a search query using a generated text embedding and returns the matching products.
+	 *
+	 * @param {Object} data - The input data for the request.
+	 * @param {string} data.prompt - The prompt to generate the text embedding from.
+	 * @return {Promise<Array>} Returns a promise that resolves to an array of products matching the conditions,
+	 * including fields: name, description, price, and $distance.
+	 */
+	async post(data) {
+		const embedding = await this.generateEmbedding(data.prompt);
+
+		return await Product.search({
+			select: ['name', 'description', 'price', '$distance'],
+			conditions: {
+				attribute: 'textEmbeddings',
+				comparator: 'lt',
+				value: SIMILARITY_THRESHOLD,
+				target: embedding[0],
+			},
+			limit: 5,
+		});
+	}
+
+	/**
+	 * Generates an embedding using the Ollama API.
+	 *
+	 * @param {string} promptData - The input data for which the embedding is to be generated.
+	 * @return {Promise<number[][]>} A promise that resolves to the generated embedding as an array of numbers.
+	 */
+	async _generateOllamaEmbedding(promptData) {
+		const embedding = await ollama.embed({
+			model: OLLAMA_EMBEDDING_MODEL,
+			input: promptData,
+		});
+		return embedding?.embeddings;
+	}
+
+	/**
+	 * Generates OpenAI embeddings based on the given prompt data.
+	 *
+	 * @param {string} promptData - The input data used for generating the embedding.
+	 * @return {Promise<number[][]>} A promise that resolves to an array of embeddings, where each embedding is an array of floats.
+	 */
+	async _generateOpenAIEmbedding(promptData) {
+		const embedding = await openai.embeddings.create({
+			model: OPENAI_EMBEDDING_MODEL,
+			input: promptData,
+			encoding_format: 'float',
+		});
+
+		let embeddings = [];
+		embedding.data.forEach((embeddingData) => {
+			embeddings.push(embeddingData.embedding);
+		});
+
+		return embeddings;
+	}
+}
+```
+
+Sample request to the `ProductSearch` resource which prompts to find "shorts for the gym":
+
+```bash
+curl -X POST "http://localhost:9926/ProductSearch/" \
+-H "accept: \
+-H "Content-Type: application/json" \
+-H "Authorization: Basic <YOUR_AUTH>" \
+-d '{"prompt": "shorts for the gym"}'
+```
+
+---
+
+## When to Use Vector Indexing
+
+Vector indexing is ideal when:
+
+- Storing embedding vectors from ML models
+- Performing semantic or similarity-based search
+- Working with high-dimensional numeric data
+- Exact-match indexes are insufficient
+
+---
+
+## Summary
+
+- Vector indexing enables fast similarity search on numeric arrays
+- Defined using `@indexed(type: "HNSW")`
+- Queried using a target vector in search sorting
+- Tunable for performance and accuracy

package/template-react-ts/AGENTS.md

@@ -16,4 +16,6 @@ This repository contains "skills" that guide AI agents in developing Harper appl
 - [Handling Binary Data](skills/handling-binary-data.md): How to store and serve binary data like images or MP3s.
 - [Serving Web Content](skills/serving-web-content): Two ways to serve web content from a Harper application.
 - [Checking Authentication](skills/checking-authentication.md): How to use sessions to verify user identity and roles.
-  [adding-tables-with-schemas.md](skills/adding-tables-with-schemas.md)
+- [Caching](skills/caching.md): How caching is defined and implemented in Harper applications.
+- [Using Blobs](skills/using-blob-datatype.md): How to store and retrieve large data in HarperDB.
+- [Vector Indexing](skills/vector-indexing.md): How to define and use vector indexes for efficient similarity search.

package/template-react-ts/skills/automatic-apis.md

@@ -47,3 +47,7 @@ This is the easiest way to add real-time capabilities to your application. For m
 The `GET /{TableName}/` and `DELETE /{TableName}/` endpoints can be filtered using query parameters. While basic equality filters are straightforward, Harper supports a rich set of operators, sorting, and pagination.
 
 For a comprehensive guide on advanced querying, see the [Querying REST APIs](querying-rest-apis.md) skill.
+
+## Customizing Resources
+
+If the automatic APIs don't behave how you need, then you can look to [customize the resources](./custom-resources.md).