create-harper 0.12.3 → 0.13.0

package/template-react-ts/skills/real-time-apps.md

@@ -1,10 +1,14 @@
-# Real-time Applications in HarperDB
+# Real-time Applications in Harper
 
-HarperDB provides built-in support for real-time data synchronization using WebSockets and a Pub/Sub mechanism. This allows clients to receive immediate updates when data changes in the database.
+Harper provides built-in support for real-time data synchronization using WebSockets and a Pub/Sub mechanism. This allows clients to receive immediate updates when data changes in the database.
+
+## Automatic WebSockets
+
+For many use cases, the [Automatic APIs](automatic-apis.md) provided by Harper are more than enough. When you `@export` a table, Harper automatically provides a WebSocket endpoint that publishes events whenever data in that table is updated.
 
 ## Implementing a WebSocket Resource
 
-To handle WebSocket connections, implement the `connect` method in your custom resource class.
+Customizing resources by implementing a `connect` method is only necessary when you want to come up with a more specific back-and-forth or custom message handling. To handle WebSocket connections, implement the `connect` method in your custom resource class.
 
 ### Example: `resources/exampleSocket.ts`
 
@@ -68,4 +72,4 @@ socket.send(JSON.stringify({ type: 'ping' }));
 
 - **Automatic Table Subscriptions**: Easily stream changes from any database table.
 - **Bi-directional Communication**: Send and receive messages in real-time.
-- **Scalable Pub/Sub**: HarperDB handles the efficient distribution of messages to subscribers.
+- **Scalable Pub/Sub**: Harper handles the efficient distribution of messages to subscribers.

package/template-react-ts/skills/typescript-type-stripping.md

@@ -1,16 +1,16 @@
 # TypeScript Type Stripping
 
-HarperDB supports using TypeScript directly without any additional build tools (like `tsc` or `esbuild`) by leveraging Node.js's native Type Stripping capability. This allows you to write `.ts` files for your Custom Resources and have them run directly in HarperDB.
+Harper supports using TypeScript directly without any additional build tools (like `tsc` or `esbuild`) by leveraging Node.js's native Type Stripping capability. This allows you to write `.ts` files for your Custom Resources and have them run directly in Harper.
 
 ## Requirements
 
 - **Node.js Version**: You must be running a version of Node.js that supports type stripping (Node.js v22.6.0 or higher).
-- **No Experimental Flags**: When running on supported Node.js versions, HarperDB can automatically handle type stripping for your resource files.
+- **No Experimental Flags**: When running on supported Node.js versions, Harper can automatically handle type stripping for your resource files.
 
 ## Benefits
 
 - **Faster Development**: No need to wait for a build step or manage complex build pipelines.
-- **Simplified Tooling**: You don't need to install or configure `ts-node`, `tsx`, or other TypeScript execution engines for your HarperDB resources.
+- **Simplified Tooling**: You don't need to install or configure `ts-node`, `tsx`, or other TypeScript execution engines for your Harper resources.
 - **Native Performance**: Leverages Node.js's built-in support for stripping types, which is highly efficient.
 
 ## Usage
@@ -44,4 +44,4 @@ jsResource:
   files: 'resources/*.ts'
 ```
 
-When HarperDB starts, it will detect the `.ts` files and, if running on a compatible Node.js version, will execute them using type stripping.
+When Harper starts, it will detect the `.ts` files and, if running on a compatible Node.js version, will execute them using type stripping.

package/template-react-ts/skills/using-blob-datatype.md

@@ -0,0 +1,131 @@
+# Blob (Binary Large Objects)
+
+Harper supports **Blobs** — binary large objects for storing unstructured or large binary data — with integrated streaming support and efficient storage. Blobs are ideal for media files, documents, and any data where size or throughput makes standard JSON fields impractical.
+
+---
+
+## What Are Blobs
+
+Blobs extend the native JavaScript `Blob` type and allow you to store **binary or arbitrary data** inside Harper tables. The blob reference is stored in the record, while the blob’s contents are streamed to and from storage.
+
+- Designed for binary data such as images, audio, and documents
+- Supports streaming reads and writes
+- Blob data is stored separately from record attributes
+- Optimized for large payloads
+
+---
+
+## Defining Blob Fields
+
+Declare a blob field using the `Blob` type in your schema:
+
+```graphql
+type MyTable @table {
+	id: ID @primaryKey
+	data: Blob
+}
+```
+
+Any record written to this field will store a reference to the blob’s contents.
+
+---
+
+## Creating and Storing Blobs
+
+### Creating a Blob from a Buffer
+
+```js
+const blob = createBlob(largeBuffer);
+await MyTable.put({ id: 'my-record', data: blob });
+```
+
+- `createBlob()` returns a blob reference
+- Data is streamed to storage asynchronously
+- Records may be committed before the blob finishes writing
+
+---
+
+### Creating a Blob from a Stream
+
+```js
+const blob = createBlob(stream);
+await MyTable.put({ id: 'streamed-record', data: blob });
+```
+
+Streaming allows large data to be written without loading it fully into memory.
+
+---
+
+## Reading Blob Data
+
+Retrieve a record and read its blob contents:
+
+```js
+const record = await MyTable.get('my-record');
+const buffer = await record.data.bytes();
+```
+
+Blob objects also support streaming interfaces for large reads.
+
+---
+
+## Blob Attributes and Events
+
+### Size
+
+The blob size may not be immediately available when streaming:
+
+```js
+if (blob.size === undefined) {
+	blob.on('size', size => {
+		console.log('Blob size:', size);
+	});
+}
+```
+
+---
+
+### saveBeforeCommit
+
+Blobs are not atomic while streaming. To ensure the blob is fully written before committing the record:
+
+```js
+const blob = createBlob(stream, { saveBeforeCommit: true });
+await MyTable.put({ id: 'safe-record', data: blob });
+```
+
+---
+
+## Error Handling
+
+Handle streaming errors by attaching an error listener:
+
+```js
+blob.on('error', () => {
+	MyTable.invalidate('my-record');
+});
+```
+
+This prevents partially written blobs from being used.
+
+---
+
+## Automatic Coercion
+
+When a field is defined as `Blob`, assigning a string or buffer automatically converts it into a blob when using `put`, `patch`, or `publish`.
+
+---
+
+## Related Skill
+
+- [Handling Binary Data with Blobs](handling-binary-data.md) How to store and serve binary data like images or MP3s using the Blob data type.
+
+---
+
+## Summary
+
+- Blobs store large or binary data efficiently
+- Blob fields reference streamed content
+- Supports buffered and streamed writes
+- Optional write-before-commit behavior
+- Integrates seamlessly with Harper tables

package/template-react-ts/skills/vector-indexing.md

@@ -0,0 +1,215 @@
+# Vector Indexing
+
+Harper supports **vector indexing** on array attributes, enabling efficient similarity search over high-dimensional vector data. This is essential for AI-powered features such as semantic search, recommendations, and embeddings-based retrieval.
+
+---
+
+## What Is Vector Indexing
+
+Vector indexing organizes numeric vectors so that Harper can efficiently find records that are closest to a given query vector using a distance metric such as cosine similarity or Euclidean distance.
+
+Unlike traditional indexes that rely on exact matches, vector indexes enable **nearest-neighbor search** across high-dimensional spaces, making them ideal for embeddings and machine learning workloads.
+
+---
+
+## Enabling a Vector Index
+
+Vector indexes are defined using the `@indexed` directive on numeric array attributes.
+
+```graphql
+type Product @table {
+	id: Long @primaryKey
+	name: String
+	description: String
+	textEmbeddings: [Float] @indexed(type: "HNSW")
+	price: Float
+}
+```
+
+- `type: "HNSW"` enables Harper’s vector index using the HNSW algorithm
+- The indexed field must be an array of numeric values
+- Vector indexes are stored and maintained automatically
+
+---
+
+## Querying with a Vector Index
+
+### Search Vectors with sort
+
+Once defined, vector indexes can be used by specifying a `sort` configuration with a target vector. To view the similarity of a result to a given query vector, use the `$distance` attribute in the `select` clause.
+
+```js
+const results = Product.search({
+	select: ['name', 'description', 'price', '$distance'],
+	sort: {
+		attribute: 'textEmbeddings',
+		target: searchVector,
+	},
+	limit: 5,
+});
+```
+
+- `attribute` is the vector index attribute
+- `target` is the vector to compare against
+- `searchVector` is the embedding to compare against
+- Results are ordered by similarity
+- Vector search can be combined with filters and limits
+- The `$distance` attribute in the `select` (optional) returns the distance between the result and the query vector
+
+### Search Vectors limited by distance
+
+Vector indexes results can be limited by distance using the `conditions` clause. In the following example, results are returned that are less than 0.1 similar to the query vector.
+The `conditions` clause can be combined with `sort` and `limit` and the `comparator` can be any of the following: `lt`, `lte`, `gt`, `gte`, `between`.
+
+```js
+const results = Product.search({
+	select: ['name', 'description', 'price', '$distance'],
+	conditions: {
+		attribute: 'textEmbeddings',
+		comparator: 'lt',
+		value: 0.1, // '0.1' is the similarity threshold
+		target: searchVector,
+	},
+});
+```
+
+- `attribute` is the vector index attribute
+- `comparator` is the comparison operator (`lt`, `lte`, `gt`, `gte`, `between` are accepted)
+- `value` is the threshold value
+- `target` is the vector to compare against
+- `searchVector` is the embedding to compare against
+- Vector search can be combined with filters, sort, and limits
+- The `$distance` attribute in the `select` (optional) returns the distance between the result and the query vector
+
+---
+
+## Vector Index Options
+
+Additional tuning options can be provided on the `@indexed` directive:
+
+| Option                 | Description                                 |
+| ---------------------- | ------------------------------------------- |
+| `distance`             | Similarity metric (`cosine` or `euclidean`) |
+| `efConstruction`       | Index build quality vs performance          |
+| `M`                    | Graph connectivity per HNSW layer           |
+| `optimizeRouting`      | Improves routing efficiency                 |
+| `efSearchConstruction` | Search breadth during queries               |
+
+These options allow fine-tuning for performance and recall tradeoffs.
+
+---
+
+## How to Generate and Search Vector Embeddings
+
+Here is a full example that generates embeddings for a set of products and then searches for similar products using vector indexes. The following example shows how to generate embeddings using OpenAI or Ollama.
+
+```js
+import { Ollama } from 'ollama';
+const ollama = new Ollama({ host: 'http://127.0.0.1:11434' });
+// The name of the ollama embedding model
+const OLLAMA_EMBEDDING_MODEL = 'nomic-embed-text';
+
+const { Product } = tables;
+
+import OpenAI from 'openai';
+const openai = new OpenAI();
+// the name of the OpenAI embedding model
+const OPENAI_EMBEDDING_MODEL = 'text-embedding-3-small';
+
+const SIMILARITY_THRESHOLD = 0.5;
+
+export class ProductSearch extends Resource {
+	// based on env variable we choose the appropriate embedding generator
+	generateEmbedding = process.env.EMBEDDING_GENERATOR === 'ollama'
+		? this._generateOllamaEmbedding
+		: this._generateOpenAIEmbedding;
+
+	/**
+	 * Executes a search query using a generated text embedding and returns the matching products.
+	 *
+	 * @param {Object} data - The input data for the request.
+	 * @param {string} data.prompt - The prompt to generate the text embedding from.
+	 * @return {Promise<Array>} Returns a promise that resolves to an array of products matching the conditions,
+	 * including fields: name, description, price, and $distance.
+	 */
+	async post(data) {
+		const embedding = await this.generateEmbedding(data.prompt);
+
+		return await Product.search({
+			select: ['name', 'description', 'price', '$distance'],
+			conditions: {
+				attribute: 'textEmbeddings',
+				comparator: 'lt',
+				value: SIMILARITY_THRESHOLD,
+				target: embedding[0],
+			},
+			limit: 5,
+		});
+	}
+
+	/**
+	 * Generates an embedding using the Ollama API.
+	 *
+	 * @param {string} promptData - The input data for which the embedding is to be generated.
+	 * @return {Promise<number[][]>} A promise that resolves to the generated embedding as an array of numbers.
+	 */
+	async _generateOllamaEmbedding(promptData) {
+		const embedding = await ollama.embed({
+			model: OLLAMA_EMBEDDING_MODEL,
+			input: promptData,
+		});
+		return embedding?.embeddings;
+	}
+
+	/**
+	 * Generates OpenAI embeddings based on the given prompt data.
+	 *
+	 * @param {string} promptData - The input data used for generating the embedding.
+	 * @return {Promise<number[][]>} A promise that resolves to an array of embeddings, where each embedding is an array of floats.
+	 */
+	async _generateOpenAIEmbedding(promptData) {
+		const embedding = await openai.embeddings.create({
+			model: OPENAI_EMBEDDING_MODEL,
+			input: promptData,
+			encoding_format: 'float',
+		});
+
+		let embeddings = [];
+		embedding.data.forEach((embeddingData) => {
+			embeddings.push(embeddingData.embedding);
+		});
+
+		return embeddings;
+	}
+}
+```
+
+Sample request to the `ProductSearch` resource which prompts to find "shorts for the gym":
+
+```bash
+curl -X POST "http://localhost:9926/ProductSearch/" \
+-H "accept: \
+-H "Content-Type: application/json" \
+-H "Authorization: Basic <YOUR_AUTH>" \
+-d '{"prompt": "shorts for the gym"}'
+```
+
+---
+
+## When to Use Vector Indexing
+
+Vector indexing is ideal when:
+
+- Storing embedding vectors from ML models
+- Performing semantic or similarity-based search
+- Working with high-dimensional numeric data
+- Exact-match indexes are insufficient
+
+---
+
+## Summary
+
+- Vector indexing enables fast similarity search on numeric arrays
+- Defined using `@indexed(type: "HNSW")`
+- Queried using a target vector in search sorting
+- Tunable for performance and accuracy

package/template-vanilla/AGENTS.md

@@ -1,11 +1,11 @@
-# HarperDB Agent Skills
+# Harper Agent Skills
 
 This repository contains "skills" that guide AI agents in developing Harper applications.
 
 ## Available Skills
 
 - [Adding Tables with Schemas](skills/adding-tables-with-schemas.md): Learn how to define schemas and enable automatic REST APIs for your database tables with schema .graphql files in Harper.
-- [Automatic REST APIs](skills/automatic-rest-apis.md): Details on the CRUD endpoints automatically generated for exported tables.
+- [Automatic APIs](skills/automatic-apis.md): Details on the CRUD endpoints automatically generated for exported tables with REST and WebSockets.
 - [Querying REST APIs](skills/querying-rest-apis.md): How to use filters, operators, sorting, and pagination in REST requests.
 - [Programmatic Table Requests](skills/programmatic-table-requests.md): How to use filters, operators, sorting, and pagination in programmatic table requests.
 - [Custom Resources](skills/custom-resources.md): How to define custom REST endpoints using JavaScript or TypeScript (Note: Paths are case-sensitive).
@@ -16,3 +16,6 @@ This repository contains "skills" that guide AI agents in developing Harper appl
 - [Handling Binary Data](skills/handling-binary-data.md): How to store and serve binary data like images or MP3s.
 - [Serving Web Content](skills/serving-web-content): Two ways to serve web content from a Harper application.
 - [Checking Authentication](skills/checking-authentication.md): How to use sessions to verify user identity and roles.
+- [Caching](skills/caching.md): How caching is defined and implemented in Harper applications.
+- [Using Blobs](skills/using-blob-datatype.md): How to store and retrieve large data in HarperDB.
+- [Vector Indexing](skills/vector-indexing.md): How to define and use vector indexes for efficient similarity search.

package/template-vanilla/resources/README.md

@@ -1,6 +1,6 @@
 # Resources
 
-The [schemas you define in .GraphQL files](../skills/adding-tables-with-schemas.md) will [automatically stand-up REST APIs](../skills/automatic-rest-apis.md).
+The [schemas you define in .GraphQL files](../skills/adding-tables-with-schemas.md) will [automatically stand-up REST APIs](../skills/automatic-apis.md).
 
 But you can [extend your tables with custom logic](../skills/extending-tables.md) and [create your own resources](../skills/custom-resources.md) in this directory.
 

package/template-vanilla/skills/adding-tables-with-schemas.md

@@ -1,4 +1,4 @@
-# Adding Tables to HarperDB
+# Adding Tables to Harper
 
 To add tables to a Harper database, follow these guidelines:
 
@@ -8,7 +8,7 @@ To add tables to a Harper database, follow these guidelines:
 
 3. **Defining Relationships**: You can link tables together using the `@relationship` directive. For more details, see the [Defining Relationships](defining-relationships.md) skill.
 
-4. **Automatic REST APIs**: If you add `@table @export` to a schema type, HarperDB automatically sets up REST APIs for basic CRUD operations against that table. For a detailed list of available endpoints and how to use them, see the [Automatic REST APIs](automatic-rest-apis.md) skill.
+4. **Automatic APIs**: If you add `@table @export` to a schema type, Harper automatically sets up REST and WebSocket APIs for basic CRUD operations against that table. For a detailed list of available endpoints and how to use them, see the [Automatic REST APIs](automatic-apis.md) skill.
 
    - `GET /{TableName}`: Describes the schema itself.
    - `GET /{TableName}/`: Lists all records (supports filtering, sorting, and pagination via query parameters). See the [Querying REST APIs](querying-rest-apis.md) skill for details.

package/template-vanilla/skills/automatic-apis.md

@@ -0,0 +1,53 @@
+# Automatic APIs in Harper
+
+When you define a GraphQL type with the `@table` and `@export` directives, Harper automatically generates a fully-functional REST API and WebSocket interface for that table. This allows for immediate CRUD (Create, Read, Update, Delete) operations and real-time updates without writing any additional code.
+
+## Enabling Automatic APIs
+
+To enable the automatic REST and WebSocket APIs for a table, ensure your GraphQL schema includes the `@export` directive:
+
+```graphql
+type MyTable @table @export {
+	id: ID @primaryKey
+	# ... other fields
+}
+```
+
+## Available REST Endpoints
+
+The following endpoints are automatically created for a table named `TableName` (Note: Paths are **case-sensitive**, so `GET /TableName/` is valid while `GET /tablename/` is not):
+
+- **Describe Schema**: `GET /{TableName}`
+  Returns the schema definition and metadata for the table.
+- **List Records**: `GET /{TableName}/`
+  Lists all records in the table. This endpoint supports advanced filtering, sorting, and pagination. For more details, see the [Querying REST APIs](querying-rest-apis.md) skill.
+- **Get Single Record**: `GET /{TableName}/{id}`
+  Retrieves a single record by its primary key (`id`).
+- **Create Record**: `POST /{TableName}/`
+  Creates a new record. The request body should be a JSON object containing the record data.
+- **Update Record (Full)**: `PUT /{TableName}/{id}`
+  Replaces the entire record at the specified `id` with the provided JSON data.
+- **Update Record (Partial)**: `PATCH /{TableName}/{id}`
+  Updates only the specified fields of the record at the given `id`.
+- **Delete All/Filtered Records**: `DELETE /{TableName}/`
+  Deletes all records in the table, or a subset of records if filtering parameters are provided.
+- **Delete Single Record**: `DELETE /{TableName}/{id}`
+  Deletes the record with the specified `id`.
+
+## Automatic WebSockets
+
+In addition to REST endpoints, Harper also stands up WebSocket interfaces for exported tables. When you connect to the table's endpoint via WebSocket, you will automatically receive events whenever updates are made to that table.
+
+- **WebSocket Endpoint**: `ws://your-harper-instance/{TableName}`
+
+This is the easiest way to add real-time capabilities to your application. For more complex real-time needs, see the [Real-time Applications](real-time-apps.md) skill.
+
+## Filtering and Querying
+
+The `GET /{TableName}/` and `DELETE /{TableName}/` endpoints can be filtered using query parameters. While basic equality filters are straightforward, Harper supports a rich set of operators, sorting, and pagination.
+
+For a comprehensive guide on advanced querying, see the [Querying REST APIs](querying-rest-apis.md) skill.
+
+## Customizing Resources
+
+If the automatic APIs don't behave how you need, then you can look to [customize the resources](./custom-resources.md).

package/template-vanilla/skills/caching.md

@@ -0,0 +1,113 @@
+# Harper Caching
+
+Harper includes integrated support for **caching data from external sources**, enabling high-performance, low-latency cache storage that is fully queryable and interoperable with your applications. With built-in caching capabilities and distributed responsiveness, Harper makes an ideal **data caching server** for both edge and centralized use cases.
+
+---
+
+## What is Harper Caching?
+
+Harper caching lets you store **cached content** in standard tables, enabling you to:
+
+- Expose cached entries as **queryable structured data** (e.g., JSON or CSV)
+- Serve data to clients with **flexible formats and custom querying**
+- Manage cache control with **timestamps and ETags** for downstream caching layers
+- Implement **active or passive caching** patterns depending on your source and invalidation strategy
+
+---
+
+## Configuring a Cache Table
+
+Define a cache table in your `schema.graphql`:
+
+```graphql
+type MyCache @table(expiration: 3600) @export {
+	id: ID @primaryKey
+}
+```
+
+- `expiration` is defined in seconds
+- Expired records are refreshed on access
+- Evicted records are removed after expiration
+
+---
+
+## Connecting an External Source
+
+Create a resource:
+
+```js
+import { Resource } from 'harperdb';
+
+export class ThirdPartyAPI extends Resource {
+	async get() {
+		const id = this.getId();
+		const response = await fetch(`https://api.example.com/items/${id}`);
+		if (!response.ok) {
+			throw new Error('Source fetch failed');
+		}
+		return await response.json();
+	}
+}
+```
+
+Attach it to your table:
+
+```js
+import { tables } from 'harperdb';
+import { ThirdPartyAPI } from './ThirdPartyAPI.js';
+
+const { MyCache } = tables;
+MyCache.sourcedFrom(ThirdPartyAPI);
+```
+
+---
+
+## Cache Behavior
+
+1. Fresh data is returned immediately
+2. Missing or stale data triggers a fetch
+3. Concurrent misses are deduplicated
+
+---
+
+## Active Caching
+
+Use `subscribe()` to proactively update or invalidate cache entries:
+
+```js
+class MyAPI extends Resource {
+	async *subscribe() {
+		// stream updates
+	}
+}
+```
+
+See [Real Time Apps](real-time-apps.md) for more details.
+
+---
+
+## Write-Through Caching
+
+Propagate updates upstream:
+
+```js
+class ThirdPartyAPI extends Resource {
+	async put(data) {
+		await fetch(`https://some-api.com/${this.getId()}`, {
+			method: 'PUT',
+			body: JSON.stringify(data),
+		});
+	}
+}
+```
+
+---
+
+## Summary
+
+Harper Caching allows you to:
+
+- Cache external APIs efficiently
+- Query cached data like native tables
+- Prevent cache stampedes
+- Build real-time or write-through caches