@harperfast/skills 1.6.0 → 1.7.0

package/harper-best-practices/rules/logging.md

@@ -1,94 +1,169 @@
 ---
 name: logging
-description: Best practices for logging in Harper, including console capture, the granular logger interface, and programmatic log retrieval.
+description: >-
+  Best practices for logging in Harper, including console capture, the granular
+  logger interface, and programmatic log retrieval.
 metadata:
-  mode: synthesized
+  mode: generate
+  sources:
+    - reference/v5/logging/overview.md
+    - reference/v5/logging/api.md
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: 46cd384598304e3b
 ---
 
-# Logging Best Practices
+# Harper Logging
 
-Harper provides a robust logging system that captures standard output and offers a granular, tagged logging interface for both local and deployed environments.
+Instructions for the agent to follow when implementing logging in Harper applications, including direct logger usage, tagged loggers, and console capture behavior.
 
-## Standard Console Logging
+## When to Use
 
-The simplest way to log in Harper is using standard JavaScript console methods. `console.log()`, `console.warn()`, `console.error()`, and `console.trace()` are automatically captured by Harper and can be viewed in the logs.
-
-- `console.log(...)`: Captured as `stdout` level in Harper logs.
-- `console.warn(...)`: Captured as `stderr` level in Harper logs.
-- `console.error(...)`: Captured as `stderr` level in Harper logs.
-- `console.trace(...)`: Captured as `stdout` level in Harper logs (includes stack trace).
-
-## Harper Logger
-
-For more granularity and better organization, use Harper's built-in `logger`. You can use the global `logger` object or import it from the `harper` package.
-
-### Log Levels
-
-The Harper `logger` supports the following levels (ordered by increasing severity):
-
-- `trace`
-- `debug`
-- `info`
-- `warn`
-- `error`
-- `fatal`
-- `notify`
-
-### Usage
-
-```typescript
-import { logger, loggerWithTag } from 'harper';
-
-// Basic logging
-logger.info('Application started');
-logger.error('An error occurred', error);
-
-// Tagged logging for better filtering (Namespacing)
-const authLogger = loggerWithTag('auth');
-authLogger.debug('User login attempt', { userId: '123' });
+Apply this rule when writing any JavaScript component, plugin, or resource that needs to emit structured log entries, filter logs by component, or capture existing `console.log` output into Harper's log system. Use it whenever you need to understand log levels, log entry format, or the `logger` global API.
+
+## How It Works
+
+1. **Use the `logger` global directly** — `logger` is available in all JavaScript components without any imports. Call the method matching the desired severity level:
+
+   ```javascript
+   logger.trace('detailed trace message');
+   logger.debug('debug info', { someContext: 'value' });
+   logger.info('informational message');
+   logger.warn('potential issue');
+   logger.error('error occurred', error);
+   logger.fatal('fatal error');
+   logger.notify('server is ready');
+   ```
+
+   Only entries at or above the configured `logging.level` (or `logging.external.level`) are written to `hdb.log`.
+
+2. **Create a tagged logger with `withTag(`** — Call `logger.withTag(tag)` once per module or class to get a `TaggedLogger` scoped to that tag. This prefixes every log entry with the tag, making log output filterable by component.
+
+   ```javascript
+   const log = logger.withTag('my-resource');
+   ```
+
+   Because `TaggedLogger` methods for disabled levels are `null`, always use optional chaining (`?.`) when calling them:
+
+   ```javascript
+   log.debug?.('Fetching record', { id });
+   log.warn?.('Record not found', { id });
+   log.error?.('Failed to update record', err);
+   ```
+
+   `TaggedLogger` does not have a `withTag()` method.
+
+3. **Understand the interface contracts** — `MainLogger` always has all methods defined:
+
+   ```typescript
+   interface MainLogger {
+   	trace(...messages: any[]): void;
+   	debug(...messages: any[]): void;
+   	info(...messages: any[]): void;
+   	warn(...messages: any[]): void;
+   	error(...messages: any[]): void;
+   	fatal(...messages: any[]): void;
+   	notify(...messages: any[]): void;
+   	withTag(tag: string): TaggedLogger;
+   }
+   ```
+
+   `TaggedLogger` methods may be `null`:
+
+   ```typescript
+   interface TaggedLogger {
+   	trace: ((...messages: any[]) => void) | null;
+   	debug: ((...messages: any[]) => void) | null;
+   	info: ((...messages: any[]) => void) | null;
+   	warn: ((...messages: any[]) => void) | null;
+   	error: ((...messages: any[]) => void) | null;
+   	fatal: ((...messages: any[]) => void) | null;
+   	notify: ((...messages: any[]) => void) | null;
+   }
+   ```
+
+4. **Know the log levels** — From least to most severe:
+
+   | Level    | Description                                                          |
+   | -------- | -------------------------------------------------------------------- |
+   | `trace`  | Highly detailed internal execution tracing.                          |
+   | `debug`  | Diagnostic information useful during development.                    |
+   | `info`   | General operational events.                                          |
+   | `warn`   | Potential issues that don't prevent normal operation.                |
+   | `error`  | Errors that affect specific operations.                              |
+   | `fatal`  | Critical errors causing process termination.                         |
+   | `notify` | Important operational milestones. Always logged regardless of level. |
+
+   The default log level is `warn`. Setting a level includes that level and all more-severe levels.
+
+5. **Enable console capture when porting existing code** — When `logging.console: true` is set, writes via `console.log`, `console.warn`, `console.error`, etc. are appended verbatim to `hdb.log`. Captured lines do **not** pass through `logger`'s level filter. Prefer `logger` directly in production code so that level filtering and tagging apply. Console capture is intended as a convenience for porting existing code and for debugging.
+
+6. **Know where logs are written** — All standard log output goes to `<ROOTPATH>/log/hdb.log` (default: `~/hdb/log/hdb.log`). To also log to `stdout`/`stderr`, set `logging.stdStreams: true`.
+
+## Examples
+
+### Basic logging in a resource
+
+```javascript
+export class MyResource extends Resource {
+	async get(id) {
+		logger.debug('Fetching record', { id });
+		const record = await super.get(id);
+		if (!record) {
+			logger.warn('Record not found', { id });
+		}
+		return record;
+	}
+
+	async put(record) {
+		logger.info('Updating record', { id: record.id });
+		try {
+			return await super.put(record);
+		} catch (err) {
+			logger.error('Failed to update record', err);
+			throw err;
+		}
+	}
+}
 ```
 
-Using `loggerWithTag` is highly recommended for grouping related logs, making them much easier to filter and analyze in the Harper Studio or via the API.
-
-## Programmatic Log Retrieval
-
-You can programmatically read logs from a deployed Harper instance using the `read_log` operation. This is useful for building custom monitoring tools or debugging dashboards.
-
-### `read_log` Operation
-
-The `read_log` operation is a POST request to the Harper instance.
-
-**Example Request:**
-
-```json
-{
-	"operation": "read_log",
-	"limit": 100,
-	"start": 0,
-	"level": "error",
-	"order": "desc",
-	"from": "2024-01-01T00:00:00.000Z",
-	"until": "2024-01-02T00:00:00.000Z"
+### Tagged logging with `withTag()`
+
+```javascript
+const log = logger.withTag('my-resource');
+
+export class MyResource extends Resource {
+	async get(id) {
+		log.debug?.('Fetching record', { id });
+		const record = await super.get(id);
+		if (!record) {
+			log.warn?.('Record not found', { id });
+		}
+		return record;
+	}
+
+	async put(record) {
+		log.info?.('Updating record', { id: record.id });
+		try {
+			return await super.put(record);
+		} catch (err) {
+			log.error?.('Failed to update record', err);
+			throw err;
+		}
+	}
 }
 ```
 
-### Parameters
-
-- `limit`: Number of log entries to return.
-- `start`: Offset for pagination.
-- `level`: Filter by log level (`info`, `error`, `warn`, `debug`, `trace`, `notify`, `fatal`, `stdout`, `stderr`).
-- `from`: ISO 8601 timestamp to start reading from.
-- `until`: ISO 8601 timestamp to stop reading at.
-- `order`: Sort order, either `asc` or `desc`.
-- `replicated`: (Boolean) Include logs from replicated nodes in a cluster.
+Tagged entries appear in `hdb.log` with the tag in the header:
 
-### Log Entry Structure
+```
+2023-03-09T14:25:05.269Z [info] [my-resource]: Updating record
+```
 
-Each log entry returned by `read_log` typically includes:
+## Notes
 
-- `level`: The severity level of the log.
-- `timestamp`: When the log was recorded.
-- `thread`: The execution thread.
-- `tags`: An array of tags (e.g., from `loggerWithTag`).
-- `node`: The node name in a Harper cluster.
-- `message`: The logged content.
+- All log output is written to `<ROOTPATH>/log/hdb.log`. The `logger` global writes to this file at the configured `logging.external` level.
+- Log entry format for `logger`: `<timestamp> [<level>] [<thread>/<id>]: <message>`
+- Log entry format for `TaggedLogger`: `<timestamp> [<level>] [<tag>]: <message>`
+- `console.log` output is only forwarded to `hdb.log` when `logging.console: true` is explicitly set; it is not forwarded by default.
+- When logging to standard streams, run Harper in the foreground (`harper`, not `harper start`).
+- `TaggedLogger` is bound to the configured log level at creation time — always use `?.` on its methods.

package/harper-best-practices/rules/querying-rest-apis.md

@@ -1,29 +1,202 @@
 ---
 name: querying-rest-apis
-description: How to use query parameters to filter, sort, and paginate Harper REST APIs.
+description: 'How to use query parameters to filter, sort, and paginate Harper REST APIs.'
 metadata:
-  mode: synthesized
+  mode: generate
+  sources:
+    - reference/v5/rest/querying.md
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: 9f8c981a629ef606
 ---
 
 # Querying REST APIs
 
-Instructions for the agent to follow when querying Harper's REST APIs.
+Instructions for the agent to filter, sort, select, and paginate Harper REST API collections using URL query parameters.
 
 ## When to Use
 
-Use this skill when you need to perform advanced data retrieval (filtering, sorting, pagination, joins) using Harper's automatic REST endpoints.
+Apply this rule when building or modifying code that queries Harper REST endpoints with filtering, sorting, field selection, or pagination. Use it whenever constructing URLs against collection paths exposed by Harper's automatic REST interface (see [automatic-apis.md](automatic-apis.md)).
 
 ## How It Works
 
-1. **Basic Filtering**: Use attribute names as query parameters: `GET /Table/?key=value`.
-2. **Use Comparison Operators**: Append operators like `gt`, `ge`, `lt`, `le`, `ne` using FIQL-style syntax: `GET /Table/?price=gt=100`.
-3. **Apply Logic and Grouping**: Use `&` for AND, `|` for OR, and `()` for grouping: `GET /Table/?(rating=5|featured=true)&price=lt=50`.
-4. **Select Specific Fields**: Use `select()` to limit returned attributes: `GET /Table/?select(name,price)`.
-5. **Paginate Results**: Use `limit(count)` or `limit(offset, count)` to set the number of records to return and skip.
-   - Example (first 10): `GET /Table/?limit(10)`
-   - Example (skip 20, return 10): `GET /Table/?limit(20, 10)`
-6. **Sort Results**: Use `sort()` with `+` (asc) or `-` (desc) before the field name. Avoid `sort=field` format.
-   - Example (asc): `GET /Table/?sort(+name)`
-   - Example (desc): `GET /Table/?sort(-price)`
-   - Example (combined): `GET /Table/?sort(-price,+name)`
-7. **Query Relationships**: Use dot syntax for tables linked with `@relationship`: `GET /Book/?author.name=Harper`.
+1. **Filter by attribute**: Add query parameters matching attribute names and values. The queried attribute must be indexed.
+
+   ```
+   GET /Product/?category=software
+   GET /Product/?category=software&inStock=true
+   ```
+
+2. **Apply comparison operators (FIQL syntax)**: Use FIQL operators directly in query parameter values.
+
+   | Operator     | Meaning                                |
+   | ------------ | -------------------------------------- |
+   | `==`         | Equal                                  |
+   | `=lt=`       | Less than                              |
+   | `=le=`       | Less than or equal                     |
+   | `=gt=`       | Greater than                           |
+   | `=ge=`       | Greater than or equal                  |
+   | `=ne=`, `!=` | Not equal                              |
+   | `=ct=`       | Contains (strings)                     |
+   | `=sw=`       | Starts with (strings)                  |
+   | `=ew=`       | Ends with (strings)                    |
+   | `=`, `===`   | Strict equality (no type conversion)   |
+   | `!==`        | Strict inequality (no type conversion) |
+
+   ```
+   GET /Product/?price=gt=100
+   GET /Product/?price=le=20
+   GET /Product/?name==Keyboard*
+   GET /Product/?category=software&price=gt=100&price=lt=200
+   ```
+
+   For date fields, URL-encode colons as `%3A`:
+
+   ```
+   GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z
+   ```
+
+3. **Chain conditions for range queries**: Omit the attribute name on the second condition to apply it to the same attribute. Only `gt`/`ge` combined with `lt`/`le` is supported.
+
+   ```
+   GET /Product/?price=gt=100&lt=200
+   ```
+
+4. **Combine conditions with OR logic**: Use `|` instead of `&`.
+
+   ```
+   GET /Product/?rating=5|featured=true
+   ```
+
+5. **Group conditions**: Use parentheses or square brackets to control order of operations. Prefer square brackets when constructing queries from user input, since standard URI encoding safely encodes `[` and `]`.
+
+   ```
+   GET /Product/?rating=5|(price=gt=100&price=lt=200)
+   GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient]
+   ```
+
+   Construct grouped queries from JavaScript:
+
+   ```javascript
+   let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`;
+   ```
+
+6. **Select specific properties with `select(`**: Use `select()` to control which fields are returned.
+
+   | Syntax                                 | Returns                                     |
+   | -------------------------------------- | ------------------------------------------- |
+   | `?select(property)`                    | Values of a single property directly        |
+   | `?select(property1,property2)`         | Objects with only the specified properties  |
+   | `?select([property1,property2])`       | Arrays of property values                   |
+   | `?select(property1,)`                  | Objects with a single specified property    |
+   | `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties |
+
+   ```
+   GET /Product/?category=software&select(name)
+   GET /Product/?brand.name=Microsoft&select(name,brand{name})
+   ```
+
+7. **Limit results with `limit(`**: Use `limit(end)` or `limit(start,end)` to paginate.
+
+   ```
+   GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20)
+   GET /Product/?rating=gt=3&limit(10,30)
+   ```
+
+8. **Sort results with `sort(`**: Use `sort(property)` or `sort(+property,-property,...)`. Prefix `+` or no prefix = ascending; `-` = descending.
+
+   ```
+   GET /Product/?rating=gt=3&sort(+name)
+   GET /Product/?sort(+rating,-price)
+   ```
+
+9. **Query across relationships**: Use dot-syntax to filter by related table attributes. Relationships must be defined in the schema using `@relation`.
+
+   ```
+   GET /Product/?brand.name=Microsoft
+   GET /Brand/?products.name=Keyboard
+   ```
+
+   Use `select()` to include relationship attributes in the response (they are not included by default):
+
+   ```
+   GET /Product/?brand.name=Microsoft&select(name,brand{name})
+   ```
+
+10. **Access a specific property by URL**: Append the property name with dot syntax to the record ID. Only works for properties declared in the schema.
+    ```
+    GET /MyTable/123.propertyName
+    ```
+
+## Examples
+
+**Range filter with select and limit:**
+
+```
+GET /Product/?category=software&price=gt=100&price=lt=200&select(name,price)&limit(20)
+```
+
+**Sort descending with multiple fields:**
+
+```
+GET /Product/?sort(+rating,-price)
+```
+
+**OR logic with grouping:**
+
+```
+GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inStock=true]
+```
+
+**Relationship join with nested select:**
+
+```
+GET /Product/?brand.name=Microsoft&select(name,brand{name,id})
+```
+
+**Schema defining a relationship for join queries:**
+
+```graphql
+type Product @table @export {
+	id: Long @primaryKey
+	name: String
+	brandId: Long @indexed
+	brand: Brand @relation(from: "brandId")
+}
+type Brand @table @export {
+	id: Long @primaryKey
+	name: String
+	products: [Product] @relation(to: "brandId")
+}
+```
+
+**Many-to-many relationship query:**
+
+```graphql
+type Product @table @export {
+	id: Long @primaryKey
+	name: String
+	resellerIds: [Long] @indexed
+	resellers: [Reseller] @relation(from: "resellerId")
+}
+```
+
+```
+GET /Product/?resellers.name=Cool Shop&select(id,name,resellers{name,id})
+```
+
+**Type conversion with explicit prefix:**
+
+```
+GET /Product/?price==number:123
+GET /Product/?active==boolean:true
+GET /Product/?listDate==date:2024-01-05T20%3A07%3A27.955Z
+```
+
+## Notes
+
+- Only indexed attributes can be used as the primary filter; additional unindexed attributes can be combined with `&` once at least one indexed attribute is present.
+- For null value queries, use `?attribute=null`. Indexes must have been created with null indexing support; existing indexes must be removed and re-added to support null queries.
+- FIQL comparators (`==`, `!=`, `=gt=`, etc.) apply automatic type conversion based on value syntax or schema-declared type. Strict operators (`=`, `===`, `!==`) skip automatic type conversion.
+- Filtering by a related attribute produces INNER JOIN behavior (only records with a matching related record are returned). Using `select()` on a relationship without a filter produces LEFT JOIN behavior.
+- The array order of foreign key values in many-to-many relationships is preserved when resolving the relationship.
+- See [automatic-apis.md](automatic-apis.md) for how Harper tables are automatically exposed as REST endpoints.

package/harper-best-practices/rules/real-time-apps.md

@@ -2,44 +2,101 @@
 name: real-time-apps
 description: How to build real-time features in Harper using WebSockets and Pub/Sub.
 metadata:
-  mode: synthesized
+  mode: generate
+  sources:
+    - reference/v5/rest/websockets.md
+  sourceCommit: b7fbddadd42eb4487190b650a9abc4bcfeef5819
+  inputHash: a8afd4d3a52f77ba
 ---
 
-# Real-time Applications
+# Real-Time Apps with WebSockets and Pub/Sub
 
-Instructions for the agent to follow when building real-time applications in Harper.
+Instructions for the agent to follow when building real-time features in Harper using WebSockets and Pub/Sub.
 
 ## When to Use
 
-Use this skill when you need to stream live updates to clients, implement chat features, or provide real-time data synchronization between the database and a frontend.
+Apply this rule when implementing any feature that requires real-time bidirectional communication, live data streaming, or push-based updates in a Harper application. This includes chat, live dashboards, sensor feeds, and any scenario where clients must receive resource changes as they happen.
 
 ## How It Works
 
-1. **Check Automatic WebSockets**: If you only need to stream table changes, use [Automatic APIs](automatic-apis.md) which provide a WebSocket endpoint for every `@export`ed table.
-2. **Implement `connect` in a Resource**: For custom bi-directional logic, implement the `connect` method.
-3. **Use Pub/Sub**: Use `tables.TableName.subscribe(query)` to listen for specific data changes and stream them to the client.
-4. **Handle SSE**: Ensure your `connect` method gracefully handles cases where `incomingMessages` is null (Server-Sent Events).
-5. **Connect from Client**: Use standard WebSockets (`new WebSocket('wss://...')`) to connect to your resource endpoint. Ensure you use the appropriate scheme (`ws://` for HTTP, `wss://` for HTTPS).
+1. **Enable WebSocket support**: WebSocket support is enabled automatically when the `rest` plugin is enabled. To explicitly disable it, set the following in your config:
 
-## Examples
+   ```yaml
+   rest:
+     webSocket: false
+   ```
 
-### Bi-directional WebSocket Resource
+2. **Connect a client to a resource**: A WebSocket connection to a resource URL automatically subscribes to that resource. When the record changes or a message is published to it, the connection receives the update.
 
-```typescript
-import { Resource, tables } from 'harper';
+   ```javascript
+   let ws = new WebSocket('wss://server/my-resource/341');
+   ws.onmessage = (event) => {
+   	let data = JSON.parse(event.data);
+   };
+   ```
 
-export class MySocket extends Resource {
-	async *connect(target, incomingMessages) {
-		// Subscribe to table changes
-		const subscription = await tables.MyTable.subscribe(target);
-		if (!incomingMessages) {
-			return subscription; // SSE mode
-		}
+   `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it.
+
+3. **Implement a custom `connect()` handler**: Override the `connect(incomingMessages)` method on a resource class to control WebSocket behavior. The method must return an async iterable (or generator) that produces messages to send to the client. See [automatic-apis.md](automatic-apis.md) for more on defining resource classes.
+
+4. **Use the default `connect()` for event-style access**: Call `super.connect()` to get a streaming iterable that provides:
+   - A `send(message)` method for pushing outgoing messages
+   - A `close` event for cleanup on disconnect
+
+5. **Handle message ordering in distributed environments**: Harper delivers messages to local subscribers immediately without inter-node coordination delay.
+
+   | Message Type                                             | Behavior                                                                |
+   | -------------------------------------------------------- | ----------------------------------------------------------------------- |
+   | Non-retained (no `retain` flag)                          | Every message delivered in order received; suitable for chat            |
+   | Retained (published with `retain`, or PUT/updated in DB) | Only the latest-timestamp message is kept; suitable for sensor readings |
 
-		// Handle incoming client messages
+6. **Use MQTT over WebSockets** when needed by setting the sub-protocol header:
+   ```
+   Sec-WebSocket-Protocol: mqtt
+   ```
+
+## Examples
+
+**Simple echo server** — override `connect(incomingMessages)` to yield each incoming message back to the client:
+
+```javascript
+export class Echo extends Resource {
+	async *connect(incomingMessages) {
 		for await (let message of incomingMessages) {
-			yield { received: message };
+			yield message; // echo each message back
 		}
 	}
 }
 ```
+
+**Custom connect with timer and event-style access** — use `super.connect()` to get the outgoing stream, push periodic messages, echo incoming messages, and clean up on disconnect:
+
+```javascript
+export class Example extends Resource {
+	connect(incomingMessages) {
+		let outgoingMessages = super.connect();
+
+		let timer = setInterval(() => {
+			outgoingMessages.send({ greeting: 'hi again!' });
+		}, 1000);
+
+		incomingMessages.on('data', (message) => {
+			outgoingMessages.send(message); // echo incoming messages
+		});
+
+		outgoingMessages.on('close', () => {
+			clearInterval(timer);
+		});
+
+		return outgoingMessages;
+	}
+}
+```
+
+## Notes
+
+- WebSocket connections target a resource URL path. By default, connecting to a resource subscribes to changes for that resource.
+- The `connect(incomingMessages)` method **must** return an async iterable or generator; returning a plain value will not work.
+- `super.connect()` returns a streaming iterable with `send(message)` and a `close` event — use this when you need to push messages outside of the incoming message loop.
+- For one-way real-time streaming without bidirectional communication, consider Server-Sent Events instead.
+- For full pub/sub capabilities, Harper also supports MQTT; set `Sec-WebSocket-Protocol: mqtt` to use MQTT over WebSockets.