@harperfast/skills 1.6.1 → 1.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -40,70 +40,153 @@ type ExamplePerson @table @export {
40
40
  }
41
41
  ```
42
42
 
43
- ### 1.2 Schema Design & Tooling
43
+ ### 1.2 Schema Design and Tooling
44
44
 
45
- Harper uses GraphQL schemas to define database tables, relationships, and APIs. To ensure the best development experience for both humans and AI agents, it's important to understand the core directives and configure your project tooling correctly.
45
+ Instructions for the agent to follow when designing Harper schemas, applying core directives, and configuring GraphQL tooling.
46
46
 
47
- #### Core Harper Directives
47
+ #### When to Use
48
48
 
49
- Harper extends GraphQL with custom directives that define database behavior. These are typically defined in `node_modules/harper/schema.graphql`. If you don't have access to that file, here is a reference of the most important ones:
49
+ Apply this rule when creating or modifying Harper schema files, configuring `graphqlSchema` in `config.yaml`, or deciding which directives to apply to tables and fields. Use it any time a component needs tables, indexes, primary keys, or exported endpoints defined.
50
50
 
51
- ##### Table Definition
51
+ #### How It Works
52
52
 
53
- - `@table`: Marks a GraphQL type as a Harper database table.
54
- - `@export`: Automatically generates REST and WebSocket APIs for the table.
55
- - `@table(expiration: Int)`: Configures a time-to-expire for records in the table (useful for caching).
53
+ 1. **Create a GraphQL schema file** with Harper-specific directives. Name it (e.g., `schema.graphql`) and place it in your component directory.
56
54
 
57
- ##### Attribute Constraints & Indexing
55
+ ```graphql
56
+ type Dog @table {
57
+ id: Long @primaryKey
58
+ name: String
59
+ breed: String
60
+ age: Int
61
+ }
58
62
 
59
- - `@primaryKey`: Specifies the unique identifier for the table.
60
- - `@indexed`: Creates a standard index on the field for faster lookups.
61
- - `@indexed(type: "HNSW", distance: "cosine" | "euclidean" | "dot")`: Creates a vector index for similarity search.
63
+ type Breed @table {
64
+ id: Long @primaryKey
65
+ name: String @indexed
66
+ }
67
+ ```
62
68
 
63
- ##### Relationships
69
+ 2. **Register the schema in `config.yaml`** using the `graphqlSchema` plugin key:
64
70
 
65
- - `@relationship(from: String)`: Defines a relationship to another table. `from` specifies the local field holding the foreign key.
71
+ ```yaml
72
+ graphqlSchema:
73
+ files: 'schema.graphql'
74
+ ```
66
75
 
67
- ##### Authentication & Authorization
76
+ Both plugins and applications can specify schemas this way.
68
77
 
69
- - `@auth(role: String)`: Restricts access to a table or field based on user roles.
78
+ 3. **Mark every table type with `@table`**. The type name becomes the table name by default. Use optional arguments to override behavior:
70
79
 
71
- #### Configuring GraphQL Tooling
80
+ | Argument | Type | Default | Description |
81
+ | -------------- | --------- | ----------------------------- | ------------------------------------------------------------- |
82
+ | `table` | `String` | type name | Override the table name |
83
+ | `database` | `String` | `"data"` | Database to place the table in |
84
+ | `expiration` | `Int` | — | Seconds until a record goes stale |
85
+ | `eviction` | `Int` | `0` | Additional seconds after `expiration` before physical removal |
86
+ | `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans |
87
+ | `replicate` | `Boolean` | `true` | Enable replication of this table |
72
88
 
73
- To get the best IDE support (autocompletion, validation) and to help AI agents understand your schema context, you should create a `graphql.config.yml` file in your project root.
89
+ 4. **Designate a primary key on every table** using `@primaryKey`. Primary keys must be unique; duplicate-key inserts are rejected. If no key is provided on insert, Harper auto-generates one:
90
+ - `String` or `ID` → UUID string
91
+ - `Int`, `Long`, or `Any` → auto-incrementing integer
74
92
 
75
- This file tells GraphQL tools where to find Harper's built-in types and directives alongside your own schema files.
93
+ Prefer `Long` or `Any` for auto-generated numeric keys; `Int` is 32-bit and may be insufficient for large tables.
76
94
 
77
- ##### Creating `graphql.config.yml`
95
+ 5. **Index fields that need fast querying** with `@indexed`. This is required for filtering by that attribute in REST queries, SQL, or NoSQL operations. If the field value is an array, each element is individually indexed.
96
+
97
+ ```graphql
98
+ type Product @table {
99
+ id: Long @primaryKey
100
+ category: String @indexed
101
+ price: Float @indexed
102
+ }
103
+ ```
78
104
 
79
- Create a file named `graphql.config.yml` in your project root with the following content:
105
+ 6. **Expose a table as an external resource endpoint** with `@export`. This makes the table accessible via REST, MQTT, and other interfaces. The optional `name` parameter sets the URL path segment; without it, the type name is used.
80
106
 
81
- ```yaml
82
- schema:
83
- - 'node_modules/harper/schema.graphql'
84
- - 'schema.graphql'
85
- - 'schemas/*.graphql'
107
+ ```graphql
108
+ type MyTable @table @export(name: "my-table") {
109
+ id: Long @primaryKey
110
+ }
111
+ ```
112
+
113
+ 7. **Restrict extra properties** with `@sealed` when records must not include attributes beyond those declared. By default, Harper allows additional properties.
114
+
115
+ ```graphql
116
+ type StrictRecord @table @sealed {
117
+ id: Long @primaryKey
118
+ name: String
119
+ }
120
+ ```
121
+
122
+ 8. **Configure expiration, eviction, and scan behavior** together when building caching tables. These three arguments control the full record lifecycle:
123
+ - `expiration` — record becomes stale; next request triggers a source fetch
124
+ - `eviction` — additional time after `expiration` before physical removal
125
+ - `scanInterval` — how often Harper scans for records to evict; clock-aligned, not startup-aligned
126
+
127
+ #### Examples
128
+
129
+ **Caching table with tuned expiration:**
130
+
131
+ ```graphql
132
+ # Expire after 5 minutes, evict after 1 hour, scan every 10 minutes
133
+ type WeatherCache @table(expiration: 300, eviction: 3300, scanInterval: 600) {
134
+ id: ID @primaryKey
135
+ temperature: Float
136
+ }
86
137
  ```
87
138
 
88
- ##### Why this is important:
139
+ **Table in a named database with expiration and an indexed field:**
89
140
 
90
- 1. **Shared Directives**: It includes `@table`, `@primaryKey`, etc., so they aren't marked as "unknown directives".
91
- 2. **Context for Agents**: When an agent reads your project, seeing this config helps it locate the core Harper definitions, leading to more accurate code generation.
92
- 3. **Consistency**: The `npm create harper@latest` command includes this by default. Manually adding it to existing projects ensures they follow the same standards.
141
+ ```graphql
142
+ type Event @table(database: "analytics", expiration: 86400) {
143
+ id: Long @primaryKey
144
+ name: String @indexed
145
+ }
146
+ ```
93
147
 
94
- #### Example Project Structure
148
+ **Session cache with auto-expiry:**
95
149
 
96
- A typical Harper project with proper schema tooling:
150
+ ```graphql
151
+ type Session @table(expiration: 3600) {
152
+ id: Long @primaryKey
153
+ userId: String
154
+ }
155
+ ```
156
+
157
+ **Table with audit timestamps:**
97
158
 
98
- ```text
99
- my-harper-app/
100
- ├── config.yaml
101
- ├── graphql.config.yml
102
- ├── package.json
103
- ├── schema.graphql
104
- └── resources.js
159
+ ```graphql
160
+ type Order @table @export(name: "orders") {
161
+ id: Long @primaryKey
162
+ createdAt: Long @createdTime
163
+ updatedAt: Long @updatedTime
164
+ status: String @indexed
165
+ }
105
166
  ```
106
167
 
168
+ **Overriding the table name and disabling replication:**
169
+
170
+ ```graphql
171
+ type Product @table(table: "products") {
172
+ id: Long @primaryKey
173
+ name: String
174
+ }
175
+
176
+ type LocalRecord @table(replicate: false) {
177
+ id: Long @primaryKey
178
+ value: String
179
+ }
180
+ ```
181
+
182
+ #### Notes
183
+
184
+ - Use unique `database` names in plugins and applications to avoid table naming collisions, since all tables default to the `"data"` database.
185
+ - Eviction removes non-indexed record data but does **not** remove a record from its secondary indexes. Indexes remain functional for evicted records; Harper fetches the full record from the source on demand when a query matches an evicted record.
186
+ - `scanInterval` is clock-aligned to the server's local timezone. The server's startup time does not affect when eviction runs.
187
+ - If replication is disabled on a table and later re-enabled, it will not catch up on writes made while replication was disabled.
188
+ - Null values are indexed by `@indexed` fields, enabling queries such as `GET /Product/?category=null`.
189
+
107
190
  ### 1.3 Defining Relationships
108
191
 
109
192
  Instructions for the agent to follow when defining relationships between Harper tables.
@@ -143,7 +226,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
143
226
 
144
227
  #### How It Works
145
228
 
146
- 1. **Declare a vector index on a `[Float]` field**: Add `@indexed(type: "HNSW")` to any `[Float]` attribute in a `@table` type. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for general schema setup.
229
+ 1. **Declare the vector index on a `[Float]` field**: Add `@indexed(type: "HNSW")` to any `[Float]` attribute in a `@table` type. See [adding-tables-with-schemas.md](adding-tables-with-schemas.md) for general schema setup.
147
230
 
148
231
  ```graphql
149
232
  type Document @table {
@@ -152,7 +235,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
152
235
  }
153
236
  ```
154
237
 
155
- 2. **Query by nearest neighbors using `sort`**: Call `Document.search()` with a `sort` object specifying `attribute` (the indexed field) and `target` (the query vector). Include `limit` to cap results.
238
+ 2. **Query by nearest neighbors using `sort`**: Call `Document.search()` with a `sort` object containing `attribute` (the indexed field name) and `target` (the query vector). Include `limit` to cap results.
156
239
 
157
240
  ```javascript
158
241
  let results = Document.search({
@@ -161,7 +244,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
161
244
  });
162
245
  ```
163
246
 
164
- 3. **Combine HNSW with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
247
+ 3. **Combine with filter conditions**: Add a `conditions` array alongside `sort` to pre-filter records before ranking by similarity.
165
248
 
166
249
  ```javascript
167
250
  let results = Document.search({
@@ -171,7 +254,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
171
254
  });
172
255
  ```
173
256
 
174
- 4. **Filter by distance threshold**: Place `target` directly on a condition (alongside `attribute`, `comparator`, and `value`) to return only records whose distance to the target vector is below a threshold. Use this form to bound result quality by a similarity cutoff rather than ranking.
257
+ 4. **Filter by distance threshold**: To return only records within a similarity cutoff (without ranking), place `target` directly on the condition alongside `comparator` and `value`. Omit `sort`.
175
258
 
176
259
  ```javascript
177
260
  let results = Document.search({
@@ -184,7 +267,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
184
267
  });
185
268
  ```
186
269
 
187
- 5. **Include computed distance in results**: Add `'$distance'` to the `select` array to return the computed distance from the target vector alongside each record. `$distance` works in both `sort`-based and `conditions`-based queries.
270
+ 5. **Include computed distance in results**: Use the special `$distance` field in `select` to return the distance from the target vector. Works with both `sort`-based and `conditions`-based queries.
188
271
 
189
272
  ```javascript
190
273
  let results = Document.search({
@@ -194,7 +277,7 @@ Apply this rule when adding a vector index to a Harper table schema or writing s
194
277
  });
195
278
  ```
196
279
 
197
- 6. **Tune HNSW parameters**: Pass additional parameters to `@indexed(type: "HNSW", ...)` to control index quality and performance:
280
+ 6. **Tune HNSW parameters**: Pass additional parameters to `@indexed(type: "HNSW", ...)` to control index quality and performance.
198
281
 
199
282
  | Parameter | Default | Description |
200
283
  | ---------------------- | ----------------- | --------------------------------------------------------------------------------------------------- |
@@ -217,7 +300,7 @@ type Document @table {
217
300
  }
218
301
  ```
219
302
 
220
- **Nearest-neighbor search with distance output:**
303
+ **Nearest-neighbor search with distance score:**
221
304
 
222
305
  ```javascript
223
306
  let results = Document.search({
@@ -242,10 +325,10 @@ let results = Document.search({
242
325
 
243
326
  #### Notes
244
327
 
245
- - The default `distance` function is `cosine`. To use Euclidean distance, set `distance: "euclidean"` in the `@indexed` directive.
246
- - `efConstruction` controls index build quality; increase it to improve recall at the cost of slower indexing.
247
- - `$distance` is a special field prefix it with `$` exactly as shown; it is not a schema attribute.
248
- - `target` is required in both `sort`-based and threshold-based condition queries to identify the reference vector for distance computation.
328
+ - The default `distance` function is `cosine`. Pass `distance: "euclidean"` to switch.
329
+ - `efConstruction` controls index build quality; raising it improves recall at the cost of build time.
330
+ - `$distance` is available in both `sort`-based ranking and `conditions`-based threshold queries.
331
+ - Use the threshold (`conditions` + `target`) form when you want to bound result quality by a similarity cutoff rather than ranking by similarity.
249
332
 
250
333
  ### 1.5 Using Blob Datatype
251
334
 
@@ -322,298 +405,617 @@ Use this skill when you need to store binary files (images, audio, etc.) in the
322
405
 
323
406
  ### 2.1 Automatic APIs
324
407
 
325
- Instructions for the agent to follow when utilizing Harper's automatic APIs.
408
+ Instructions for the agent to follow when enabling and using Harper's automatically generated REST and WebSocket APIs.
326
409
 
327
410
  #### When to Use
328
411
 
329
- Use this skill when you want to interact with Harper tables via REST or WebSockets without writing custom resource logic. This is ideal for basic CRUD operations and real-time updates.
412
+ Apply this rule when adding REST or WebSocket API access to Harper tables or custom resources. Use it when configuring `config.yaml` to expose endpoints, mapping HTTP methods to resource operations, or implementing real-time WebSocket connections on a resource class.
330
413
 
331
414
  #### How It Works
332
415
 
333
- 1. **Enable REST in `config.yaml`**: REST endpoints are **not active by default**. You must explicitly enable them:
416
+ 1. **Enable the REST plugin**: Add `rest: true` to your application's `config.yaml`. This activates the HTTP REST interface and enables WebSocket support by default.
417
+
334
418
  ```yaml
335
419
  rest: true
336
420
  ```
337
- Without this, `@export`ed tables will not respond to HTTP requests.
338
- 2. **Enable Automatic APIs**: Ensure your GraphQL schema includes the `@export` directive for the table.
339
- 3. **Access REST Endpoints**: Use the standard endpoints for your table (Note: Paths are case-sensitive).
340
- 4. **Use Automatic WebSockets**: Connect to `wss://your-harper-instance/{TableName}` to receive events whenever updates are made to that table. This is the easiest way to add real-time capabilities. (Use `ws://` for local development without SSL). For more complex needs, see [Real-time Apps](real-time-apps.md).
341
- 5. **Apply Filtering and Querying**: Use query parameters with `GET /{TableName}/` and `DELETE /{TableName}/`. See the [Querying REST APIs](querying-rest-apis.md) skill for advanced details.
342
- 6. **Customize if Needed**: If the automatic APIs don't meet your requirements, [customize the resources](./custom-resources.md).
421
+
422
+ To configure optional behavior:
423
+
424
+ ```yaml
425
+ rest:
426
+ lastModified: true # enables Last-Modified response header support
427
+ webSocket: false # disables automatic WebSocket support (enabled by default)
428
+ ```
429
+
430
+ 2. **Export your resource in the schema**: Tables are not exposed by default. Use the `@export` directive in your schema definition to make a table available as a REST endpoint. The exported name defines the base URL path, served on the application HTTP server port (default `9926`).
431
+
432
+ 3. **Use the correct URL structure**: The REST interface follows a consistent path convention.
433
+
434
+ | Path | Description |
435
+ | -------------------------------------------- | ---------------------------------------------------------------------------------- |
436
+ | `/my-resource` | Returns a description of the resource (e.g., table metadata) |
437
+ | `/my-resource/` | Trailing slash — represents the full collection; append query parameters to search |
438
+ | `/my-resource/record-id` | A specific record identified by its primary key |
439
+ | `/my-resource/record-id/` | Trailing slash — collection of records with the given id prefix |
440
+ | `/my-resource/record-id/with/multiple/parts` | Record id with multiple path segments |
441
+
442
+ 4. **Map HTTP methods to operations**: Each HTTP method maps to a resource method and operation.
443
+ - **GET** — Retrieve a record or search. Calls `get()`.
444
+
445
+ ```
446
+ GET /MyTable/123
447
+ GET /MyTable/?name=Harper
448
+ GET /MyTable/123.propertyName
449
+ ```
450
+
451
+ Responses include an `ETag` header. Clients may send `If-None-Match` to receive `304 Not Modified` when the record is unchanged.
452
+
453
+ - **PUT** — Create or replace a record (upsert). Calls `put(record)`. Properties not in the body are removed.
454
+
455
+ ```
456
+ PUT /MyTable/123
457
+ Content-Type: application/json
458
+
459
+ { "name": "some data" }
460
+ ```
461
+
462
+ - **POST** — Create a new record without specifying a primary key. Calls `post(data)`. The assigned key is returned in the `Location` response header.
463
+
464
+ ```
465
+ POST /MyTable/
466
+ Content-Type: application/json
467
+
468
+ { "name": "some data" }
469
+ ```
470
+
471
+ - **PATCH** — Partially update a record, merging only provided properties. Unspecified properties are preserved.
472
+
473
+ ```
474
+ PATCH /MyTable/123
475
+ Content-Type: application/json
476
+
477
+ { "status": "active" }
478
+ ```
479
+
480
+ - **DELETE** — Delete a record or all records matching a query.
481
+ ```
482
+ DELETE /MyTable/123
483
+ DELETE /MyTable/?status=archived
484
+ ```
485
+
486
+ 5. **Access the auto-generated OpenAPI spec**: Harper generates an OpenAPI specification for all exported resources. Retrieve it at:
487
+
488
+ ```
489
+ GET /openapi
490
+ ```
491
+
492
+ 6. **Connect via WebSocket**: When `rest` is enabled, WebSocket support is on by default. Connect to a resource URL to subscribe to change events for that resource.
493
+
494
+ ```javascript
495
+ let ws = new WebSocket('wss://server/my-resource/341');
496
+ ws.onmessage = (event) => {
497
+ let data = JSON.parse(event.data);
498
+ };
499
+ ```
500
+
501
+ Connecting to `wss://server/my-resource/341` accesses the `my-resource` resource with record id `341` and subscribes to it. When the record changes or a message is published to it, the WebSocket connection receives the update.
502
+
503
+ 7. **Implement a custom `connect()` handler**: Override `connect(incomingMessages)` on a resource class to control WebSocket behavior. The method must return an async iterable or generator that produces messages to send to the client.
343
504
 
344
505
  #### Examples
345
506
 
346
- ##### Schema Configuration
507
+ **Simple echo server using an async generator**:
347
508
 
348
- ```graphql
349
- type MyTable @table @export {
350
- id: ID @primaryKey
351
- name: String
509
+ ```javascript
510
+ export class Echo extends Resource {
511
+ async *connect(incomingMessages) {
512
+ for await (let message of incomingMessages) {
513
+ yield message; // echo each message back
514
+ }
515
+ }
352
516
  }
353
517
  ```
354
518
 
355
- ##### Common REST Operations
519
+ **Using the default `connect()` with event-style access and a timer**:
356
520
 
357
- - **List Records**: `GET /MyTable/`
358
- - **Create Record**: `POST /MyTable/`
359
- - **Update Record**: `PATCH /MyTable/{id}`
521
+ ```javascript
522
+ export class Example extends Resource {
523
+ connect(incomingMessages) {
524
+ let outgoingMessages = super.connect();
525
+
526
+ let timer = setInterval(() => {
527
+ outgoingMessages.send({ greeting: 'hi again!' });
528
+ }, 1000);
529
+
530
+ incomingMessages.on('data', (message) => {
531
+ outgoingMessages.send(message); // echo incoming messages
532
+ });
533
+
534
+ outgoingMessages.on('close', () => {
535
+ clearInterval(timer);
536
+ });
537
+
538
+ return outgoingMessages;
539
+ }
540
+ }
541
+ ```
542
+
543
+ **Minimal `config.yaml` enabling REST with WebSocket disabled**:
544
+
545
+ ```yaml
546
+ rest:
547
+ webSocket: false
548
+ ```
549
+
550
+ #### Notes
551
+
552
+ - Tables must be explicitly exported using `@export` in the schema — they are not exposed by default.
553
+ - `rest: true` is the minimal configuration to enable both REST and WebSocket support. See [real-time-apps.md](real-time-apps.md) for patterns around real-time WebSocket usage.
554
+ - For full query syntax on `GET` and `DELETE` with query parameters, see [querying-rest-apis.md](querying-rest-apis.md).
555
+ - The default `connect()` returns an iterable with a `send(message)` method and a `close` event for cleanup on disconnect.
556
+ - For MQTT over WebSockets, set the sub-protocol header `Sec-WebSocket-Protocol: mqtt`.
557
+ - In distributed environments, non-retained messages are delivered in the order received per node; retained messages (PUT/updated records) keep only the latest-timestamp version as the winning record across the cluster.
558
+ - Use the `Content-Type` request header to specify body format and the `Accept` header to request a specific response format.
360
559
 
361
560
  ### 2.2 Querying REST APIs
362
561
 
363
- Instructions for the agent to follow when querying Harper's REST APIs.
562
+ Instructions for the agent to filter, sort, select, and paginate Harper REST API collections using URL query parameters.
364
563
 
365
564
  #### When to Use
366
565
 
367
- Use this skill when you need to perform advanced data retrieval (filtering, sorting, pagination, joins) using Harper's automatic REST endpoints.
566
+ Apply this rule when building or modifying code that queries Harper REST endpoints with filtering, sorting, field selection, or pagination. Use it whenever constructing URLs against collection paths exposed by Harper's automatic REST interface (see [automatic-apis.md](automatic-apis.md)).
368
567
 
369
568
  #### How It Works
370
569
 
371
- 1. **Basic Filtering**: Use attribute names as query parameters: `GET /Table/?key=value`.
372
- 2. **Use Comparison Operators**: Append operators like `gt`, `ge`, `lt`, `le`, `ne` using FIQL-style syntax: `GET /Table/?price=gt=100`.
373
- 3. **Apply Logic and Grouping**: Use `&` for AND, `|` for OR, and `()` for grouping: `GET /Table/?(rating=5|featured=true)&price=lt=50`.
374
- 4. **Select Specific Fields**: Use `select()` to limit returned attributes: `GET /Table/?select(name,price)`.
375
- 5. **Paginate Results**: Use `limit(count)` or `limit(offset, count)` to set the number of records to return and skip.
376
- - Example (first 10): `GET /Table/?limit(10)`
377
- - Example (skip 20, return 10): `GET /Table/?limit(20, 10)`
378
- 6. **Sort Results**: Use `sort()` with `+` (asc) or `-` (desc) before the field name. Avoid `sort=field` format.
379
- - Example (asc): `GET /Table/?sort(+name)`
380
- - Example (desc): `GET /Table/?sort(-price)`
381
- - Example (combined): `GET /Table/?sort(-price,+name)`
382
- 7. **Query Relationships**: Use dot syntax for tables linked with `@relationship`: `GET /Book/?author.name=Harper`.
570
+ 1. **Filter by attribute**: Add query parameters matching attribute names and values. The queried attribute must be indexed.
383
571
 
384
- ### 2.3 Real-time Applications
572
+ ```
573
+ GET /Product/?category=software
574
+ GET /Product/?category=software&inStock=true
575
+ ```
385
576
 
386
- Instructions for the agent to follow when building real-time applications in Harper.
577
+ 2. **Apply comparison operators (FIQL syntax)**: Use FIQL operators directly in query parameter values.
578
+
579
+ | Operator | Meaning |
580
+ | ------------ | -------------------------------------- |
581
+ | `==` | Equal |
582
+ | `=lt=` | Less than |
583
+ | `=le=` | Less than or equal |
584
+ | `=gt=` | Greater than |
585
+ | `=ge=` | Greater than or equal |
586
+ | `=ne=`, `!=` | Not equal |
587
+ | `=ct=` | Contains (strings) |
588
+ | `=sw=` | Starts with (strings) |
589
+ | `=ew=` | Ends with (strings) |
590
+ | `=`, `===` | Strict equality (no type conversion) |
591
+ | `!==` | Strict inequality (no type conversion) |
387
592
 
388
- #### When to Use
593
+ ```
594
+ GET /Product/?price=gt=100
595
+ GET /Product/?price=le=20
596
+ GET /Product/?name==Keyboard*
597
+ GET /Product/?category=software&price=gt=100&price=lt=200
598
+ ```
389
599
 
390
- Use this skill when you need to stream live updates to clients, implement chat features, or provide real-time data synchronization between the database and a frontend.
600
+ For date fields, URL-encode colons as `%3A`:
391
601
 
392
- #### How It Works
602
+ ```
603
+ GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z
604
+ ```
605
+
606
+ 3. **Chain conditions for range queries**: Omit the attribute name on the second condition to apply it to the same attribute. Only `gt`/`ge` combined with `lt`/`le` is supported.
607
+
608
+ ```
609
+ GET /Product/?price=gt=100&lt=200
610
+ ```
611
+
612
+ 4. **Combine conditions with OR logic**: Use `|` instead of `&`.
613
+
614
+ ```
615
+ GET /Product/?rating=5|featured=true
616
+ ```
617
+
618
+ 5. **Group conditions**: Use parentheses or square brackets to control order of operations. Prefer square brackets when constructing queries from user input, since standard URI encoding safely encodes `[` and `]`.
619
+
620
+ ```
621
+ GET /Product/?rating=5|(price=gt=100&price=lt=200)
622
+ GET /Product/?rating=5&[tag=fast|tag=scalable|tag=efficient]
623
+ ```
624
+
625
+ Construct grouped queries from JavaScript:
626
+
627
+ ```javascript
628
+ let url = `/Product/?rating=5&[${tags.map(encodeURIComponent).join('|')}]`;
629
+ ```
630
+
631
+ 6. **Select specific properties with `select(`**: Use `select()` to control which fields are returned.
632
+
633
+ | Syntax | Returns |
634
+ | -------------------------------------- | ------------------------------------------- |
635
+ | `?select(property)` | Values of a single property directly |
636
+ | `?select(property1,property2)` | Objects with only the specified properties |
637
+ | `?select([property1,property2])` | Arrays of property values |
638
+ | `?select(property1,)` | Objects with a single specified property |
639
+ | `?select(property{subProp1,subProp2})` | Nested objects with specific sub-properties |
640
+
641
+ ```
642
+ GET /Product/?category=software&select(name)
643
+ GET /Product/?brand.name=Microsoft&select(name,brand{name})
644
+ ```
645
+
646
+ 7. **Limit results with `limit(`**: Use `limit(end)` or `limit(start,end)` to paginate.
647
+
648
+ ```
649
+ GET /Product/?rating=gt=3&inStock=true&select(rating,name)&limit(20)
650
+ GET /Product/?rating=gt=3&limit(10,30)
651
+ ```
652
+
653
+ 8. **Sort results with `sort(`**: Use `sort(property)` or `sort(+property,-property,...)`. Prefix `+` or no prefix = ascending; `-` = descending.
654
+
655
+ ```
656
+ GET /Product/?rating=gt=3&sort(+name)
657
+ GET /Product/?sort(+rating,-price)
658
+ ```
659
+
660
+ 9. **Query across relationships**: Use dot-syntax to filter by related table attributes. Relationships must be defined in the schema using `@relation`.
661
+
662
+ ```
663
+ GET /Product/?brand.name=Microsoft
664
+ GET /Brand/?products.name=Keyboard
665
+ ```
666
+
667
+ Use `select()` to include relationship attributes in the response (they are not included by default):
668
+
669
+ ```
670
+ GET /Product/?brand.name=Microsoft&select(name,brand{name})
671
+ ```
393
672
 
394
- 1. **Check Automatic WebSockets**: If you only need to stream table changes, use [Automatic APIs](automatic-apis.md) which provide a WebSocket endpoint for every `@export`ed table.
395
- 2. **Implement `connect` in a Resource**: For custom bi-directional logic, implement the `connect` method.
396
- 3. **Use Pub/Sub**: Use `tables.TableName.subscribe(query)` to listen for specific data changes and stream them to the client.
397
- 4. **Handle SSE**: Ensure your `connect` method gracefully handles cases where `incomingMessages` is null (Server-Sent Events).
398
- 5. **Connect from Client**: Use standard WebSockets (`new WebSocket('wss://...')`) to connect to your resource endpoint. Ensure you use the appropriate scheme (`ws://` for HTTP, `wss://` for HTTPS).
673
+ 10. **Access a specific property by URL**: Append the property name with dot syntax to the record ID. Only works for properties declared in the schema.
674
+ ```
675
+ GET /MyTable/123.propertyName
676
+ ```
399
677
 
400
678
  #### Examples
401
679
 
402
- ##### Bi-directional WebSocket Resource
680
+ **Range filter with select and limit:**
403
681
 
404
- ```typescript
405
- import { Resource, tables } from 'harper';
406
-
407
- export class MySocket extends Resource {
408
- async *connect(target, incomingMessages) {
409
- // Subscribe to table changes
410
- const subscription = await tables.MyTable.subscribe(target);
411
- if (!incomingMessages) {
412
- return subscription; // SSE mode
413
- }
682
+ ```
683
+ GET /Product/?category=software&price=gt=100&price=lt=200&select(name,price)&limit(20)
684
+ ```
414
685
 
415
- // Handle incoming client messages
416
- for await (let message of incomingMessages) {
417
- yield { received: message };
418
- }
419
- }
686
+ **Sort descending with multiple fields:**
687
+
688
+ ```
689
+ GET /Product/?sort(+rating,-price)
690
+ ```
691
+
692
+ **OR logic with grouping:**
693
+
694
+ ```
695
+ GET /Product/?price=lt=100|[rating=5&[tag=fast|tag=scalable|tag=efficient]&inStock=true]
696
+ ```
697
+
698
+ **Relationship join with nested select:**
699
+
700
+ ```
701
+ GET /Product/?brand.name=Microsoft&select(name,brand{name,id})
702
+ ```
703
+
704
+ **Schema defining a relationship for join queries:**
705
+
706
+ ```graphql
707
+ type Product @table @export {
708
+ id: Long @primaryKey
709
+ name: String
710
+ brandId: Long @indexed
711
+ brand: Brand @relation(from: "brandId")
712
+ }
713
+ type Brand @table @export {
714
+ id: Long @primaryKey
715
+ name: String
716
+ products: [Product] @relation(to: "brandId")
420
717
  }
421
718
  ```
422
719
 
423
- ### 2.4 Checking Authentication
720
+ **Many-to-many relationship query:**
721
+
722
+ ```graphql
723
+ type Product @table @export {
724
+ id: Long @primaryKey
725
+ name: String
726
+ resellerIds: [Long] @indexed
727
+ resellers: [Reseller] @relation(from: "resellerId")
728
+ }
729
+ ```
424
730
 
425
- Instructions for the agent to follow when handling authentication and sessions.
731
+ ```
732
+ GET /Product/?resellers.name=Cool Shop&select(id,name,resellers{name,id})
733
+ ```
734
+
735
+ **Type conversion with explicit prefix:**
736
+
737
+ ```
738
+ GET /Product/?price==number:123
739
+ GET /Product/?active==boolean:true
740
+ GET /Product/?listDate==date:2024-01-05T20%3A07%3A27.955Z
741
+ ```
742
+
743
+ #### Notes
744
+
745
+ - Only indexed attributes can be used as the primary filter; additional unindexed attributes can be combined with `&` once at least one indexed attribute is present.
746
+ - For null value queries, use `?attribute=null`. Indexes must have been created with null indexing support; existing indexes must be removed and re-added to support null queries.
747
+ - FIQL comparators (`==`, `!=`, `=gt=`, etc.) apply automatic type conversion based on value syntax or schema-declared type. Strict operators (`=`, `===`, `!==`) skip automatic type conversion.
748
+ - Filtering by a related attribute produces INNER JOIN behavior (only records with a matching related record are returned). Using `select()` on a relationship without a filter produces LEFT JOIN behavior.
749
+ - The array order of foreign key values in many-to-many relationships is preserved when resolving the relationship.
750
+ - See [automatic-apis.md](automatic-apis.md) for how Harper tables are automatically exposed as REST endpoints.
751
+
752
+ ### 2.3 Real-Time Apps with WebSockets and Pub/Sub
753
+
754
+ Instructions for the agent to follow when building real-time features in Harper using WebSockets and Pub/Sub.
426
755
 
427
756
  #### When to Use
428
757
 
429
- Use this skill when you need to implement sign-in/sign-out functionality, protect specific resource endpoints, or identify the currently logged-in user in a Harper application.
758
+ Apply this rule when implementing any feature that requires real-time bidirectional communication, live data streaming, or push-based updates in a Harper application. This includes chat, live dashboards, sensor feeds, and any scenario where clients must receive resource changes as they happen.
430
759
 
431
760
  #### How It Works
432
761
 
433
- 1. **Configure Harper for Sessions**: Ensure `harper-config.yaml` has sessions enabled and local auto-authorization disabled for testing:
762
+ 1. **Enable WebSocket support**: WebSocket support is enabled automatically when the `rest` plugin is enabled. To explicitly disable it, set the following in your config:
763
+
434
764
  ```yaml
435
- authentication:
436
- authorizeLocal: false
437
- enableSessions: true
765
+ rest:
766
+ webSocket: false
438
767
  ```
439
- 2. **Implement Sign In**: Use `this.getContext().login(username, password)` to create a session:
440
- ```typescript
441
- async post(_target, data) {
442
- const context = this.getContext();
443
- try {
444
- await context.login(data.username, data.password);
445
- } catch {
446
- return new Response('Invalid credentials', { status: 403 });
447
- }
448
- return new Response('Logged in', { status: 200 });
449
- }
768
+
769
+ 2. **Connect a client to a resource**: A WebSocket connection to a resource URL automatically subscribes to that resource. When the record changes or a message is published to it, the connection receives the update.
770
+
771
+ ```javascript
772
+ let ws = new WebSocket('wss://server/my-resource/341');
773
+ ws.onmessage = (event) => {
774
+ let data = JSON.parse(event.data);
775
+ };
450
776
  ```
451
- 3. **Identify Current User**: Use `this.getCurrentUser()` to access session data:
452
- ```typescript
453
- async get() {
454
- const user = this.getCurrentUser?.();
777
+
778
+ `new WebSocket('wss://server/my-resource/341')` accesses the resource defined for `my-resource` with record id `341` and subscribes to it.
779
+
780
+ 3. **Implement a custom `connect()` handler**: Override the `connect(incomingMessages)` method on a resource class to control WebSocket behavior. The method must return an async iterable (or generator) that produces messages to send to the client. See [automatic-apis.md](automatic-apis.md) for more on defining resource classes.
781
+
782
+ 4. **Use the default `connect()` for event-style access**: Call `super.connect()` to get a streaming iterable that provides:
783
+ - A `send(message)` method for pushing outgoing messages
784
+ - A `close` event for cleanup on disconnect
785
+
786
+ 5. **Handle message ordering in distributed environments**: Harper delivers messages to local subscribers immediately without inter-node coordination delay.
787
+
788
+ | Message Type | Behavior |
789
+ | -------------------------------------------------------- | ----------------------------------------------------------------------- |
790
+ | Non-retained (no `retain` flag) | Every message delivered in order received; suitable for chat |
791
+ | Retained (published with `retain`, or PUT/updated in DB) | Only the latest-timestamp message is kept; suitable for sensor readings |
792
+
793
+ 6. **Use MQTT over WebSockets** when needed by setting the sub-protocol header:
794
+ ```
795
+ Sec-WebSocket-Protocol: mqtt
796
+ ```
797
+
798
+ #### Examples
799
+
800
+ **Simple echo server** — override `connect(incomingMessages)` to yield each incoming message back to the client:
801
+
802
+ ```javascript
803
+ export class Echo extends Resource {
804
+ async *connect(incomingMessages) {
805
+ for await (let message of incomingMessages) {
806
+ yield message; // echo each message back
807
+ }
808
+ }
809
+ }
810
+ ```
811
+
812
+ **Custom connect with timer and event-style access** — use `super.connect()` to get the outgoing stream, push periodic messages, echo incoming messages, and clean up on disconnect:
813
+
814
+ ```javascript
815
+ export class Example extends Resource {
816
+ connect(incomingMessages) {
817
+ let outgoingMessages = super.connect();
818
+
819
+ let timer = setInterval(() => {
820
+ outgoingMessages.send({ greeting: 'hi again!' });
821
+ }, 1000);
822
+
823
+ incomingMessages.on('data', (message) => {
824
+ outgoingMessages.send(message); // echo incoming messages
825
+ });
826
+
827
+ outgoingMessages.on('close', () => {
828
+ clearInterval(timer);
829
+ });
830
+
831
+ return outgoingMessages;
832
+ }
833
+ }
834
+ ```
835
+
836
+ #### Notes
837
+
838
+ - WebSocket connections target a resource URL path. By default, connecting to a resource subscribes to changes for that resource.
839
+ - The `connect(incomingMessages)` method **must** return an async iterable or generator; returning a plain value will not work.
840
+ - `super.connect()` returns a streaming iterable with `send(message)` and a `close` event — use this when you need to push messages outside of the incoming message loop.
841
+ - For one-way real-time streaming without bidirectional communication, consider Server-Sent Events instead.
842
+ - For full pub/sub capabilities, Harper also supports MQTT; set `Sec-WebSocket-Protocol: mqtt` to use MQTT over WebSockets.
843
+
844
+ ### 2.4 Checking Authentication
845
+
846
+ Instructions for the agent to follow when handling user authentication and session management inside Harper Resources.
847
+
848
+ #### When to Use
849
+
850
+ Apply this rule when implementing authentication checks, login/logout flows, or token issuance inside a custom Resource. Use it any time a Resource needs to identify the current user, establish a session, or issue JWTs to clients. See [custom-resources.md](custom-resources.md) for the general Resource authoring pattern.
851
+
852
+ #### How It Works
853
+
854
+ 1. **Check the current user** with `getCurrentUser()`. Call it inside any Resource method to retrieve the authenticated user or `undefined` if no user is authenticated. Guard protected endpoints by returning a `401` when the result is `undefined`.
855
+
856
+ ```javascript
857
+ async get(target) {
858
+ const user = this.getCurrentUser();
455
859
  if (!user) return new Response(null, { status: 401 });
456
860
  return { username: user.username, role: user.role };
457
861
  }
458
862
  ```
459
- 4. **Implement Sign Out**: Use `this.getContext().logout()` or delete the session from context:
460
- ```typescript
461
- async post() {
462
- const context = this.getContext();
463
- await context.session?.delete?.(context.session.id);
464
- return new Response('Logged out', { status: 200 });
465
- }
466
- ```
467
- 5. **Protect Routes**: In your Resource, use `allowRead()`, `allowUpdate()`, etc., to enforce authorization logic based on `this.getCurrentUser()`. For privileged actions, verify `user.role.permission.super_user`.
468
863
 
469
- #### Examples
470
-
471
- ##### Sign In Implementation
864
+ The returned object exposes `username`, `role`, and `role.permission` flags.
472
865
 
473
- ```typescript
474
- async post(_target, data) {
475
- const context = this.getContext();
476
- try {
477
- await context.login(data.username, data.password);
478
- } catch {
479
- return new Response('Invalid credentials', { status: 403 });
480
- }
481
- return new Response('Logged in', { status: 200 });
482
- }
483
- ```
866
+ 2. **Enable sessions** before using session-based login. Set `authentication.enableSessions: true` in `harperdb-config.yaml`:
484
867
 
485
- ##### Identify Current User
868
+ ```yaml
869
+ authentication:
870
+ enableSessions: true
871
+ ```
486
872
 
487
- ```typescript
488
- async get() {
489
- const user = this.getCurrentUser?.();
490
- if (!user) return new Response(null, { status: 401 });
491
- return { username: user.username, role: user.role };
492
- }
493
- ```
873
+ 3. **Access login and session helpers** via `getContext()`. The context object exposes `context.login` and `context.session` for sign-in/out flows.
874
+ - Call `context.login(username, password)` to verify credentials and establish a session cookie on success.
875
+ - To end a session, delete it via `context.session.delete(context.session.id)`.
494
876
 
495
- ##### Sign Out Implementation
877
+ 4. **Implement sign-in and sign-out Resources** using the context helpers:
496
878
 
497
- ```typescript
498
- async post() {
499
- const context = this.getContext();
500
- await context.session?.delete?.(context.session.id);
501
- return new Response('Logged out', { status: 200 });
502
- }
503
- ```
879
+ ```javascript
880
+ export class SignIn extends Resource {
881
+ async post(_target, data) {
882
+ const context = this.getContext();
883
+ try {
884
+ await context.login(data.username, data.password);
885
+ } catch {
886
+ return new Response('Invalid credentials', { status: 403 });
887
+ }
888
+ return new Response('Logged in', { status: 200 });
889
+ }
890
+ }
504
891
 
505
- #### Status code conventions used here
892
+ export class SignOut extends Resource {
893
+ async post() {
894
+ const context = this.getContext();
895
+ if (!context.session) return new Response(null, { status: 401 });
896
+ await context.session.delete(context.session.id);
897
+ return new Response('Logged out', { status: 200 });
898
+ }
899
+ }
900
+ ```
506
901
 
507
- - 200: Successful operation. For `GET /me`, a `200` with empty body means “not signed in”.
508
- - 400: Missing required fields (e.g., username/password on sign-in).
509
- - 401: No current session for an action that requires one (e.g., sign out when not signed in).
510
- - 403: Authenticated but not authorized (bad credentials on login attempt, or insufficient privileges).
902
+ 5. **Issue JWTs for non-browser clients** (CLI tools, mobile apps, service-to-service). Cookie-based sessions are intended for browser clients. For other clients, mint tokens programmatically using `server.operation()`:
511
903
 
512
- #### Client considerations
904
+ ```javascript
905
+ import { Resource, server } from 'harper';
906
+
907
+ export class IssueTokens extends Resource {
908
+ static async get(_target, context) {
909
+ const { operation_token, refresh_token } = await server.operation(
910
+ { operation: 'create_authentication_tokens' },
911
+ context,
912
+ true,
913
+ );
914
+ return { operation_token, refresh_token };
915
+ }
513
916
 
514
- - Sessions are cookie-based; the server handles setting and reading the cookie via Harper. If you make cross-origin requests, ensure the appropriate `credentials` mode and CORS settings.
515
- - If developing locally, double-check the server config still has `authentication.authorizeLocal: false` to avoid accidental superuser bypass.
917
+ static async post(_target, data) {
918
+ const { username, password } = await data;
919
+ if (!username || !password) {
920
+ return new Response('username and password required', { status: 400 });
921
+ }
922
+ const { operation_token, refresh_token } = await server.operation({
923
+ operation: 'create_authentication_tokens',
924
+ username,
925
+ password,
926
+ });
927
+ return { operation_token, refresh_token };
928
+ }
929
+ }
516
930
 
517
- #### Token-based auth (JWT + refresh token) for non-browser clients
931
+ export class RefreshJWT extends Resource {
932
+ static async post(_target, data) {
933
+ const { refresh_token } = await data;
934
+ if (!refresh_token) {
935
+ return new Response('refresh_token required', { status: 400 });
936
+ }
937
+ const { operation_token } = await server.operation({
938
+ operation: 'refresh_operation_token',
939
+ refresh_token,
940
+ });
941
+ return { operation_token };
942
+ }
943
+ }
944
+ ```
518
945
 
519
- Cookie-backed sessions are great for browser flows. For CLI tools, mobile apps, or other non-browser clients, it’s often easier to use **explicit tokens**:
946
+ Pass `true` as the third argument to `server.operation()` when the operation should run as the current authenticated user. Omit it or pass `false` when the operation supplies its own credentials.
520
947
 
521
- - **JWT (`operation_token`)**: short-lived bearer token used to authorize API requests.
522
- - **Refresh token (`refresh_token`)**: longer-lived token used to mint a new JWT when it expires.
948
+ 6. **Configure JWT token expiry** in `harperdb-config.yaml` under the `authentication` section:
523
949
 
524
- This project includes two Resource patterns for that flow:
950
+ ```yaml
951
+ authentication:
952
+ operationTokenTimeout: 1d
953
+ refreshTokenTimeout: 30d
954
+ ```
525
955
 
526
- ##### Issuing tokens: `IssueTokens`
956
+ Duration strings follow the `jsonwebtoken` package format (e.g., `1d`, `12h`, `60m`).
527
957
 
528
- **Description / use case:** Generate `{ refreshToken, jwt }` either:
958
+ #### Examples
529
959
 
530
- - with an existing Authorization token (either Basic Auth or a JWT) and you want to issue new tokens, or
531
- - from an explicit `{ username, password }` payload (useful for direct “login” from a CLI/mobile client).
960
+ **Protecting a resource endpoint and returning user info:**
532
961
 
533
962
  ```javascript
534
- export class IssueTokens extends Resource {
535
- static loadAsInstance = false;
536
-
537
- async get(target) {
538
- const { refresh_token: refreshToken, operation_token: jwt } =
539
- await databases.system.hdb_user.operation(
540
- { operation: 'create_authentication_tokens' },
541
- this.getContext(),
542
- );
543
- return { refreshToken, jwt };
544
- }
963
+ async get(target) {
964
+ const user = this.getCurrentUser();
965
+ if (!user) return new Response(null, { status: 401 });
966
+ return { username: user.username, role: user.role };
967
+ }
968
+ ```
545
969
 
546
- async post(target, data) {
547
- if (!data.username || !data.password) {
548
- throw new Error('username and password are required');
970
+ **Full session-based sign-in/sign-out flow:**
971
+
972
+ ```javascript
973
+ export class SignIn extends Resource {
974
+ async post(_target, data) {
975
+ const context = this.getContext();
976
+ try {
977
+ await context.login(data.username, data.password);
978
+ } catch {
979
+ return new Response('Invalid credentials', { status: 403 });
549
980
  }
981
+ return new Response('Logged in', { status: 200 });
982
+ }
983
+ }
550
984
 
551
- const { refresh_token: refreshToken, operation_token: jwt } =
552
- await databases.system.hdb_user.operation({
553
- operation: 'create_authentication_tokens',
554
- username: data.username,
555
- password: data.password,
556
- });
557
- return { refreshToken, jwt };
985
+ export class SignOut extends Resource {
986
+ async post() {
987
+ const context = this.getContext();
988
+ if (!context.session) return new Response(null, { status: 401 });
989
+ await context.session.delete(context.session.id);
990
+ return new Response('Logged out', { status: 200 });
558
991
  }
559
992
  }
560
993
  ```
561
994
 
562
- **Recommended documentation notes to include:**
563
-
564
- - `GET` variant: intended for “I already have an Authorization token, give me new tokens”.
565
- - `POST` variant: intended for “I have credentials, give me tokens”.
566
- - Response shape:
567
- - `refreshToken`: store securely (long-lived).
568
- - `jwt`: attach to requests (short-lived).
569
-
570
- ##### Refreshing a JWT: `RefreshJWT`
571
-
572
- **Description / use case:** When the JWT expires, the client uses the refresh token to get a new JWT without re-supplying username/password.
995
+ **JWT token refresh endpoint:**
573
996
 
574
997
  ```javascript
575
998
  export class RefreshJWT extends Resource {
576
- static loadAsInstance = false;
577
-
578
- async post(target, data) {
579
- if (!data.refreshToken) {
580
- throw new Error('refreshToken is required');
999
+ static async post(_target, data) {
1000
+ const { refresh_token } = await data;
1001
+ if (!refresh_token) {
1002
+ return new Response('refresh_token required', { status: 400 });
581
1003
  }
582
-
583
- const { operation_token: jwt } = await databases.system.hdb_user.operation({
1004
+ const { operation_token } = await server.operation({
584
1005
  operation: 'refresh_operation_token',
585
- refresh_token: data.refreshToken,
1006
+ refresh_token,
586
1007
  });
587
- return { jwt };
1008
+ return { operation_token };
588
1009
  }
589
1010
  }
590
1011
  ```
591
1012
 
592
- **Recommended documentation notes to include:**
593
-
594
- - Requires `refreshToken` in the request body.
595
- - Returns a new `{ jwt }`.
596
- - If refresh fails (expired/revoked), client must re-authenticate (e.g., call `IssueTokens.post` again).
597
-
598
- ##### Suggested client flow (high-level)
599
-
600
- 1. **Sign in (token flow)**
601
- - POST /IssueTokens/ with a body of `{ "username": "your username", "password": "your password" }` or GET /IssueTokens/ with an existing Authorization token.
602
- - Receive `{ jwt, refreshToken }` in the response
603
- 2. **Call protected APIs**
604
- - Send the JWT with each request in the Authorization header (as your auth mechanism expects)
605
- 3. **JWT expires**
606
- - POST /RefreshJWT/ with a body of `{ "refreshToken": "your refresh token" }`.
607
- - Receive `{ jwt }` in the response and continue
608
-
609
- #### Quick checklist
1013
+ #### Notes
610
1014
 
611
- - [ ] Public endpoints explicitly `allowRead`/`allowCreate` as needed.
612
- - [ ] Sign-in uses `context.login` and handles 400/403 correctly.
613
- - [ ] Protected routes call `ensureSuperUser(this.getCurrentUser())` (or another role check) before doing work.
614
- - [ ] Sign-out verifies a session and deletes it.
615
- - [ ] `authentication.authorizeLocal` is `false` and `enableSessions` is `true` in Harper config.
616
- - [ ] If using tokens: `IssueTokens` issues `{ jwt, refreshToken }`, `RefreshJWT` refreshes `{ jwt }` with a `refreshToken`.
1015
+ - `getCurrentUser()` and `getContext()` are instance methods; call them with `this` inside non-static Resource methods.
1016
+ - `enableSessions` must be `true` in config before `context.login` or `context.session` will function.
1017
+ - Cookie-based sessions target browser clients. Use JWT issuance via `server.operation()` for all other client types.
1018
+ - When both `operation_token` and `refresh_token` have expired, the client must call `create_authentication_tokens` again with credentials.
617
1019
 
618
1020
  ## 3. Logic & Extension
619
1021
 
@@ -819,179 +1221,323 @@ for await (const record of tables.Product.search({
819
1221
 
820
1222
  Be very careful when performing updates and deletions! You may be dealing with live production data. The wrong request to delete, without approval from a human, could be devastating to a business. Always use the proper approval process.
821
1223
 
822
- ### 3.4 TypeScript Type Stripping
1224
+ ### 3.4 TypeScript Type Stripping in Harper
823
1225
 
824
- Instructions for the agent to follow when using TypeScript in Harper.
1226
+ Instructions for the agent to run `.ts` files directly in Harper without a build step using Node.js's built-in type stripping.
825
1227
 
826
1228
  #### When to Use
827
1229
 
828
- Use this skill when you want to write Harper Resources in TypeScript and have them execute directly in Node.js without an intermediate build or compilation step.
1230
+ Apply this rule when writing Harper resource files in TypeScript. Use it any time you need to reference `.ts` source files from `config.yaml` or import between local TypeScript modules in a Harper project.
829
1231
 
830
1232
  #### How It Works
831
1233
 
832
- 1. **Verify Node.js Version**: Ensure you are using Node.js v22.6.0 or higher.
833
- 2. **Name Files with `.ts`**: Create your resource files in the `resources/` directory with a `.ts` extension.
834
- 3. **Use TypeScript Syntax**: Write your resource classes using standard TypeScript (interfaces, types, etc.).
835
- ```typescript
836
- import { Resource } from 'harper';
837
- export class MyResource extends Resource {
838
- async get(): Promise<{ message: string }> {
839
- return { message: 'Running TS directly!' };
840
- }
841
- }
842
- ```
843
- 4. **Use Explicit Extensions in Imports**: When importing other local modules, include the `.ts` extension: `import { helper } from './helper.ts'`.
844
- 5. **Configure `config.yaml`**: Ensure `jsResource` points to your `.ts` files:
1234
+ 1. **Ensure Node.js version**: Require Node.js 22.6 or later. Type stripping is unavailable on earlier versions.
1235
+
1236
+ 2. **Point `jsResource` at `.ts` files**: The `jsResource` plugin loads both `.js` and `.ts` files. Set its `files` glob in `config.yaml` to target your `.ts` source files:
1237
+
845
1238
  ```yaml
846
1239
  jsResource:
847
1240
  files: 'resources/*.ts'
848
1241
  ```
849
1242
 
850
- ### 3.5 Caching
1243
+ 3. **Use explicit `.ts` extensions in local imports**: Node's loader does not resolve `'./helper'` to `'./helper.ts'`, so always include the full extension:
1244
+
1245
+ ```typescript
1246
+ import { helper } from './helper.ts';
1247
+ ```
1248
+
1249
+ 4. **Stay within type-stripping limits**: Only type annotations and declarations are removed. Do not use enums with runtime values, namespaces with runtime semantics, or any other features that require code transformation beyond type stripping.
1250
+
1251
+ #### Examples
1252
+
1253
+ A complete Harper resource written in TypeScript, using imports from the `harper` package:
1254
+
1255
+ ```typescript
1256
+ import { type RequestTargetOrId, Resource, tables } from 'harper';
1257
+
1258
+ export class MyResource extends Resource {
1259
+ async get(target?: RequestTargetOrId): Promise<{ message: string }> {
1260
+ return { message: 'Hello from TS' };
1261
+ }
1262
+ }
1263
+ ```
1264
+
1265
+ Paired `config.yaml` entry loading the file via `jsResource`:
1266
+
1267
+ ```yaml
1268
+ jsResource:
1269
+ files: 'resources/*.ts'
1270
+ ```
1271
+
1272
+ #### Notes
1273
+
1274
+ - No build step or transpiler is required — Harper runs `.ts` files directly.
1275
+ - Type imports (e.g., `import { type RequestTargetOrId }`) from the `harper` package work as usual.
1276
+ - Unsupported TypeScript features include: enums with runtime values, namespaces with runtime semantics, and anything requiring code transformation beyond simple type stripping.
1277
+
1278
+ ### 3.5 Caching External Data Sources in Harper
851
1279
 
852
- Instructions for the agent to follow when implementing caching in Harper.
1280
+ Instructions for the agent to implement integrated data caching in Harper by wrapping external sources with a cache table and `sourcedFrom`.
853
1281
 
854
1282
  #### When to Use
855
1283
 
856
- Use this skill when you need high-performance, low-latency storage for data from external sources. It's ideal for reducing API calls to third-party services, preventing cache stampedes, and making external data queryable as if it were native Harper tables.
1284
+ Apply this rule when a Harper application needs to cache responses from an external API, microservice, or database to avoid repeated slow or expensive upstream calls. Use it whenever you need to define TTL-based cache expiration, observe ETag-based conditional responses, or manually invalidate cached entries.
857
1285
 
858
1286
  #### How It Works
859
1287
 
860
- 1. **Configure a Cache Table**: Define a table in your `schema.graphql` with an `expiration` (in seconds).
861
- 2. **Define an External Source**: Create a Resource class that fetches the data from your source.
862
- 3. **Attach Source to Table**: Use `sourcedFrom` to link your resource to the table.
863
- 4. **Implement Active Caching (Optional)**: Use `subscribe()` for proactive updates. See [Real-Time Apps](real-time-apps.md).
864
- 5. **Implement Write-Through Caching (Optional)**: Define `put` or `post` in your resource to propagate updates upstream.
1288
+ 1. **Define a cache table with `expiration`**: In `schema.graphql`, add the `expiration` argument to `@table`. The value is in seconds. Any record older than this threshold is considered stale and will be re-fetched on next access.
1289
+
1290
+ ```graphql
1291
+ type JokeCache @table(expiration: 60) @export {
1292
+ id: ID @primaryKey
1293
+ setup: String
1294
+ punchline: String
1295
+ }
1296
+ ```
1297
+
1298
+ 2. **Wrap the external source in `resources.js`**: Create an object with a `get(id)` method that fetches from the upstream source. Then call `sourcedFrom` on the table to register it.
1299
+
1300
+ ```javascript
1301
+ const jokeAPI = {
1302
+ async get(id) {
1303
+ const response = await fetch(`https://official-joke-api.appspot.com/jokes/${id}`);
1304
+ return response.json();
1305
+ },
1306
+ };
1307
+
1308
+ tables.JokeCache.sourcedFrom(jokeAPI);
1309
+ ```
1310
+
1311
+ Harper's caching behavior after `sourcedFrom` is registered:
1312
+ - A request arrives for `/JokeCache/1`.
1313
+ - Harper checks if the record with id `1` exists in `JokeCache` and is not stale.
1314
+ - If fresh, Harper returns it immediately.
1315
+ - If missing or stale, Harper calls `jokeAPI.get()`, stores the result in `JokeCache`, and returns it.
1316
+ - Multiple simultaneous requests for the same missing or stale record wait on a single upstream call — Harper prevents cache stampedes automatically.
1317
+
1318
+ 3. **Configure plugins in `config.yaml`**: Enable the schema, REST API, and JS resource plugins.
1319
+
1320
+ ```yaml
1321
+ graphqlSchema:
1322
+ files: 'schema.graphql'
1323
+ rest: true
1324
+ jsResource:
1325
+ files: 'resources.js'
1326
+ ```
1327
+
1328
+ 4. **Observe caching via ETags**: Harper automatically computes an ETag from the record's last-modified timestamp. On the first request you receive a `200` with an `etag` header. Pass that value back in `If-None-Match` on subsequent requests; Harper returns `304 Not Modified` with an empty body if the record is unchanged.
1329
+
1330
+ ```bash
1331
+ curl -i 'http://localhost:9926/JokeCache/1' \
1332
+ -H 'If-None-Match: "abCDefGHij"'
1333
+ ```
1334
+
1335
+ 5. **Force a cache bypass**: Send `Cache-Control: no-cache` to make Harper skip the local cache and always call the upstream source, regardless of TTL.
1336
+
1337
+ ```bash
1338
+ curl -i 'http://localhost:9926/JokeCache/1' \
1339
+ -H 'Cache-Control: no-cache'
1340
+ ```
1341
+
1342
+ 6. **Invalidate a cache entry on demand**: Remove `@export` from the schema type, then export a class of the same name in `resources.js` that extends the table and implements a `post` handler calling `this.invalidate(target)`.
1343
+
1344
+ ```graphql
1345
+ type JokeCache @table(expiration: 60) {
1346
+ id: ID @primaryKey
1347
+ setup: String
1348
+ punchline: String
1349
+ }
1350
+ ```
1351
+
1352
+ ```javascript
1353
+ export class JokeCache extends tables.JokeCache {
1354
+ static async post(target, data) {
1355
+ const body = await data;
1356
+ if (body?.action === 'invalidate') {
1357
+ this.invalidate(target);
1358
+ return { status: 200, data: { message: 'invalidated' } };
1359
+ }
1360
+ }
1361
+ }
1362
+ ```
1363
+
1364
+ Trigger invalidation with a `POST`:
1365
+
1366
+ ```bash
1367
+ curl -X POST 'http://localhost:9926/JokeCache/1' \
1368
+ -H 'Content-Type: application/json' \
1369
+ -d '{"action": "invalidate"}'
1370
+ ```
1371
+
1372
+ The next `GET /JokeCache/1` will fetch fresh data from the upstream source regardless of TTL.
865
1373
 
866
1374
  #### Examples
867
1375
 
868
- ##### Schema Configuration
1376
+ Complete `schema.graphql` and `resources.js` for a cached external API with on-demand invalidation:
869
1377
 
870
1378
  ```graphql
871
- type MyCache @table(expiration: 3600) @export {
1379
+ type JokeCache @table(expiration: 60) {
872
1380
  id: ID @primaryKey
1381
+ setup: String
1382
+ punchline: String
873
1383
  }
874
1384
  ```
875
1385
 
876
- ##### Resource Implementation
877
-
878
- ```js
879
- import { Resource, tables } from 'harper';
1386
+ ```javascript
1387
+ // resources.js
880
1388
 
881
- export class ThirdPartyAPI extends Resource {
1389
+ const jokeAPI = {
882
1390
  async get() {
883
1391
  const id = this.getId();
884
- const response = await fetch(`https://api.example.com/items/${id}`);
885
- if (!response.ok) {
886
- throw new Error('Source fetch failed');
1392
+ const response = await fetch(`https://official-joke-api.appspot.com/jokes/${id}`);
1393
+ return response.json();
1394
+ },
1395
+ };
1396
+
1397
+ tables.JokeCache.sourcedFrom(jokeAPI);
1398
+
1399
+ export class JokeCache extends tables.JokeCache {
1400
+ static async post(target, data) {
1401
+ const body = await data;
1402
+ if (body?.action === 'invalidate') {
1403
+ this.invalidate(target);
1404
+ return { status: 200, data: { message: 'invalidated' } };
887
1405
  }
888
- return await response.json();
889
1406
  }
890
1407
  }
1408
+ ```
1409
+
1410
+ First request — cache miss, upstream is called, `200` returned:
1411
+
1412
+ ```bash
1413
+ curl -i 'http://localhost:9926/JokeCache/1'
1414
+ ```
1415
+
1416
+ Second request with ETag — cache hit, `304 Not Modified`:
891
1417
 
892
- // Attach source to table
893
- tables.MyCache.sourcedFrom(ThirdPartyAPI);
1418
+ ```bash
1419
+ curl -i 'http://localhost:9926/JokeCache/1' \
1420
+ -H 'If-None-Match: "abCDefGHij"'
894
1421
  ```
895
1422
 
1423
+ #### Notes
1424
+
1425
+ - `expiration` is measured in seconds. Harper also supports separate `eviction` and `scanInterval` arguments on `@table` for fine-grained control over physical record removal.
1426
+ - The `@export` directive on the schema type is not required when you export a Resource class of the same name from `resources.js` — the class export serves as the endpoint registration. See [custom-resources.md](custom-resources.md) for details on building Resource classes.
1427
+ - Harper's REST layer automatically exposes `@export`-ed tables and Resource classes as HTTP endpoints. See [automatic-apis.md](automatic-apis.md) for how endpoints are structured and named.
1428
+ - ETag values include their double quotes as part of the value — include them verbatim when passing the value in `If-None-Match`.
1429
+ - `sourcedFrom` must be called after the table reference (`tables.JokeCache`) is available, which is guaranteed when the call is at the top level of `resources.js`.
1430
+
896
1431
  ## 4. Infrastructure & Ops
897
1432
 
898
1433
  ### 4.1 Deploying to Harper Fabric
899
1434
 
900
- Instructions for the agent to follow when deploying to Harper Fabric.
1435
+ Instructions for the agent to follow when deploying a Harper application to the Harper Fabric cloud using the Harper CLI.
901
1436
 
902
1437
  #### When to Use
903
1438
 
904
- Use this skill when you are ready to move your Harper application from local development to a cloud-hosted environment.
1439
+ Apply this rule when deploying a Harper application to a remote Harper instance or Harper Fabric cluster. This covers interactive deployments, CI/CD pipelines, and any scenario where the agent must push a local or remote package to a target environment.
905
1440
 
906
1441
  #### How It Works
907
1442
 
908
- 1. **Sign up**: Follow the [creating-a-fabric-account-and-cluster](creating-a-fabric-account-and-cluster.md) rule to create a Harper Fabric account, organization, and cluster.
909
- 2. **Configure Environment**: Add your cluster credentials and cluster application URL to `.env`:
1443
+ 1. **Authenticate with the remote target**: Run `harper login` once to store an authentication token. The CLI writes `HARPER_CLI_TARGET` to a local `.env` so subsequent commands do not need credentials repeated. Find the **Application URL** on the cluster's **Config → Overview** page (see [creating-a-fabric-account-and-cluster.md](creating-a-fabric-account-and-cluster.md)).
1444
+
910
1445
  ```bash
911
- CLI_TARGET_USERNAME='YOUR_CLUSTER_USERNAME'
912
- CLI_TARGET_PASSWORD='YOUR_CLUSTER_PASSWORD'
913
- CLI_TARGET='YOUR_CLUSTER_URL'
1446
+ harper login <Application URL>
1447
+ # Provide cluster username and password when prompted
914
1448
  ```
915
- 3. **Deploy From Local Environment**: Run `npm run deploy`.
916
- 4. **Set up CI/CD**: Configure `.github/workflows/deploy.yaml` and set repository secrets for automated deployments.
917
1449
 
918
- #### Manual Setup for Existing Apps
1450
+ 2. **Deploy the application**: Run `harper deploy` with the required parameters. After logging in, no credentials are needed inline.
1451
+
1452
+ ```bash
1453
+ harper deploy \
1454
+ project=<name> \
1455
+ package=<package> \
1456
+ target=<remote> \
1457
+ restart=true \
1458
+ replicated=true
1459
+ ```
919
1460
 
920
- If your application was not created with `npm create harper`, you'll need to manually configure the deployment scripts and CI/CD workflow.
1461
+ 3. **Choose a package source**: Set the `package` parameter to any valid npm dependency value, or omit it to package and deploy the current local directory.
921
1462
 
922
- ##### 1. Update `package.json`
1463
+ | Value | Effect |
1464
+ | ---------------------------------------------------- | ------------------------------------------------ |
1465
+ | _(omitted)_ | Packages and deploys the current local directory |
1466
+ | `"@harperdb/status-check"` | npm package |
1467
+ | `"HarperDB/status-check"` | GitHub repo (short form) |
1468
+ | `"https://github.com/HarperDB/status-check"` | GitHub repo (full URL) |
1469
+ | `"git+ssh://git@github.com:HarperDB/secret-app.git"` | Private repo via SSH |
1470
+ | `"https://example.com/application.tar.gz"` | Remote tarball |
923
1471
 
924
- Add the following scripts and dependencies to your `package.json`:
1472
+ For git tags, use the `semver` directive for reliable versioning:
925
1473
 
926
- ```json
927
- {
928
- "scripts": {
929
- "deploy": "dotenv -- npm run deploy:component",
930
- "deploy:component": "harper deploy_component . restart=rolling replicated=true"
931
- },
932
- "devDependencies": {
933
- "dotenv-cli": "^11.0.0",
934
- "harper": "^5.0.0"
935
- }
936
- }
937
- ```
1474
+ ```
1475
+ HarperDB/application-template#semver:v1.0.0
1476
+ ```
938
1477
 
939
- ###### Why split the scripts?
1478
+ 4. **Authenticate for CI/CD pipelines**: Use environment variables instead of interactive login. Set credentials before running `harper deploy`.
940
1479
 
941
- The `deploy` script is separated from `deploy:component` to ensure environment variables from your `.env` file are properly loaded and passed to the Harper CLI.
1480
+ ```bash
1481
+ export HARPER_CLI_USERNAME=<username>
1482
+ export HARPER_CLI_PASSWORD=<password>
1483
+ harper deploy \
1484
+ project=<name> \
1485
+ package=<package> \
1486
+ target=<remote> \
1487
+ restart=true \
1488
+ replicated=true
1489
+ ```
942
1490
 
943
- - `deploy`: Uses `dotenv-cli` to load environment variables (like `CLI_TARGET`, `CLI_TARGET_USERNAME`, and `CLI_TARGET_PASSWORD`) before executing the next command.
944
- - `deploy:component`: The actual command that performs the deployment.
1491
+ 5. **Register SSH keys for private repos**: Before deploying from an SSH-based private repository, use the Add SSH Key operation to register the key with the remote instance.
1492
+
1493
+ #### Examples
945
1494
 
946
- By using `dotenv -- npm run deploy:component`, the environment variables are correctly set in the shell session before `harper deploy_component` is called, allowing it to authenticate with your cluster.
1495
+ **Interactive login then deploy (recommended):**
947
1496
 
948
- ##### 2. Configure GitHub Actions
1497
+ ```bash
1498
+ # Log in once
1499
+ harper login <remote>
1500
+ # Provide your username and password when prompted
1501
+
1502
+ # Subsequently deploy without credentials
1503
+ harper deploy \
1504
+ project=<name> \
1505
+ package=<package> \
1506
+ target=<remote> \
1507
+ restart=true \
1508
+ replicated=true
1509
+ ```
949
1510
 
950
- Create a `.github/workflows/deploy.yaml` file with the following content:
1511
+ **Deploy with inline credentials (not recommended for production):**
951
1512
 
952
- ```yaml
953
- name: Deploy to Harper Fabric
954
- on:
955
- workflow_dispatch:
956
- # push:
957
- # branches:
958
- # - main
959
- concurrency:
960
- group: main
961
- cancel-in-progress: false
962
- jobs:
963
- deploy:
964
- runs-on: ubuntu-latest
965
- steps:
966
- - name: Checkout code
967
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
968
- with:
969
- fetch-depth: 0
970
- fetch-tags: true
971
- - name: Set up Node.js
972
- uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0
973
- with:
974
- cache: 'npm'
975
- node-version-file: '.nvmrc'
976
- - name: Install dependencies
977
- run: npm ci
978
- - name: Run unit tests
979
- run: npm test
980
- - name: Run lint
981
- run: npm run lint
982
- - name: Deploy
983
- run: npm run deploy
984
- env:
985
- CLI_TARGET: ${{ secrets.CLI_TARGET }}
986
- CLI_TARGET_USERNAME: ${{ secrets.CLI_TARGET_USERNAME }}
987
- CLI_TARGET_PASSWORD: ${{ secrets.CLI_TARGET_PASSWORD }}
1513
+ ```bash
1514
+ harper deploy \
1515
+ project=<name> \
1516
+ package=<package> \
1517
+ username=<username> \
1518
+ password=<password> \
1519
+ target=<remote> \
1520
+ restart=true \
1521
+ replicated=true
988
1522
  ```
989
1523
 
990
- Be sure to set the following repository secrets in your GitHub repository's /settings/secrets/actions:
1524
+ **Deploy a specific GitHub release by semver tag:**
991
1525
 
992
- - `CLI_TARGET`
993
- - `CLI_TARGET_USERNAME`
994
- - `CLI_TARGET_PASSWORD`
1526
+ ```bash
1527
+ harper deploy \
1528
+ project=my-app \
1529
+ package="HarperDB/application-template#semver:v1.0.0" \
1530
+ target=<remote> \
1531
+ restart=true \
1532
+ replicated=true
1533
+ ```
1534
+
1535
+ #### Notes
1536
+
1537
+ - Always prefer `harper login` for interactive use and environment variables (`HARPER_CLI_USERNAME`, `HARPER_CLI_PASSWORD`) for CI/CD. Avoid inline `username`/`password` parameters in production.
1538
+ - Omitting `package` causes the CLI to package the current local directory. Specifying a local file path creates a symlink, so changes are picked up between restarts without redeploying.
1539
+ - Harper generates a `package.json` from component configurations and resolves dependencies using a form of `npm install`.
1540
+ - For SSH-based private repos, register keys with the Add SSH Key operation before deploying.
995
1541
 
996
1542
  ### 4.2 Creating a Harper Fabric Account and Cluster
997
1543
 
@@ -1019,11 +1565,14 @@ CLI_TARGET='YOUR_CLUSTER_URL'
1019
1565
 
1020
1566
  ### 4.3 Creating Harper Applications
1021
1567
 
1022
- The fastest way to start a new Harper project is using the `create-harper` CLI tool. This command initializes a project with a standard folder structure, essential configuration files, and basic schema definitions.
1568
+ The fastest way to start a new Harper project is using the `create-harper` CLI tool. This command
1569
+ initializes a project with a standard folder structure, essential configuration files, and basic
1570
+ schema definitions.
1023
1571
 
1024
1572
  #### When to Use
1025
1573
 
1026
- Use this command when starting a new Harper application or adding a new Harper microservice to an existing architecture.
1574
+ Use this command when starting a new Harper application or adding a new Harper microservice to an
1575
+ existing architecture.
1027
1576
 
1028
1577
  #### Commands
1029
1578
 
@@ -1122,90 +1671,246 @@ Use this skill when you need to serve a frontend (HTML, CSS, JS, or a React app)
1122
1671
  ```
1123
1672
  Then in production, the "Static Plugin" option will performantly and securely serve your assets. `npm create harper@latest` scaffolds all of this for you.
1124
1673
 
1125
- ### 4.5 Logging Best Practices
1674
+ ### 4.5 Harper Logging
1675
+
1676
+ Instructions for the agent to follow when implementing logging in Harper applications, including direct logger usage, tagged loggers, and console capture behavior.
1677
+
1678
+ #### When to Use
1679
+
1680
+ Apply this rule when writing any JavaScript component, plugin, or resource that needs to emit structured log entries, filter logs by component, or capture existing `console.log` output into Harper's log system. Use it whenever you need to understand log levels, log entry format, or the `logger` global API.
1126
1681
 
1127
- Harper provides a robust logging system that captures standard output and offers a granular, tagged logging interface for both local and deployed environments.
1682
+ #### How It Works
1128
1683
 
1129
- #### Standard Console Logging
1684
+ 1. **Use the `logger` global directly** — `logger` is available in all JavaScript components without any imports. Call the method matching the desired severity level:
1130
1685
 
1131
- The simplest way to log in Harper is using standard JavaScript console methods. `console.log()`, `console.warn()`, `console.error()`, and `console.trace()` are automatically captured by Harper and can be viewed in the logs.
1686
+ ```javascript
1687
+ logger.trace('detailed trace message');
1688
+ logger.debug('debug info', { someContext: 'value' });
1689
+ logger.info('informational message');
1690
+ logger.warn('potential issue');
1691
+ logger.error('error occurred', error);
1692
+ logger.fatal('fatal error');
1693
+ logger.notify('server is ready');
1694
+ ```
1132
1695
 
1133
- - `console.log(...)`: Captured as `stdout` level in Harper logs.
1134
- - `console.warn(...)`: Captured as `stderr` level in Harper logs.
1135
- - `console.error(...)`: Captured as `stderr` level in Harper logs.
1136
- - `console.trace(...)`: Captured as `stdout` level in Harper logs (includes stack trace).
1696
+ Only entries at or above the configured `logging.level` (or `logging.external.level`) are written to `hdb.log`.
1137
1697
 
1138
- #### Harper Logger
1698
+ 2. **Create a tagged logger with `withTag(`** — Call `logger.withTag(tag)` once per module or class to get a `TaggedLogger` scoped to that tag. This prefixes every log entry with the tag, making log output filterable by component.
1139
1699
 
1140
- For more granularity and better organization, use Harper's built-in `logger`. You can use the global `logger` object or import it from the `harper` package.
1700
+ ```javascript
1701
+ const log = logger.withTag('my-resource');
1702
+ ```
1141
1703
 
1142
- ##### Log Levels
1704
+ Because `TaggedLogger` methods for disabled levels are `null`, always use optional chaining (`?.`) when calling them:
1143
1705
 
1144
- The Harper `logger` supports the following levels (ordered by increasing severity):
1706
+ ```javascript
1707
+ log.debug?.('Fetching record', { id });
1708
+ log.warn?.('Record not found', { id });
1709
+ log.error?.('Failed to update record', err);
1710
+ ```
1145
1711
 
1146
- - `trace`
1147
- - `debug`
1148
- - `info`
1149
- - `warn`
1150
- - `error`
1151
- - `fatal`
1152
- - `notify`
1712
+ `TaggedLogger` does not have a `withTag()` method.
1153
1713
 
1154
- ##### Usage
1714
+ 3. **Understand the interface contracts** — `MainLogger` always has all methods defined:
1155
1715
 
1156
- ```typescript
1157
- import { logger, loggerWithTag } from 'harper';
1716
+ ```typescript
1717
+ interface MainLogger {
1718
+ trace(...messages: any[]): void;
1719
+ debug(...messages: any[]): void;
1720
+ info(...messages: any[]): void;
1721
+ warn(...messages: any[]): void;
1722
+ error(...messages: any[]): void;
1723
+ fatal(...messages: any[]): void;
1724
+ notify(...messages: any[]): void;
1725
+ withTag(tag: string): TaggedLogger;
1726
+ }
1727
+ ```
1158
1728
 
1159
- // Basic logging
1160
- logger.info('Application started');
1161
- logger.error('An error occurred', error);
1729
+ `TaggedLogger` methods may be `null`:
1162
1730
 
1163
- // Tagged logging for better filtering (Namespacing)
1164
- const authLogger = loggerWithTag('auth');
1165
- authLogger.debug('User login attempt', { userId: '123' });
1166
- ```
1731
+ ```typescript
1732
+ interface TaggedLogger {
1733
+ trace: ((...messages: any[]) => void) | null;
1734
+ debug: ((...messages: any[]) => void) | null;
1735
+ info: ((...messages: any[]) => void) | null;
1736
+ warn: ((...messages: any[]) => void) | null;
1737
+ error: ((...messages: any[]) => void) | null;
1738
+ fatal: ((...messages: any[]) => void) | null;
1739
+ notify: ((...messages: any[]) => void) | null;
1740
+ }
1741
+ ```
1167
1742
 
1168
- Using `loggerWithTag` is highly recommended for grouping related logs, making them much easier to filter and analyze in the Harper Studio or via the API.
1743
+ 4. **Know the log levels** From least to most severe:
1169
1744
 
1170
- #### Programmatic Log Retrieval
1745
+ | Level | Description |
1746
+ | -------- | -------------------------------------------------------------------- |
1747
+ | `trace` | Highly detailed internal execution tracing. |
1748
+ | `debug` | Diagnostic information useful during development. |
1749
+ | `info` | General operational events. |
1750
+ | `warn` | Potential issues that don't prevent normal operation. |
1751
+ | `error` | Errors that affect specific operations. |
1752
+ | `fatal` | Critical errors causing process termination. |
1753
+ | `notify` | Important operational milestones. Always logged regardless of level. |
1171
1754
 
1172
- You can programmatically read logs from a deployed Harper instance using the `read_log` operation. This is useful for building custom monitoring tools or debugging dashboards.
1755
+ The default log level is `warn`. Setting a level includes that level and all more-severe levels.
1173
1756
 
1174
- ##### `read_log` Operation
1757
+ 5. **Enable console capture when porting existing code** — When `logging.console: true` is set, writes via `console.log`, `console.warn`, `console.error`, etc. are appended verbatim to `hdb.log`. Captured lines do **not** pass through `logger`'s level filter. Prefer `logger` directly in production code so that level filtering and tagging apply. Console capture is intended as a convenience for porting existing code and for debugging.
1175
1758
 
1176
- The `read_log` operation is a POST request to the Harper instance.
1759
+ 6. **Know where logs are written** — All standard log output goes to `<ROOTPATH>/log/hdb.log` (default: `~/hdb/log/hdb.log`). To also log to `stdout`/`stderr`, set `logging.stdStreams: true`.
1177
1760
 
1178
- **Example Request:**
1761
+ #### Examples
1762
+
1763
+ ##### Basic logging in a resource
1764
+
1765
+ ```javascript
1766
+ export class MyResource extends Resource {
1767
+ async get(id) {
1768
+ logger.debug('Fetching record', { id });
1769
+ const record = await super.get(id);
1770
+ if (!record) {
1771
+ logger.warn('Record not found', { id });
1772
+ }
1773
+ return record;
1774
+ }
1775
+
1776
+ async put(record) {
1777
+ logger.info('Updating record', { id: record.id });
1778
+ try {
1779
+ return await super.put(record);
1780
+ } catch (err) {
1781
+ logger.error('Failed to update record', err);
1782
+ throw err;
1783
+ }
1784
+ }
1785
+ }
1786
+ ```
1787
+
1788
+ ##### Tagged logging with `withTag()`
1789
+
1790
+ ```javascript
1791
+ const log = logger.withTag('my-resource');
1792
+
1793
+ export class MyResource extends Resource {
1794
+ async get(id) {
1795
+ log.debug?.('Fetching record', { id });
1796
+ const record = await super.get(id);
1797
+ if (!record) {
1798
+ log.warn?.('Record not found', { id });
1799
+ }
1800
+ return record;
1801
+ }
1179
1802
 
1180
- ```json
1181
- {
1182
- "operation": "read_log",
1183
- "limit": 100,
1184
- "start": 0,
1185
- "level": "error",
1186
- "order": "desc",
1187
- "from": "2024-01-01T00:00:00.000Z",
1188
- "until": "2024-01-02T00:00:00.000Z"
1803
+ async put(record) {
1804
+ log.info?.('Updating record', { id: record.id });
1805
+ try {
1806
+ return await super.put(record);
1807
+ } catch (err) {
1808
+ log.error?.('Failed to update record', err);
1809
+ throw err;
1810
+ }
1811
+ }
1189
1812
  }
1190
1813
  ```
1191
1814
 
1192
- ##### Parameters
1815
+ Tagged entries appear in `hdb.log` with the tag in the header:
1816
+
1817
+ ```
1818
+ 2023-03-09T14:25:05.269Z [info] [my-resource]: Updating record
1819
+ ```
1820
+
1821
+ #### Notes
1822
+
1823
+ - All log output is written to `<ROOTPATH>/log/hdb.log`. The `logger` global writes to this file at the configured `logging.external` level.
1824
+ - Log entry format for `logger`: `<timestamp> [<level>] [<thread>/<id>]: <message>`
1825
+ - Log entry format for `TaggedLogger`: `<timestamp> [<level>] [<tag>]: <message>`
1826
+ - `console.log` output is only forwarded to `hdb.log` when `logging.console: true` is explicitly set; it is not forwarded by default.
1827
+ - When logging to standard streams, run Harper in the foreground (`harper`, not `harper start`).
1828
+ - `TaggedLogger` is bound to the configured log level at creation time — always use `?.` on its methods.
1829
+
1830
+ ### 4.6 Load Environment Variables with loadEnv
1831
+
1832
+ Instructions for the agent to follow when loading environment variables from `.env` files into a Harper application using the `loadEnv` plugin.
1833
+
1834
+ #### When to Use
1835
+
1836
+ Apply this rule when a Harper application needs to load secrets or configuration values from `.env` files into `process.env` at startup. Use it whenever you need to configure `loadEnv` in `config.yaml`, control load order, handle multiple files, or manage override behavior.
1837
+
1838
+ #### How It Works
1839
+
1840
+ 1. **Declare `loadEnv` in `config.yaml`**: Add `loadEnv` to your `config.yaml` with a `files` key pointing to the `.env` file. `loadEnv` is built into Harper and does not need to be installed separately.
1841
+
1842
+ ```yaml
1843
+ loadEnv:
1844
+ files: '.env'
1845
+ ```
1846
+
1847
+ This loads the specified file from the root of your component directory into `process.env`.
1848
+
1849
+ 2. **Place `loadEnv` first**: Always declare `loadEnv` before any other components in `config.yaml` so environment variables are available before dependent components start. Because Harper is single-process, variables loaded onto `process.env` are shared across all components.
1850
+
1851
+ ```yaml
1852
+ # config.yaml — loadEnv must come first
1853
+ loadEnv:
1854
+ files: '.env'
1855
+
1856
+ rest: true
1857
+
1858
+ myApp:
1859
+ files: './src/*.js'
1860
+ ```
1861
+
1862
+ 3. **Control override behavior**: By default, existing shell or container environment variables take precedence over values in `.env` files. To force `.env` values to overwrite existing variables, set `override: true`.
1863
+
1864
+ ```yaml
1865
+ loadEnv:
1866
+ files: '.env'
1867
+ override: true
1868
+ ```
1869
+
1870
+ 4. **Load multiple files**: Provide a list of files or a glob pattern under `files`. Files are loaded in the order specified.
1871
+ ```yaml
1872
+ loadEnv:
1873
+ files:
1874
+ - '.env'
1875
+ - '.env.local'
1876
+ ```
1877
+ Or using a glob pattern:
1878
+ ```yaml
1879
+ loadEnv:
1880
+ files: 'env-vars/*'
1881
+ ```
1882
+
1883
+ #### Examples
1884
+
1885
+ A complete `config.yaml` using `loadEnv` with multiple files and override enabled:
1886
+
1887
+ ```yaml
1888
+ # config.yaml — loadEnv must come first
1889
+ loadEnv:
1890
+ files:
1891
+ - '.env'
1892
+ - '.env.local'
1893
+ override: true
1894
+
1895
+ rest: true
1896
+
1897
+ myApp:
1898
+ files: './src/*.js'
1899
+ ```
1900
+
1901
+ A minimal setup loading a single `.env` file:
1193
1902
 
1194
- - `limit`: Number of log entries to return.
1195
- - `start`: Offset for pagination.
1196
- - `level`: Filter by log level (`info`, `error`, `warn`, `debug`, `trace`, `notify`, `fatal`, `stdout`, `stderr`).
1197
- - `from`: ISO 8601 timestamp to start reading from.
1198
- - `until`: ISO 8601 timestamp to stop reading at.
1199
- - `order`: Sort order, either `asc` or `desc`.
1200
- - `replicated`: (Boolean) Include logs from replicated nodes in a cluster.
1903
+ ```yaml
1904
+ loadEnv:
1905
+ files: '.env'
1201
1906
 
1202
- ##### Log Entry Structure
1907
+ myApp:
1908
+ files: './src/*.js'
1909
+ ```
1203
1910
 
1204
- Each log entry returned by `read_log` typically includes:
1911
+ #### Notes
1205
1912
 
1206
- - `level`: The severity level of the log.
1207
- - `timestamp`: When the log was recorded.
1208
- - `thread`: The execution thread.
1209
- - `tags`: An array of tags (e.g., from `loggerWithTag`).
1210
- - `node`: The node name in a Harper cluster.
1211
- - `message`: The logged content.
1913
+ - `loadEnv` is built into Harper declare it in `config.yaml` only; do not install it as a separate package.
1914
+ - The `files` value accepts either a single string, a list of strings, or a glob pattern.
1915
+ - Without `override: true`, variables already present in the environment are never overwritten by values in `.env` files.
1916
+ - `process.env` is shared across all Harper components in the same process, so load order in `config.yaml` determines availability.