@venizia/ignis-docs 0.2.1-0 → 0.2.1-1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/content/best-practices/architectural-patterns.md +3 -3
- package/content/best-practices/code-style-standards/naming-conventions.md +1 -1
- package/content/best-practices/contribution-workflow.md +2 -2
- package/content/best-practices/error-handling.md +94 -89
- package/content/best-practices/security-guidelines.md +5 -5
- package/content/extensions/components/api-reference.md +22 -21
- package/content/extensions/components/authentication/api.md +64 -28
- package/content/extensions/components/authentication/errors.md +19 -5
- package/content/extensions/components/authentication/index.md +19 -18
- package/content/extensions/components/authentication/usage.md +55 -30
- package/content/extensions/components/authorization/api.md +351 -84
- package/content/extensions/components/authorization/errors.md +51 -17
- package/content/extensions/components/authorization/getting-started.md +227 -0
- package/content/extensions/components/authorization/index.md +24 -16
- package/content/extensions/components/authorization/usage.md +45 -21
- package/content/extensions/components/health-check.md +15 -9
- package/content/extensions/components/index.md +24 -90
- package/content/extensions/components/mail/api.md +105 -54
- package/content/extensions/components/mail/errors.md +12 -10
- package/content/extensions/components/mail/index.md +32 -13
- package/content/extensions/components/mail/usage.md +26 -18
- package/content/extensions/components/request-tracker.md +18 -14
- package/content/extensions/components/socket-io/api.md +377 -882
- package/content/extensions/components/socket-io/errors.md +49 -51
- package/content/extensions/components/socket-io/index.md +72 -88
- package/content/extensions/components/socket-io/usage.md +107 -117
- package/content/extensions/components/static-asset/api.md +83 -31
- package/content/extensions/components/static-asset/errors.md +18 -7
- package/content/extensions/components/static-asset/index.md +18 -11
- package/content/extensions/components/static-asset/usage.md +11 -6
- package/content/extensions/components/template/index.md +3 -3
- package/content/extensions/components/websocket/api.md +58 -27
- package/content/extensions/components/websocket/errors.md +3 -3
- package/content/extensions/components/websocket/index.md +8 -7
- package/content/extensions/components/websocket/usage.md +21 -8
- package/content/extensions/helpers/cron/index.md +8 -7
- package/content/extensions/helpers/crypto/index.md +16 -8
- package/content/extensions/helpers/crypto/reference.md +96 -24
- package/content/extensions/helpers/env/index.md +14 -10
- package/content/extensions/helpers/error/index.md +99 -23
- package/content/extensions/helpers/index.md +61 -47
- package/content/extensions/helpers/inversion/index.md +23 -6
- package/content/extensions/helpers/inversion/reference.md +30 -22
- package/content/extensions/helpers/kafka/admin.md +3 -2
- package/content/extensions/helpers/kafka/compile-binary.md +69 -44
- package/content/extensions/helpers/kafka/consumer.md +24 -21
- package/content/extensions/helpers/kafka/examples.md +22 -234
- package/content/extensions/helpers/kafka/index.md +30 -62
- package/content/extensions/helpers/kafka/producer.md +31 -26
- package/content/extensions/helpers/kafka/schema-registry.md +19 -14
- package/content/extensions/helpers/logger/hf-logger.md +49 -22
- package/content/extensions/helpers/logger/index.md +37 -12
- package/content/extensions/helpers/logger/pino.md +31 -11
- package/content/extensions/helpers/logger/reference.md +243 -52
- package/content/extensions/helpers/network/api.md +65 -30
- package/content/extensions/helpers/network/index.md +32 -11
- package/content/extensions/helpers/queue/index.md +17 -6
- package/content/extensions/helpers/queue/reference.md +52 -25
- package/content/extensions/helpers/redis/index.md +29 -11
- package/content/extensions/helpers/redis/reference.md +85 -28
- package/content/extensions/helpers/secrets/index.md +82 -12
- package/content/extensions/helpers/socket-io/api.md +40 -21
- package/content/extensions/helpers/socket-io/index.md +26 -10
- package/content/extensions/helpers/storage/api.md +42 -24
- package/content/extensions/helpers/storage/index.md +10 -7
- package/content/extensions/helpers/types/index.md +20 -7
- package/content/extensions/helpers/types/reference.md +53 -14
- package/content/extensions/helpers/uid/index.md +166 -10
- package/content/extensions/helpers/websocket/api.md +52 -13
- package/content/extensions/helpers/websocket/index.md +7 -4
- package/content/extensions/helpers/worker-thread/index.md +9 -5
- package/content/extensions/helpers/worker-thread/reference.md +20 -20
- package/content/extensions/index.md +38 -39
- package/content/extensions/src-details/mcp-server.md +96 -548
- package/content/guides/core-concepts/persistent/datasources.md +2 -2
- package/content/guides/core-concepts/persistent/index.md +5 -1
- package/content/guides/core-concepts/persistent/pglite.md +218 -0
- package/content/guides/core-concepts/persistent/postgres-drivers.md +21 -15
- package/content/guides/core-concepts/persistent/search-meilisearch.md +22 -12
- package/content/guides/core-concepts/persistent/search-typesense.md +26 -14
- package/content/guides/core-concepts/persistent/sqlite.md +296 -0
- package/content/guides/core-concepts/persistent/transactions.md +12 -11
- package/content/guides/get-started/5-minute-quickstart.md +59 -206
- package/content/guides/get-started/philosophy.md +135 -670
- package/content/guides/get-started/setup.md +53 -74
- package/content/guides/migrations/redis-helpers-migration.md +4 -3
- package/content/guides/migrations/scoped-rbac-migration.md +5 -5
- package/content/guides/migrations/unified-connectors-migration.md +7 -6
- package/content/guides/tutorials/realtime-chat.md +1 -1
- package/content/references/base/application.md +63 -21
- package/content/references/base/components.md +3 -3
- package/content/references/base/connectors.md +14 -14
- package/content/references/base/controllers.md +12 -12
- package/content/references/base/datasources-reference.md +32 -31
- package/content/references/base/datasources.md +6 -6
- package/content/references/base/dependency-injection.md +12 -11
- package/content/references/base/filter-system/application-usage.md +54 -30
- package/content/references/base/filter-system/array-operators.md +24 -46
- package/content/references/base/filter-system/comparison-operators.md +47 -67
- package/content/references/base/filter-system/default-filter.md +59 -53
- package/content/references/base/filter-system/fields-order-pagination.md +92 -146
- package/content/references/base/filter-system/index.md +25 -13
- package/content/references/base/filter-system/json-filtering.md +45 -184
- package/content/references/base/filter-system/list-operators.md +23 -53
- package/content/references/base/filter-system/logical-operators.md +63 -121
- package/content/references/base/filter-system/null-operators.md +34 -104
- package/content/references/base/filter-system/pattern-matching.md +40 -55
- package/content/references/base/filter-system/quick-reference.md +86 -198
- package/content/references/base/filter-system/range-operators.md +18 -46
- package/content/references/base/filter-system/tips.md +6 -6
- package/content/references/base/filter-system/use-cases.md +33 -15
- package/content/references/base/grpc-controllers.md +53 -17
- package/content/references/base/index.md +5 -3
- package/content/references/base/middlewares.md +11 -10
- package/content/references/base/models-reference.md +17 -17
- package/content/references/base/models.md +4 -3
- package/content/references/base/providers.md +8 -8
- package/content/references/base/repositories/advanced.md +224 -326
- package/content/references/base/repositories/index.md +19 -6
- package/content/references/base/repositories/mixins.md +5 -5
- package/content/references/base/repositories/relations.md +160 -293
- package/content/references/base/repositories/soft-deletable.md +16 -6
- package/content/references/base/secrets.md +17 -13
- package/content/references/base/services.md +6 -4
- package/content/references/configuration/environment-variables.md +31 -23
- package/content/references/configuration/index.md +6 -4
- package/content/references/index.md +1 -1
- package/content/references/utilities/duration.md +85 -0
- package/content/references/utilities/index.md +5 -1
- package/content/references/utilities/jsx-reference.md +11 -11
- package/content/references/utilities/jsx.md +2 -2
- package/content/references/utilities/module.md +78 -25
- package/content/references/utilities/request.md +2 -1
- package/content/references/utilities/retry.md +139 -0
- package/content/references/utilities/schema.md +2 -2
- package/content/references/utilities/statuses-reference.md +4 -4
- package/content/references/utilities/statuses.md +5 -5
- package/dist/mcp-server/index.js +0 -0
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts.map +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.js +3 -3
- package/dist/mcp-server/tools/github/search-code.tool.js +1 -1
- package/dist/mcp-server/tools/github/verify-dependencies.tool.js +1 -1
- package/dist/mcp-server/tools/github/view-source-file.tool.js +1 -1
- package/package.json +17 -16
|
@@ -6,7 +6,7 @@ difficulty: intermediate
|
|
|
6
6
|
|
|
7
7
|
# Kafka
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
IGNIS wraps `@platformatic/kafka` in four scoped helpers: producer, consumer, admin, and schema registry. Each adds health tracking, graceful shutdown, and IGNIS-style scoped logging over the raw client.
|
|
10
10
|
|
|
11
11
|
## In one example
|
|
12
12
|
|
|
@@ -29,75 +29,43 @@ await producer.getProducer().send({
|
|
|
29
29
|
await producer.close();
|
|
30
30
|
```
|
|
31
31
|
|
|
32
|
-
`getProducer()` returns the full `@platformatic/kafka` `Producer`. Every helper follows
|
|
32
|
+
`getProducer()` returns the full `@platformatic/kafka` `Producer`. Every helper follows the same three-step pattern:
|
|
33
33
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|---|---|---|
|
|
40
|
-
| `KafkaProducerHelper` | `Producer` | Publish messages, run transactions |
|
|
41
|
-
| `KafkaConsumerHelper` | `Consumer` | Consume via consumer groups, monitor lag |
|
|
42
|
-
| `KafkaAdminHelper` | `Admin` | Manage topics, partitions, groups, ACLs, configs |
|
|
43
|
-
| `KafkaSchemaRegistryHelper` | `ConfluentSchemaRegistry` | Schema-validated serialization (Avro/Protobuf/JSON Schema) |
|
|
44
|
-
|
|
45
|
-
- **The three connected helpers share an identical health & close API**, regardless of which client they wrap:
|
|
46
|
-
|
|
47
|
-
```typescript
|
|
48
|
-
const producer = KafkaProducerHelper.newInstance({ bootstrapBrokers, clientId });
|
|
49
|
-
const consumer = KafkaConsumerHelper.newInstance({ bootstrapBrokers, clientId, groupId });
|
|
50
|
-
const admin = KafkaAdminHelper.newInstance({ bootstrapBrokers, clientId });
|
|
51
|
-
|
|
52
|
-
producer.isHealthy(); // true once at least one broker is connected
|
|
53
|
-
consumer.isReady(); // true once connected AND consumer.isActive()
|
|
54
|
-
admin.getHealthStatus(); // 'connected' | 'disconnected' | 'unknown'
|
|
55
|
-
|
|
56
|
-
await Promise.all([producer.close(), consumer.close(), admin.close()]); // graceful, force fallback
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
```
|
|
60
|
-
BaseHelper (scoped logging, identifier)
|
|
61
|
-
+-- BaseKafkaHelper<TClient> (health tracking, broker events, graceful shutdown)
|
|
62
|
-
| +-- KafkaProducerHelper<K,V,HK,HV>
|
|
63
|
-
| +-- KafkaConsumerHelper<K,V,HK,HV>
|
|
64
|
-
| +-- KafkaAdminHelper
|
|
65
|
-
|
|
|
66
|
-
+-- KafkaSchemaRegistryHelper<K,V,HK,HV> (no broker connection)
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
- **`BaseKafkaHelper` is the shared base** for the three connected helpers (everything but the schema registry). It tracks per-broker connection state (`host:port` keys), so one idle disconnect never flips `isHealthy()` to `false` - only when every broker is gone. `isReady()`, `getHealthStatus()`, and `getConnectedBrokerCount()` read the same state, and `close({ isForce })` gives every connected helper the same two-phase graceful-then-force shutdown.
|
|
70
|
-
- **Health status follows broker events automatically.** `client:broker:connect` marks a broker connected; `client:broker:disconnect` and `client:broker:failed` remove it and only flip the status to `'disconnected'` once every broker is gone; `close()` clears everything. You never set `healthStatus` yourself.
|
|
71
|
-
- **`KafkaSchemaRegistryHelper` extends `BaseHelper` directly.** It opens no broker connection, so it has no health tracking - it's a configuration wrapper you hand to a producer or consumer via `registry`.
|
|
72
|
-
- **Everything lives at the `/kafka` sub-path, never the root barrel.** `@platformatic/kafka` is an optional peer dependency (`^2.1.0` - `bun add @platformatic/kafka`); keeping it off the root barrel lets apps that never touch Kafka tree-shake it away entirely.
|
|
73
|
-
- **Generics default to `string`.** All four classes accept `<KeyType, ValueType, HeaderKeyType, HeaderValueType>` generics. Without serializers/deserializers, messages travel as raw `Buffer`.
|
|
74
|
-
- **`newInstance()` is the documented entry point.** Every class also exposes a public constructor, but `newInstance()` is what every example on these pages uses, and it carries the same generic inference: `KafkaProducerHelper.newInstance<string, MyEvent>({ ... })`.
|
|
75
|
-
- **Compiling to a single binary needs one extra step.** `bun build --compile` crashes at startup with `ENOENT: native.wasm` unless the build registers `platformaticWasmPlugin()` - see [Compiling to a Single Binary](./compile-binary).
|
|
34
|
+
| Step | Call |
|
|
35
|
+
|---|---|
|
|
36
|
+
| Construct | `newInstance()` |
|
|
37
|
+
| Reach the native client | `getProducer()` / `getConsumer()` / `getAdmin()` |
|
|
38
|
+
| Close | through the helper, not the native client |
|
|
76
39
|
|
|
77
|
-
|
|
40
|
+
## Which helper do I need
|
|
78
41
|
|
|
79
|
-
|
|
|
42
|
+
| Class | Wraps | Use it to |
|
|
80
43
|
|---|---|---|
|
|
81
|
-
| `
|
|
82
|
-
| `
|
|
83
|
-
| `
|
|
84
|
-
| `
|
|
85
|
-
|
|
44
|
+
| `KafkaProducerHelper` | `Producer` | Publish messages, run transactions |
|
|
45
|
+
| `KafkaConsumerHelper` | `Consumer` | Consume via consumer groups, monitor lag |
|
|
46
|
+
| `KafkaAdminHelper` | `Admin` | Manage topics, partitions, groups, ACLs, configs |
|
|
47
|
+
| `KafkaSchemaRegistryHelper` | `ConfluentSchemaRegistry` | Schema-validated serialization (Avro/Protobuf/JSON Schema) |
|
|
48
|
+
|
|
49
|
+
A few facts hold across all four:
|
|
86
50
|
|
|
87
|
-
|
|
51
|
+
- **Producer, consumer, and admin share one health and close API.** `isHealthy()`, `isReady()`, `getHealthStatus()`, and `close({ isForce })` mean the same thing on every class. Each page documents the exact return values.
|
|
52
|
+
- **Schema registry opens no broker connection.** It extends `BaseHelper` directly, not the shared connected-helper base. It has no health tracking - it's a configuration wrapper you hand to a producer or consumer via `registry`.
|
|
53
|
+
- **Everything lives under `/kafka`, never the root barrel.** Install the optional peer yourself: `bun add @platformatic/kafka` (`^2.6.1`). An app that never touches Kafka tree-shakes it away entirely.
|
|
54
|
+
- **Compiling to a single binary needs one extra build step.** Skip it, and the compiled app crashes at startup with `ENOENT: native.wasm` or `Cannot find package 'ajv-draft-04'` - see [Compiling to a Single Binary](./compile-binary).
|
|
55
|
+
- **Defaults and enum-like values ship as exported constants**, not magic numbers - `KafkaDefaults`, `KafkaAcks`, `KafkaGroupProtocol`, `KafkaHealthStatuses`. Each page's options table names the constant it uses.
|
|
88
56
|
|
|
89
|
-
|
|
57
|
+
## Find what you need
|
|
90
58
|
|
|
91
|
-
|
|
|
59
|
+
| You want to | Go to |
|
|
92
60
|
|---|---|
|
|
93
|
-
|
|
|
94
|
-
|
|
|
95
|
-
|
|
|
96
|
-
| [Schema Registry](./schema-registry) |
|
|
97
|
-
|
|
|
98
|
-
|
|
|
99
|
-
|
|
100
|
-
Start with [Producer](./producer) or [Consumer](./consumer) if you're wiring up your first topic
|
|
61
|
+
| Publish messages, set up SASL/TLS, run transactions | [Producer](./producer) |
|
|
62
|
+
| Consume messages, monitor lag, handle reconnects | [Consumer](./consumer) |
|
|
63
|
+
| Create or delete topics, inspect consumer groups, manage ACLs | [Admin](./admin) |
|
|
64
|
+
| Validate message shape with Avro, Protobuf, or JSON Schema | [Schema Registry](./schema-registry) |
|
|
65
|
+
| See end-to-end examples or fix a connection error | [Examples & Troubleshooting](./examples) |
|
|
66
|
+
| Ship an app that imports a Kafka helper as a single binary | [Compiling to a Single Binary](./compile-binary) |
|
|
67
|
+
|
|
68
|
+
Start with [Producer](./producer) or [Consumer](./consumer) if you're wiring up your first topic. Start with [Compiling to a Single Binary](./compile-binary) if an existing app started crashing at startup after a `bun build --compile`.
|
|
101
69
|
|
|
102
70
|
## See also
|
|
103
71
|
|
|
@@ -40,17 +40,17 @@ interface IKafkaProducerOptions<KeyType, ValueType, HeaderKeyType, HeaderValueTy
|
|
|
40
40
|
| Option | Type | Default | Description |
|
|
41
41
|
|--------|------|---------|-------------|
|
|
42
42
|
| `identifier` | `string` | `'kafka-producer'` | Scoped logging identifier |
|
|
43
|
-
| `serializers` | `Partial<Serializers<K,V,HK,HV>>` |
|
|
44
|
-
| `compression` | `CompressionAlgorithmValue` |
|
|
45
|
-
| `acks` | `TKafkaAcks` |
|
|
46
|
-
| `idempotent` | `boolean` |
|
|
47
|
-
| `transactionalId` | `string` |
|
|
48
|
-
| `strict` | `boolean` | `true` | Strict mode
|
|
43
|
+
| `serializers` | `Partial<Serializers<K,V,HK,HV>>` | - | Key/value/header serializers |
|
|
44
|
+
| `compression` | `CompressionAlgorithmValue` | - | `'none'`, `'gzip'`, `'snappy'`, `'lz4'`, `'zstd'` |
|
|
45
|
+
| `acks` | `TKafkaAcks` | - | Acknowledgment level: `0` (none), `1` (leader), `-1` (all) |
|
|
46
|
+
| `idempotent` | `boolean` | - | Enable idempotent producer (exactly-once within partition) |
|
|
47
|
+
| `transactionalId` | `string` | - | Transactional ID for exactly-once across partitions |
|
|
48
|
+
| `strict` | `boolean` | `true` | Strict mode - fail on unknown topics |
|
|
49
49
|
| `autocreateTopics` | `boolean` | `false` | Auto-create topics on first produce |
|
|
50
50
|
| `shutdownTimeout` | `number` | `30000` | Graceful shutdown timeout in ms |
|
|
51
|
-
| `registry` | `SchemaRegistry` |
|
|
52
|
-
| `onBrokerConnect` | `TKafkaBrokerEventCallback` |
|
|
53
|
-
| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` |
|
|
51
|
+
| `registry` | `SchemaRegistry` | - | Schema registry for auto ser/deser |
|
|
52
|
+
| `onBrokerConnect` | `TKafkaBrokerEventCallback` | - | Called when broker connects |
|
|
53
|
+
| `onBrokerDisconnect` | `TKafkaBrokerEventCallback` | - | Called when broker disconnects |
|
|
54
54
|
|
|
55
55
|
Plus the shared [Connection & Authentication](#connection--authentication) options below, all inherited from `IKafkaConnectionOptions`.
|
|
56
56
|
|
|
@@ -60,14 +60,14 @@ Plus the shared [Connection & Authentication](#connection--authentication) optio
|
|
|
60
60
|
|
|
61
61
|
| Option | Type | Default | Description |
|
|
62
62
|
|--------|------|---------|-------------|
|
|
63
|
-
| `bootstrapBrokers` | `string[]` |
|
|
64
|
-
| `clientId` | `string` |
|
|
63
|
+
| `bootstrapBrokers` | `string[]` | - | Broker addresses (`host:port`). **Required** |
|
|
64
|
+
| `clientId` | `string` | - | Unique client identifier. **Required** |
|
|
65
65
|
| `retries` | `number` | `3` | Connection retries before failing |
|
|
66
66
|
| `retryDelay` | `number` | `1000` | Delay between retries (ms) |
|
|
67
|
-
| `sasl` | `SASLOptions` |
|
|
68
|
-
| `tls` / `ssl` | `TLSConnectionOptions` |
|
|
69
|
-
| `connectTimeout` | `number` |
|
|
70
|
-
| `requestTimeout` | `number` |
|
|
67
|
+
| `sasl` | `SASLOptions` | - | SASL authentication |
|
|
68
|
+
| `tls` / `ssl` | `TLSConnectionOptions` | - | TLS options (`ssl` is an alias for `tls`) |
|
|
69
|
+
| `connectTimeout` | `number` | - | TCP connection timeout (ms) |
|
|
70
|
+
| `requestTimeout` | `number` | - | Kafka request timeout (ms) |
|
|
71
71
|
|
|
72
72
|
**SASL mechanisms** (`@platformatic/kafka` supports five):
|
|
73
73
|
|
|
@@ -75,7 +75,7 @@ Plus the shared [Connection & Authentication](#connection--authentication) optio
|
|
|
75
75
|
|-----------|----------|
|
|
76
76
|
| `PLAIN` | Username/password (pair with `tls` in production) |
|
|
77
77
|
| `SCRAM-SHA-256` / `SCRAM-SHA-512` | Challenge-response - password never sent in plaintext |
|
|
78
|
-
| `OAUTHBEARER` | Token-based
|
|
78
|
+
| `OAUTHBEARER` | Token-based - Azure Event Hubs, Confluent Cloud. `token` accepts a string or an async function that returns one |
|
|
79
79
|
| `GSSAPI` | Kerberos authentication |
|
|
80
80
|
|
|
81
81
|
```typescript
|
|
@@ -142,7 +142,7 @@ await helper.close({ isForce: true });
|
|
|
142
142
|
|
|
143
143
|
## Serialization
|
|
144
144
|
|
|
145
|
-
`@platformatic/kafka`'s default wire format is `Buffer`. The helpers default the generic types to `string`, but
|
|
145
|
+
`@platformatic/kafka`'s default wire format is `Buffer`. The helpers default the generic types to `string`, but that's a type default only. Pass serializers explicitly, or messages travel as raw `Buffer`.
|
|
146
146
|
|
|
147
147
|
| Export | Type | Description |
|
|
148
148
|
|--------|------|-------------|
|
|
@@ -264,13 +264,15 @@ The callback receives an `IKafkaTransactionContext`:
|
|
|
264
264
|
|
|
265
265
|
## Graceful Shutdown
|
|
266
266
|
|
|
267
|
-
`close()`
|
|
267
|
+
`close()` runs a two-phase shutdown:
|
|
268
268
|
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
269
|
+
| Phase | When it runs | What happens |
|
|
270
|
+
|---|---|---|
|
|
271
|
+
| Graceful | Default | Calls `close(true)` on the underlying producer, capped at `shutdownTimeout` (default 30s) |
|
|
272
|
+
| Force fallback | Graceful times out | Force-closes automatically, no further waiting |
|
|
273
|
+
| Force | `close({ isForce: true })` | Calls `close(true)` immediately, no timeout |
|
|
272
274
|
|
|
273
|
-
After `close()`, all broker tracking is cleared and `healthStatus`
|
|
275
|
+
After `close()`, all broker tracking is cleared and `healthStatus` becomes `'disconnected'`.
|
|
274
276
|
|
|
275
277
|
```typescript
|
|
276
278
|
// Graceful (recommended)
|
|
@@ -373,11 +375,13 @@ Close the producer connection.
|
|
|
373
375
|
|
|
374
376
|
## Key Partitioning
|
|
375
377
|
|
|
376
|
-
By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to
|
|
378
|
+
By default, `@platformatic/kafka` uses **murmur2 hashing** on the message key to pick the target partition:
|
|
377
379
|
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
380
|
+
| Key | Target partition |
|
|
381
|
+
|---|---|
|
|
382
|
+
| Same key, every time | Same partition - ordering is guaranteed per key |
|
|
383
|
+
| `undefined` | Round-robin across partitions |
|
|
384
|
+
| Explicit `partition` field | That partition, overriding the partitioner |
|
|
381
385
|
|
|
382
386
|
```typescript
|
|
383
387
|
// Custom partitioner
|
|
@@ -395,6 +399,7 @@ await producer.send({
|
|
|
395
399
|
- [Consumer](./consumer) - the receiving side, including message callbacks and lag monitoring
|
|
396
400
|
- [Schema Registry](./schema-registry) - schema-validated serialization instead of manual `serializers`
|
|
397
401
|
- [Compiling to a Single Binary](./compile-binary) - required when this helper ships inside a `bun build --compile` binary
|
|
402
|
+
- [Examples & Troubleshooting](./examples) - IoC wiring and the common connection-error lookup table
|
|
398
403
|
|
|
399
404
|
**Files:**
|
|
400
405
|
|
|
@@ -18,7 +18,10 @@ class KafkaSchemaRegistryHelper<
|
|
|
18
18
|
```
|
|
19
19
|
|
|
20
20
|
> [!NOTE]
|
|
21
|
-
> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`)
|
|
21
|
+
> `KafkaSchemaRegistryHelper` extends `BaseHelper` directly (not `BaseKafkaHelper`) - it has no broker connection or health tracking. It's a configuration wrapper, not a client.
|
|
22
|
+
|
|
23
|
+
> [!WARNING]
|
|
24
|
+
> Compiling to a standalone binary needs `platformaticRequirePlugin()`. `ConfluentSchemaRegistry` resolves `ajv-draft-04` through `createRequire` at module load, which `bun build --compile` cannot see through. See [Compiling to a Single Binary](./compile-binary).
|
|
22
25
|
|
|
23
26
|
## Helper API
|
|
24
27
|
|
|
@@ -39,10 +42,10 @@ interface IKafkaSchemaRegistryOptions extends ConfluentSchemaRegistryOptions {
|
|
|
39
42
|
|
|
40
43
|
| Option | Type | Default | Description |
|
|
41
44
|
|--------|------|---------|-------------|
|
|
42
|
-
| `url` | `string` |
|
|
43
|
-
| `auth` | `{ username: string; password: string }` |
|
|
44
|
-
| `protobufTypeMapper` | `ProtobufTypeMapper` |
|
|
45
|
-
| `jsonValidateSend` | `boolean` |
|
|
45
|
+
| `url` | `string` | - | Schema registry URL. **Required** |
|
|
46
|
+
| `auth` | `{ username: string; password: string }` | - | Basic auth credentials |
|
|
47
|
+
| `protobufTypeMapper` | `ProtobufTypeMapper` | - | Custom Protobuf type mapper |
|
|
48
|
+
| `jsonValidateSend` | `boolean` | - | Validate JSON schema on produce |
|
|
46
49
|
| `identifier` | `string` | `'kafka-schema-registry'` | Scoped logging identifier |
|
|
47
50
|
|
|
48
51
|
## What it solves
|
|
@@ -50,23 +53,24 @@ interface IKafkaSchemaRegistryOptions extends ConfluentSchemaRegistryOptions {
|
|
|
50
53
|
Without a schema registry, producers and consumers must agree on message shape out-of-band. If the producer changes the shape of `value` (adds/removes fields), consumers break silently at runtime.
|
|
51
54
|
|
|
52
55
|
- **Schema Registry is a centralized server** (Confluent Schema Registry) that stores and validates schemas.
|
|
53
|
-
- **It enforces a contract
|
|
56
|
+
- **It enforces a contract on the way out.** The producer says "I want to send this shape." The registry validates that shape against the registered schema before the message reaches Kafka.
|
|
57
|
+
- **It enforces a contract on the way in.** The consumer asks "what shape is this?" The registry answers with how to deserialize it.
|
|
54
58
|
|
|
55
59
|
| | Without registry | With registry |
|
|
56
60
|
|---|---|---|
|
|
57
61
|
| **Message format** | Raw string, manual `JSON.stringify`/`JSON.parse` | Typed object, auto ser/deser |
|
|
58
|
-
| **Validation** | None
|
|
62
|
+
| **Validation** | None - runtime crashes on shape drift | Schema validated before send |
|
|
59
63
|
| **Schema evolution** | Breaks consumers silently | Backward/forward compatibility enforced |
|
|
60
64
|
| **Where schemas live** | Nowhere (tribal knowledge) | Centralized server, e.g. `http://registry:8081` |
|
|
61
65
|
|
|
62
|
-
Use it when you need schema enforcement and compatibility checks across producers
|
|
66
|
+
Use it when you need schema enforcement and compatibility checks across producers and consumers, especially in multi-team environments. Skip it for simple string or JSON messages where one team controls both sides - coordinate format changes by hand instead.
|
|
63
67
|
|
|
64
68
|
## Without vs. with the registry
|
|
65
69
|
|
|
66
70
|
Without a registry, serialization is entirely manual and unchecked:
|
|
67
71
|
|
|
68
72
|
```typescript
|
|
69
|
-
// Producer
|
|
73
|
+
// Producer - manually serialize
|
|
70
74
|
const producer = KafkaProducerHelper.newInstance({
|
|
71
75
|
bootstrapBrokers: ['127.0.0.1:29092'],
|
|
72
76
|
clientId: 'order-producer',
|
|
@@ -76,11 +80,11 @@ await producer.getProducer().send({
|
|
|
76
80
|
messages: [{
|
|
77
81
|
topic: 'orders',
|
|
78
82
|
key: 'order-1',
|
|
79
|
-
value: JSON.stringify({ id: 1, total: 99.99 }), // <-
|
|
83
|
+
value: JSON.stringify({ id: 1, total: 99.99 }), // <- plain string, no validation
|
|
80
84
|
}],
|
|
81
85
|
});
|
|
82
86
|
|
|
83
|
-
// Consumer
|
|
87
|
+
// Consumer - manually deserialize, hope the shape is correct
|
|
84
88
|
const consumer = KafkaConsumerHelper.newInstance({
|
|
85
89
|
bootstrapBrokers: ['127.0.0.1:29092'],
|
|
86
90
|
clientId: 'order-consumer',
|
|
@@ -103,13 +107,13 @@ import {
|
|
|
103
107
|
KafkaConsumerHelper,
|
|
104
108
|
} from '@venizia/ignis-helpers/kafka';
|
|
105
109
|
|
|
106
|
-
// 1. Create registry
|
|
110
|
+
// 1. Create registry - points to Confluent Schema Registry server
|
|
107
111
|
const registry = KafkaSchemaRegistryHelper.newInstance({
|
|
108
112
|
url: 'http://localhost:8081',
|
|
109
113
|
// auth: { username: 'user', password: 'pass' }, // optional
|
|
110
114
|
});
|
|
111
115
|
|
|
112
|
-
// 2. Producer
|
|
116
|
+
// 2. Producer - pass registry, it auto-serializes values using the registered schema
|
|
113
117
|
const producer = KafkaProducerHelper.newInstance({
|
|
114
118
|
bootstrapBrokers: ['127.0.0.1:29092'],
|
|
115
119
|
clientId: 'order-producer',
|
|
@@ -125,7 +129,7 @@ await producer.getProducer().send({
|
|
|
125
129
|
});
|
|
126
130
|
// If the value doesn't match the registered schema -> error BEFORE sending to Kafka
|
|
127
131
|
|
|
128
|
-
// 3. Consumer
|
|
132
|
+
// 3. Consumer - pass the same registry, it auto-deserializes
|
|
129
133
|
const consumer = KafkaConsumerHelper.newInstance({
|
|
130
134
|
bootstrapBrokers: ['127.0.0.1:29092'],
|
|
131
135
|
clientId: 'order-consumer',
|
|
@@ -182,6 +186,7 @@ const consumer = KafkaConsumerHelper.newInstance({
|
|
|
182
186
|
- [Kafka Overview](./) - the four helpers, shared health/close API, and the compile-binary caveat
|
|
183
187
|
- [Producer](./producer) - `registry` as a producer option, plus the manual `serializers` alternative
|
|
184
188
|
- [Consumer](./consumer) - `registry` as a consumer option, plus the manual `deserializers` alternative
|
|
189
|
+
- [Examples & Troubleshooting](./examples) - IoC wiring and the common connection-error lookup table
|
|
185
190
|
|
|
186
191
|
**Files:**
|
|
187
192
|
|
|
@@ -6,16 +6,22 @@ difficulty: advanced
|
|
|
6
6
|
|
|
7
7
|
# HfLogger - High-Frequency Logging Guide
|
|
8
8
|
|
|
9
|
-
`HfLogger` is a fixed-layout ring-buffer logger for hot paths where even the standard `Logger` is too expensive. A log call writes a 256-byte binary entry into a lazily-allocated buffer
|
|
9
|
+
`HfLogger` is a fixed-layout ring-buffer logger for hot paths where even the standard `Logger` is too expensive. A log call writes a 256-byte binary entry into a lazily-allocated buffer. Nothing runs on the fast path: no string formatting, no transport I/O, no per-call allocation. A separate `HfLogFlusher` drains entries later, off the hot path.
|
|
10
10
|
|
|
11
|
-
It implements `ILogger` (`AbstractLogger`), so it
|
|
11
|
+
It implements `ILogger` (`AbstractLogger`), so it's a drop-in replacement anywhere an `ILogger` is expected. Every level method works - `debug`, `info`, `warn`, `error`, `emerg`, plus `log` and `for`. But it stays entirely separate from the Winston-backed `Logger` pipeline: no formatters, no transports, no `APP_ENV_LOGGER_*` variables apply to it.
|
|
12
|
+
|
|
13
|
+
That separation buys enqueue speed - roughly 14x faster than pino on the same machine. See [Performance characteristics](#performance-characteristics) below for the measured numbers.
|
|
12
14
|
|
|
13
15
|
> [!IMPORTANT]
|
|
14
16
|
> Reach for `HfLogger` only when a profiler shows logging itself in your hot path - order engines, market-data ticks, per-packet paths at 100k+ events/sec. For everything else, the standard scoped `Logger` is the right tool: it formats, redacts, rotates files, and ships UDP. Most services never need this module.
|
|
15
17
|
|
|
16
18
|
## Mental model
|
|
17
19
|
|
|
18
|
-
One ring buffer per process holds 65,536 entries of 256 bytes each
|
|
20
|
+
One ring buffer per process holds 65,536 entries of 256 bytes each - 16MB total. It allocates lazily, on the first `HfLogger.get()` call, never at module import.
|
|
21
|
+
|
|
22
|
+
Writers stamp entries in. The ring never blocks and never grows.
|
|
23
|
+
|
|
24
|
+
When the ring is full, the next write overwrites the oldest entry. The buffer trades completeness for a bounded, allocation-free hot path. But every overwrite is counted and reported, never silent - see [Lap accounting](#lap-accounting) below.
|
|
19
25
|
|
|
20
26
|
**Entry layout (256 bytes):**
|
|
21
27
|
|
|
@@ -28,7 +34,9 @@ One ring buffer per process holds 65,536 entries of 256 bytes each (16MB), alloc
|
|
|
28
34
|
| 42 | 1 byte | Message length (0-213) |
|
|
29
35
|
| 43-255 | 213 bytes | Message bytes |
|
|
30
36
|
|
|
31
|
-
The two length bytes are what make reads exact
|
|
37
|
+
The two length bytes are what make reads exact. The flusher decodes only the bytes a field actually holds, never the fixed-width remainder. So there's no NUL padding, no stale tail leaking from whatever a reused slot last held.
|
|
38
|
+
|
|
39
|
+
Because entries are fixed-width binary, everything you log must fit the layout above. Anything longer than the scope or message cap is truncated, not rejected.
|
|
32
40
|
|
|
33
41
|
## Setup - everything happens at initialization time
|
|
34
42
|
|
|
@@ -60,7 +68,7 @@ orderLogger.log('error', MSG_ORDER_REJECTED);
|
|
|
60
68
|
```
|
|
61
69
|
|
|
62
70
|
```typescript
|
|
63
|
-
// -- Or drain manually (
|
|
71
|
+
// -- Or drain manually (for example, at a batch boundary or before shutdown) --
|
|
64
72
|
await flusher.flush();
|
|
65
73
|
|
|
66
74
|
// -- And stop the interval when the logger is no longer needed --
|
|
@@ -69,7 +77,7 @@ flusher.stop();
|
|
|
69
77
|
|
|
70
78
|
## The ILogger surface - and its cost model
|
|
71
79
|
|
|
72
|
-
`HfLogger` implements the same `ILogger` methods as the standard `Logger`, so it can be typed and passed around as `ILogger`. But each method sits on a different point of the cost curve
|
|
80
|
+
`HfLogger` implements the same `ILogger` methods as the standard `Logger`, so it can be typed and passed around as `ILogger`. But each method sits on a different point of the cost curve. Picking the right one is the whole point of this module:
|
|
73
81
|
|
|
74
82
|
```typescript
|
|
75
83
|
import { HfLogger } from '@venizia/ignis-helpers';
|
|
@@ -98,19 +106,35 @@ const MSG_ORDER_SENT = HfLogger.encodeMessage('Order sent');
|
|
|
98
106
|
logger.log('info', MSG_ORDER_SENT);
|
|
99
107
|
```
|
|
100
108
|
|
|
101
|
-
Rule of thumb:
|
|
109
|
+
Rule of thumb:
|
|
110
|
+
|
|
111
|
+
| When | Use | Cost |
|
|
112
|
+
|---|---|---|
|
|
113
|
+
| A per-event, per-tick, per-packet path | Pre-encode, then call the bytes overload | ~59ns/op |
|
|
114
|
+
| A small fixed set of facts you didn't pre-encode | The no-args string call | ~66ns/op |
|
|
115
|
+
| Off the true hot path only | The args form (dynamic `%s` data) | Correct - nothing is ever silently dropped - but not free |
|
|
102
116
|
|
|
103
117
|
## The rules that keep it fast and correct
|
|
104
118
|
|
|
105
|
-
**1. Never encode in the hot path.** `HfLogger.encodeMessage`
|
|
119
|
+
**1. Never encode in the hot path.** `HfLogger.encodeMessage` remembers every distinct string it sees. So does the no-args string call, since it shares the same cache. That cache is FIFO-bounded at 4096 entries.
|
|
120
|
+
|
|
121
|
+
Calling it with dynamic strings (`encodeMessage('order ' + id)`) still puts UTF-8 encoding on your hot path. Worse, it evicts the oldest cached message once you cross the cap - corrupting the fixed vocabulary you rely on elsewhere.
|
|
122
|
+
|
|
123
|
+
If a value varies per event, it doesn't belong in an `HfLogger` message. Log the static fact here, and the variable detail through the standard `Logger` at a lower frequency. Or use the args form, off the hot path.
|
|
124
|
+
|
|
125
|
+
**2. A fixed vocabulary of messages.** The bytes path targets a finite set of pre-encoded facts: "Order sent", "Tick received", "Risk check failed".
|
|
126
|
+
|
|
127
|
+
If you find yourself needing free-form text on the hot path, you're in the wrong module. Use the args form instead, off the hot path.
|
|
128
|
+
|
|
129
|
+
**3. Size the flush interval against your write rate.** The ring holds 65,536 entries. Write more than that between two flushes, and the oldest unflushed entries get overwritten. The flusher reports exactly how many via `dropped` on the sink batch - never silently (see below).
|
|
106
130
|
|
|
107
|
-
|
|
131
|
+
Pick the interval so `writeRate x interval < 65,536`, with comfortable margin. At 100k logs/sec, a 100ms interval accumulates ~10k entries per drain - safe. At 1M logs/sec, you need ~30ms or faster.
|
|
108
132
|
|
|
109
|
-
**
|
|
133
|
+
**4. One process, one thread.** `HfLogger` is safe only on a single thread within a single process. The write index is a plain counter - not shared, not atomic.
|
|
110
134
|
|
|
111
|
-
|
|
135
|
+
Don't log to it from worker threads. Each worker that imports the module gets its own independent ring, and nothing coordinates them. This is a documented design point, not an accident.
|
|
112
136
|
|
|
113
|
-
**5. Flush before shutdown.** Entries live only in memory. An exiting process loses everything not yet flushed
|
|
137
|
+
**5. Flush before shutdown.** Entries live only in memory. An exiting process loses everything not yet flushed. Call `await flusher.flush()` in your shutdown path, and `flusher.stop()` to clear the interval.
|
|
114
138
|
|
|
115
139
|
## The flusher
|
|
116
140
|
|
|
@@ -136,7 +160,7 @@ const customFlusher = new HfLogFlusher({
|
|
|
136
160
|
});
|
|
137
161
|
|
|
138
162
|
flusher.start(100); // interval-based draining, unref'd so it never blocks process exit
|
|
139
|
-
await flusher.flush(); // one-shot drain,
|
|
163
|
+
await flusher.flush(); // one-shot drain, for example before shutdown
|
|
140
164
|
flusher.stop(); // clears the interval; start() again to resume
|
|
141
165
|
```
|
|
142
166
|
|
|
@@ -150,24 +174,27 @@ A rendered line looks like this:
|
|
|
150
174
|
|
|
151
175
|
### Lap accounting
|
|
152
176
|
|
|
153
|
-
Every batch the flusher hands to its sink carries `dropped: number
|
|
177
|
+
Every batch the flusher hands to its sink carries `dropped: number`. That's the count of entries the ring overwrote before the flusher could read them, since the previous batch. The default sink emits a `warn` marker line ahead of the batch when `dropped > 0`:
|
|
154
178
|
|
|
155
179
|
```
|
|
156
180
|
2026-07-18T09:41:03.200Z [warn] HfLogFlusher ring lapped - 342 entries overwritten before they could be read
|
|
157
181
|
```
|
|
158
182
|
|
|
159
|
-
A custom `sink` gets the same `dropped` count on `batch.dropped` and decides how to surface it. This replaces the old silent behavior where a lapped ring
|
|
183
|
+
A custom `sink` gets the same `dropped` count on `batch.dropped` and decides how to surface it. This replaces the old silent behavior, where a lapped ring emitted whatever currently sat in each slot with no warning.
|
|
160
184
|
|
|
161
185
|
## Current limitations
|
|
162
186
|
|
|
163
187
|
These are real behaviors of the current implementation - design around them:
|
|
164
188
|
|
|
165
|
-
- **Single-thread only.** The write index is not shared or atomic across threads
|
|
166
|
-
- **The ring overwrites the oldest entry when lapped.** If the producer writes faster than the flusher drains (rule 3 above), unflushed entries
|
|
167
|
-
- **
|
|
168
|
-
- **
|
|
169
|
-
- **
|
|
170
|
-
- **
|
|
189
|
+
- **Single-thread only.** The write index is not shared or atomic across threads. Each worker thread that imports the module gets an independent ring. Don't log to `HfLogger` from worker threads expecting a shared buffer.
|
|
190
|
+
- **The ring overwrites the oldest entry when lapped.** If the producer writes faster than the flusher drains (see rule 3 above), unflushed entries get silently overwritten in memory.
|
|
191
|
+
- **The loss is visible, not invisible.** The flusher counts and reports every overwritten entry via `dropped`.
|
|
192
|
+
- **213-byte message cap, 32-byte scope cap.** Both are truncation-only - a longer value is cut, not rejected.
|
|
193
|
+
- **Truncation happens at a byte boundary, not a character boundary.** It can split a multibyte UTF-8 character - the truncated tail then renders as the U+FFFD replacement character.
|
|
194
|
+
- **Run one flusher per process.** Each `HfLogFlusher` tracks its own read position from the start of the ring - not a shared cursor. A second flusher re-emits entries the first one already drained.
|
|
195
|
+
- **A fixed, pre-encoded vocabulary is still the right pattern for the bytes path.** `HfLogger.encodeMessage` and the no-args string call exist to make the ENCODE cost a one-time expense.
|
|
196
|
+
- **The args form is correct, but it's the slow path by design.** Reserve it for control-flow events, off the hot path.
|
|
197
|
+
- **The encode cache is FIFO-bounded at 4096 entries.** That's well within a real fixed vocabulary. But a hot path that generates many distinct dynamic strings, through the no-args string call, will start evicting and re-encoding.
|
|
171
198
|
|
|
172
199
|
## Performance characteristics
|
|
173
200
|
|
|
@@ -179,7 +206,7 @@ Measured on Bun 1.3.14, 1M-iteration median:
|
|
|
179
206
|
| String, no args (`info('Order sent')`) | 66.0ns/op | Cache-hit encode lookup + bytes-path write |
|
|
180
207
|
| pino, sync, `/dev/null` | 831ns/op | ~14x slower than the `HfLogger` bytes path on the same machine |
|
|
181
208
|
|
|
182
|
-
Heap growth measured at 0.0MB over 1M bytes-path logs - the hot path does not allocate. That buys an ENQUEUE, not a durable log line
|
|
209
|
+
Heap growth measured at 0.0MB over 1M bytes-path logs - the hot path does not allocate. That buys an ENQUEUE, not a durable log line. The flusher still pays rendering cost later, off the hot path.
|
|
183
210
|
|
|
184
211
|
## See also
|
|
185
212
|
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Logger
|
|
3
|
-
description: Scoped, cached Winston
|
|
3
|
+
description: Scoped, cached logging via LoggerFactory - Winston by default, with console, daily-rotating file, and UDP transports built in
|
|
4
4
|
difficulty: beginner
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
# Logger
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
IGNIS gives every helper a scoped `ILogger`. `LoggerFactory` builds it from one registered provider - Winston by default, with console, daily-rotating file, and UDP transports built in.
|
|
10
10
|
|
|
11
11
|
## In one example
|
|
12
12
|
|
|
@@ -24,16 +24,17 @@ logger.info('User created');
|
|
|
24
24
|
|
|
25
25
|
## How it works
|
|
26
26
|
|
|
27
|
-
- **Typed against `ILogger`.** Every consumer
|
|
28
|
-
- **Provider-based.** `LoggerFactory.use({ provider })` selects the app's logger engine once at the entrypoint
|
|
29
|
-
- **Scoped and cached.** `LoggerFactory.getLogger(scopes)` joins the scopes with `-` and caches per scope
|
|
27
|
+
- **Typed against `ILogger`.** Every consumer - including `BaseHelper.logger` - gets the `ILogger` interface, never a concrete class. Winston is the default provider behind it, selected in `factory.ts`.
|
|
28
|
+
- **Provider-based.** `LoggerFactory.use({ provider })` selects the app's logger engine once, at the entrypoint. Winston is the default; [pino](/extensions/helpers/logger/pino) is the throughput option. Every factory-issued logger follows the registration, even one captured at import time.
|
|
29
|
+
- **Scoped and cached.** `LoggerFactory.getLogger(scopes)` joins the scopes with `-` and caches the result per scope. The same scope always returns the same instance. `BaseHelper` calls this in its constructor, so every helper's `this.logger` comes pre-scoped.
|
|
30
|
+
- **Custom-backed loggers are the exception.** `Logger.get(scope, customWinstonLogger)` (from the `/winston` sub-path) is NOT cached. Each call returns a fresh wrapper over the instance you passed in.
|
|
30
31
|
- **Method scoping.** `.for(methodName)` returns a child logger scoped to `<scope>-<methodName>` (also cached), so each line shows where it came from.
|
|
31
|
-
- **Level floor.** `APP_ENV_LOGGER_LEVEL` (default `debug`) sets the logger-level floor
|
|
32
|
-
- **`debug()` is gated.** It emits only when `DEBUG=true` and `NODE_ENV` is unset or in `Environment.COMMON_ENVS
|
|
32
|
+
- **Level floor.** `APP_ENV_LOGGER_LEVEL` (default `debug`) sets the logger-level floor. Transports without their own level inherit it.
|
|
33
|
+
- **`debug()` is gated.** It emits only when `DEBUG=true` and `NODE_ENV` is unset or listed in `Environment.COMMON_ENVS`. Extend that set via `APP_ENV_EXTRA_LOG_ENVS`. The check runs once at module load - runtime env changes need a restart.
|
|
33
34
|
|
|
34
35
|
**Log levels**
|
|
35
36
|
|
|
36
|
-
Five levels, each with a direct method: `debug`, `info`, `warn`, `error`, `emerg`. The generic `.log(level, ...)` remains for picking the level dynamically. What each level
|
|
37
|
+
Five levels, each with a direct method: `debug`, `info`, `warn`, `error`, `emerg`. The generic `.log(level, ...)` remains for picking the level dynamically. What each level means, and when to use it, is in the [level guide](/extensions/helpers/logger/reference#what-each-level-means).
|
|
37
38
|
|
|
38
39
|
**Transports**
|
|
39
40
|
|
|
@@ -43,7 +44,9 @@ Five levels, each with a direct method: `debug`, `info`, `warn`, `error`, `emerg
|
|
|
43
44
|
| Daily-rotating file | `APP_ENV_LOGGER_FOLDER_PATH` is set |
|
|
44
45
|
| UDP (`DgramTransport`) | All four UDP `APP_ENV_LOGGER_DGRAM_*` variables are set |
|
|
45
46
|
|
|
46
|
-
Output shape (plain text or JSON) follows `APP_ENV_LOGGER_FORMAT`. Color codes appear only on the console
|
|
47
|
+
Output shape (plain text or JSON) follows `APP_ENV_LOGGER_FORMAT`. Color codes appear only on the console, and only in a development `NODE_ENV` - file and UDP output never carries ANSI escapes. See [Color](./reference#color) to override.
|
|
48
|
+
|
|
49
|
+
For extreme hot paths, `HfLogger` is a separate ring-buffer logger outside this pipeline, with its own [usage guide](/extensions/helpers/logger/hf-logger). The [Full reference](/extensions/helpers/logger/reference) covers everything else, including the `ApplicationLogger` facade.
|
|
47
50
|
|
|
48
51
|
## Common tasks
|
|
49
52
|
|
|
@@ -75,12 +78,28 @@ class UserService {
|
|
|
75
78
|
|
|
76
79
|
### Log an Error with `%s`, never `%j`
|
|
77
80
|
|
|
78
|
-
`
|
|
81
|
+
`%s` routes the error through `ErrorPrettier`, which projects it down to identity, cause and frames. `%j` keeps every enumerable own property instead, so a driver error takes its whole query along and a `jose` error its whole payload. Always pair an `Error` argument with `%s`.
|
|
79
82
|
|
|
80
83
|
```typescript
|
|
81
84
|
logger.error('Failed to create user: %s', error); // prints message + stack
|
|
82
85
|
```
|
|
83
86
|
|
|
87
|
+
A mistaken `%j` is no longer a silent loss: the formatter projects `message` and `stack` in before `JSON.stringify` runs, which on its own would drop both (they are non-enumerable). It is still the wrong placeholder for an error.
|
|
88
|
+
|
|
89
|
+
### Keep a driver error readable with `ErrorPrettier`
|
|
90
|
+
|
|
91
|
+
`%s` prints the whole object. A `pg` or `drizzle` failure carries the statement in `message`, again in `stack`, and again in `query` - one failure floods the log with the same SQL several times. Wrap it:
|
|
92
|
+
|
|
93
|
+
```typescript
|
|
94
|
+
import { ErrorPrettier } from '@venizia/ignis-helpers';
|
|
95
|
+
|
|
96
|
+
logger.error('Failed to create user | %s', ErrorPrettier.format({ error }));
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
You get the identity, the root `cause` with its code, the driver's `hint`, the full message and the top stack frames - each on its own line, with the message's real newlines intact. The duplicated statement and the noisy driver internals are gone.
|
|
100
|
+
|
|
101
|
+
Pass `includeStack: false` when the error is one you raised yourself and the frames add nothing. For a JSON sink, `ErrorPrettier.summarize({ error })` returns the same projection as a typed object instead of a string.
|
|
102
|
+
|
|
84
103
|
### Switch the output format
|
|
85
104
|
|
|
86
105
|
`APP_ENV_LOGGER_FORMAT` controls plain text (default) vs. JSON output.
|
|
@@ -94,14 +113,20 @@ The `[APP]` label comes from `APP_ENV_APPLICATION_NAME` (defaults to `'APP'`).
|
|
|
94
113
|
|
|
95
114
|
### Enable daily file rotation
|
|
96
115
|
|
|
97
|
-
Point `APP_ENV_LOGGER_FOLDER_PATH` at a directory
|
|
116
|
+
Point `APP_ENV_LOGGER_FOLDER_PATH` at a directory. Rotation frequency, size cap, and retention are also env-driven. Without this variable, no log files are written - console (and UDP, if configured) remain the only outputs.
|
|
98
117
|
|
|
99
118
|
```bash
|
|
100
119
|
APP_ENV_LOGGER_FOLDER_PATH=./app_data/logs
|
|
101
120
|
APP_ENV_LOGGER_FILE_MAX_FILES=30d
|
|
102
121
|
```
|
|
103
122
|
|
|
104
|
-
|
|
123
|
+
| Setting | Default |
|
|
124
|
+
|---|---|
|
|
125
|
+
| Rotation frequency | `1h` |
|
|
126
|
+
| Max file size | `100m` |
|
|
127
|
+
| Retention | `5d` |
|
|
128
|
+
|
|
129
|
+
Full programmatic configuration - custom prefixes, custom retention - is in the [Full reference](/extensions/helpers/logger/reference).
|
|
105
130
|
|
|
106
131
|
### Forward logs over UDP
|
|
107
132
|
|