@eqxjs/kafka-server-confluent-kafka 0.0.1-alpha.0

package/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+## v2.2.0
+
+- **Feature:** Topic monitor polling interval configurable via `KAFKA_TOPIC_MONITOR_INTERVAL_MS` env var (default: `300000` ms / 5 minutes)
+- **Feature:** `KAFKA_DISABLE_TOPIC_MONITOR=true` disables the topic monitor entirely; a warning is logged at startup when set
+- **Refactor:** `monitorTopicNewChange` rewritten from a `while(true)` async loop to `setInterval`, stored in `topicMonitorInterval` (public property, same pattern as `consumeInterval`)
+- **Fix:** `monitorTopicNewChange` now clears any existing interval before starting a new one, preventing duplicate monitors on reconnect
+- **Fix:** `close()` now clears `topicMonitorInterval` before disconnecting, ensuring the monitor loop is stopped cleanly on shutdown
+
+## v2.1.0
+
+- **Feature:** Heap-based consumer back-pressure — consumer automatically pauses and resumes based on Node.js heap usage. Configure threshold via 4th constructor argument or `KAFKA_HEAP_LIMIT_PERCENT` env var (default `85%`, clamped `10–99`)
+- **Feature:** Heap calculation extracted to `utils/get-mem.ts` as `getHeapUsage()` — returns `usedPercent`, `usedMB`, `limitMB`, `isOverLimit(threshold)`, and `format()` helpers
+- **Feature:** Consumer `ready` event now logs `client name`, `group.id`, `assignment.strategy`, `bootstrap broker`, and full `broker list` sourced from the server response metadata
+- **Feature:** Rebalance callback now logs current partition assignment state on both ASSIGN and REVOKE events, showing `before` / `assigned` / `revoking` / `remaining` maps per topic
+- **Fix:** Cooperative rebalance now correctly maintains `memberAssignment` and `assignment` incrementally (merge on ASSIGN, filter on REVOKE) instead of replacing the full state with the delta
+- **Fix:** `group.id` is always set to a stable value (`groupId` config or `"nestjs-kafka-consumer"`), preventing the "different consumer group" behaviour caused by librdkafka generating a random group ID when none is provided
+- **Fix:** `group.instance.id` automatically set to `os.hostname()` to enable static membership and reduce unnecessary rebalances on pod/container restarts
+
+## v2.0.2
+
+- **Fix:** `rb_callback` now supports both **EAGER** and **COOPERATIVE** rebalance protocols. When `partition.assignment.strategy` is set to `cooperative-sticky` (or any cooperative variant), the callback automatically detects the protocol on the first rebalance event and switches to `incremental_assign(assignment)` / `incremental_unassign(assignment)` for all subsequent rebalances. The protocol is auto-detected at runtime — no configuration change required.
+
+## v2.0.1
+
+- **Fix:** `consumer.assign()` / `consumer.unassign()` now guarded with `isConnected()` and wrapped in try/catch inside `rb_callback` to prevent "Local: Erroneous state" (`ERR__STATE`) crash during partition rebalance when the consumer is in a transitional (disconnecting / reconnecting) state.

package/FEATURES.md

@@ -0,0 +1,278 @@
+# Features
+
+## Stable Consumer Group Identity
+
+`group.id` is always written to `consumerConfig`, defaulting to `"nestjs-kafka-consumer"` when not provided. This ensures offsets are persisted across restarts and the consumer always resumes from where it left off.
+
+Additionally, `group.instance.id` is automatically set to `os.hostname()`, enabling **static membership**. This prevents unnecessary rebalances when a container restarts — Kafka recognises the returning member by its static ID instead of treating it as a new joiner.
+
+**Relevant config:**
+```typescript
+consumer: { groupId: 'my-service-consumer' }
+// or
+consumerConfig['group.id'] = 'my-service-consumer';
+```
+
+---
+
+## Consumer Connected Log
+
+On the `ready` event, the consumer logs a one-line summary built from the **server response metadata** — not just local config:
+
+```
+Kafka consumer connected — client=rdkafka#consumer-1, group.id=my-group, assignment.strategy=range, bootstrap.broker=kafka-1:9092, brokers=[kafka-1:9092, kafka-2:9092, kafka-3:9092]
+```
+
+| Field | Source |
+|---|---|
+| `client` | `ReadyInfo.name` — librdkafka-assigned client identity |
+| `group.id` | Configured `consumerConfig["group.id"]` |
+| `assignment.strategy` | Configured `consumerConfig["partition.assignment.strategy"]` |
+| `bootstrap.broker` | `Metadata.orig_broker_name` — first broker the client connected to |
+| `brokers` | `Metadata.brokers[]` — full broker list advertised by the cluster |
+
+---
+
+## Rebalance Logging with Partition State
+
+Every rebalance event logs the consumer's full partition state before and after the event, per topic.
+
+**ASSIGN** — logs what is being assigned and the resulting full assignment:
+```
+Rebalance ASSIGN [Assign] — group=my-group, strategy=range | assigned={"orders":[0,1]}
+Rebalance ASSIGN — current assignment: {"orders":[0,1],"events":[0]}
+```
+
+**REVOKE** — logs the current assignment, what is being revoked, and what remains:
+```
+Rebalance REVOKE [Revoke] — group=my-group, strategy=range | before={"orders":[0,1],"events":[0]} | revoking={"events":[0]}
+Rebalance REVOKE — remaining assignment: {"orders":[0,1]}
+```
+
+The current assignment is also always readable at runtime:
+```typescript
+const server = CustomServerConfluentKafka.getInstance();
+console.log(server.memberAssignment); // { 'orders': [0, 1], 'events': [0] }
+console.log(server.assignment);       // TopicPartition[]
+```
+
+---
+
+## EAGER and COOPERATIVE Rebalance Protocol Support
+
+The rebalance callback supports both protocols, auto-detected from `partition.assignment.strategy` at construction time.
+
+| Protocol | Strategies | Behaviour |
+|---|---|---|
+| **Eager** | `range`, `roundrobin` | Full revoke then full reassign — `assign()` / `unassign()` |
+| **Cooperative** | `cooperative-sticky` | Incremental delta only — `incrementalAssign()` / `incrementalUnassign()` |
+
+For **cooperative**, `memberAssignment` and `assignment` are maintained incrementally:
+- **ASSIGN**: incoming partitions are merged into the existing assignment
+- **REVOKE**: only the revoked partitions are removed; the rest remain untouched
+
+All assign/unassign calls are guarded with `isConnected()` and wrapped in try/catch to prevent `ERR__STATE` crashes during shutdown or reconnection.
+
+**Configuration:**
+```typescript
+consumerConfig['partition.assignment.strategy'] = 'cooperative-sticky';
+```
+
+---
+
+## Heap-based Consumer Back-pressure
+
+Every second, before calling `consumer.consume()`, the server checks Node.js heap usage via `v8.getHeapStatistics()`. If `used_heap_size / heap_size_limit` exceeds the configured threshold, the consume tick is **skipped** — no messages are fetched until heap recovers.
+
+The consumer logs `PAUSED` once on the first over-limit tick, then `RESUMED` when it drops back below the threshold, avoiding log spam on sustained pressure.
+
+```
+Consumer PAUSED — heap usage 87.3% (used=694.1 MB, limit=796.0 MB) exceeds limit 85%
+Consumer RESUMED — heap usage recovered to 81.2% (used=646.4 MB, limit=796.0 MB)
+```
+
+> The pause does **not** unsubscribe or trigger a rebalance. The consumer stays in the group and simply stops fetching until memory recovers.
+
+**Configuration:**
+```typescript
+// 4th constructor argument (default: 85, clamped to [10, 99])
+new CustomServerConfluentKafka(options, consumerConfig, producerConfig, 80);
+
+// Or via environment variable
+KAFKA_HEAP_LIMIT_PERCENT=80
+```
+
+**Heap utility (`utils/get-mem`):**
+```typescript
+import { getHeapUsage } from '@eqxjs/kafka-server-confluent-kafka/utils/get-mem';
+
+const heap = getHeapUsage();
+heap.usedPercent       // 87.3
+heap.usedMB            // 694.1
+heap.limitMB           // 796.0
+heap.isOverLimit(85)   // true
+heap.format()          // "87.3% (used=694.1 MB, limit=796.0 MB)"
+```
+
+---
+
+## librdkafka Config via Environment Variables
+
+All `ConsumerGlobalConfig` and `ProducerGlobalConfig` keys can be provided via environment variables without writing any code.
+
+**Convention:** `KAFKA_CONSUMER_{KEY}` → consumer config, `KAFKA_PRODUCER_{KEY}` → producer config, where `KEY` is the librdkafka config key name with `.` replaced by `_` and uppercased.
+
+```
+KAFKA_CONSUMER_GROUP_ID            → group.id
+KAFKA_CONSUMER_SESSION_TIMEOUT_MS  → session.timeout.ms
+KAFKA_CONSUMER_BOOTSTRAP_SERVERS   → bootstrap.servers
+KAFKA_PRODUCER_COMPRESSION_TYPE    → compression.type
+KAFKA_PRODUCER_LINGER_MS           → linger.ms
+```
+
+Env vars are merged before explicit constructor arguments, so code-level config always takes precedence. Values are automatically coerced — `"true"`/`"false"` → `boolean`, numeric strings → `number`, everything else stays a `string`.
+
+For a full list of all `ConsumerGlobalConfig` and `ProducerGlobalConfig` keys, defaults, and descriptions see **[KAFKA-CONFIG.md](KAFKA-CONFIG.md)**.
+
+```bash
+# Boot a consumer with no code-level config at all
+KAFKA_CONSUMER_BOOTSTRAP_SERVERS=kafka:9092
+KAFKA_CONSUMER_GROUP_ID=my-service
+KAFKA_CONSUMER_SESSION_TIMEOUT_MS=30000
+KAFKA_PRODUCER_COMPRESSION_TYPE=snappy
+```
+
+---
+
+## Automatic Topic Monitoring
+
+After the consumer connects, a `setInterval` loop (stored in the public `topicMonitorInterval` property) checks for topic changes at a configurable interval:
+
+- **New topics**: if a topic matching a registered `@EventPattern` handler appears on the broker, the consumer unsubscribes and re-subscribes to include it
+- **Deleted topics**: if a subscribed topic disappears from broker metadata, it is removed from the subscription and logged as an error
+
+This means handlers registered at startup will automatically begin consuming newly created topics without a restart. The monitor logs its interval at startup:
+```
+Topic monitor started — interval=300000ms
+```
+
+Before starting, any existing monitor interval is cleared to prevent duplicate monitors on reconnect. `close()` also clears the interval as part of graceful shutdown.
+
+**Disable via environment variable:**
+```bash
+KAFKA_DISABLE_TOPIC_MONITOR=true
+```
+When disabled, a warning is logged at startup and the consumer subscribes only to topics available at connect time.
+
+**Configure polling interval:**
+```bash
+KAFKA_TOPIC_MONITOR_INTERVAL_MS=60000   # default: 300000 (5 minutes)
+```
+
+---
+
+## Consumer Throughput Control
+
+Two environment variables jointly control how fast messages are consumed:
+
+**Polling interval** — how often the consume tick fires:
+```bash
+KAFKA_CONSUME_INTERVAL_MS=500   # default: 1000 (1 second)
+```
+
+**Batch size** — how many messages are fetched per tick:
+```bash
+KAFKA_CONSUME_MESSAGES_PER_INTERVAL=50   # default: 10
+```
+
+Combined with heap back-pressure, these provide three independent levers for rate control.
+
+---
+
+## Dual Producer Support
+
+Every instance initialises two producers:
+
+| Producer | Class | Use case |
+|---|---|---|
+| `highProducer` | `HighLevelProducer` | `produce()` — async with per-message callback, returns committed offset |
+| `producer` | `Producer` | `sendMessage()` — fire-and-forget, lower overhead |
+
+Both producers emit delivery reports, which feed into optional success/error callbacks:
+
+```typescript
+kafkaServer.setSuccessCallback((err, report) => { /* ... */ });
+kafkaServer.setErrorCallback((err, report) => { /* ... */ });
+```
+
+---
+
+## Authentication Support
+
+### SASL (PLAIN / SCRAM)
+
+```typescript
+sasl: {
+  mechanism: 'plain',           // or 'scram-sha-256', 'scram-sha-512'
+  username: 'my-username',
+  password: 'my-password',
+}
+```
+
+Maps to librdkafka: `sasl.username`, `sasl.password`, `sasl.mechanisms`, `security.protocol=sasl_plaintext`.
+
+### SSL / TLS
+
+```typescript
+ssl: true
+// or
+ssl: {
+  ca: fs.readFileSync('./ca-cert.pem'),
+  cert: fs.readFileSync('./client-cert.pem'),
+  key: fs.readFileSync('./client-key.pem'),
+}
+```
+
+Maps to: `ssl.ca.pem`, `ssl.certificate.pem`, `ssl.key.pem`, `security.protocol=sasl_ssl`.
+
+---
+
+## Native Config Passthrough
+
+Any librdkafka option not exposed via `KafkaOptions` can be set directly via the `consumerConfig` and `producerConfig` constructor arguments. These are merged with (and take priority over) options derived from the first `KafkaOptions` parameter.
+
+```typescript
+new CustomServerConfluentKafka(
+  kafkaOptions,
+  { 'fetch.wait.max.ms': 100, 'max.poll.interval.ms': 300000 },
+  { 'compression.type': 'snappy', 'linger.ms': 10 },
+);
+```
+
+See the [librdkafka configuration reference](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) for all available options.
+
+---
+
+## Singleton Access
+
+The server registers itself as a static singleton at construction time, allowing access from anywhere in the application without dependency injection:
+
+```typescript
+const server = CustomServerConfluentKafka.getInstance();
+server.produce('my-topic', message);
+server.memberAssignment;
+server.isKafkaConnected();
+```
+
+---
+
+## Direct Client Access
+
+Use `unwrap()` to get the underlying librdkafka consumer and producer instances for operations not covered by this wrapper:
+
+```typescript
+const { consumer, producer } = kafkaServer.unwrap<{
+  consumer: KafkaConsumer;
+  producer: Producer;
+}>();
+```

package/KAFKA-CONFIG.md

@@ -0,0 +1,304 @@
+# Kafka Configuration Reference
+
+Configuration for `ConsumerGlobalConfig` and `ProducerGlobalConfig`, base on **librdkafka 2.13.2**.
+
+Properties are passed directly to the `CustomServerConfluentKafka` constructor or set via environment variables (see [Environment Variables via Env](#environment-variables-via-env)).
+
+---
+
+## Table of Contents
+
+- [Shared (GlobalConfig)](#shared-globalconfig)
+  - [Core](#core)
+  - [Network & Socket](#network--socket)
+  - [SSL / TLS](#ssl--tls)
+  - [SASL Authentication](#sasl-authentication)
+  - [Metadata & Topic](#metadata--topic)
+  - [Retry & Backoff](#retry--backoff)
+  - [Logging & Debug](#logging--debug)
+  - [Misc](#misc)
+- [ConsumerGlobalConfig](#consumerglobalconfig)
+  - [Group & Rebalance](#group--rebalance)
+  - [Offset & Commit](#offset--commit)
+  - [Fetch & Queue](#fetch--queue)
+  - [Consumer Misc](#consumer-misc)
+- [ProducerGlobalConfig](#producerglobalconfig)
+  - [Delivery & Reliability](#delivery--reliability)
+  - [Batching & Throughput](#batching--throughput)
+  - [Producer Misc](#producer-misc)
+- [Environment Variables via Env](#environment-variables-via-env)
+
+---
+
+## Shared (GlobalConfig)
+
+Properties available to **both** consumer and producer.
+
+### Core
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `client.id` | `string` | `rdkafka` | Client identifier |
+| `bootstrap.servers` | `string` | — | Initial broker list as CSV of `host` or `host:port`. Alias: `metadata.broker.list` |
+| `security.protocol` | `'plaintext'\|'ssl'\|'sasl_plaintext'\|'sasl_ssl'` | `plaintext` | Protocol used to communicate with brokers |
+| `client.rack` | `string` | — | Rack identifier for this client (matches broker `broker.rack`) |
+| `client.dns.lookup` | `'use_all_dns_ips'\|'resolve_canonical_bootstrap_servers_only'` | `use_all_dns_ips` | Controls how the client uses DNS lookups |
+| `allow.auto.create.topics` | `boolean` | `false` | Allow automatic topic creation on subscribe/assign when broker also has it enabled |
+| `builtin.features` | `string` | *(build-dependent)* | Read-only: lists features compiled into librdkafka |
+
+### Network & Socket
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `socket.timeout.ms` | `number` | `60000` | Default timeout for network requests |
+| `socket.send.buffer.bytes` | `number` | `0` | Broker socket send buffer size (0 = system default) |
+| `socket.receive.buffer.bytes` | `number` | `0` | Broker socket receive buffer size (0 = system default) |
+| `socket.keepalive.enable` | `boolean` | `false` | Enable TCP keep-alives (`SO_KEEPALIVE`) |
+| `socket.nagle.disable` | `boolean` | `true` | Disable Nagle algorithm (`TCP_NODELAY`) |
+| `socket.max.fails` | `number` | `1` | Disconnect from broker after this many send failures (0 = disable) |
+| `socket.connection.setup.timeout.ms` | `number` | `30000` | Max time for broker connection setup including SSL/SASL handshake |
+| `connections.max.idle.ms` | `number` | `0` | Close idle broker connections after this ms (0 = disable) |
+| `broker.address.ttl` | `number` | `1000` | How long to cache broker address resolution results (ms) |
+| `broker.address.family` | `'any'\|'v4'\|'v6'` | `any` | Allowed broker IP address families |
+| `reconnect.backoff.ms` | `number` | `100` | Initial wait before reconnecting to a broker |
+| `reconnect.backoff.max.ms` | `number` | `10000` | Maximum wait before reconnecting to a broker |
+| `max.in.flight.requests.per.connection` | `number` | `1000000` | Maximum in-flight requests per broker connection. Alias: `max.in.flight` |
+| `message.max.bytes` | `number` | `1000000` | Maximum Kafka protocol request message size |
+| `receive.message.max.bytes` | `number` | `100000000` | Maximum Kafka protocol response message size |
+
+### SSL / TLS
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `ssl.key.location` | `string` | — | Path to client private key (PEM) |
+| `ssl.key.password` | `string` | — | Private key passphrase |
+| `ssl.key.pem` | `string` | — | Client private key string (PEM format) |
+| `ssl.certificate.location` | `string` | — | Path to client public key (PEM) |
+| `ssl.certificate.pem` | `string` | — | Client public key string (PEM format) |
+| `ssl.ca.location` | `string` | — | Path to CA certificate(s) for verifying broker key |
+| `ssl.ca.pem` | `string` | — | CA certificate string (PEM format) for verifying broker key |
+| `ssl.ca.certificate.stores` | `string` | `Root` | Windows Certificate stores to load CA certificates from |
+| `ssl.crl.location` | `string` | — | Path to CRL for verifying broker certificate validity |
+| `ssl.keystore.location` | `string` | — | Path to client keystore (PKCS#12) |
+| `ssl.keystore.password` | `string` | — | Client keystore password |
+| `ssl.cipher.suites` | `string` | — | TLS cipher suite list |
+| `ssl.curves.list` | `string` | — | Supported curves extension for TLS ClientHello |
+| `ssl.sigalgs.list` | `string` | — | Signature/hash algorithm pairs for TLS ClientHello |
+| `ssl.providers` | `string` | — | Comma-separated OpenSSL 3.0.x implementation providers |
+| `ssl.engine.id` | `string` | `dynamic` | OpenSSL engine id |
+| `enable.ssl.certificate.verification` | `boolean` | `true` | Enable OpenSSL's built-in broker certificate verification |
+| `ssl.endpoint.identification.algorithm` | `'none'\|'https'` | `https` | Endpoint identification algorithm to validate broker hostname |
+| `https.ca.location` | `string` | — | Path to CA certificate(s) for verifying HTTPS endpoints (e.g. OAUTHBEARER) |
+| `https.ca.pem` | `string` | — | CA certificate string (PEM) for verifying HTTPS endpoints |
+
+### SASL Authentication
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `sasl.mechanisms` | `string` | `GSSAPI` | SASL mechanism: `GSSAPI`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`. Alias: `sasl.mechanism` |
+| `sasl.username` | `string` | — | SASL username (PLAIN / SCRAM) |
+| `sasl.password` | `string` | — | SASL password (PLAIN / SCRAM) |
+| `sasl.kerberos.service.name` | `string` | `kafka` | Kerberos principal name Kafka runs as |
+| `sasl.kerberos.principal` | `string` | `kafkaclient` | This client's Kerberos principal name |
+| `sasl.kerberos.keytab` | `string` | — | Path to Kerberos keytab file |
+| `sasl.kerberos.kinit.cmd` | `string` | *(kinit command)* | Shell command to refresh/acquire Kerberos ticket |
+| `sasl.kerberos.min.time.before.relogin` | `number` | `60000` | Minimum ms between Kerberos key refresh attempts (0 = disable) |
+| `sasl.oauthbearer.config` | `string` | — | SASL/OAUTHBEARER configuration string |
+| `sasl.oauthbearer.method` | `'default'\|'oidc'` | `default` | Login method for OAUTHBEARER |
+| `sasl.oauthbearer.client.id` | `string` | — | OIDC client ID. Alias: `sasl.oauthbearer.client.credentials.client.id` |
+| `sasl.oauthbearer.client.secret` | `string` | — | OIDC client secret. Alias: `sasl.oauthbearer.client.credentials.client.secret` |
+| `sasl.oauthbearer.scope` | `string` | — | OIDC access scope |
+| `sasl.oauthbearer.extensions` | `string` | — | Additional OIDC key=value pairs |
+| `sasl.oauthbearer.token.endpoint.url` | `string` | — | OIDC token endpoint URL |
+| `sasl.oauthbearer.grant.type` | `'client_credentials'\|'urn:ietf:params:oauth:grant-type:jwt-bearer'` | `client_credentials` | OAuth grant type |
+| `sasl.oauthbearer.assertion.algorithm` | `'RS256'\|'ES256'` | `RS256` | Algorithm for JWT assertion signing |
+| `sasl.oauthbearer.assertion.private.key.file` | `string` | — | Path to private key (PEM) for JWT assertion |
+| `sasl.oauthbearer.assertion.private.key.pem` | `string` | — | Private key (PEM string) for JWT assertion |
+| `sasl.oauthbearer.assertion.private.key.passphrase` | `string` | — | Passphrase for JWT assertion private key |
+| `sasl.oauthbearer.assertion.file` | `string` | — | Path to assertion file |
+| `sasl.oauthbearer.assertion.claim.aud` | `string` | — | JWT audience claim |
+| `sasl.oauthbearer.assertion.claim.iss` | `string` | — | JWT issuer claim |
+| `sasl.oauthbearer.assertion.claim.sub` | `string` | — | JWT subject claim |
+| `sasl.oauthbearer.assertion.claim.exp.seconds` | `number` | `300` | JWT assertion expiration in seconds |
+| `sasl.oauthbearer.assertion.claim.nbf.seconds` | `number` | `60` | JWT assertion not-before time in seconds |
+| `sasl.oauthbearer.assertion.claim.jti.include` | `boolean` | `false` | Include random UUID as JWT ID claim |
+| `sasl.oauthbearer.assertion.jwt.template.file` | `string` | — | Path to JWT template file |
+| `sasl.oauthbearer.metadata.authentication.type` | `'none'\|'azure_imds'` | `none` | Metadata-based authentication type |
+| `enable.sasl.oauthbearer.unsecure.jwt` | `boolean` | `false` | Enable built-in unsecure JWT handler (dev/test only) |
+
+### Metadata & Topic
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `topic.metadata.refresh.interval.ms` | `number` | `300000` | Interval for proactive topic and broker metadata refresh (-1 = disable) |
+| `metadata.max.age.ms` | `number` | `900000` | Metadata cache max age |
+| `topic.metadata.refresh.fast.interval.ms` | `number` | `100` | Fast metadata refresh interval after a leader is lost |
+| `topic.metadata.refresh.sparse` | `boolean` | `true` | Sparse metadata requests (reduces network bandwidth) |
+| `topic.metadata.propagation.max.ms` | `number` | `30000` | Delay before marking a newly created topic as non-existent |
+| `topic.blacklist` | `any` | — | Comma-separated regex list of topics to ignore in broker metadata |
+| `metadata.recovery.strategy` | `'none'\|'rebootstrap'` | `rebootstrap` | Client recovery strategy when no brokers are available |
+| `metadata.recovery.rebootstrap.trigger.ms` | `number` | `300000` | Interval before triggering rebootstrap when metadata is unavailable |
+
+### Retry & Backoff
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `retry.backoff.ms` | `number` | `100` | Initial backoff before retrying a protocol request |
+| `retry.backoff.max.ms` | `number` | `1000` | Maximum backoff for exponentially backed-off requests |
+
+### Logging & Debug
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `debug` | `string` | — | Comma-separated debug contexts. Producer: `broker,topic,msg`. Consumer: `consumer,cgrp,topic,fetch` |
+| `log_level` | `number` | `6` | Logging level (syslog levels 0–7) |
+| `log.queue` | `boolean` | `false` | Enqueue log messages instead of spontaneous log callbacks |
+| `log.thread.name` | `boolean` | `true` | Print internal thread name in log messages |
+| `log.connection.close` | `boolean` | `true` | Log broker disconnects |
+| `statistics.interval.ms` | `number` | `0` | Stats emit interval in ms (0 = disable) |
+
+### Misc
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `enable.random.seed` | `boolean` | `true` | Initialize PRNG with current time on first `rd_kafka_new()` |
+| `enable.metrics.push` | `boolean` | `true` | Enable pushing client metrics to the cluster |
+| `api.version.request` | `boolean` | `true` | Request broker supported API versions |
+| `api.version.request.timeout.ms` | `number` | `10000` | Timeout for broker API version requests |
+| `internal.termination.signal` | `number` | `0` | Signal used to quickly terminate on destroy |
+| `plugin.library.paths` | `string` | — | Semicolon-separated list of plugin libraries to load |
+
+---
+
+## ConsumerGlobalConfig
+
+Properties specific to the **consumer** (extends `GlobalConfig`).
+
+### Group & Rebalance
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `group.id` | `string` | — | **Required.** Consumer group ID. All consumers sharing the same `group.id` form one group |
+| `group.instance.id` | `string` | — | Static member ID for static group membership (avoids rebalance on restart within `session.timeout.ms`) |
+| `partition.assignment.strategy` | `string` | `range,roundrobin` | Comma-separated partition assignment strategies. Options: `range`, `roundrobin`, `cooperative-sticky` |
+| `session.timeout.ms` | `number` | `45000` | Group session timeout. Consumer is removed from group if no heartbeat is received within this interval |
+| `heartbeat.interval.ms` | `number` | `3000` | Frequency of heartbeats to the broker |
+| `max.poll.interval.ms` | `number` | `300000` | Maximum time between `consume()` calls before the consumer is considered failed |
+| `coordinator.query.interval.ms` | `number` | `600000` | How often to query for the current group coordinator |
+| `group.protocol` | `'classic'\|'consumer'` | `classic` | Group protocol to use (`classic` = original, `consumer` = KIP-848) |
+| `group.protocol.type` | `string` | `consumer` | Group protocol type (classic protocol only) |
+| `group.remote.assignor` | `string` | — | Server-side assignor for `group.protocol=consumer` (`uniform` or `range`) |
+
+### Offset & Commit
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `enable.auto.commit` | `boolean` | `true` | Automatically commit offsets in the background |
+| `auto.commit.interval.ms` | `number` | `5000` | Frequency (ms) for automatic offset commits (0 = disable) |
+| `enable.auto.offset.store` | `boolean` | `true` | Automatically store offset of last message delivered to the application |
+| `isolation.level` | `'read_uncommitted'\|'read_committed'` | `read_committed` | Controls visibility of transactional messages |
+
+### Fetch & Queue
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `fetch.min.bytes` | `number` | `1` | Minimum bytes the broker responds with per fetch |
+| `fetch.wait.max.ms` | `number` | `500` | Maximum time the broker waits to fill the fetch response |
+| `fetch.message.max.bytes` | `number` | `1048576` | Initial max bytes per partition per fetch request. Alias: `max.partition.fetch.bytes` |
+| `fetch.max.bytes` | `number` | `52428800` | Maximum bytes returned for a single Fetch request (50 MB default) |
+| `fetch.error.backoff.ms` | `number` | `500` | How long to postpone the next fetch after an error |
+| `fetch.queue.backoff.ms` | `number` | `1000` | How long to postpone the next fetch when queue thresholds are exceeded |
+| `queued.min.messages` | `number` | `100000` | Minimum messages per topic+partition librdkafka tries to maintain in the local queue |
+| `queued.max.messages.kbytes` | `number` | `65536` | Maximum kilobytes of queued pre-fetched messages in the local consumer queue |
+
+### Consumer Misc
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `enable.partition.eof` | `boolean` | `false` | Emit `ERR__PARTITION_EOF` event when consumer reaches end of partition |
+| `check.crcs` | `boolean` | `false` | Verify CRC32 of consumed messages |
+| `offset.store.method` | `'none'\|'file'\|'broker'` | `broker` | Offset commit store method (**`file` is deprecated**) |
+
+---
+
+## ProducerGlobalConfig
+
+Properties specific to the **producer** (extends `GlobalConfig`).
+
+### Delivery & Reliability
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `enable.idempotence` | `boolean` | `false` | Enable exactly-once delivery and original produce order |
+| `transactional.id` | `string` | — | Transactional producer ID. Enables transactional semantics (requires broker >= 0.11) |
+| `transaction.timeout.ms` | `number` | `60000` | Transaction coordinator timeout. Adjusts `message.timeout.ms` and `socket.timeout.ms` automatically |
+| `message.send.max.retries` | `number` | `2147483647` | How many times to retry a failing message. Alias: `retries` |
+| `enable.gapless.guarantee` | `boolean` | `false` | *(Experimental)* Raise fatal error on any gap in produced message sequence |
+| `dr_cb` | `boolean\|Function` | — | Delivery report callback |
+
+### Batching & Throughput
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `queue.buffering.max.messages` | `number` | `100000` | Maximum messages on the producer queue (0 = unlimited) |
+| `queue.buffering.max.kbytes` | `number` | `1048576` | Maximum total message size on the producer queue (1 GB default) |
+| `queue.buffering.max.ms` | `number` | `5` | Delay (ms) to accumulate messages before transmitting (linger). Alias: `linger.ms` |
+| `queue.buffering.backpressure.threshold` | `number` | `1` | Outstanding un-transmitted requests threshold before backpressure kicks in |
+| `batch.num.messages` | `number` | `10000` | Maximum messages per MessageSet batch |
+| `batch.size` | `number` | `1000000` | Maximum bytes per MessageSet batch (1 MB default) |
+| `sticky.partitioning.linger.ms` | `number` | `10` | How long to use the same sticky partition for null-key messages |
+| `compression.codec` | `'none'\|'gzip'\|'snappy'\|'lz4'\|'zstd'` | `none` | Compression codec for messages. Alias: `compression.type` |
+| `delivery.report.only.error` | `boolean` | `false` | Only emit delivery reports for failed messages |
+
+### Producer Misc
+
+| Property | Type | Default | Description |
+|----------|------|---------|-------------|
+| `allow.auto.create.topics` | `boolean` | `true` | Allow automatic topic creation on produce (producer default differs from consumer) |
+
+---
+
+## Environment Variables via Env
+
+Any property above can be configured without code using the `KAFKA_CONSUMER_*` / `KAFKA_PRODUCER_*` environment variable convention supported by this library.
+
+**Conversion rule:** strip prefix → lowercase → replace `_` with `.`
+
+```
+KAFKA_CONSUMER_GROUP_ID                  → group.id
+KAFKA_CONSUMER_SESSION_TIMEOUT_MS        → session.timeout.ms
+KAFKA_CONSUMER_PARTITION_ASSIGNMENT_STRATEGY → partition.assignment.strategy
+KAFKA_CONSUMER_ENABLE_AUTO_COMMIT        → enable.auto.commit
+KAFKA_CONSUMER_ISOLATION_LEVEL          → isolation.level
+KAFKA_PRODUCER_COMPRESSION_TYPE         → compression.type (= compression.codec)
+KAFKA_PRODUCER_LINGER_MS                → linger.ms (= queue.buffering.max.ms)
+KAFKA_PRODUCER_BATCH_SIZE               → batch.size
+KAFKA_PRODUCER_ENABLE_IDEMPOTENCE       → enable.idempotence
+KAFKA_PRODUCER_TRANSACTIONAL_ID         → transactional.id
+```
+
+Values are automatically coerced:
+- `"true"` / `"false"` → `boolean`
+- Numeric strings (e.g. `"30000"`) → `number`
+- All other strings remain as `string`
+
+Code-level config passed to the constructor always takes precedence over env vars.
+
+```bash
+# Example .env
+KAFKA_CONSUMER_BOOTSTRAP_SERVERS=kafka:9092
+KAFKA_CONSUMER_GROUP_ID=my-service
+KAFKA_CONSUMER_SESSION_TIMEOUT_MS=30000
+KAFKA_CONSUMER_MAX_POLL_INTERVAL_MS=300000
+KAFKA_CONSUMER_ENABLE_AUTO_COMMIT=true
+KAFKA_CONSUMER_AUTO_COMMIT_INTERVAL_MS=5000
+KAFKA_CONSUMER_ISOLATION_LEVEL=read_committed
+KAFKA_CONSUMER_FETCH_MAX_BYTES=52428800
+KAFKA_PRODUCER_BOOTSTRAP_SERVERS=kafka:9092
+KAFKA_PRODUCER_COMPRESSION_TYPE=snappy
+KAFKA_PRODUCER_LINGER_MS=10
+KAFKA_PRODUCER_BATCH_SIZE=65536
+KAFKA_PRODUCER_ENABLE_IDEMPOTENCE=false
+```
+
+> For the full librdkafka configuration reference, see the [upstream CONFIGURATION.md](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).