kafka-console 2.0.76 → 2.1.0

package/README.md

@@ -1,181 +1,421 @@
-# Kafka CLI tool
+# Kafka Console CLI
 
-Command line tool to sufficiently and easy work with Kafka
+A powerful and easy-to-use command-line interface for Apache Kafka operations.
 
 [![NPM version][npm-image]][npm-url]
 [![Downloads][downloads-image]][npm-url]
 
 ## Table of Contents
 
-  - [Features](#features)
-  - [Installing](#installing)
-  - [Examples](#examples)
-    - [Consumer](#consumer)
-    - [Producer](#producer)
-    - [Formatters](#formatters)
-  - [Environment](#environment)
-  - [License](#license)
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Commands](#commands)
+  - [Consuming Messages](#consuming-messages)
+  - [Producing Messages](#producing-messages)
+  - [Topic Management](#topic-management)
+  - [Cluster Information](#cluster-information)
+- [Authentication](#authentication)
+- [Message Formats](#message-formats)
+- [Environment Variables](#environment-variables)
+- [Common Use Cases](#common-use-cases)
+- [Troubleshooting](#troubleshooting)
+- [License](#license)
 
 ## Features
 
-- Producer
-- Consumer groups with seek and timeout
-- Built-in message encoders/decoders with types: json, js, raw
-- Custom message encoders/decoders as a js module
-- Message headers
-- GZIP compression
-- Plain, SSL and SASL_SSL implementations
-- Admin client
-- TypeScript support
-
-## Installing
-
-```sh
+- ✅ **Consumer & Producer** - Full support for consuming and producing messages
+- ✅ **Multiple Authentication Methods** - Plain, SCRAM-SHA-256/512, AWS IAM, OAuth Bearer
+- ✅ **Flexible Message Formats** - JSON, JavaScript, raw text, or custom formatters
+- ✅ **Consumer Groups** - Full consumer group support with offset management
+- ✅ **Time-based Consumption** - Read messages from specific timestamps
+- ✅ **SSL/TLS Support** - Secure connections to Kafka clusters
+- ✅ **Topic Management** - Create, delete, and inspect topics
+- ✅ **Headers Support** - Read and write message headers
+- ✅ **GZIP Compression** - Automatic compression support
+- ✅ **TypeScript** - Full TypeScript support
+
+## Installation
+
+### Global Installation (Recommended)
+```bash
 npm install -g kafka-console
 ```
 
-## Examples
+### Local Installation
+```bash
+npm install kafka-console
+```
 
-### Common options
+### Using without Installation
+```bash
+npx kafka-console [command]
 ```
-  -b, --brokers <brokers>                bootstrap server host (default: "localhost:9092")
-  -l, --log-level <logLevel>             log level
-  -t, --timeout <timeout>                set a timeout of operation (default: "0")
-  -p, --pretty                           pretty print (default: false)
-  --ssl                                  enable ssl (default: false)
-  --mechanism <mechanism>                sasl mechanism
-  --username <username>                  sasl username
-  --password <password>                  sasl password
-  --auth-id <authId>                     sasl aws authorization identity
-  --access-key-id <accessKeyId>          sasl aws access key id
-  --secret-access-key <secretAccessKey>  sasl aws secret access key
-  --session-token <seccionToken>         sasl aws session token
-  --oauth-bearer <oauthBearer>           sasl oauth bearer token
-  -V, --version                          output the version number
-  -h, --help                             display help for command
+
+## Quick Start
+
+### 1. List all topics
+```bash
+kafka-console list --brokers localhost:9092
 ```
 
-### Commands
+### 2. Consume messages from a topic
+```bash
+kafka-console consume my-topic --brokers localhost:9092
 ```
-  consume [options] <topic>              Consume kafka topic events
-  produce [options] <topic>              Produce kafka topic events
-  metadata                               Displays kafka server metadata
-  list|ls [options]                      Lists kafka topics
-  config [options]                       Describes config for specific resource
-  topic:create <topic>                   Creates kafka topic
-  topic:delete <topic>                   Deletes kafka topic
-  topic:offsets <topic> [timestamp]      Shows kafka topic offsets
-  help [command]                         display help for command
+
+### 3. Produce a message to a topic
+```bash
+echo '{"message": "Hello Kafka!"}' | kafka-console produce my-topic --brokers localhost:9092
 ```
 
-### Consumer
+## Commands
+
+### Consuming Messages
 
-`npx kafka-console consume [options] <topic>`
+```bash
+kafka-console consume <topic> [options]
+```
 
 #### Options
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-g, --group <group>` | Consumer group name | `kafka-console-consumer-{timestamp}` |
+| `-f, --from <from>` | Start position (timestamp/ISO date/0 for beginning) | latest |
+| `-c, --count <count>` | Number of messages to read | unlimited |
+| `-s, --skip <skip>` | Number of messages to skip | 0 |
+| `-o, --output <file>` | Write output to file | stdout |
+| `-d, --data-format <format>` | Message format (json/js/raw/custom) | json |
+| `-p, --pretty` | Pretty print JSON output | false |
+
+#### Examples
+
+**Consume from beginning and pretty print:**
+```bash
+kafka-console consume my-topic --from 0 --pretty
+```
+
+**Consume last 10 messages:**
+```bash
+kafka-console consume my-topic --count 10
 ```
-  -g, --group <group>              consumer group name (default: "kafka-console-consumer-TIMESTAMP")
-  -d, --data-format <data-format>  messages data-format: json, js, raw (default: "json")
-  -o, --output <filename>          write output to specified filename
-  -f, --from <from>                read messages from the specific timestamp in milliseconds or ISO 8601 format. Set 0 to read from the beginning
-  -c, --count <count>              a number of messages to read (default: null)
-  -s, --skip <skip>                a number of messages to skip (default: 0)
-  -h, --help                       display help for command
+
+**Consume from specific timestamp:**
+```bash
+kafka-console consume my-topic --from "2024-01-01T00:00:00Z"
 ```
 
-General usage with authentication
-```sh
-npx kafka-console --brokers $KAFKA_BROKERS --ssl --mechanism plain --username $KAFKA_USERNAME --password $KAFKA_PASSWORD consume $KAFKA_TOPIC --group $KAFKA_TOPIC_GROUP
+**Consume with specific consumer group:**
+```bash
+kafka-console consume my-topic --group my-consumer-group
 ```
 
-Stdout from timestamp `jq` example
-```sh
-npx kafka-console consume $KAFKA_TOPIC --from 0 | jq .value
+**Save output to file:**
+```bash
+kafka-console consume my-topic --output messages.json
 ```
 
-Custom data formatter example
-```sh
-npx kafka-console consume $KAFKA_TOPIC --data-format ./formatter/avro.js | jq
+**Extract specific fields with jq:**
+```bash
+kafka-console consume my-topic | jq '.value.userId'
 ```
 
-### Producer
+### Producing Messages
 
-```sh
-npx kafka-console produce [options] <topic>
+```bash
+kafka-console produce <topic> [options]
 ```
 
 #### Options
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-i, --input <file>` | Read input from file | stdin |
+| `-d, --data-format <format>` | Message format (json/js/raw/custom) | json |
+| `-h, --header <header>` | Add message header (format: key:value) | none |
+| `-w, --wait <ms>` | Wait time between messages | 0 |
+
+#### Examples
+
+**Produce single message:**
+```bash
+echo '{"user": "john", "action": "login"}' | kafka-console produce my-topic
+```
+
+**Produce from file:**
+```bash
+kafka-console produce my-topic --input messages.json
+```
+
+**Produce with headers:**
+```bash
+echo '{"data": "test"}' | kafka-console produce my-topic --header "source:api" --header "version:1.0"
 ```
-  -d, --data-format <data-format>  messages data-format: json, js, raw (default: "json")
-  -i, --input <filename>           input filename
-  -w, --wait <wait>                wait the time in ms after sending a message (default: 0)
-  -h, --header <header>            set a static header (default: [])
-  --help                           display help for command
+
+**Produce multiple messages from JSON array:**
+```bash
+cat users.json | jq -c '.[]' | kafka-console produce my-topic
 ```
 
-General usage
-```sh
-npx kafka-console produce $KAFKA_TOPIC -b $KAFKA_BROKERS --ssl --mechanism plain --username $KAFKA_USERNAME --password $KAFKA_PASSWORD
+**Produce with key (for partitioning):**
+```bash
+echo '{"key": "user123", "value": {"name": "John"}}' | kafka-console produce my-topic
 ```
 
-Produce a json data from stdin with custom formatter
-```sh
-npx kafka-console payload.txt|kcli produce $KAFKA_TOPIC --data-format ./formatter/avro.js
+### Topic Management
+
+#### Create Topic
+```bash
+kafka-console topic:create my-new-topic
 ```
 
-Produce a json data from stdin
-```sh
-node payloadGenerator.js|npx kafka-console produce $KAFKA_TOPIC
+#### Delete Topic
+```bash
+kafka-console topic:delete old-topic
 ```
 
-Produce a json array data from stdin
-```sh
-cat payload.json|jq -r -c .[]|npx kafka-console produce $KAFKA_TOPIC
+#### Show Topic Offsets
+```bash
+kafka-console topic:offsets my-topic
 ```
 
-Payload single message input interface
-```typescript
-interface Payload {
-  key?: string; // kafka
-  value: any;
-  headers?: { [key: string]: value };
-}
+#### Show Topic Offsets for Specific Timestamp
+```bash
+kafka-console topic:offsets my-topic "2024-01-01T00:00:00Z"
 ```
 
-### Formatters
+### Cluster Information
 
-```typescript
-export interface Encoder<T> {
-  (value: T): Promise<string | Buffer> | string | Buffer;
-}
+#### List All Topics
+```bash
+kafka-console list
+```
 
-export interface Decoder<T> {
-  (value: Buffer): Promise<T> | T;
-}
+#### List Including Internal Topics
+```bash
+kafka-console list --all
+```
 
-export interface Formatter<T> {
-  encode: Encoder<T>;
-  decode: Decoder<T>;
-}
+#### Show Cluster Metadata
+```bash
+kafka-console metadata
 ```
 
-## Supported Environment Variables
+#### Show Topic Configuration
+```bash
+kafka-console config --resource topic --resourceName my-topic
+```
 
- - KAFKA_BROKERS
- - KAFKA_TIMEOUT
- - KAFKA_MECHANISM
- - KAFKA_USERNAME
- - KAFKA_PASSWORD
- - KAFKA_AUTH_ID
- - KAFKA_ACCESS_KEY_ID
- - KAFKA_SECRET_ACCESS_KEY
- - KAFKA_SESSION_TOKEN
- - KAFKA_OAUTH_BEARER
+## Authentication
+
+### SSL/TLS Connection
+```bash
+kafka-console consume my-topic \
+  --brokers broker1:9093,broker2:9093 \
+  --ssl
+```
+
+### SASL/PLAIN
+```bash
+kafka-console consume my-topic \
+  --brokers broker:9093 \
+  --ssl \
+  --mechanism plain \
+  --username myuser \
+  --password mypassword
+```
+
+### SASL/SCRAM-SHA-256
+```bash
+kafka-console consume my-topic \
+  --brokers broker:9093 \
+  --ssl \
+  --mechanism scram-sha-256 \
+  --username myuser \
+  --password mypassword
+```
+
+### AWS IAM
+```bash
+kafka-console consume my-topic \
+  --brokers broker:9093 \
+  --ssl \
+  --mechanism aws \
+  --access-key-id AKIAXXXXXXXX \
+  --secret-access-key XXXXXXXXXX \
+  --session-token XXXXXXXXXX
+```
+
+### OAuth Bearer
+```bash
+kafka-console consume my-topic \
+  --brokers broker:9093 \
+  --ssl \
+  --mechanism oauthbearer \
+  --oauth-bearer "eyJhbGciOiJIUzI1NiIs..."
+```
+
+## Message Formats
+
+### JSON Format (Default)
+Messages are parsed as JSON:
+```bash
+echo '{"name": "Alice", "age": 30}' | kafka-console produce my-topic
+```
+
+### Raw Format
+Messages are sent as plain text:
+```bash
+echo "Plain text message" | kafka-console produce my-topic --data-format raw
+```
+
+### JavaScript Format
+Messages can contain JavaScript exports:
+```bash
+echo 'module.exports = { timestamp: Date.now() }' | kafka-console produce my-topic --data-format js
+```
+
+### Custom Formatter
+Create a custom formatter module:
+
+```javascript
+// formatter/custom.js
+module.exports = {
+  encode: (value) => Buffer.from(JSON.stringify(value)),
+  decode: (buffer) => JSON.parse(buffer.toString())
+};
+```
+
+Use the custom formatter:
+```bash
+kafka-console consume my-topic --data-format ./formatter/custom.js
+```
+
+## Environment Variables
+
+Set environment variables to avoid repeating common options:
+
+```bash
+export KAFKA_BROKERS=broker1:9092,broker2:9092
+export KAFKA_USERNAME=myuser
+export KAFKA_PASSWORD=mypassword
+export KAFKA_MECHANISM=plain
+export KAFKA_TIMEOUT=30000
+```
+
+All supported environment variables:
+- `KAFKA_BROKERS` - Comma-separated list of brokers
+- `KAFKA_TIMEOUT` - Operation timeout in milliseconds
+- `KAFKA_MECHANISM` - SASL mechanism
+- `KAFKA_USERNAME` - SASL username
+- `KAFKA_PASSWORD` - SASL password
+- `KAFKA_AUTH_ID` - AWS authorization identity
+- `KAFKA_ACCESS_KEY_ID` - AWS access key ID
+- `KAFKA_SECRET_ACCESS_KEY` - AWS secret access key
+- `KAFKA_SESSION_TOKEN` - AWS session token
+- `KAFKA_OAUTH_BEARER` - OAuth bearer token
+
+## Common Use Cases
+
+### Monitor Topic in Real-time
+```bash
+kafka-console consume logs --group monitor-group --pretty
+```
+
+### Replay Messages from Yesterday
+```bash
+kafka-console consume events --from "$(date -d yesterday --iso-8601)"
+```
+
+### Copy Messages Between Topics
+```bash
+kafka-console consume source-topic | kafka-console produce destination-topic
+```
+
+### Filter Messages
+```bash
+kafka-console consume all-events | jq 'select(.value.type == "ERROR")' 
+```
+
+### Count Messages in Topic
+```bash
+kafka-console consume my-topic --from 0 | wc -l
+```
+
+### Sample Messages
+```bash
+kafka-console consume large-topic --count 100 --pretty
+```
+
+### Debug Message Headers
+```bash
+kafka-console consume my-topic | jq '.headers'
+```
+
+## Troubleshooting
+
+### Connection Issues
+
+**Problem:** Cannot connect to Kafka broker
+```bash
+Error: KafkaJSConnectionError: Connection timeout
+```
+**Solution:** 
+- Verify broker addresses are correct
+- Check network connectivity: `telnet broker-host 9092`
+- Ensure security groups/firewalls allow connection
+- For Docker: use host network or proper port mapping
+
+### Authentication Failures
+
+**Problem:** Authentication failed
+```bash
+Error: KafkaJSProtocolError: SASL authentication failed
+```
+**Solution:**
+- Verify credentials are correct
+- Check SASL mechanism matches broker configuration
+- Ensure SSL is enabled if required: `--ssl`
+
+### Consumer Group Issues
+
+**Problem:** Not receiving messages
+**Solution:**
+- Check consumer group offset: `kafka-console topic:offsets my-topic --group my-group`
+- Reset to beginning: `--from 0`
+- Use a new consumer group name
+
+### Message Format Errors
+
+**Problem:** JSON parsing errors
+```bash
+SyntaxError: Unexpected token...
+```
+**Solution:**
+- Verify message format matches specified data-format
+- Use `--data-format raw` for non-JSON messages
+- Check for malformed JSON with: `jq . < input.json`
+
+### Performance Issues
+
+**Problem:** Slow message consumption
+**Solution:**
+- Increase batch size in consumer configuration
+- Use multiple consumer instances with same group
+- Check network latency to brokers
+
+### SSL/TLS Issues
+
+**Problem:** SSL handshake failed
+**Solution:**
+- Ensure `--ssl` flag is used
+- Verify broker SSL port (usually 9093)
+- Check certificate validity
 
 ## License
+
 License [The MIT License](http://opensource.org/licenses/MIT)
 Copyright (c) 2024 Ivan Zakharchanka
 
 [npm-url]: https://www.npmjs.com/package/kafka-console
 [downloads-image]: https://img.shields.io/npm/dw/kafka-console.svg?maxAge=43200
-[npm-image]: https://img.shields.io/npm/v/kafka-console.svg?maxAge=43200
+[npm-image]: https://img.shields.io/npm/v/kafka-console.svg?maxAge=43200

package/build/commands/fetchTopicOffsets.js

@@ -24,7 +24,7 @@ exports.default = fetchTopicOffset;
 const kafka_1 = require("../utils/kafka");
 function fetchTopicOffset(topic_1, timestamp_1, opts_1, _a) {
     return __awaiter(this, arguments, void 0, function* (topic, timestamp, opts, { parent }) {
-        const _b = Object.assign(Object.assign({}, parent.opts()), opts), { brokers, logLevel, ssl, pretty } = _b, rest = __rest(_b, ["brokers", "logLevel", "ssl", "pretty"]);
+        const _b = Object.assign(Object.assign({}, parent.opts()), opts), { brokers, group, logLevel, ssl, pretty } = _b, rest = __rest(_b, ["brokers", "group", "logLevel", "ssl", "pretty"]);
         const space = pretty ? 2 : 0;
         const sasl = (0, kafka_1.getSASL)(rest);
         const client = (0, kafka_1.createClient)(brokers, ssl, sasl, logLevel);
@@ -37,6 +37,10 @@ function fetchTopicOffset(topic_1, timestamp_1, opts_1, _a) {
             const topicOffsets = yield admin.fetchTopicOffsetsByTimestamp(topic, unixTimestamp);
             console.log(JSON.stringify(topicOffsets, null, space));
         }
+        if (group) {
+            const topicOffsets = yield admin.fetchOffsets({ groupId: group, topics: [topic] });
+            console.log(JSON.stringify(topicOffsets, null, space));
+        }
         else {
             const topicOffsets = yield admin.fetchTopicOffsets(topic);
             console.log(JSON.stringify(topicOffsets, null, space));

package/build/index.js

@@ -83,6 +83,7 @@ commander
 commander
     .command('topic:offsets <topic> [timestamp]')
     .description('Shows kafka topic offsets')
+    .option('-g, --group <group>', 'consumer group name', `kafka-console-consumer-${Date.now()}`)
     .action(fetchTopicOffsets_1.default);
 commander.on('--help', function () {
     [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kafka-console",
-  "version": "2.0.76",
+  "version": "2.1.0",
   "description": "Kafka CLI tool",
   "main": "index.js",
   "bin": {

package/codecov.SHA256SUM

@@ -1 +0,0 @@
-5cc89518864eaae49f3819e42ba353f9a3c676912d4472351d31970643f1a49a  codecov

package/codecov.SHA256SUM.sig

@@ -1,16 +0,0 @@
------BEGIN PGP SIGNATURE-----
-
-iQIzBAABCgAdFiEEJwNOf9uFDgu8LGL/gGuyiu13mGkFAmiKiI8ACgkQgGuyiu13
-mGlk4xAAmsinowQbB0DMmXRHBW5lHO4dS8WQDq8vUsAO++eomoi6w3dH0DWdv1+2
-PY3v2UmCgEqS42fm+avl5LkIEyhE8n2MCwPevgtj72KGEeswoLSXk7T8XB/E5fbj
-TAAUjxeES/0r8ugbs3JuXw7ttl9Tt2XoGb3sMBAWSCLN1UxUEh5oc22UKjDaNSve
-NRirNwhO6BndacABO2Kt+Wu1UbBznyvy4XQ9jmF1wPdGUZeyHMROcePUof8xc3At
-IUIbdlAIIEG7o3WXzCHoRwKZ7P1PMXNdXBTprIeUNGueMi9OJdo/5tFSDyOkQl8z
-BbK6NqljjMXDaCH1gXS4OAyxHevUd/BDOlkKzKT9krThDQ5KMb36jag21VNI3WSi
-NASgbl83raq2taqPCOLOpQZoROuXtTLyYc1623D+3nrv+wL7lwJTR1b+BJ0Zyhgf
-mKQ3uu94cQHxn1fXPranBeZwD47xXpLIPHpsLXPgK1T/Ge6odW33rnhasQvWkJdu
-+MDVvBcKKzGILlpmHRXCNTqcDWO/Pe6I4I1fP1vGH3XRELLHeSiCDKYrY/PH28zj
-xJTPXd89xu6IEeSU9PtIvQ4oCeUahOnSbBUbCMzl5/2FNu1pIPPR6yv6XA313Zdi
-OPkrKo/0qtm/z0ZVn44zNzqbUgx7QiKBinFNuAJ2N6mMqkE4wdM=
-=9CDH
------END PGP SIGNATURE-----