dgkafka 1.0.0__tar.gz

dgkafka-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dgkafka-1.0.0/PKG-INFO

@@ -0,0 +1,239 @@
+Metadata-Version: 2.4
+Name: dgkafka
+Version: 1.0.0
+Summary: Kafka clients
+Home-page: https://gitlab.com/gng-group/dgkafka.git
+Author: Malanris
+Author-email: Roman Rasputin <admin@roro.su>
+License: MIT License
+Project-URL: Homepage, https://gitlab.com/gng-group/dgkafka
+Project-URL: BugTracker, https://gitlab.com/gng-group/dgkafka/issues
+Keywords: kafka,client,confluent,avro,fastapi,logging
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Logging
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: confluent-kafka>=2.1.1
+Requires-Dist: dglog>=1.0.0
+Requires-Dist: pydantic
+Provides-Extra: avro
+Requires-Dist: requests; extra == "avro"
+Requires-Dist: fastavro<2; extra == "avro"
+Requires-Dist: avro<2,>=1.11.1; extra == "avro"
+Requires-Dist: attrs; extra == "avro"
+Requires-Dist: cachetools; extra == "avro"
+Requires-Dist: httpx>=0.26; extra == "avro"
+Requires-Dist: authlib; extra == "avro"
+Provides-Extra: json
+Requires-Dist: pyrsistent; extra == "json"
+Requires-Dist: jsonschema; extra == "json"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+
+# dgkafka
+
+Python package for working with Apache Kafka supporting multiple data formats.
+
+## Installation
+
+```bash
+pip install dgkafka
+```
+
+For Avro support (requires additional dependencies):
+
+```bash
+pip install dgkafka[avro]
+```
+
+For Json support (requires additional dependencies):
+
+```bash
+pip install dgkafka[json]
+```
+
+## Features
+
+- Producers and consumers for different data formats:
+  - Raw messages (bytes/strings)
+  - JSON
+  - Avro (with Schema Registry integration)
+- Robust error handling
+- Comprehensive operation logging
+- Context manager support
+- Flexible configuration
+
+## Quick Start
+
+### Basic Producer/Consumer
+
+```python
+from dgkafka import KafkaProducer, KafkaConsumer
+
+# Producer
+with KafkaProducer(bootstrap_servers='localhost:9092') as producer:
+    producer.produce('test_topic', 'Hello, Kafka!')
+
+# Consumer
+with KafkaConsumer(bootstrap_servers='localhost:9092', group_id='test_group') as consumer:
+    consumer.subscribe(['test_topic'])
+    for msg in consumer.consume():
+        print(msg.value())
+```
+
+### JSON Support
+
+```python
+from dgkafka import JsonKafkaProducer, JsonKafkaConsumer
+
+# Producer
+with JsonKafkaProducer(bootstrap_servers='localhost:9092') as producer:
+    producer.produce('json_topic', {'key': 'value'})
+
+# Consumer
+with JsonKafkaConsumer(bootstrap_servers='localhost:9092', group_id='json_group') as consumer:
+    consumer.subscribe(['json_topic'])
+    for msg in consumer.consume():
+        print(msg.value())  # Automatically deserialized JSON
+```
+
+### Avro Support
+
+```python
+from dgkafka import AvroKafkaProducer, AvroKafkaConsumer
+
+# Producer
+value_schema = {
+    "type": "record",
+    "name": "User",
+    "fields": [
+        {"name": "name", "type": "string"},
+        {"name": "age", "type": "int"}
+    ]
+}
+
+with AvroKafkaProducer(
+    schema_registry_url='http://localhost:8081',
+    bootstrap_servers='localhost:9092',
+    default_value_schema=value_schema
+) as producer:
+    producer.produce('avro_topic', {'name': 'Alice', 'age': 30})
+
+# Consumer
+with AvroKafkaConsumer(
+    schema_registry_url='http://localhost:8081',
+    bootstrap_servers='localhost:9092',
+    group_id='avro_group'
+) as consumer:
+    consumer.subscribe(['avro_topic'])
+    for msg in consumer.consume():
+        print(msg.value())  # Automatically deserialized Avro object
+```
+
+## Classes
+
+### Base Classes
+
+- `KafkaProducer` - base message producer
+- `KafkaConsumer` - base message consumer
+
+### Specialized Classes
+
+- `JsonKafkaProducer` - JSON message producer (inherits from `KafkaProducer`)
+- `JsonKafkaConsumer` - JSON message consumer (inherits from `KafkaConsumer`)
+- `AvroKafkaProducer` - Avro message producer (inherits from `KafkaProducer`)
+- `AvroKafkaConsumer` - Avro message consumer (inherits from `KafkaConsumer`)
+
+## Configuration
+
+All classes accept standard Kafka configuration parameters:
+
+```python
+config = {
+    'bootstrap.servers': 'localhost:9092',
+    'group.id': 'my_group',
+    'auto.offset.reset': 'earliest'
+}
+```
+
+Avro classes require additional parameter:
+- `schema_registry_url` - Schema Registry URL
+
+## Logging
+
+All classes use `dglog.Logger` for logging. You can provide a custom logger:
+
+```python
+from dglog import Logger
+
+logger = Logger()
+producer = KafkaProducer(logger_=logger, ...)
+```
+
+## Best Practices
+
+1. Always use context managers (`with`) for proper resource cleanup
+2. Implement error handling and retry logic for production use
+3. Pre-register Avro schemas in Schema Registry
+4. Configure appropriate `acks` and `retries` parameters for producers
+5. Monitor consumer lag and producer throughput
+
+## Advanced Usage
+
+### Custom Serialization
+
+```python
+# Custom Avro serializer
+class CustomAvroProducer(AvroKafkaProducer):
+    def _serialize_value(self, value):
+        # Custom serialization logic
+        return super()._serialize_value(value)
+```
+
+### Message Headers
+
+```python
+# Adding headers to messages
+headers = {
+    'correlation_id': '12345',
+    'message_type': 'user_update'
+}
+
+producer.produce(
+    topic='events',
+    value=message_data,
+    headers=headers
+)
+```
+
+### Error Handling
+
+```python
+from confluent_kafka import KafkaException
+
+try:
+    with AvroKafkaProducer(...) as producer:
+        producer.produce(...)
+except KafkaException as e:
+    print(f"Kafka error occurred: {e}")
+```
+
+## Performance Tips
+
+1. Batch messages when possible (`batch.num.messages` config)
+2. Adjust `linger.ms` for better batching
+3. Use `compression.type` (lz4, snappy, or gzip)
+4. Tune `fetch.max.bytes` and `max.partition.fetch.bytes` for consumers
+
+## License
+
+MIT

dgkafka-1.0.0/README.md

@@ -0,0 +1,198 @@
+# dgkafka
+
+Python package for working with Apache Kafka supporting multiple data formats.
+
+## Installation
+
+```bash
+pip install dgkafka
+```
+
+For Avro support (requires additional dependencies):
+
+```bash
+pip install dgkafka[avro]
+```
+
+For Json support (requires additional dependencies):
+
+```bash
+pip install dgkafka[json]
+```
+
+## Features
+
+- Producers and consumers for different data formats:
+  - Raw messages (bytes/strings)
+  - JSON
+  - Avro (with Schema Registry integration)
+- Robust error handling
+- Comprehensive operation logging
+- Context manager support
+- Flexible configuration
+
+## Quick Start
+
+### Basic Producer/Consumer
+
+```python
+from dgkafka import KafkaProducer, KafkaConsumer
+
+# Producer
+with KafkaProducer(bootstrap_servers='localhost:9092') as producer:
+    producer.produce('test_topic', 'Hello, Kafka!')
+
+# Consumer
+with KafkaConsumer(bootstrap_servers='localhost:9092', group_id='test_group') as consumer:
+    consumer.subscribe(['test_topic'])
+    for msg in consumer.consume():
+        print(msg.value())
+```
+
+### JSON Support
+
+```python
+from dgkafka import JsonKafkaProducer, JsonKafkaConsumer
+
+# Producer
+with JsonKafkaProducer(bootstrap_servers='localhost:9092') as producer:
+    producer.produce('json_topic', {'key': 'value'})
+
+# Consumer
+with JsonKafkaConsumer(bootstrap_servers='localhost:9092', group_id='json_group') as consumer:
+    consumer.subscribe(['json_topic'])
+    for msg in consumer.consume():
+        print(msg.value())  # Automatically deserialized JSON
+```
+
+### Avro Support
+
+```python
+from dgkafka import AvroKafkaProducer, AvroKafkaConsumer
+
+# Producer
+value_schema = {
+    "type": "record",
+    "name": "User",
+    "fields": [
+        {"name": "name", "type": "string"},
+        {"name": "age", "type": "int"}
+    ]
+}
+
+with AvroKafkaProducer(
+    schema_registry_url='http://localhost:8081',
+    bootstrap_servers='localhost:9092',
+    default_value_schema=value_schema
+) as producer:
+    producer.produce('avro_topic', {'name': 'Alice', 'age': 30})
+
+# Consumer
+with AvroKafkaConsumer(
+    schema_registry_url='http://localhost:8081',
+    bootstrap_servers='localhost:9092',
+    group_id='avro_group'
+) as consumer:
+    consumer.subscribe(['avro_topic'])
+    for msg in consumer.consume():
+        print(msg.value())  # Automatically deserialized Avro object
+```
+
+## Classes
+
+### Base Classes
+
+- `KafkaProducer` - base message producer
+- `KafkaConsumer` - base message consumer
+
+### Specialized Classes
+
+- `JsonKafkaProducer` - JSON message producer (inherits from `KafkaProducer`)
+- `JsonKafkaConsumer` - JSON message consumer (inherits from `KafkaConsumer`)
+- `AvroKafkaProducer` - Avro message producer (inherits from `KafkaProducer`)
+- `AvroKafkaConsumer` - Avro message consumer (inherits from `KafkaConsumer`)
+
+## Configuration
+
+All classes accept standard Kafka configuration parameters:
+
+```python
+config = {
+    'bootstrap.servers': 'localhost:9092',
+    'group.id': 'my_group',
+    'auto.offset.reset': 'earliest'
+}
+```
+
+Avro classes require additional parameter:
+- `schema_registry_url` - Schema Registry URL
+
+## Logging
+
+All classes use `dglog.Logger` for logging. You can provide a custom logger:
+
+```python
+from dglog import Logger
+
+logger = Logger()
+producer = KafkaProducer(logger_=logger, ...)
+```
+
+## Best Practices
+
+1. Always use context managers (`with`) for proper resource cleanup
+2. Implement error handling and retry logic for production use
+3. Pre-register Avro schemas in Schema Registry
+4. Configure appropriate `acks` and `retries` parameters for producers
+5. Monitor consumer lag and producer throughput
+
+## Advanced Usage
+
+### Custom Serialization
+
+```python
+# Custom Avro serializer
+class CustomAvroProducer(AvroKafkaProducer):
+    def _serialize_value(self, value):
+        # Custom serialization logic
+        return super()._serialize_value(value)
+```
+
+### Message Headers
+
+```python
+# Adding headers to messages
+headers = {
+    'correlation_id': '12345',
+    'message_type': 'user_update'
+}
+
+producer.produce(
+    topic='events',
+    value=message_data,
+    headers=headers
+)
+```
+
+### Error Handling
+
+```python
+from confluent_kafka import KafkaException
+
+try:
+    with AvroKafkaProducer(...) as producer:
+        producer.produce(...)
+except KafkaException as e:
+    print(f"Kafka error occurred: {e}")
+```
+
+## Performance Tips
+
+1. Batch messages when possible (`batch.num.messages` config)
+2. Adjust `linger.ms` for better batching
+3. Use `compression.type` (lz4, snappy, or gzip)
+4. Tune `fetch.max.bytes` and `max.partition.fetch.bytes` for consumers
+
+## License
+
+MIT

dgkafka-1.0.0/dgkafka/__init__.py

@@ -0,0 +1,11 @@
+from .consumer import KafkaConsumer
+from .producer import KafkaProducer
+try:
+    from .avro_consumer import AvroKafkaConsumer
+    from .avro_producer import AvroKafkaProducer
+except ImportError:
+    pass
+try:
+    from .json_consumer import JsonKafkaConsumer
+except ImportError:
+    pass

dgkafka-1.0.0/dgkafka/avro_consumer.py

@@ -0,0 +1,74 @@
+from typing import Any, Iterator
+
+from dgkafka.consumer import KafkaConsumer
+
+from confluent_kafka import Message
+from confluent_kafka.avro import AvroConsumer
+from confluent_kafka.avro.serializer import SerializerError
+from confluent_kafka.avro.cached_schema_registry_client import CachedSchemaRegistryClient
+
+import logging
+import dglog
+
+
+class AvroKafkaConsumer(KafkaConsumer):
+    """Kafka consumer with Avro schema support using Schema Registry."""
+
+    def __init__(self, logger_: logging.Logger | dglog.Logger | None = None, **configs: Any) -> None:
+        """
+        Initialize Avro consumer.
+
+        Args:
+            schema_registry_url: URL of Schema Registry
+            logger_: Optional logger instance
+            configs: Kafka consumer configuration
+        """
+        self.schema_registry_url = configs.get('schema.registry.url')
+        assert self.schema_registry_url is not None, "schema.registry.url is required"
+
+        self.schema_registry_client = CachedSchemaRegistryClient(url=self.schema_registry_url)
+        super().__init__(logger_=logger_, **configs)
+
+    def _init_consumer(self, **configs: Any) -> None:
+        """Initialize AvroConsumer instance."""
+        try:
+            self.consumer = AvroConsumer(configs)
+            self.logger.info("[*] Avro consumer initialized successfully")
+        except Exception as ex:
+            self.logger.error(f"[x] Failed to initialize avro consumer: {ex}")
+            raise
+
+    def consume(self, num_messages: int = 1, timeout: float = 1.0, decode_: bool = False, **kwargs: Any) -> Iterator[str | bytes | Message | None]:
+        """
+        Consume Avro-encoded messages.
+
+        Args:
+            num_messages: Maximum number of messages to consume
+            timeout: Poll timeout in seconds
+            kwargs: Additional arguments
+
+        Yields:
+            Deserialized Avro messages as dictionaries or Message objects on error
+        """
+        consumer = self._ensure_consumer()
+
+        for _ in range(num_messages):
+            msg = self._consume(consumer, timeout)
+            try:
+                if msg is None:
+                    continue
+                yield msg.value() if decode_ else msg
+            except SerializerError as e:
+                self.logger.error(f"[x] Avro deserialization failed: {e}")
+                yield msg  # Return raw message on deserialization error
+            except Exception as ex:
+                self.logger.error(f"[!] Unexpected error: {ex}")
+                continue
+
+    def get_schema(self, subject: str, version: int = 1) -> dict[str, Any]:
+        """Get Avro schema from Schema Registry."""
+        return self.schema_registry_client.get_schema(subject, version)
+
+    def get_latest_schema(self, subject: str) -> dict[str, Any]:
+        """Get latest version of schema for given subject."""
+        return self.schema_registry_client.get_latest_schema(subject)[1]

dgkafka-1.0.0/dgkafka/avro_producer.py

@@ -0,0 +1,138 @@
+from typing import Optional, Union, Dict, Any
+from confluent_kafka.avro import AvroProducer
+from confluent_kafka.avro.serializer import SerializerError
+from confluent_kafka.avro.cached_schema_registry_client import CachedSchemaRegistryClient
+
+import dglog
+import logging
+
+from dgkafka.producer import KafkaProducer
+
+
+class AvroKafkaProducer(KafkaProducer):
+    """Kafka producer with Avro schema support using Schema Registry."""
+
+    def __init__(
+            self,
+            default_key_schema: str | None = None,
+            default_value_schema: str | None = None,
+            logger_: logging.Logger | dglog.Logger | None = None,
+            **configs: Any
+    ) -> None:
+        """
+        Initialize Avro producer.
+
+        Args:
+            schema_registry_url: URL of Schema Registry
+            default_key_schema: Default Avro schema for message keys
+            default_value_schema: Default Avro schema for message values
+            logger_: Optional logger instance
+            configs: Kafka producer configuration
+        """
+        self.schema_registry_url = configs.get('schema.registry.url')
+        assert self.schema_registry_url is not None, "schema.registry.url is required"
+
+        self.default_key_schema = default_key_schema
+        self.default_value_schema = default_value_schema
+        self.schema_registry_client = CachedSchemaRegistryClient(url=self.schema_registry_url)
+        super().__init__(logger_=logger_, **configs)
+
+    def _init_producer(self, **configs: Any) -> None:
+        """Initialize AvroProducer instance."""
+        try:
+            self.producer = AvroProducer(
+                config=configs,
+                default_key_schema=self.default_key_schema,
+                default_value_schema=self.default_value_schema
+            )
+            self.logger.info("[*] Avro producer initialized successfully")
+        except Exception as ex:
+            self.logger.error(f"[x] Failed to initialize avro producer: {ex}")
+            raise
+
+    def produce(
+            self,
+            topic: str,
+            value: dict[str, Any] | Any,
+            key: dict[str, Any] | str | None  = None,
+            value_schema: dict[str, Any] | None = None,
+            key_schema: dict[str, Any] | None = None,
+            partition: int | None = None,
+            headers: dict[str, bytes] | None = None,
+            flush: bool = True
+    ) -> None:
+        """
+        Produce Avro-encoded message to Kafka.
+
+        Args:
+            topic: Target topic name
+            value: Message value (must match Avro schema)
+            key: Message key (optional)
+            value_schema: Avro schema for message value (optional)
+            key_schema: Avro schema for message key (optional)
+            partition: Specific partition (optional)
+            headers: Message headers (optional)
+            flush: Immediately flush after producing (default: True)
+        """
+        producer = self._ensure_producer()
+        producer.poll(0)
+
+        self._delivery_status['success'] = None
+
+        # Prepare headers
+        headers_list = None
+        if headers:
+            headers_list = [(k, v if isinstance(v, bytes) else str(v).encode('utf-8'))
+                            for k, v in headers.items()]
+
+        try:
+            if not partition:
+                producer.produce(
+                    topic=topic,
+                    value=value,
+                    value_schema=value_schema,
+                    key=key,
+                    key_schema=key_schema,
+                    on_delivery=self.delivery_report,
+                    headers=headers_list
+                )
+            else:
+                producer.produce(
+                    topic=topic,
+                    value=value,
+                    value_schema=value_schema,
+                    key=key,
+                    key_schema=key_schema,
+                    partition=partition,
+                    on_delivery=self.delivery_report,
+                    headers=headers_list
+                )
+
+            if flush:
+                remaining = producer.flush(timeout)
+                if remaining > 0:
+                    self.logger.warning(f"[!] {remaining} messages remain undelivered after flush timeout")
+                    return False
+
+            # Если flush=True, статус должен быть установлен к этому моменту
+            if flush and self._delivery_status['success'] is not None:
+                return self._delivery_status['success']
+
+            # Если flush=False, мы не можем гарантировать доставку, возвращаем True
+            # (так как технически ошибки пока нет)
+            return True
+
+        except SerializerError as ex:
+            self.logger.error(f"[x] Avro serialization failed: {ex}")
+            return False
+        except Exception as ex:
+            self.logger.error(f"[x] Failed to produce Avro message: {ex}")
+            return False
+
+    def get_schema(self, subject: str, version: int = 1) -> Dict[str, Any]:
+        """Get Avro schema from Schema Registry."""
+        return self.schema_registry_client.get_schema(subject, version)
+
+    def get_latest_schema(self, subject: str) -> Dict[str, Any]:
+        """Get latest version of schema for given subject."""
+        return self.schema_registry_client.get_latest_schema(subject)[1]