cledar-sdk 2.0.2__py3-none-any.whl → 2.1.0__py3-none-any.whl

cledar/kafka/tests/README.md

@@ -0,0 +1,216 @@
+# Kafka Service Tests
+
+This directory contains the test suite for the Kafka service, organized into unit and integration tests.
+
+## Directory Structure
+
+```
+tests/
+├── conftest.py              # Test-wide teardown (cleans Kafka client threads)
+├── README.md
+├── unit/                    # Unit tests (176 tests)
+│   ├── test_base_kafka_client.py
+│   ├── test_config_validation.py
+│   ├── test_dead_letter_handler.py
+│   ├── test_error_handling.py
+│   ├── test_input_parser.py
+│   ├── test_input_parser_comprehensive.py
+│   ├── test_utils.py
+│   ├── test_utils_comprehensive.py
+│   └── requirements-test.txt
+└── integration/             # Integration tests (41 tests)
+    ├── conftest.py          # Shared Kafka fixtures (container, configs, clients)
+    ├── helpers.py           # E2EData, consume_until, ensure_topic_and_subscribe
+    ├── test_integration.py
+    ├── test_producer_integration.py
+    ├── test_consumer_integration.py
+    └── test_producer_consumer_interaction.py
+```
+
+## Test Categories
+
+### Unit Tests (`unit/`)
+Unit tests focus on testing individual components in isolation using mocks and stubs. They are fast, reliable, and don't require external dependencies.
+
+- **Base Client Tests**: Test the base Kafka client functionality
+- **Config Validation**: Test configuration validation and schema validation
+- **Dead Letter Handler**: Test dead letter queue handling with mocked producers
+- **Error Handling**: Test error scenarios and exception handling
+- **Input Parser**: Test message parsing and validation
+- **Utils**: Test utility functions and helper methods
+
+### Integration Tests (`integration/`)
+Integration tests use real external dependencies (like Kafka via testcontainers) to test the complete flow of the system.
+
+- **Real Kafka Integration**: Tests with actual Kafka instance using testcontainers
+- **Producer Integration**: Real producer operations and message sending
+- **Consumer Integration**: Real consumer operations and message consumption
+- **Producer-Consumer Interaction**: Real interaction patterns between producer and consumer
+- **End-to-End Flows**: Complete producer-consumer workflows
+- **Connection Recovery**: Real connection failure and recovery scenarios
+- **Performance Tests**: Stress tests and large message handling
+
+## Running Tests
+
+### Run All Tests
+```bash
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/
+```
+
+### Run Only Unit Tests
+```bash
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/unit/
+```
+
+### Run Only Integration Tests
+```bash
+# Run all integration tests
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/
+
+# Run specific integration test file
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_producer_integration.py
+
+# Run single test
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_integration.py::test_end_to_end_message_flow -v
+```
+
+### Run Specific Test Files
+```bash
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/unit/test_config_validation.py
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_integration.py
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_producer_integration.py
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_consumer_integration.py
+```
+
+## Test Requirements
+
+- **Unit Tests**: No external dependencies required
+- **Integration Tests**: Requires Docker to be running for testcontainers
+- **Slow Integration Tests**: Marked as skipped by default due to execution time (2-5 minutes each)
+
+## Performance Notes
+
+- **Unit Tests**: Fast execution (~10–15 seconds for all 176 tests)
+- **Integration Tests**: Moderate execution (~2–2.5 minutes for 41 tests)
+- Helpers reduce flakiness: `consume_until()` polls with timeout instead of fixed sleeps
+
+## Docker Setup for Integration Tests
+
+The integration tests use testcontainers to spin up real Kafka instances for testing. This requires Docker to be installed and running.
+
+### Prerequisites
+
+1. **Install Docker Desktop** (recommended):
+   - [Docker Desktop for Mac](https://docs.docker.com/desktop/mac/install/)
+   - [Docker Desktop for Windows](https://docs.docker.com/desktop/windows/install/)
+   - [Docker Desktop for Linux](https://docs.docker.com/desktop/linux/install/)
+
+2. **Or install Docker Engine** (alternative):
+   ```bash
+   # macOS (using Homebrew)
+   brew install docker
+   
+   # Ubuntu/Debian
+   sudo apt-get update
+   sudo apt-get install docker.io
+   
+   # CentOS/RHEL
+   sudo yum install docker
+   ```
+
+### Starting Docker
+
+#### Docker Desktop
+1. Launch Docker Desktop application
+2. Wait for Docker to start (you'll see the Docker whale icon in your system tray)
+3. Verify Docker is running:
+   ```bash
+   docker --version
+   docker ps
+   ```
+
+#### Docker Engine (Linux)
+```bash
+# Start Docker service
+sudo systemctl start docker
+sudo systemctl enable docker
+
+# Add your user to docker group (optional, to avoid sudo)
+sudo usermod -aG docker $USER
+# Log out and back in for group changes to take effect
+
+# Verify Docker is running
+docker --version
+docker ps
+```
+
+### Running Integration Tests
+
+Once Docker is running, you can execute the integration tests:
+
+```bash
+# Run all integration tests
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/
+
+# Run a specific integration test
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/test_integration.py::test_producer_consumer_basic_flow
+
+# Run integration tests with verbose output
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/ -v
+
+# Run integration tests and show logs
+PYTHONPATH=. uv run pytest cledar/kafka_service/tests/integration/ -s
+```
+
+### Troubleshooting Docker Issues
+
+#### Docker not running
+```bash
+# Check if Docker is running
+docker ps
+# If you get "Cannot connect to the Docker daemon", Docker is not running
+
+# Start Docker Desktop or Docker service
+# Docker Desktop: Launch the application
+# Docker Engine: sudo systemctl start docker
+```
+
+#### Permission denied errors
+```bash
+# Add user to docker group (Linux)
+sudo usermod -aG docker $USER
+# Log out and back in
+
+# Or run with sudo (not recommended)
+sudo docker ps
+```
+
+#### Port conflicts
+If you have Kafka running locally on port 9092, the testcontainers will automatically use different ports. No action needed.
+
+#### Resource constraints
+If tests fail due to memory/CPU constraints:
+```bash
+# Check Docker resource limits in Docker Desktop settings
+# Increase memory allocation if needed (recommended: 4GB+)
+```
+
+### Test Container Details
+
+The integration tests use:
+- **Kafka Image**: `confluentinc/cp-kafka:7.4.0`
+- **Automatic Port Assignment**: testcontainers handles port conflicts
+- **Automatic Cleanup**: containers are removed after tests complete
+- **Session Scope**: Kafka container is shared across all integration tests in a session
+
+## Test Statistics
+
+- **Total Tests**: 217
+- **Unit Tests**: 176
+- **Integration Tests**: 41
+
+## Notes
+
+- All tests use `PYTHONPATH=.` to ensure proper module imports
+- Integration tests use shared fixtures in `integration/conftest.py` and helpers in `integration/helpers.py`
+- Test-wide teardown in `tests/conftest.py` ensures Kafka client threads don’t block process exit

cledar/kafka/tests/conftest.py

@@ -0,0 +1,104 @@
+"""
+Pytest configuration and fixtures to ensure proper teardown of Kafka clients and
+their background threads during tests.
+"""
+
+import threading
+import time
+import weakref
+from collections.abc import Callable, Generator
+from typing import Any, cast
+
+import pytest
+
+from cledar.kafka.clients.base import BaseKafkaClient
+from cledar.kafka.logger import logger
+
+# Weak registry of all created BaseKafkaClient instances (does not keep them alive)
+_active_clients: "weakref.WeakSet[BaseKafkaClient]" = weakref.WeakSet()
+
+
+def _wrap_post_init() -> tuple[
+    Callable[[BaseKafkaClient], None], Callable[[BaseKafkaClient], None]
+]:
+    """Monkeypatch BaseKafkaClient.__post_init__ to register instances."""
+    original = BaseKafkaClient.__post_init__
+
+    def wrapped(self: BaseKafkaClient) -> None:
+        original(self)
+        try:
+            _active_clients.add(self)
+        except Exception:
+            # Best-effort registration only for tests
+            pass
+
+    return original, wrapped
+
+
+def _wrap_start_connection_check_thread() -> tuple[
+    Callable[[BaseKafkaClient], None], Callable[[BaseKafkaClient], None]
+]:
+    """Monkeypatch start_connection_check_thread to use daemon threads in tests.
+
+    This prevents non-daemon threads from blocking interpreter shutdown when tests
+    forget to call shutdown() explicitly.
+    """
+    original = BaseKafkaClient.start_connection_check_thread
+
+    def wrapped(self: BaseKafkaClient) -> None:
+        if self.connection_check_thread is None:
+            self.connection_check_thread = threading.Thread(
+                target=self._monitor_connection
+            )
+            # Ensure test background threads never block process exit
+            self.connection_check_thread.daemon = True
+            self.connection_check_thread.start()
+            logger.info(
+                f"Started {self.__class__.__name__} connection check thread.",
+                extra={"interval": self.config.kafka_connection_check_interval_sec},
+            )
+
+    return original, wrapped
+
+
+def _cleanup_all_clients() -> None:
+    """Shutdown all known clients; ignore errors during cleanup."""
+    for client in list(_active_clients):
+        try:
+            client.shutdown()
+        except Exception:
+            pass
+
+
+@pytest.fixture(scope="session", autouse=True)
+def _session_monkeypatch() -> Generator[None, None, None]:
+    """Apply monkeypatches for the entire test session and ensure final cleanup."""
+    # Monkeypatch __post_init__ to register instances
+    orig_post_init, wrapped_post_init = _wrap_post_init()
+    cast(Any, BaseKafkaClient).__post_init__ = wrapped_post_init
+
+    # Monkeypatch start_connection_check_thread to create daemon threads
+    orig_start, wrapped_start = _wrap_start_connection_check_thread()
+    cast(Any, BaseKafkaClient).start_connection_check_thread = wrapped_start
+
+    try:
+        yield
+    finally:
+        # Restore originals
+        cast(Any, BaseKafkaClient).__post_init__ = orig_post_init
+        cast(Any, BaseKafkaClient).start_connection_check_thread = orig_start
+
+        # Final cleanup at session end
+        _cleanup_all_clients()
+
+        # Give threads a small grace period to finish
+        time.sleep(0.1)
+
+
+@pytest.fixture(autouse=True)
+def _per_test_cleanup() -> Generator[None, None, None]:
+    """Ensure all clients are shut down after each individual test."""
+    yield
+    _cleanup_all_clients()
+    # Small grace to allow quick thread exit
+    time.sleep(0.05)

cledar/kafka/tests/integration/__init__.py

@@ -0,0 +1 @@
+"""Integration tests for kafka_service."""

cledar/kafka/tests/integration/conftest.py

@@ -0,0 +1,78 @@
+from collections.abc import Generator
+
+import pytest
+from testcontainers.kafka import KafkaContainer
+
+from cledar.kafka.clients.consumer import KafkaConsumer
+from cledar.kafka.clients.producer import KafkaProducer
+from cledar.kafka.config.schemas import KafkaConsumerConfig, KafkaProducerConfig
+
+
+@pytest.fixture(scope="session")
+def kafka_container() -> Generator[KafkaContainer, None, None]:
+    kafka = KafkaContainer("confluentinc/cp-kafka:7.4.0")
+    kafka = kafka.with_env("KAFKA_AUTO_CREATE_TOPICS_ENABLE", "true")
+    kafka = kafka.with_env("KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR", "1")
+    kafka = kafka.with_env("KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR", "1")
+    kafka = kafka.with_env("KAFKA_TRANSACTION_STATE_LOG_MIN_ISR", "1")
+    kafka = kafka.with_env("KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS", "0")
+    kafka = kafka.with_env("KAFKA_LOG_RETENTION_HOURS", "1")
+    with kafka as container:
+        yield container
+
+
+@pytest.fixture
+def kafka_bootstrap_servers(kafka_container: KafkaContainer) -> str:
+    # testcontainers returns a str, but mypy sees Any without stubs
+    server: str = str(kafka_container.get_bootstrap_server())
+    return server
+
+
+@pytest.fixture
+def producer_config(kafka_bootstrap_servers: str) -> KafkaProducerConfig:
+    return KafkaProducerConfig(
+        kafka_servers=kafka_bootstrap_servers,
+        kafka_group_id="integration-test-producer",
+        kafka_topic_prefix="integration-test.",
+        kafka_block_buffer_time_sec=1,
+        kafka_connection_check_timeout_sec=5,
+        kafka_connection_check_interval_sec=10,
+    )
+
+
+@pytest.fixture
+def consumer_config(kafka_bootstrap_servers: str) -> KafkaConsumerConfig:
+    return KafkaConsumerConfig(
+        kafka_servers=kafka_bootstrap_servers,
+        kafka_group_id="integration-test-consumer",
+        kafka_offset="earliest",
+        kafka_topic_prefix="integration-test.",
+        kafka_block_consumer_time_sec=1,
+        kafka_connection_check_timeout_sec=5,
+        kafka_auto_commit_interval_ms=1000,
+        kafka_connection_check_interval_sec=10,
+    )
+
+
+@pytest.fixture
+def producer(
+    producer_config: KafkaProducerConfig,
+) -> Generator[KafkaProducer, None, None]:
+    p = KafkaProducer(producer_config)
+    p.connect()
+    try:
+        yield p
+    finally:
+        p.shutdown()
+
+
+@pytest.fixture
+def consumer(
+    consumer_config: KafkaConsumerConfig,
+) -> Generator[KafkaConsumer, None, None]:
+    c = KafkaConsumer(consumer_config)
+    c.connect()
+    try:
+        yield c
+    finally:
+        c.shutdown()

cledar/kafka/tests/integration/helpers.py

@@ -0,0 +1,47 @@
+import time
+
+from pydantic import BaseModel
+
+from cledar.kafka.clients.consumer import KafkaConsumer
+from cledar.kafka.clients.producer import KafkaProducer
+from cledar.kafka.models.message import KafkaMessage
+
+
+class E2EData(BaseModel):
+    id: str
+    message: str
+    timestamp: float
+
+    def to_json(self) -> str:
+        return self.model_dump_json()
+
+
+def ensure_topic_and_subscribe(
+    producer: KafkaProducer,
+    consumer: KafkaConsumer,
+    topic: str,
+    init_payload: str = '{"id":"init","message":"init"}',
+    create_wait: float = 2.0,
+    subscribe_wait: float = 1.0,
+) -> None:
+    producer.send(topic=topic, value=init_payload, key="init-key")
+    time.sleep(create_wait)
+    consumer.subscribe([topic])
+    time.sleep(subscribe_wait)
+
+
+def consume_until(
+    consumer: KafkaConsumer,
+    expected_count: int,
+    timeout_seconds: float = 10.0,
+    idle_sleep: float = 0.2,
+) -> list[KafkaMessage]:
+    deadline = time.time() + timeout_seconds
+    received: list[KafkaMessage] = []
+    while len(received) < expected_count and time.time() < deadline:
+        msg = consumer.consume_next()
+        if msg is not None:
+            received.append(msg)
+        else:
+            time.sleep(idle_sleep)
+    return received