kafka-python 2.0.2__tar.gz → 2.0.3__tar.gz

{kafka-python-2.0.2 → kafka-python-2.0.3}/CHANGES.md

@@ -1,3 +1,55 @@
+# 2.0.3 (Feb 12, 2025)
+
+Improvements
+* Add optional compression libs to extras_require (#2123, #2387)
+* KafkaConsumer: Exit poll if consumer is closed (#2152)
+* Support configuration of custom kafka client for Admin/Consumer/Producer (#2144)
+* Core Protocol: Add support for flexible versions (#2151)
+* (Internal) Allow disabling thread wakeup in _send_request_to_node (#2335)
+* Change loglevel of cancelled errors to info (#2467)
+* Strip trailing dot off hostname for SSL validation. (#2472)
+* Log connection close(error) at ERROR level (#2473)
+* Support DescribeLogDirs admin api (#2475)
+
+Compatibility
+* Support for python 3.12 (#2379, #2382)
+* Kafka 2.5 / 2.6 (#2162)
+* Try collections.abc imports in vendored selectors34 (#2394)
+* Catch OSError when checking for gssapi import for windows compatibility (#2407)
+* Update vendored six to 1.16.0 (#2398)
+
+Documentation
+* Update usage.rst (#2308, #2334)
+* Fix typos (#2319, #2207, #2178)
+* Fix links to the compatibility page (#2295, #2226)
+* Cleanup install instructions for optional libs (#2139)
+* Update license_file to license_files (#2462)
+* Update some RST documentation syntax (#2463)
+* Add .readthedocs.yaml; update copyright date (#2474)
+
+Fixes
+* Use isinstance in builtin crc32 (#2329)
+* Use six.viewitems instead of six.iteritems to avoid encoding problems in StickyPartitionAssignor (#2154)
+* Fix array encoding TypeError: object of type 'dict_itemiterator' has no len() (#2167)
+* Only try to update sensors fetch lag if the unpacked list contains elements (#2158)
+* Avoid logging errors during test fixture cleanup (#2458)
+* Release coordinator lock before calling maybe_leave_group (#2460)
+* Dont raise RuntimeError for dead process in SpawnedService.wait_for() (#2461)
+* Cast the size of a MemoryRecordsBuilder object (#2438)
+* Fix DescribeConfigsResponse_v1 config_source (#2464)
+* Fix base class of DescribeClientQuotasResponse_v0 (#2465)
+* Update socketpair w/ CVE-2024-3219 fix (#2468)
+
+Testing
+* Transition CI/CD to GitHub Workflows (#2378, #2392, #2381, #2406, #2419, #2418, #2417, #2456)
+* Refactor Makefile (#2457)
+* Use assert_called_with in client_async tests (#2375)
+* Cover sticky assignor's metadata method with tests (#2161)
+* Update fixtures.py to check "127.0.0.1" for auto port assignment (#2384)
+* Use -Djava.security.manager=allow for Java 23 sasl tests (#2469)
+* Test with Java 23 (#2470)
+* Update kafka properties template; disable group rebalance delay (#2471)
+
 # 2.0.2 (Sep 29, 2020)
 
 Consumer

{kafka-python-2.0.2 → kafka-python-2.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: kafka-python
-Version: 2.0.2
+Version: 2.0.3
 Summary: Pure Python client for Apache Kafka
 Home-page: https://github.com/dpkp/kafka-python
 Author: Dana Powers
@@ -9,16 +9,22 @@ License: Apache License 2.0
 Description: Kafka Python client
         ------------------------
         
-        .. image:: https://img.shields.io/badge/kafka-2.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
+        .. image:: https://img.shields.io/badge/kafka-2.6%2C%202.5%2C%202.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
             :target: https://kafka-python.readthedocs.io/en/master/compatibility.html
         .. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
             :target: https://pypi.python.org/pypi/kafka-python
         .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
             :target: https://coveralls.io/github/dpkp/kafka-python?branch=master
-        .. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master
-            :target: https://travis-ci.org/dpkp/kafka-python
         .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
             :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
+        .. image:: https://img.shields.io/pypi/dw/kafka-python.svg
+            :target: https://pypistats.org/packages/kafka-python
+        .. image:: https://img.shields.io/pypi/v/kafka-python.svg
+            :target: https://pypi.org/project/kafka-python
+        .. image:: https://img.shields.io/pypi/implementation/kafka-python
+            :target: https://github.com/dpkp/kafka-python/blob/master/setup.py
+        
+        
         
         Python client for the Apache Kafka distributed stream processing system.
         kafka-python is designed to function much like the official java client, with a
@@ -34,13 +40,16 @@ Description: Kafka Python client
         achieve something similar by manually assigning different partitions to each
         consumer instance with config management tools like chef, ansible, etc. This
         approach will work fine, though it does not support rebalancing on failures.
-        See <https://kafka-python.readthedocs.io/en/master/compatibility.html>
+        See https://kafka-python.readthedocs.io/en/master/compatibility.html
         for more details.
         
         Please note that the master branch may contain unreleased features. For release
         documentation, please see readthedocs and/or python's inline help.
         
-        >>> pip install kafka-python
+        .. code-block:: bash
+        
+            $ pip install kafka-python
+        
         
         KafkaConsumer
         *************
@@ -49,88 +58,120 @@ Description: Kafka Python client
         as possible to the official java client. Full support for coordinated
         consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+.
         
-        See <https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html>
+        See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html
         for API and configuration details.
         
         The consumer iterator returns ConsumerRecords, which are simple namedtuples
         that expose basic message attributes: topic, partition, offset, key, and value:
         
-        >>> from kafka import KafkaConsumer
-        >>> consumer = KafkaConsumer('my_favorite_topic')
-        >>> for msg in consumer:
-        ...     print (msg)
-        
-        >>> # join a consumer group for dynamic partition assignment and offset commits
-        >>> from kafka import KafkaConsumer
-        >>> consumer = KafkaConsumer('my_favorite_topic', group_id='my_favorite_group')
-        >>> for msg in consumer:
-        ...     print (msg)
-        
-        >>> # manually assign the partition list for the consumer
-        >>> from kafka import TopicPartition
-        >>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
-        >>> consumer.assign([TopicPartition('foobar', 2)])
-        >>> msg = next(consumer)
-        
-        >>> # Deserialize msgpack-encoded values
-        >>> consumer = KafkaConsumer(value_deserializer=msgpack.loads)
-        >>> consumer.subscribe(['msgpackfoo'])
-        >>> for msg in consumer:
-        ...     assert isinstance(msg.value, dict)
-        
-        >>> # Access record headers. The returned value is a list of tuples
-        >>> # with str, bytes for key and value
-        >>> for msg in consumer:
-        ...     print (msg.headers)
-        
-        >>> # Get consumer metrics
-        >>> metrics = consumer.metrics()
+        .. code-block:: python
+        
+            from kafka import KafkaConsumer
+            consumer = KafkaConsumer('my_favorite_topic')
+            for msg in consumer:
+                print (msg)
+        
+        .. code-block:: python
+        
+            # join a consumer group for dynamic partition assignment and offset commits
+            from kafka import KafkaConsumer
+            consumer = KafkaConsumer('my_favorite_topic', group_id='my_favorite_group')
+            for msg in consumer:
+                print (msg)
+        
+        .. code-block:: python
+        
+            # manually assign the partition list for the consumer
+            from kafka import TopicPartition
+            consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
+            consumer.assign([TopicPartition('foobar', 2)])
+            msg = next(consumer)
+        
+        .. code-block:: python
+        
+            # Deserialize msgpack-encoded values
+            consumer = KafkaConsumer(value_deserializer=msgpack.loads)
+            consumer.subscribe(['msgpackfoo'])
+            for msg in consumer:
+                assert isinstance(msg.value, dict)
+        
+        .. code-block:: python
+        
+            # Access record headers. The returned value is a list of tuples
+            # with str, bytes for key and value
+            for msg in consumer:
+                print (msg.headers)
+        
+        .. code-block:: python
+        
+            # Get consumer metrics
+            metrics = consumer.metrics()
+        
         
         KafkaProducer
         *************
         
         KafkaProducer is a high-level, asynchronous message producer. The class is
         intended to operate as similarly as possible to the official java client.
-        See <https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html>
+        See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html
         for more details.
         
-        >>> from kafka import KafkaProducer
-        >>> producer = KafkaProducer(bootstrap_servers='localhost:1234')
-        >>> for _ in range(100):
-        ...     producer.send('foobar', b'some_message_bytes')
+        .. code-block:: python
         
-        >>> # Block until a single message is sent (or timeout)
-        >>> future = producer.send('foobar', b'another_message')
-        >>> result = future.get(timeout=60)
+            from kafka import KafkaProducer
+            producer = KafkaProducer(bootstrap_servers='localhost:1234')
+            for _ in range(100):
+                producer.send('foobar', b'some_message_bytes')
         
-        >>> # Block until all pending messages are at least put on the network
-        >>> # NOTE: This does not guarantee delivery or success! It is really
-        >>> # only useful if you configure internal batching using linger_ms
-        >>> producer.flush()
+        .. code-block:: python
         
-        >>> # Use a key for hashed-partitioning
-        >>> producer.send('foobar', key=b'foo', value=b'bar')
+            # Block until a single message is sent (or timeout)
+            future = producer.send('foobar', b'another_message')
+            result = future.get(timeout=60)
         
-        >>> # Serialize json messages
-        >>> import json
-        >>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
-        >>> producer.send('fizzbuzz', {'foo': 'bar'})
+        .. code-block:: python
         
-        >>> # Serialize string keys
-        >>> producer = KafkaProducer(key_serializer=str.encode)
-        >>> producer.send('flipflap', key='ping', value=b'1234')
+            # Block until all pending messages are at least put on the network
+            # NOTE: This does not guarantee delivery or success! It is really
+            # only useful if you configure internal batching using linger_ms
+            producer.flush()
         
-        >>> # Compress messages
-        >>> producer = KafkaProducer(compression_type='gzip')
-        >>> for i in range(1000):
-        ...     producer.send('foobar', b'msg %d' % i)
+        .. code-block:: python
         
-        >>> # Include record headers. The format is list of tuples with string key
-        >>> # and bytes value.
-        >>> producer.send('foobar', value=b'c29tZSB2YWx1ZQ==', headers=[('content-encoding', b'base64')])
+            # Use a key for hashed-partitioning
+            producer.send('foobar', key=b'foo', value=b'bar')
+        
+        .. code-block:: python
+        
+            # Serialize json messages
+            import json
+            producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
+            producer.send('fizzbuzz', {'foo': 'bar'})
+        
+        .. code-block:: python
+        
+            # Serialize string keys
+            producer = KafkaProducer(key_serializer=str.encode)
+            producer.send('flipflap', key='ping', value=b'1234')
+        
+        .. code-block:: python
+        
+            # Compress messages
+            producer = KafkaProducer(compression_type='gzip')
+            for i in range(1000):
+                producer.send('foobar', b'msg %d' % i)
+        
+        .. code-block:: python
+        
+            # Include record headers. The format is list of tuples with string key
+            # and bytes value.
+            producer.send('foobar', value=b'c29tZSB2YWx1ZQ==', headers=[('content-encoding', b'base64')])
+        
+        .. code-block:: python
+        
+            # Get producer performance metrics
+            metrics = producer.metrics()
         
-        >>> # Get producer performance metrics
-        >>> metrics = producer.metrics()
         
         Thread safety
         *************
@@ -141,14 +182,20 @@ Description: Kafka Python client
         While it is possible to use the KafkaConsumer in a thread-local manner,
         multiprocessing is recommended.
         
+        
         Compression
         ***********
         
-        kafka-python supports gzip compression/decompression natively. To produce or consume lz4
-        compressed messages, you should install python-lz4 (pip install lz4).
-        To enable snappy compression/decompression install python-snappy (also requires snappy library).
-        See <https://kafka-python.readthedocs.io/en/master/install.html#optional-snappy-install>
-        for more information.
+        kafka-python supports the following compression formats:
+        
+        - gzip
+        - LZ4
+        - Snappy
+        - Zstandard (zstd)
+        
+        gzip is supported natively, the others require installing additional libraries.
+        See https://kafka-python.readthedocs.io/en/master/install.html for more information.
+        
         
         Optimized CRC32 Validation
         **************************
@@ -156,7 +203,9 @@ Description: Kafka Python client
         Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
         python implementation for compatibility. To improve performance for high-throughput
         applications, kafka-python will use `crc32c` for optimized native code if installed.
-        See https://pypi.org/project/crc32c/
+        See https://kafka-python.readthedocs.io/en/master/install.html for installation instructions.
+        See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
+        
         
         Protocol
         ********
@@ -166,9 +215,9 @@ Description: Kafka Python client
         testing, probing, and general experimentation. The protocol support is
         leveraged to enable a KafkaClient.check_version() method that
         probes a kafka broker and attempts to identify which version it is running
-        (0.8.0 to 2.4+).
+        (0.8.0 to 2.6+).
         
-Keywords: apache kafka
+Keywords: apache kafka,kafka
 Platform: UNKNOWN
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
@@ -182,6 +231,14 @@ Classifier: Programming Language :: Python :: 3.5
 Classifier: Programming Language :: Python :: 3.6
 Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Provides-Extra: zstd
 Provides-Extra: crc32c
+Provides-Extra: snappy
+Provides-Extra: lz4

kafka-python-2.0.3/README.rst

@@ -0,0 +1,210 @@
+Kafka Python client
+------------------------
+
+.. image:: https://img.shields.io/badge/kafka-2.6%2C%202.5%2C%202.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
+    :target: https://kafka-python.readthedocs.io/en/master/compatibility.html
+.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
+    :target: https://pypi.python.org/pypi/kafka-python
+.. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
+    :target: https://coveralls.io/github/dpkp/kafka-python?branch=master
+.. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
+    :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
+.. image:: https://img.shields.io/pypi/dw/kafka-python.svg
+    :target: https://pypistats.org/packages/kafka-python
+.. image:: https://img.shields.io/pypi/v/kafka-python.svg
+    :target: https://pypi.org/project/kafka-python
+.. image:: https://img.shields.io/pypi/implementation/kafka-python
+    :target: https://github.com/dpkp/kafka-python/blob/master/setup.py
+
+
+
+Python client for the Apache Kafka distributed stream processing system.
+kafka-python is designed to function much like the official java client, with a
+sprinkling of pythonic interfaces (e.g., consumer iterators).
+
+kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with
+older versions (to 0.8.0). Some features will only be enabled on newer brokers.
+For example, fully coordinated consumer groups -- i.e., dynamic partition
+assignment to multiple consumers in the same group -- requires use of 0.9+ kafka
+brokers. Supporting this feature for earlier broker releases would require
+writing and maintaining custom leadership election and membership / health
+check code (perhaps using zookeeper or consul). For older brokers, you can
+achieve something similar by manually assigning different partitions to each
+consumer instance with config management tools like chef, ansible, etc. This
+approach will work fine, though it does not support rebalancing on failures.
+See https://kafka-python.readthedocs.io/en/master/compatibility.html
+for more details.
+
+Please note that the master branch may contain unreleased features. For release
+documentation, please see readthedocs and/or python's inline help.
+
+.. code-block:: bash
+
+    $ pip install kafka-python
+
+
+KafkaConsumer
+*************
+
+KafkaConsumer is a high-level message consumer, intended to operate as similarly
+as possible to the official java client. Full support for coordinated
+consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+.
+
+See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html
+for API and configuration details.
+
+The consumer iterator returns ConsumerRecords, which are simple namedtuples
+that expose basic message attributes: topic, partition, offset, key, and value:
+
+.. code-block:: python
+
+    from kafka import KafkaConsumer
+    consumer = KafkaConsumer('my_favorite_topic')
+    for msg in consumer:
+        print (msg)
+
+.. code-block:: python
+
+    # join a consumer group for dynamic partition assignment and offset commits
+    from kafka import KafkaConsumer
+    consumer = KafkaConsumer('my_favorite_topic', group_id='my_favorite_group')
+    for msg in consumer:
+        print (msg)
+
+.. code-block:: python
+
+    # manually assign the partition list for the consumer
+    from kafka import TopicPartition
+    consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
+    consumer.assign([TopicPartition('foobar', 2)])
+    msg = next(consumer)
+
+.. code-block:: python
+
+    # Deserialize msgpack-encoded values
+    consumer = KafkaConsumer(value_deserializer=msgpack.loads)
+    consumer.subscribe(['msgpackfoo'])
+    for msg in consumer:
+        assert isinstance(msg.value, dict)
+
+.. code-block:: python
+
+    # Access record headers. The returned value is a list of tuples
+    # with str, bytes for key and value
+    for msg in consumer:
+        print (msg.headers)
+
+.. code-block:: python
+
+    # Get consumer metrics
+    metrics = consumer.metrics()
+
+
+KafkaProducer
+*************
+
+KafkaProducer is a high-level, asynchronous message producer. The class is
+intended to operate as similarly as possible to the official java client.
+See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html
+for more details.
+
+.. code-block:: python
+
+    from kafka import KafkaProducer
+    producer = KafkaProducer(bootstrap_servers='localhost:1234')
+    for _ in range(100):
+        producer.send('foobar', b'some_message_bytes')
+
+.. code-block:: python
+
+    # Block until a single message is sent (or timeout)
+    future = producer.send('foobar', b'another_message')
+    result = future.get(timeout=60)
+
+.. code-block:: python
+
+    # Block until all pending messages are at least put on the network
+    # NOTE: This does not guarantee delivery or success! It is really
+    # only useful if you configure internal batching using linger_ms
+    producer.flush()
+
+.. code-block:: python
+
+    # Use a key for hashed-partitioning
+    producer.send('foobar', key=b'foo', value=b'bar')
+
+.. code-block:: python
+
+    # Serialize json messages
+    import json
+    producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
+    producer.send('fizzbuzz', {'foo': 'bar'})
+
+.. code-block:: python
+
+    # Serialize string keys
+    producer = KafkaProducer(key_serializer=str.encode)
+    producer.send('flipflap', key='ping', value=b'1234')
+
+.. code-block:: python
+
+    # Compress messages
+    producer = KafkaProducer(compression_type='gzip')
+    for i in range(1000):
+        producer.send('foobar', b'msg %d' % i)
+
+.. code-block:: python
+
+    # Include record headers. The format is list of tuples with string key
+    # and bytes value.
+    producer.send('foobar', value=b'c29tZSB2YWx1ZQ==', headers=[('content-encoding', b'base64')])
+
+.. code-block:: python
+
+    # Get producer performance metrics
+    metrics = producer.metrics()
+
+
+Thread safety
+*************
+
+The KafkaProducer can be used across threads without issue, unlike the
+KafkaConsumer which cannot.
+
+While it is possible to use the KafkaConsumer in a thread-local manner,
+multiprocessing is recommended.
+
+
+Compression
+***********
+
+kafka-python supports the following compression formats:
+
+- gzip
+- LZ4
+- Snappy
+- Zstandard (zstd)
+
+gzip is supported natively, the others require installing additional libraries.
+See https://kafka-python.readthedocs.io/en/master/install.html for more information.
+
+
+Optimized CRC32 Validation
+**************************
+
+Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
+python implementation for compatibility. To improve performance for high-throughput
+applications, kafka-python will use `crc32c` for optimized native code if installed.
+See https://kafka-python.readthedocs.io/en/master/install.html for installation instructions.
+See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
+
+
+Protocol
+********
+
+A secondary goal of kafka-python is to provide an easy-to-use protocol layer
+for interacting with kafka brokers via the python repl. This is useful for
+testing, probing, and general experimentation. The protocol support is
+leveraged to enable a KafkaClient.check_version() method that
+probes a kafka broker and attempts to identify which version it is running
+(0.8.0 to 2.6+).

{kafka-python-2.0.2 → kafka-python-2.0.3}/kafka/__init__.py

@@ -4,7 +4,7 @@ __title__ = 'kafka'
 from kafka.version import __version__
 __author__ = 'Dana Powers'
 __license__ = 'Apache License 2.0'
-__copyright__ = 'Copyright 2016 Dana Powers, David Arthur, and Contributors'
+__copyright__ = 'Copyright 2025 Dana Powers, David Arthur, and Contributors'
 
 # Set default logging handler to avoid "No handler found" warnings.
 import logging