kafka-python 2.0.1__tar.gz → 2.0.3__tar.gz

{kafka-python-2.0.1 → kafka-python-2.0.3}/CHANGES.md

@@ -1,3 +1,97 @@
+# 2.0.3 (Feb 12, 2025)
+
+Improvements
+* Add optional compression libs to extras_require (#2123, #2387)
+* KafkaConsumer: Exit poll if consumer is closed (#2152)
+* Support configuration of custom kafka client for Admin/Consumer/Producer (#2144)
+* Core Protocol: Add support for flexible versions (#2151)
+* (Internal) Allow disabling thread wakeup in _send_request_to_node (#2335)
+* Change loglevel of cancelled errors to info (#2467)
+* Strip trailing dot off hostname for SSL validation. (#2472)
+* Log connection close(error) at ERROR level (#2473)
+* Support DescribeLogDirs admin api (#2475)
+
+Compatibility
+* Support for python 3.12 (#2379, #2382)
+* Kafka 2.5 / 2.6 (#2162)
+* Try collections.abc imports in vendored selectors34 (#2394)
+* Catch OSError when checking for gssapi import for windows compatibility (#2407)
+* Update vendored six to 1.16.0 (#2398)
+
+Documentation
+* Update usage.rst (#2308, #2334)
+* Fix typos (#2319, #2207, #2178)
+* Fix links to the compatibility page (#2295, #2226)
+* Cleanup install instructions for optional libs (#2139)
+* Update license_file to license_files (#2462)
+* Update some RST documentation syntax (#2463)
+* Add .readthedocs.yaml; update copyright date (#2474)
+
+Fixes
+* Use isinstance in builtin crc32 (#2329)
+* Use six.viewitems instead of six.iteritems to avoid encoding problems in StickyPartitionAssignor (#2154)
+* Fix array encoding TypeError: object of type 'dict_itemiterator' has no len() (#2167)
+* Only try to update sensors fetch lag if the unpacked list contains elements (#2158)
+* Avoid logging errors during test fixture cleanup (#2458)
+* Release coordinator lock before calling maybe_leave_group (#2460)
+* Dont raise RuntimeError for dead process in SpawnedService.wait_for() (#2461)
+* Cast the size of a MemoryRecordsBuilder object (#2438)
+* Fix DescribeConfigsResponse_v1 config_source (#2464)
+* Fix base class of DescribeClientQuotasResponse_v0 (#2465)
+* Update socketpair w/ CVE-2024-3219 fix (#2468)
+
+Testing
+* Transition CI/CD to GitHub Workflows (#2378, #2392, #2381, #2406, #2419, #2418, #2417, #2456)
+* Refactor Makefile (#2457)
+* Use assert_called_with in client_async tests (#2375)
+* Cover sticky assignor's metadata method with tests (#2161)
+* Update fixtures.py to check "127.0.0.1" for auto port assignment (#2384)
+* Use -Djava.security.manager=allow for Java 23 sasl tests (#2469)
+* Test with Java 23 (#2470)
+* Update kafka properties template; disable group rebalance delay (#2471)
+
+# 2.0.2 (Sep 29, 2020)
+
+Consumer
+* KIP-54: Implement sticky partition assignment strategy (aynroot / PR #2057)
+* Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR #2064)
+
+Compatibility
+* Python 3.8 support (Photonios / PR #2088)
+
+Cleanups
+* Bump dev requirements (jeffwidman / PR #2129)
+* Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR #2128)
+* Lint cleanup (jeffwidman / PR #2126)
+* Fix initialization order in KafkaClient (pecalleja / PR #2119)
+* Allow installing crc32c via extras (mishas / PR #2069)
+* Remove unused imports (jameslamb / PR #2046)
+
+Admin Client
+* Merge _find_coordinator_id methods (jeffwidman / PR #2127)
+* Feature: delete consumergroups (swenzel / PR #2040)
+* Allow configurable timeouts in admin client check version (sunnyakaxd / PR #2107)
+* Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR #2035)
+
+Protocol
+* Add support for zstd compression (gabriel-tincu / PR #2021)
+* Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR #2038)
+* Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR #2020)
+* Fix parsing NULL header values (kvfi / PR #2024)
+
+Tests
+* Add 2.5.0 to automated CI tests (gabriel-tincu / PR #2038)
+* Add 2.1.1 to build_integration (gabriel-tincu / PR #2019)
+
+Documentation / Logging / Errors
+* Disable logging during producer object gc (gioele / PR #2043)
+* Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR #2081)
+* Fix typo in exception message (haracejacob / PR #2096)
+* Add kafka.structs docstrings (Mostafa-Elmenbawy / PR #2080)
+* Fix broken compatibility page link (anuragrana / PR #2045)
+* Rename README to README.md (qhzxc0015 / PR #2055)
+* Fix docs by adding SASL mention (jeffwidman / #1990)
+
 # 2.0.1 (Feb 19, 2020)
 
 Admin Client

kafka-python-2.0.3/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.1
+Name: kafka-python
+Version: 2.0.3
+Summary: Pure Python client for Apache Kafka
+Home-page: https://github.com/dpkp/kafka-python
+Author: Dana Powers
+Author-email: dana.powers@gmail.com
+License: Apache License 2.0
+Description: Kafka Python client
+        ------------------------
+        
+        .. image:: https://img.shields.io/badge/kafka-2.6%2C%202.5%2C%202.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
+            :target: https://kafka-python.readthedocs.io/en/master/compatibility.html
+        .. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
+            :target: https://pypi.python.org/pypi/kafka-python
+        .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
+            :target: https://coveralls.io/github/dpkp/kafka-python?branch=master
+        .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
+            :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
+        .. image:: https://img.shields.io/pypi/dw/kafka-python.svg
+            :target: https://pypistats.org/packages/kafka-python
+        .. image:: https://img.shields.io/pypi/v/kafka-python.svg
+            :target: https://pypi.org/project/kafka-python
+        .. image:: https://img.shields.io/pypi/implementation/kafka-python
+            :target: https://github.com/dpkp/kafka-python/blob/master/setup.py
+        
+        
+        
+        Python client for the Apache Kafka distributed stream processing system.
+        kafka-python is designed to function much like the official java client, with a
+        sprinkling of pythonic interfaces (e.g., consumer iterators).
+        
+        kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with
+        older versions (to 0.8.0). Some features will only be enabled on newer brokers.
+        For example, fully coordinated consumer groups -- i.e., dynamic partition
+        assignment to multiple consumers in the same group -- requires use of 0.9+ kafka
+        brokers. Supporting this feature for earlier broker releases would require
+        writing and maintaining custom leadership election and membership / health
+        check code (perhaps using zookeeper or consul). For older brokers, you can
+        achieve something similar by manually assigning different partitions to each
+        consumer instance with config management tools like chef, ansible, etc. This
+        approach will work fine, though it does not support rebalancing on failures.
+        See https://kafka-python.readthedocs.io/en/master/compatibility.html
+        for more details.
+        
+        Please note that the master branch may contain unreleased features. For release
+        documentation, please see readthedocs and/or python's inline help.
+        
+        .. code-block:: bash
+        
+            $ pip install kafka-python
+        
+        
+        KafkaConsumer
+        *************
+        
+        KafkaConsumer is a high-level message consumer, intended to operate as similarly
+        as possible to the official java client. Full support for coordinated
+        consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+.
+        
+        See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html
+        for API and configuration details.
+        
+        The consumer iterator returns ConsumerRecords, which are simple namedtuples
+        that expose basic message attributes: topic, partition, offset, key, and value:
+        
+        .. code-block:: python
+        
+            from kafka import KafkaConsumer
+            consumer = KafkaConsumer('my_favorite_topic')
+            for msg in consumer:
+                print (msg)
+        
+        .. code-block:: python
+        
+            # join a consumer group for dynamic partition assignment and offset commits
+            from kafka import KafkaConsumer
+            consumer = KafkaConsumer('my_favorite_topic', group_id='my_favorite_group')
+            for msg in consumer:
+                print (msg)
+        
+        .. code-block:: python
+        
+            # manually assign the partition list for the consumer
+            from kafka import TopicPartition
+            consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
+            consumer.assign([TopicPartition('foobar', 2)])
+            msg = next(consumer)
+        
+        .. code-block:: python
+        
+            # Deserialize msgpack-encoded values
+            consumer = KafkaConsumer(value_deserializer=msgpack.loads)
+            consumer.subscribe(['msgpackfoo'])
+            for msg in consumer:
+                assert isinstance(msg.value, dict)
+        
+        .. code-block:: python
+        
+            # Access record headers. The returned value is a list of tuples
+            # with str, bytes for key and value
+            for msg in consumer:
+                print (msg.headers)
+        
+        .. code-block:: python
+        
+            # Get consumer metrics
+            metrics = consumer.metrics()
+        
+        
+        KafkaProducer
+        *************
+        
+        KafkaProducer is a high-level, asynchronous message producer. The class is
+        intended to operate as similarly as possible to the official java client.
+        See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html
+        for more details.
+        
+        .. code-block:: python
+        
+            from kafka import KafkaProducer
+            producer = KafkaProducer(bootstrap_servers='localhost:1234')
+            for _ in range(100):
+                producer.send('foobar', b'some_message_bytes')
+        
+        .. code-block:: python
+        
+            # Block until a single message is sent (or timeout)
+            future = producer.send('foobar', b'another_message')
+            result = future.get(timeout=60)
+        
+        .. code-block:: python
+        
+            # Block until all pending messages are at least put on the network
+            # NOTE: This does not guarantee delivery or success! It is really
+            # only useful if you configure internal batching using linger_ms
+            producer.flush()
+        
+        .. code-block:: python
+        
+            # Use a key for hashed-partitioning
+            producer.send('foobar', key=b'foo', value=b'bar')
+        
+        .. code-block:: python
+        
+            # Serialize json messages
+            import json
+            producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
+            producer.send('fizzbuzz', {'foo': 'bar'})
+        
+        .. code-block:: python
+        
+            # Serialize string keys
+            producer = KafkaProducer(key_serializer=str.encode)
+            producer.send('flipflap', key='ping', value=b'1234')
+        
+        .. code-block:: python
+        
+            # Compress messages
+            producer = KafkaProducer(compression_type='gzip')
+            for i in range(1000):
+                producer.send('foobar', b'msg %d' % i)
+        
+        .. code-block:: python
+        
+            # Include record headers. The format is list of tuples with string key
+            # and bytes value.
+            producer.send('foobar', value=b'c29tZSB2YWx1ZQ==', headers=[('content-encoding', b'base64')])
+        
+        .. code-block:: python
+        
+            # Get producer performance metrics
+            metrics = producer.metrics()
+        
+        
+        Thread safety
+        *************
+        
+        The KafkaProducer can be used across threads without issue, unlike the
+        KafkaConsumer which cannot.
+        
+        While it is possible to use the KafkaConsumer in a thread-local manner,
+        multiprocessing is recommended.
+        
+        
+        Compression
+        ***********
+        
+        kafka-python supports the following compression formats:
+        
+        - gzip
+        - LZ4
+        - Snappy
+        - Zstandard (zstd)
+        
+        gzip is supported natively, the others require installing additional libraries.
+        See https://kafka-python.readthedocs.io/en/master/install.html for more information.
+        
+        
+        Optimized CRC32 Validation
+        **************************
+        
+        Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
+        python implementation for compatibility. To improve performance for high-throughput
+        applications, kafka-python will use `crc32c` for optimized native code if installed.
+        See https://kafka-python.readthedocs.io/en/master/install.html for installation instructions.
+        See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
+        
+        
+        Protocol
+        ********
+        
+        A secondary goal of kafka-python is to provide an easy-to-use protocol layer
+        for interacting with kafka brokers via the python repl. This is useful for
+        testing, probing, and general experimentation. The protocol support is
+        leveraged to enable a KafkaClient.check_version() method that
+        probes a kafka broker and attempts to identify which version it is running
+        (0.8.0 to 2.6+).
+        
+Keywords: apache kafka,kafka
+Platform: UNKNOWN
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 2
+Classifier: Programming Language :: Python :: 2.7
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.4
+Classifier: Programming Language :: Python :: 3.5
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Provides-Extra: zstd
+Provides-Extra: crc32c
+Provides-Extra: snappy
+Provides-Extra: lz4

kafka-python-2.0.3/README.rst

@@ -0,0 +1,210 @@
+Kafka Python client
+------------------------
+
+.. image:: https://img.shields.io/badge/kafka-2.6%2C%202.5%2C%202.4%2C%202.3%2C%202.2%2C%202.1%2C%202.0%2C%201.1%2C%201.0%2C%200.11%2C%200.10%2C%200.9%2C%200.8-brightgreen.svg
+    :target: https://kafka-python.readthedocs.io/en/master/compatibility.html
+.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
+    :target: https://pypi.python.org/pypi/kafka-python
+.. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
+    :target: https://coveralls.io/github/dpkp/kafka-python?branch=master
+.. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
+    :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
+.. image:: https://img.shields.io/pypi/dw/kafka-python.svg
+    :target: https://pypistats.org/packages/kafka-python
+.. image:: https://img.shields.io/pypi/v/kafka-python.svg
+    :target: https://pypi.org/project/kafka-python
+.. image:: https://img.shields.io/pypi/implementation/kafka-python
+    :target: https://github.com/dpkp/kafka-python/blob/master/setup.py
+
+
+
+Python client for the Apache Kafka distributed stream processing system.
+kafka-python is designed to function much like the official java client, with a
+sprinkling of pythonic interfaces (e.g., consumer iterators).
+
+kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with
+older versions (to 0.8.0). Some features will only be enabled on newer brokers.
+For example, fully coordinated consumer groups -- i.e., dynamic partition
+assignment to multiple consumers in the same group -- requires use of 0.9+ kafka
+brokers. Supporting this feature for earlier broker releases would require
+writing and maintaining custom leadership election and membership / health
+check code (perhaps using zookeeper or consul). For older brokers, you can
+achieve something similar by manually assigning different partitions to each
+consumer instance with config management tools like chef, ansible, etc. This
+approach will work fine, though it does not support rebalancing on failures.
+See https://kafka-python.readthedocs.io/en/master/compatibility.html
+for more details.
+
+Please note that the master branch may contain unreleased features. For release
+documentation, please see readthedocs and/or python's inline help.
+
+.. code-block:: bash
+
+    $ pip install kafka-python
+
+
+KafkaConsumer
+*************
+
+KafkaConsumer is a high-level message consumer, intended to operate as similarly
+as possible to the official java client. Full support for coordinated
+consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+.
+
+See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html
+for API and configuration details.
+
+The consumer iterator returns ConsumerRecords, which are simple namedtuples
+that expose basic message attributes: topic, partition, offset, key, and value:
+
+.. code-block:: python
+
+    from kafka import KafkaConsumer
+    consumer = KafkaConsumer('my_favorite_topic')
+    for msg in consumer:
+        print (msg)
+
+.. code-block:: python
+
+    # join a consumer group for dynamic partition assignment and offset commits
+    from kafka import KafkaConsumer
+    consumer = KafkaConsumer('my_favorite_topic', group_id='my_favorite_group')
+    for msg in consumer:
+        print (msg)
+
+.. code-block:: python
+
+    # manually assign the partition list for the consumer
+    from kafka import TopicPartition
+    consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
+    consumer.assign([TopicPartition('foobar', 2)])
+    msg = next(consumer)
+
+.. code-block:: python
+
+    # Deserialize msgpack-encoded values
+    consumer = KafkaConsumer(value_deserializer=msgpack.loads)
+    consumer.subscribe(['msgpackfoo'])
+    for msg in consumer:
+        assert isinstance(msg.value, dict)
+
+.. code-block:: python
+
+    # Access record headers. The returned value is a list of tuples
+    # with str, bytes for key and value
+    for msg in consumer:
+        print (msg.headers)
+
+.. code-block:: python
+
+    # Get consumer metrics
+    metrics = consumer.metrics()
+
+
+KafkaProducer
+*************
+
+KafkaProducer is a high-level, asynchronous message producer. The class is
+intended to operate as similarly as possible to the official java client.
+See https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html
+for more details.
+
+.. code-block:: python
+
+    from kafka import KafkaProducer
+    producer = KafkaProducer(bootstrap_servers='localhost:1234')
+    for _ in range(100):
+        producer.send('foobar', b'some_message_bytes')
+
+.. code-block:: python
+
+    # Block until a single message is sent (or timeout)
+    future = producer.send('foobar', b'another_message')
+    result = future.get(timeout=60)
+
+.. code-block:: python
+
+    # Block until all pending messages are at least put on the network
+    # NOTE: This does not guarantee delivery or success! It is really
+    # only useful if you configure internal batching using linger_ms
+    producer.flush()
+
+.. code-block:: python
+
+    # Use a key for hashed-partitioning
+    producer.send('foobar', key=b'foo', value=b'bar')
+
+.. code-block:: python
+
+    # Serialize json messages
+    import json
+    producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
+    producer.send('fizzbuzz', {'foo': 'bar'})
+
+.. code-block:: python
+
+    # Serialize string keys
+    producer = KafkaProducer(key_serializer=str.encode)
+    producer.send('flipflap', key='ping', value=b'1234')
+
+.. code-block:: python
+
+    # Compress messages
+    producer = KafkaProducer(compression_type='gzip')
+    for i in range(1000):
+        producer.send('foobar', b'msg %d' % i)
+
+.. code-block:: python
+
+    # Include record headers. The format is list of tuples with string key
+    # and bytes value.
+    producer.send('foobar', value=b'c29tZSB2YWx1ZQ==', headers=[('content-encoding', b'base64')])
+
+.. code-block:: python
+
+    # Get producer performance metrics
+    metrics = producer.metrics()
+
+
+Thread safety
+*************
+
+The KafkaProducer can be used across threads without issue, unlike the
+KafkaConsumer which cannot.
+
+While it is possible to use the KafkaConsumer in a thread-local manner,
+multiprocessing is recommended.
+
+
+Compression
+***********
+
+kafka-python supports the following compression formats:
+
+- gzip
+- LZ4
+- Snappy
+- Zstandard (zstd)
+
+gzip is supported natively, the others require installing additional libraries.
+See https://kafka-python.readthedocs.io/en/master/install.html for more information.
+
+
+Optimized CRC32 Validation
+**************************
+
+Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
+python implementation for compatibility. To improve performance for high-throughput
+applications, kafka-python will use `crc32c` for optimized native code if installed.
+See https://kafka-python.readthedocs.io/en/master/install.html for installation instructions.
+See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
+
+
+Protocol
+********
+
+A secondary goal of kafka-python is to provide an easy-to-use protocol layer
+for interacting with kafka brokers via the python repl. This is useful for
+testing, probing, and general experimentation. The protocol support is
+leveraged to enable a KafkaClient.check_version() method that
+probes a kafka broker and attempts to identify which version it is running
+(0.8.0 to 2.6+).

{kafka-python-2.0.1 → kafka-python-2.0.3}/kafka/__init__.py

@@ -4,7 +4,7 @@ __title__ = 'kafka'
 from kafka.version import __version__
 __author__ = 'Dana Powers'
 __license__ = 'Apache License 2.0'
-__copyright__ = 'Copyright 2016 Dana Powers, David Arthur, and Contributors'
+__copyright__ = 'Copyright 2025 Dana Powers, David Arthur, and Contributors'
 
 # Set default logging handler to avoid "No handler found" warnings.
 import logging