karafka-rdkafka 0.21.0.rc1-x86_64-linux-gnu → 0.21.0.rc2-x86_64-linux-gnu

data/README.md

@@ -1,163 +1,63 @@
-# Rdkafka
+# Karafka-Rdkafka
 
-[![Build Status](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_x86_64_gnu.yml)
+[![Build Status](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_x86_64_gnu.yml)
 [![Gem Version](https://badge.fury.io/rb/karafka-rdkafka.svg)](https://badge.fury.io/rb/karafka-rdkafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
 > [!NOTE]
-> The `rdkafka-ruby` gem was created and developed by [AppSignal](https://www.appsignal.com/). Their impactful contributions have significantly shaped the Ruby Kafka and Karafka ecosystems. For robust monitoring, we highly recommend AppSignal.
+> **Karafka-Rdkafka** is a fork of the [rdkafka-ruby](https://github.com/karafka/rdkafka-ruby) gem, specifically maintained for the [Karafka ecosystem](https://karafka.io). This fork exists to provide Karafka-specific patches and optimizations that are not generic enough for inclusion in the upstream rdkafka-ruby library.
 
 ---
 
-The `rdkafka` gem is a modern Kafka client library for Ruby based on
-[librdkafka](https://github.com/confluentinc/librdkafka/).
-It wraps the production-ready C client using the [ffi](https://github.com/ffi/ffi)
-gem and targets Kafka 1.0+ and Ruby versions under security or
-active maintenance. We remove a Ruby version from our CI builds when they 
-become EOL.
+## Why This Fork Exists
 
-`rdkafka` was written because of the need for a reliable Ruby client for Kafka that supports modern Kafka at [AppSignal](https://appsignal.com). AppSignal runs it in production on very high-traffic systems.
+While rdkafka-ruby serves as an excellent general-purpose Kafka client for Ruby, Karafka requires specific optimizations and patches that are:
 
-The most essential pieces of a Kafka client are implemented, and we aim to provide all relevant consumer, producer, and admin APIs.
+- **Karafka-specific**: Tailored for Karafka's unique processing patterns and requirements
+- **Performance-oriented**: Focused on high-throughput, low-latency scenarios that Karafka specializes in
+- **Framework-integrated**: Designed to work seamlessly with Karafka's architecture and features
 
-## Table of content
+These modifications are not suitable for the upstream rdkafka-ruby library because they are either too specific to Karafka's use cases or might introduce breaking changes for other users of rdkafka-ruby.
 
-- [Project Scope](#project-scope)
-- [Installation](#installation)
-- [Usage](#usage)
-  * [Consuming Messages](#consuming-messages)
-  * [Producing Messages](#producing-messages)
-- [Higher Level Libraries](#higher-level-libraries)
-  * [Message Processing Frameworks](#message-processing-frameworks)
-  * [Message Publishing Libraries](#message-publishing-libraries)
-- [Forking](#forking)
-- [Development](#development)
-- [Example](#example)
-- [Versions](#versions)
+## Maintenance and Synchronization
 
-## Project Scope
+This fork is actively maintained and kept in sync with the upstream rdkafka-ruby repository as much as possible. We:
 
-While rdkafka-ruby aims to simplify the use of librdkafka in Ruby applications, it's important to understand the limitations of this library:
+- **Regularly merge** upstream changes from rdkafka-ruby
+- **Maintain compatibility** with the rdkafka-ruby API wherever possible
+- **Apply minimal patches** to avoid diverging significantly from upstream
+- **Merge back generic improvements** from karafka-rdkafka to rdkafka-ruby when they benefit the broader community
+- **Test thoroughly** to ensure stability within the Karafka ecosystem
 
-- **No Complex Producers/Consumers**: This library does not intend to offer complex producers or consumers. The aim is to stick closely to the functionalities provided by librdkafka itself.
+## Long-term Plan
 
-- **Focus on librdkafka Capabilities**: Features that can be achieved directly in Ruby, without specific needs from librdkafka, are outside the scope of this library.
+Our long-term goal is to work with the rdkafka-ruby maintainers to eventually merge the beneficial changes back upstream. This would allow us to:
 
-- **Existing High-Level Functionalities**: Certain high-level functionalities like producer metadata cache and simple consumer are already part of the library. Although they fall slightly outside the primary goal, they will remain part of the contract, given their existing usage.
+- Reduce maintenance overhead
+- Benefit the broader Ruby Kafka community
+- Simplify the Karafka ecosystem dependencies
 
+However, until such integration is possible, this fork ensures that Karafka users get the best possible performance and reliability.
 
-## Installation
+### If You're Using Standalone rdkafka
 
-When installed, this gem downloads and compiles librdkafka. If you have any problems installing the gem, please open an issue.
+You should use the [original rdkafka-ruby gem](https://github.com/karafka/rdkafka-ruby) for general Kafka client needs. This fork is specifically designed for Karafka and may not be suitable for other use cases.
 
-## Usage
+### Reporting Issues
 
-Please see the [documentation](https://karafka.io/docs/code/rdkafka-ruby/) for full details on how to use this gem. Below are two quick examples.
+For issues related to this fork, please report them in the [rdkafka-ruby repository](https://github.com/karafka/rdkafka-ruby/issues) rather than here. This helps us:
 
-Unless you are seeking specific low-level capabilities, we **strongly** recommend using [Karafka](https://github.com/karafka/karafka) and [WaterDrop](https://github.com/karafka/waterdrop) when working with Kafka. These are higher-level libraries also maintained by us based on rdkafka-ruby.
+- Maintain a single place for issue tracking
+- Ensure upstream compatibility
+- Provide better support for all users
 
-### Consuming Messages
+### Contributing
 
-Subscribe to a topic and get messages. Kafka will automatically spread
-the available partitions over consumers with the same group id.
+Contributions should generally be made to the upstream [rdkafka-ruby repository](https://github.com/karafka/rdkafka-ruby). Changes to this fork are only made when:
 
-```ruby
-config = {
-  :"bootstrap.servers" => "localhost:9092",
-  :"group.id" => "ruby-test"
-}
-consumer = Rdkafka::Config.new(config).consumer
-consumer.subscribe("ruby-test-topic")
-
-consumer.each do |message|
-  puts "Message received: #{message}"
-end
-```
-
-### Producing Messages
-
-Produce several messages, put the delivery handles in an array, and
-wait for them before exiting. This way the messages will be batched and
-efficiently sent to Kafka.
-
-```ruby
-config = {:"bootstrap.servers" => "localhost:9092"}
-producer = Rdkafka::Config.new(config).producer
-delivery_handles = []
-
-100.times do |i|
-  puts "Producing message #{i}"
-  delivery_handles << producer.produce(
-      topic:   "ruby-test-topic",
-      payload: "Payload #{i}",
-      key:     "Key #{i}"
-  )
-end
-
-delivery_handles.each(&:wait)
-```
-
-Note that creating a producer consumes some resources that will not be released until it `#close` is explicitly called, so be sure to call `Config#producer` only as necessary.
-
-## Higher Level Libraries
-
-Currently, there are two actively developed frameworks based on `rdkafka-ruby`, that provide higher-level API that can be used to work with Kafka messages and one library for publishing messages.
-
-### Message Processing Frameworks
-
-* [Karafka](https://github.com/karafka/karafka) - Ruby and Rails efficient Kafka processing framework.
-* [Racecar](https://github.com/zendesk/racecar) - A simple framework for Kafka consumers in Ruby 
-
-### Message Publishing Libraries
-
-* [WaterDrop](https://github.com/karafka/waterdrop) – Standalone Karafka library for producing Kafka messages.
-
-## Forking
-
-When working with `rdkafka-ruby`, it's essential to know that the underlying `librdkafka` library does not support fork-safe operations, even though it is thread-safe. Forking a process after initializing librdkafka clients can lead to unpredictable behavior due to inherited file descriptors and memory states. This limitation requires careful handling, especially in Ruby applications that rely on forking.
-
-To address this, it's highly recommended to:
-
-- Never initialize any `rdkafka-ruby` producers or consumers before forking to avoid state corruption.
-- Before forking, always close any open producers or consumers if you've opened any.
-- Use high-level libraries like [WaterDrop](https://github.com/karafka/waterdrop) and [Karafka](https://github.com/karafka/karafka/), which provide abstractions for handling librdkafka's intricacies.
-
-## Development
-
-Contributors are encouraged to focus on enhancements that align with the core goal of the library. We appreciate contributions but will likely not accept pull requests for features that:
-
-- Implement functionalities that can achieved using standard Ruby capabilities without changes to the underlying rdkafka-ruby bindings.
-- Deviate significantly from the primary aim of providing librdkafka bindings with Ruby-friendly interfaces.
-
-A Docker Compose file is included to run Kafka. To run that:
-
-```
-docker-compose up
-```
-
-Run `bundle` and `cd ext && bundle exec rake && cd ..` to download and compile `librdkafka`.
-
-You can then run `bundle exec rspec` to run the tests. To see rdkafka debug output:
-
-```
-DEBUG_PRODUCER=true bundle exec rspec
-DEBUG_CONSUMER=true bundle exec rspec
-```
-
-After running the tests, you can bring the cluster down to start with a clean slate:
-
-```
-docker-compose down
-```
-
-## Example
-
-To see everything working, run these in separate tabs:
-
-```
-bundle exec rake consume_messages
-bundle exec rake produce_messages
-```
+- They are specific to Karafka's requirements
+- They cannot be generalized for upstream inclusion
+- They are temporary while working on upstream integration
 
 ## Versions
 

data/docker-compose-ssl.yml

@@ -0,0 +1,35 @@
+services:
+  kafka:
+    container_name: kafka
+    image: confluentinc/cp-kafka:8.0.0
+    ports:
+      - 9092:9092 # Support PLAINTEXT so we can run one docker setup for SSL and PLAINTEXT
+      - 9093:9093
+    volumes:
+      - ./ssl:/etc/kafka/secrets
+    environment:
+      CLUSTER_ID: kafka-docker-cluster-1
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_LISTENERS: PLAINTEXT://:9092,SSL://:9093,CONTROLLER://:9094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092,SSL://127.0.0.1:9093
+      KAFKA_BROKER_ID: 1
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9094
+      ALLOW_PLAINTEXT_LISTENER: 'yes'
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
+      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
+
+      # SSL Configuration
+      KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
+      KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_keystore_creds
+      KAFKA_SSL_KEY_CREDENTIALS: kafka_ssl_key_creds
+      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
+      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka_truststore_creds
+      KAFKA_SSL_CLIENT_AUTH: none
+      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""

data/ext/build_common.sh

@@ -94,12 +94,25 @@ secure_download() {
     local url="$1"
     local filename="$2"
 
+    # Check if file already exists in current directory (may have been already downloaded)
     if [ -f "$filename" ]; then
         log "File $filename already exists, verifying checksum..."
         verify_checksum "$filename"
         return 0
     fi
 
+    # Check dist directory relative to script location
+    local script_dir
+    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+    local dist_file="$script_dir/../dist/$filename"
+
+    if [ -f "$dist_file" ]; then
+        log "Using distributed $filename from dist/"
+        cp "$dist_file" "$filename"
+        verify_checksum "$filename"
+        return 0
+    fi
+
     log "Downloading $filename from $url..."
 
     # Use platform-appropriate download command
@@ -344,7 +357,9 @@ get_zstd_url() {
 }
 
 get_krb5_url() {
-    echo "https://kerberos.org/dist/krb5/${KRB5_VERSION%.*}/krb5-${KRB5_VERSION}.tar.gz"
+    # Using MIT mirror since kerberos.org is down
+    # echo "https://kerberos.org/dist/krb5/${KRB5_VERSION%.*}/krb5-${KRB5_VERSION}.tar.gz"
+    echo "https://web.mit.edu/kerberos/dist/krb5/${KRB5_VERSION%.*}/krb5-${KRB5_VERSION}.tar.gz"
 }
 
 # Export functions and variables that scripts will need

data/ext/build_linux_aarch64_gnu.sh

@@ -0,0 +1,326 @@
+#!/usr/bin/env bash
+#
+# Build self-contained librdkafka.so for Linux aarch64 GNU with checksum verification
+# Usage: ./build_linux_aarch64_gnu.sh
+#
+# Expected directory structure:
+#   ext/build_linux_aarch64_gnu.sh            (this script)
+#   ext/build_common.sh                       (shared functions)
+#   dist/librdkafka-*.tar.gz                  (librdkafka source tarball)
+#   dist/patches/*.patch                      (optional Ruby-specific patches)
+#
+set -euo pipefail
+
+# Source common functions and constants
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/build_common.sh"
+
+# Platform-specific paths
+DIST_DIR="$SCRIPT_DIR/../dist"
+PATCHES_DIR="$DIST_DIR/patches"
+BUILD_DIR="$(pwd)/build-tmp"
+DEPS_PREFIX="/tmp"
+
+# Check common dependencies
+check_common_dependencies
+
+# Linux-specific dependency check
+log "Checking Linux aarch64 GNU-specific build dependencies..."
+command -v gcc &> /dev/null || error "gcc not found. Install with: apt-get install build-essential"
+
+# Verify we're on ARM64 or cross-compiling
+ARCH=$(uname -m)
+if [[ "$ARCH" != "aarch64" && "$ARCH" != "arm64" ]]; then
+    warn "Not running on aarch64 ($ARCH detected). Ensure you have aarch64 cross-compilation tools if needed."
+fi
+
+# Auto-detect librdkafka tarball
+log "Looking for librdkafka tarball in $DIST_DIR..."
+LIBRDKAFKA_TARBALL=$(find_librdkafka_tarball "$DIST_DIR")
+log "Found librdkafka tarball: $LIBRDKAFKA_TARBALL"
+
+# Verify librdkafka tarball checksum if available
+verify_librdkafka_checksum "$LIBRDKAFKA_TARBALL"
+
+# Find patches
+PATCHES_FOUND=()
+find_patches "$PATCHES_DIR" PATCHES_FOUND
+
+security_log "Starting secure build with checksum verification enabled"
+log "Building self-contained librdkafka.so for Linux aarch64 GNU"
+log "Dependencies to build:"
+log "  - OpenSSL: $OPENSSL_VERSION"
+log "  - Cyrus SASL: $CYRUS_SASL_VERSION"
+log "  - MIT Kerberos: $KRB5_VERSION"
+log "  - zlib: $ZLIB_VERSION"
+log "  - ZStd: $ZSTD_VERSION"
+log "librdkafka source: $LIBRDKAFKA_TARBALL"
+log "Build directory: $BUILD_DIR"
+
+# Create build directory
+mkdir -p "$BUILD_DIR"
+cd "$BUILD_DIR"
+
+# Build OpenSSL
+log "Building OpenSSL $OPENSSL_VERSION for ARM64..."
+OPENSSL_PREFIX="$DEPS_PREFIX/static-openssl-$OPENSSL_VERSION-arm64"
+OPENSSL_TARBALL="openssl-$OPENSSL_VERSION.tar.gz"
+OPENSSL_DIR="openssl-$OPENSSL_VERSION"
+
+secure_download "$(get_openssl_url)" "$OPENSSL_TARBALL"
+extract_if_needed "$OPENSSL_TARBALL" "$OPENSSL_DIR"
+cd "$OPENSSL_DIR"
+
+# Check if OpenSSL lib directory exists (lib or lib64)
+if [ ! -f "$OPENSSL_PREFIX/lib/libssl.a" ] && [ ! -f "$OPENSSL_PREFIX/lib64/libssl.a" ]; then
+    log "Configuring and building OpenSSL for ARM64..."
+    export CFLAGS="-fPIC"
+    # Use linux-aarch64 target for ARM64
+    ./Configure linux-aarch64 \
+        no-shared \
+        no-dso \
+        --prefix="$OPENSSL_PREFIX"
+    make clean || true
+    make -j$(get_cpu_count)
+    make install
+    unset CFLAGS
+    log "OpenSSL built successfully"
+else
+    log "OpenSSL already built, skipping..."
+fi
+
+# Determine OpenSSL lib directory
+if [ -f "$OPENSSL_PREFIX/lib64/libssl.a" ]; then
+    OPENSSL_LIB_DIR="$OPENSSL_PREFIX/lib64"
+else
+    OPENSSL_LIB_DIR="$OPENSSL_PREFIX/lib"
+fi
+log "OpenSSL libraries in: $OPENSSL_LIB_DIR"
+
+cd "$BUILD_DIR"
+
+# Build MIT Kerberos (krb5)
+log "Building MIT Kerberos $KRB5_VERSION for ARM64..."
+KRB5_PREFIX="$DEPS_PREFIX/static-krb5-$KRB5_VERSION-arm64"
+KRB5_TARBALL="krb5-$KRB5_VERSION.tar.gz"
+KRB5_DIR="krb5-$KRB5_VERSION"
+
+secure_download "$(get_krb5_url)" "$KRB5_TARBALL"
+extract_if_needed "$KRB5_TARBALL" "$KRB5_DIR"
+cd "$KRB5_DIR/src"
+
+if [ ! -f "$KRB5_PREFIX/lib/libgssapi_krb5.a" ]; then
+    log "Configuring and building MIT Kerberos for ARM64..."
+    make clean 2>/dev/null || true
+    ./configure --disable-shared --enable-static --prefix="$KRB5_PREFIX" \
+        --without-ldap --without-tcl --without-keyutils \
+        --disable-rpath --without-system-verto \
+        CFLAGS="-fPIC" CXXFLAGS="-fPIC"
+
+    # Build everything except the problematic kadmin tools
+    log "Building Kerberos (will ignore kadmin build failures)..."
+    make -j$(get_cpu_count) || {
+        log "Full build failed (expected due to kadmin), continuing with libraries..."
+        # The libraries should be built even if kadmin fails
+        true
+    }
+
+    # Install what was successfully built
+    make install || {
+        log "Full install failed, installing individual components..."
+        # Try to install the core libraries manually
+        make install-mkdirs 2>/dev/null || true
+        make -C util install 2>/dev/null || true
+        make -C lib install 2>/dev/null || true
+        make -C plugins/kdb/db2 install 2>/dev/null || true
+    }
+
+    # Verify we got the essential libraries
+    if [ ! -f "$KRB5_PREFIX/lib/libgssapi_krb5.a" ]; then
+        error "Failed to build essential Kerberos libraries"
+    fi
+
+    log "MIT Kerberos libraries built successfully"
+else
+    log "MIT Kerberos already built, skipping..."
+fi
+
+cd "$BUILD_DIR"
+
+# Build SASL
+log "Building Cyrus SASL $CYRUS_SASL_VERSION for ARM64..."
+SASL_PREFIX="$DEPS_PREFIX/static-sasl-$CYRUS_SASL_VERSION-arm64"
+SASL_TARBALL="cyrus-sasl-$CYRUS_SASL_VERSION.tar.gz"
+SASL_DIR="cyrus-sasl-$CYRUS_SASL_VERSION"
+
+secure_download "$(get_sasl_url)" "$SASL_TARBALL"
+extract_if_needed "$SASL_TARBALL" "$SASL_DIR"
+cd "$SASL_DIR"
+
+if [ ! -f "$SASL_PREFIX/lib/libsasl2.a" ]; then
+    log "Configuring and building SASL for ARM64..."
+    make clean 2>/dev/null || true
+    ./configure --disable-shared --enable-static --prefix="$SASL_PREFIX" \
+        --without-dblib --disable-gdbm \
+        --enable-gssapi="$KRB5_PREFIX" \
+        CFLAGS="-fPIC" CXXFLAGS="-fPIC" \
+        CPPFLAGS="-I$KRB5_PREFIX/include" \
+        LDFLAGS="-L$KRB5_PREFIX/lib"
+    make -j$(get_cpu_count)
+    make install
+    log "SASL built successfully"
+else
+    log "SASL already built, skipping..."
+fi
+
+cd "$BUILD_DIR"
+
+# Build zlib
+log "Building zlib $ZLIB_VERSION for ARM64..."
+ZLIB_PREFIX="$DEPS_PREFIX/static-zlib-$ZLIB_VERSION-arm64"
+ZLIB_TARBALL="zlib-$ZLIB_VERSION.tar.gz"
+ZLIB_DIR="zlib-$ZLIB_VERSION"
+
+secure_download "$(get_zlib_url)" "$ZLIB_TARBALL"
+extract_if_needed "$ZLIB_TARBALL" "$ZLIB_DIR"
+cd "$ZLIB_DIR"
+
+if [ ! -f "$ZLIB_PREFIX/lib/libz.a" ]; then
+    log "Configuring and building zlib for ARM64..."
+    make clean 2>/dev/null || true
+    export CFLAGS="-fPIC"
+    ./configure --prefix="$ZLIB_PREFIX" --static
+    make -j$(get_cpu_count)
+    make install
+    unset CFLAGS
+    log "zlib built successfully"
+else
+    log "zlib already built, skipping..."
+fi
+
+cd "$BUILD_DIR"
+
+# Build ZStd
+log "Building ZStd $ZSTD_VERSION for ARM64..."
+ZSTD_PREFIX="$DEPS_PREFIX/static-zstd-$ZSTD_VERSION-arm64"
+ZSTD_TARBALL="zstd-$ZSTD_VERSION.tar.gz"
+ZSTD_DIR="zstd-$ZSTD_VERSION"
+
+secure_download "$(get_zstd_url)" "$ZSTD_TARBALL"
+extract_if_needed "$ZSTD_TARBALL" "$ZSTD_DIR"
+cd "$ZSTD_DIR"
+
+if [ ! -f "$ZSTD_PREFIX/lib/libzstd.a" ]; then
+    log "Building ZStd for ARM64..."
+    make clean 2>/dev/null || true
+    make lib-mt CFLAGS="-fPIC" PREFIX="$ZSTD_PREFIX" -j$(get_cpu_count)
+    # Use standard install target - install-pc may not exist in all versions
+    make install PREFIX="$ZSTD_PREFIX"
+    log "ZStd built successfully"
+else
+    log "ZStd already built, skipping..."
+fi
+
+cd "$BUILD_DIR"
+
+# Extract and patch librdkafka
+log "Extracting librdkafka..."
+tar xzf "$LIBRDKAFKA_TARBALL"
+cd "librdkafka-$LIBRDKAFKA_VERSION"
+
+# Fix permissions and apply patches
+fix_configure_permissions
+apply_patches PATCHES_FOUND
+
+# Configure librdkafka
+log "Configuring librdkafka for ARM64..."
+
+if [ -f configure ]; then
+    log "Using standard configure (autotools)"
+    # Export environment variables for configure to pick up
+    export CPPFLAGS="-I$KRB5_PREFIX/include"
+    export LDFLAGS="-L$KRB5_PREFIX/lib"
+
+    ./configure --enable-static --disable-shared --disable-curl \
+        --enable-gssapi
+
+    # Clean up environment variables
+    unset CPPFLAGS LDFLAGS
+else
+    error "No configure script found (checked: configure.self, configure)"
+fi
+
+# Build librdkafka
+log "Compiling librdkafka for ARM64..."
+make clean || true
+make -j$(get_cpu_count)
+
+# Verify librdkafka.a exists
+if [ ! -f src/librdkafka.a ]; then
+    error "librdkafka.a not found after build"
+fi
+
+log "librdkafka.a built successfully"
+
+# Create self-contained shared library
+log "Creating self-contained librdkafka.so for ARM64..."
+
+# Write the export map
+cat > export.map <<'EOF'
+{
+  global:
+    rd_kafka_*;
+  local:
+    *;
+};
+EOF
+
+# Link everything statically, expose only rd_kafka_* symbols
+aarch64-linux-gnu-gcc -shared -fPIC \
+    -Wl,--version-script=export.map \
+    -Wl,--whole-archive src/librdkafka.a -Wl,--no-whole-archive \
+    -o librdkafka.so \
+    "$SASL_PREFIX/lib/libsasl2.a" \
+    "$KRB5_PREFIX/lib/libgssapi_krb5.a" \
+    "$KRB5_PREFIX/lib/libkrb5.a" \
+    "$KRB5_PREFIX/lib/libk5crypto.a" \
+    "$KRB5_PREFIX/lib/libcom_err.a" \
+    "$KRB5_PREFIX/lib/libkrb5support.a" \
+    "$OPENSSL_LIB_DIR/libssl.a" \
+    "$OPENSSL_LIB_DIR/libcrypto.a" \
+    "$ZLIB_PREFIX/lib/libz.a" \
+    "$ZSTD_PREFIX/lib/libzstd.a" \
+    -lpthread -lm -ldl -lresolv
+
+if [ ! -f librdkafka.so ]; then
+    error "Failed to create librdkafka.so"
+fi
+
+log "librdkafka.so created successfully"
+
+# Verify the build
+log "Verifying build..."
+file librdkafka.so
+
+log "Checking dependencies with ldd:"
+ldd librdkafka.so
+
+log "Checking for external dependencies (should only show system libraries):"
+EXTERNAL_DEPS=$(nm -D librdkafka.so | grep " U " | grep -v "@GLIBC" || true)
+if [ -n "$EXTERNAL_DEPS" ]; then
+    error "Found external dependencies - library is not self-contained: $EXTERNAL_DEPS"
+else
+    log "✅ No external dependencies found - library is self-contained!"
+fi
+
+# Copy to output directory
+OUTPUT_DIR="$SCRIPT_DIR"
+cp librdkafka.so "$OUTPUT_DIR/"
+log "librdkafka.so copied to: $OUTPUT_DIR/librdkafka.so"
+
+# Print summaries
+print_security_summary
+print_build_summary "Linux" "aarch64 GNU" "$OUTPUT_DIR" "librdkafka.so"
+
+# Cleanup
+cleanup_build_dir "$BUILD_DIR"

data/ext/build_linux_x86_64_gnu.sh

@@ -258,7 +258,18 @@ log "librdkafka.a built successfully"
 # Create self-contained shared library
 log "Creating self-contained librdkafka.so..."
 
-gcc -shared -fPIC -Wl,--whole-archive src/librdkafka.a -Wl,--no-whole-archive \
+echo '
+{
+  global:
+    rd_kafka_*;
+  local:
+    *;
+};
+' > export.map
+
+gcc -shared -fPIC \
+    -Wl,--version-script=export.map \
+    -Wl,--whole-archive src/librdkafka.a -Wl,--no-whole-archive \
     -o librdkafka.so \
     "$SASL_PREFIX/lib/libsasl2.a" \
     "$KRB5_PREFIX/lib/libgssapi_krb5.a" \

data/ext/build_linux_x86_64_musl.sh

@@ -625,10 +625,20 @@ do
     fi
 done
 
+echo '
+{
+  global:
+    rd_kafka_*;
+  local:
+    *;
+};
+' > export.map
+
 gcc -shared -fPIC \
-    -Wl,--whole-archive src/librdkafka.a -Wl,--no-whole-archive \
-    -o librdkafka.so \
-    -Wl,-Bstatic \
+  -Wl,--version-script=export.map \
+  -Wl,--whole-archive src/librdkafka.a -Wl,--no-whole-archive \
+  -o librdkafka.so \
+  -Wl,-Bstatic \
     "$SASL_PREFIX/lib/libsasl2.a" \
     "$KRB5_PREFIX/lib/libgssapi_krb5.a" \
     "$KRB5_PREFIX/lib/libkrb5.a" \
@@ -639,11 +649,11 @@ gcc -shared -fPIC \
     "$OPENSSL_LIB_DIR/libcrypto.a" \
     "$ZLIB_PREFIX/lib/libz.a" \
     "$ZSTD_PREFIX/lib/libzstd.a" \
-    -Wl,-Bdynamic \
-    -lpthread -lm -ldl -lc \
-    -static-libgcc \
-    -Wl,--as-needed \
-    -Wl,--no-undefined
+  -Wl,-Bdynamic \
+  -lpthread -lm -ldl -lc \
+  -static-libgcc \
+  -Wl,--as-needed \
+  -Wl,--no-undefined
 
 if [ ! -f librdkafka.so ]; then
     error "Failed to create librdkafka.so"